+ jKI2Rt^RIt^RIHt^RIHt^RIt^RIHt^RIH t ^RI H t ^RI Ht^RIHt^R IHt^R IHtHt^R IHt^R IHtHtHt^R IHtHtHtH t ^RI!H"t"] PF!]$4t%R]RRllt&!RR]PN4t(!RR]PN4t)R^RRllt*R_RRllt+!RR]PN4t,!RR]PN4t-!RR]PN4t.]!RR ]44t/!R!R"]PN4t0!R#R$]PN4t1!R%R&]/4t2]!R'R(7]!R)R*]444t3]!R+R(7]!R,R-]444t4]!R.R(7]!R/R0]444t5]!R1R(7]!R2R3]444t6]!R4R(7]!R5R6]444t7]!R7R(7]!R8R9]444t8R:R;lt9R`R<R=llt:!R>R?]PN4t;!R@RA]PN4t<!RBRC]PN4t=!RDRE]PN4t>]!RFRG]/44t?!RHRI]PN4t@]!RJR(7!RKRL]/44tA!RMRN]PN4tB]!ROR(7!RPRQ]/44tC]!RRR(7!RSRT]PN44tD]!RUR(7!RVRW]/44tE!RXRY]PN4tF]!RZR(7!R[R\]/44tG.RaOtHR#)bzPyTorch PatchTST model.N)Callable) dataclass)nn)initialization)ACT2CLS)is_deepspeed_zero3_enabled)FlashAttentionKwargs)BaseModelOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)NegativeBinomialOutput NormalOutputStudentTOutput) ModelOutputTransformersKwargsauto_docstringloggingPatchTSTConfigcV^8dQhR\PR\PR\PR\PR\PR,R\R,R\R \ \ ,/#) modulequerykeyvalueattention_maskNscalingdropoutkwargs)rModuletorchTensorfloatr r)formats"o/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/patchtst/modeling_patchtst.py __annotate__r&'s{%% II% <<% % << % LL4' % T\ %%'(%c VfVPR4R,p\P!WP^^44V,pVe W,p\P P VRR7p\P PWVPR7p\P!W4p V P^^4P4p W3#)Ndim)ptraining) sizer!matmul transposer functionalsoftmaxrr, contiguous) rrrrrrrr attn_weights attn_outputs &&&&&&&, r%eager_attention_forwardr7's**R.D(<<}}Q':;gEL!#4 ==((2(>L==((6??([L,,|3K''1-88:K  $$r'c\aa]tRt^DtoRtRV3RlV3RllltRV3RlRlltRtVtV;t #) PatchTSTAttentionz=Multi-headed attention from 'Attention Is All You Need' papercR<V^8dQhRS[RS[RS[RS[RS[RS[RS[R,/#) r embed_dim num_headsr is_decoderbias is_causalconfigN)intr#boolr)r$ __classdict__s"r%r&PatchTSTAttention.__annotate__Gs_CCCC C  C  CC%Cr'c<\SV`4WnW nW0nW,VnWpnVP V,VP8wd\RVP RV R24hVP R,VnW@n W`n \P!WVR7Vn \P!WVR7Vn\P!WVR7Vn\P!WVR7VnR#)z;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: ).r>Nr.)super__init__r;r<rhead_dimr@ ValueErrorrr=r?rLineark_projv_projq_projout_proj) selfr;r<rr=r>r?r@ __class__s &&&&&&&&r%rIPatchTSTAttention.__init__Gs "" !.  MMI %$.. 8MdnnM]$YKr3 }}d* $"ii 4@ ii 4@ ii 4@  )TB r'c<V^8dQhRS[PRS[PR,RS[PR,RS[R,RS[S[,RS[S[PS[PR,3,/#)r hidden_stateskey_value_statesNroutput_attentionsrreturn)r!r"rBr rtuple)r$rCs"r%r&rDfs~0)0)||0) ,,-0) t+ 0)  $; 0)-.0) u||U\\D00 10)r'c VRJpVPRRp.VORNVPN5pVPV4PV4P ^^4p V'dTMTp .V PRRORNVPN5p VP V 4PV 4P ^^4p VP V 4PV 4P ^^4p \P!VPP\4pT!VV V V V3RVP'gRM VPRVPRV/VBwppVP!.VORN5!P!4pVP#V4pVV3#)z#Input shape: Batch x Time x ChannelNrrrWr-)shaperJrOviewr1rMrNr get_interfacer@_attn_implementationr7r,rrreshaper4rP)rQrUrVrrWris_cross_attention input_shape hidden_shape query_statescurrent_stateskv_shape key_states value_statesattention_interfacer6r5s&&&&&, r%forwardPatchTSTAttention.forwardfs.T9$))#2. 88b8$--8 {{=166|DNNqRST -?)]B^))#2.BBDMMB[[055h?II!QO {{>277AKKAqQ (?(M(M KK , ,.E) %8      % $}}}C$,, % LL % 0 %  % ! \"));;;;FFH mmK0 L((r') r@rr;rJr?r=rMr<rPrOrrN)r[FTFN)NNF __name__ __module__ __qualname____firstlineno____doc__rIrj__static_attributes____classdictcell__ __classcell__rRrCs@@r%r9r9Ds#GCC>0)0)0)r'r9cTaa]tRt^toRtV3RlV3RlltV3RlRltRtVtV;t #)PatchTSTBatchNormzH Compute batch normalization over the sequence length (time) dimension. c <V^8dQhRS[/#rr@r)r$rCs"r%r&PatchTSTBatchNorm.__annotate__sMM~Mr'c<\SV`4\P!VPVP R7VnR#)epsN)rHrIr BatchNorm1dd_modelnorm_eps batchnormrQr@rRs&&r%rIPatchTSTBatchNorm.__init__s( FOOLr'c4<V^8dQhRS[P/#)rinputsr!r")r$rCs"r%r&rzs & &ell &r'c lVP^^4pVPV4pVP^^4#)z Parameters: inputs (`torch.Tensor` of shape `(batch_size, sequence_length, d_model)`): input for Batch norm calculation Returns: `torch.Tensor` of shape `(batch_size, sequence_length, d_model)` )r1r)rQroutputs&& r%rjPatchTSTBatchNorm.forwards7!!!Q''1%%r')rrlrus@@r%rwrws%MM & & &r'rwc vV^8dQhR\PR\R\R,R\R\ /#)rr mask_ratiounmasked_channel_indicesNchannel_consistent_masking mask_value)r!r"r#listrBrA)r$s"r%r&r&sB7%7% LL7%7%#Tk7%!% 7%  7%r'cV^8gV^8d\RV R24hVPwrVrxVPp \V^V, ,4p V'd.\P !V^WyR7p V P ^V^4p M\P !WVWyR7p \P!WVWyR7p ^V RRRV 13&\P!V RR7p \P!V RR7p\P!V RVR7p V PR4P ^^^V4p Ve ^V RVRR3&VPV P4V4pWR ,3#) arandom_masking: Mask the input considering the control variables. Args: inputs (`torch.Tensor` of shape `(batch_size, num_channels, sequence_length, num_features)`): The input tensor to mask. mask_ratio (`float`): Masking ratio applied to mask the input data during random pretraining. It is the number between 0 and 1. unmasked_channel_indices (list, *optional*): Indices of channels that will not be masked. channel_consistent_masking (bool, *optional*, defaults to `False`): When true, masking will be same across all channels of a timeseries. Otherwise, masking positions will vary across channels. mask_value (int, *optional*, defaults to 0): Define the value of masked patches for pretraining. Returns: `tuple(torch.Tensor)`: inputs_mask, masked input, same shape as input Tensor and mask tensor of shape [bs x c x n] z Mask ratio z has to be between 0 and 1.deviceNNNNr))r*indexr-.) rKr\rrAr!randrepeatonesargsortgather unsqueeze masked_fillrB)rrrrr batch_size num_channelssequence_length num_featuresrlen_keepnoisemask ids_shuffle ids_restore inputs_masks&&&&& r%random_maskingrsA4A~q;zl2MNOO>Dll;Jo ]]F?a*n56H! :q/I Q a0 :_T ::j ODDAyy--2.K-- 4K <<"K 8D >>"  $ $Q1l ;D+23Q(!Q ./$$TYY[*=K V $$r'cV^8dQhR\PR\\,R\R,R\/#)rrnum_forecast_mask_patchesrNr)r!r"rrA)r$s"r%r&r&sBA%A% LLA%#czA%#TkA% A%r'c\V\4'dV.pVUu.uFp^NK ppVPwrgr\P!WgWP R7p .p ^p \ V4p \W4FUwrV^8:gW8d\RV R24h\Wo,V , 4pV PWV.4V V, p KW \V RR7p W8d)V ^,^,Wl, ,V ^,^&M-W8d(V R,^,W, ,V R,^&^pV FwpppVV,p^V VV1RV)R13&TpK! \P!V P^,4pV V,p V PR4P^^^V 4p Ve ^V RVRR3&VPV P4V4pVV R ,3#uupi) aiForecast masking that masks the last K patches where K is from the num_forecast_mask_patches. If num_forecast_mask_patches is a list, samples in the batch will be randomly masked by numbers defined in the list. Parameters: inputs (`torch.Tensor`): Input of shape `(bs, num_channels, num_patch, patch_length)` num_forecast_mask_patches (`list`): Number of patches to be masked at the end of each batch sample. e.g. 4 or [3, 5]. unmasked_channel_indices (`list`, *optional*): Indices of channels that are not masked. mask_value (`int`, *optional*, defaults to 0): Values in the masked patches will be filled by `mask_value`. Returns: `tuple(torch.Tensor)`: inputs_mask, masked input, same shape as inputs Tensor and Mask tensor of shape `(bs, num_channels , num_patch)` or `(bs, tsg1, tsg2, num_channels, num_patch)` rznum_forecast_mask_patches z6 should be greater than 0 and less than total patches.cV^,#)r)xs&r%"forecast_masking..s!A$r')rrNr-r) isinstancerAr\r!zerosrsumziprKappendsortedrandpermrrrrB)rrrr_forecast_mask_ratiosrrrrrt_list total_length total_ratio patch_lengthratiotemp_lenbatch1 patch_lenbatch2permrs&&&& r%forecast_maskingrs0+S11%>$?!'@A'@!A'@A>Dll;Jo ;;z WD FL*+K"#<S 1  ?,\N:pq z)K78 |H56  TF /F ay|z'@Aq !  "r 1 )BCr 1 F"( 1h("./VF]A z{ *+#) >>$**Q- (D :D >>"  $ $Q1l ;D+23Q(!Q ./$$TYY[*=K V $$OBs G*cTaa]tRtRtoRtV3RlV3RlltV3RlRltRtVtV;t #)PatchTSTPatchifyi-z A class to patchify the time series sequence into different patches Returns: `torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)` c <V^8dQhRS[/#ryr)r$rCs"r%r&PatchTSTPatchify.__annotate__5sII~Ir'cB<\SV`4VPVnVPVnVP VnVPVP8:d'\ RVP RVP R24h\VPVP4VP, VP ,^,VnVPVP VP^, ,,pVPV, Vn R#)zSequence length (z+) has to be greater than the patch length ()N) rHrIcontext_lengthrr patch_striderKmax num_patchessequence_start)rQr@new_sequence_lengthrRs&& r%rIPatchTSTPatchify.__init__5s %44"//"//   4#4#4 4#D$8$8#99deievevdwwxy    4 4d6G6GH4K\K\\aeararruvv"//$2C2CtGWGWZ[G[2\\"225HHr'c4<V^8dQhRS[P/#)r past_valuesr)r$rCs"r%r&rFs5<<r'c DVPR,pW P8wd\RV RVP R24hVRVPR1R3,pVP RVP VP R7pVPRR4P4pV#) z Parameters: past_values (`torch.Tensor` of shape `(batch_size, sequence_length, num_channels)`, *required*): Input for patchification Returns: `torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)` zInput sequence length (z%) doesn't match model configuration (rFrN) dimensionr/step) r\rrKrunfoldrrr1r4)rQrrrs&& r%rjPatchTSTPatchify.forwardFs&++B/ 22 2)/)::_`d`t`t_uuwx Q 3 3 5q89$2C2C$J[J[\!!"b)446 r')rrrrrrlrus@@r%rr-s%II"r'rcTaa]tRtRtoRtV3RlV3RlltV3RlRltRtVtV;t #)PatchTSTMaskingi]al Class to perform random or forecast masking. Parameters: config (`PatchTSTConfig`): model config Returns: x_mask (`torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)`) Masked patched input mask (`torch.Tensor` of shape `(batch_size, num_channels, num_patches)`) Bool tensor indicating True on masked points c <V^8dQhRS[/#ryr)r$rCs"r%r&PatchTSTMasking.__annotate__js R R~ Rr'cD<\SV`4VPVnVPVnVPVnVP VnVP VnVPVnVP e\VP 4VnR#R#N) rHrIrandom_mask_ratior mask_typerrrrrs&&r%rIPatchTSTMasking.__init__js !'!9!9*0*K*K')))/)I)I&(.(G(G% ++  ( ( 4,243P3P,QD ) 5r'c4<V^8dQhRS[P/#r patch_inputr)r$rCs"r%r&rus!"!"5<<!"r'c tVPR8Xd<\VVPVPVPVP R7wr#MZVPR8Xd1\ VVPVPVP R7wr#M\RVP R24hVP4pW#3#)a Parameters: patch_input (`torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)`, *required*): Patch input Return: masked_input (`torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)`) Masked patched input mask (`torch.Tensor` of shape `(batch_size, num_channels, num_patches)`) Bool tensor indicating True on masked points random)rrrrrforecast)rrrrzInvalid mask type .) rrrrrrrrrKrB)rQr masked_inputrs&& r%rjPatchTSTMasking.forwardus >>X %!/"11)-)F)F+/+J+J?? " L$^^z )!1"*.*H*H)-)F)F?? " L$1$..1ACD Dyy{!!r')rrrrrrrlrus@@r%rr]s%  R R!"!"!"r'rcXaa]tRtRtoRtV3RlV3RlltRV3RlRlltRtVtV;t #) PatchTSTEncoderLayeriz PatchTST encoder layer c <V^8dQhRS[/#ryr)r$rCs"r%r&!PatchTSTEncoderLayer.__annotate__s0(0(~0(r'c l<\SV`4VPVn\VPVP VP VR7VnVP^8d!\P!VP4M\P!4Vn VPR8Xd\V4VnM[VPR8Xd3\P !VPVP"R7VnM\%VP R24hVP'dVP^8d!\P!VP4M\P!4VnVPR8Xd\V4VnM[VPR8Xd3\P !VPVP"R7VnM\%VP R24h\P*!\P,!VPVP.VP0R7\2VP4,!4VP6^8d!\P!VP64M\P!4\P,!VP.VPVP0R74VnVP^8d!\P!VP4M\P!4VnVPR8Xd\V4VnM[VPR8Xd3\P !VPVP"R7VnM\%VP R24hVP>VnR#))r;r<rr@r layernormr|z$ is not a supported norm layer type.rGN) rHrIchannel_attentionr9rnum_attention_headsattention_dropout self_attn path_dropoutrDropoutIdentity dropout_path1 norm_typerwnorm_sublayer1 LayerNormrrK dropout_path2norm_sublayer2 SequentialrLffn_dimr>ractivation_function ff_dropoutff dropout_path3norm_sublayer3pre_normrs&&r%rIPatchTSTEncoderLayer.__init__s !'!9!9*nn00,,  AG@S@SVW@WRZZ(;(;<]_]h]h]j   { *"3F";D     ,"$,,v~~6??"SD  0 011UVW W  ! ! !DJDWDWZ[D[F,?,?!@acalalanD ;.&7&?#!![0&(ll6>>v&W# F$4$4#55Y!Z[[-- IIfnnfnn6;; G F.. / 1-3->->-BBJJv(( ) IIfnnfnn6;; G  AG@S@SVW@WRZZ(;(;<]_]h]h]j   { *"3F";D     ,"$,,v~~6??"SD  0 011UVW W r'cH<V^8dQhRS[PRS[R,/#)r hidden_staterWN)r!r"rB)r$rCs"r%r&rs'OOELLOTD[Or'c 2VPwr4rVVPW4,WV4pVP'd=VPVP V4VR7wrxWP V4,pM:VPWR7wrxVP WP V4,4pVP W4WV4pVP'dVP^^4P4pVPW5,WF4pVP'd=VPVPV4VR7wryWPV4,pM:VPWR7wryVPWPV4,4pVP W5WF4pVP^^4P4pVPW4,WV4pVP'd7WPVPVPV444,pM5VPWPVPV44,4pVP W4WV4pV3p V'd YP'dVX 3MV3, p V #)ao Parameters: hidden_state (`torch.Tensor` of shape `(batch_size, num_channels, sequence_length, d_model)`, *required*): Past values of the time series output_attentions (`bool`, *optional*): Whether or not to return the output attention of all layers Return: `torch.Tensor` of shape `(batch_size, num_channels, sequence_length, d_model)` )rUrW)r\r]rrrrr`rr1r4rrrrr) rQr rWrnum_input_channelsrrr6r5channel_attn_weightsoutputss &&& r%rjPatchTSTEncoderLayer.forwardseDPCUCU@ $(()H/c ===(,"11,?Sd)7) %K(*<*<[*IIL)-\(w %K..|>P>PQ\>]/]^L$++JOe   ! ! !'11!Q7BBDL',,Z-IK]gL}}}48NN"&"5"5l"CWh5C51  ,.@.@.MM 59NN".5C51  $22P>PQUQXQXYeQf>g/ghL$++JOe /  ?U?U?U &:;\h[j jGr') rrrrrrrrrrrrlrus@@r%rrs'0(0(dOOOr'rcaa]tRtRtoRtRtR tRtRtRt Rt ] P!4V3RlV3Rll4t R RltV3R ltR tVtV;t#) PatchTSTPreTrainedModeli"modelrFTc4<V^8dQhRS[P/#)rr)rr )r$rCs"r%r&$PatchTSTPreTrainedModel.__annotate__.s>>BII>r'c <\SV`V4\V\4'Ed\ VP P VP P4VP P, VP P,^,pVP P'd,\P!VPRR7V^, pVPVP V4p\4'd^RIpVP P#VP$RR7;_uu_4VP$P'4^8d"\P(!VP$V4RRR4R#\P(!VP$V4R#R# +'giR#;i)z Initialize weights g{Gz?)stdN) modifier_rank)rH _init_weightsrPatchTSTPositionalEncodingrr@rrr use_cls_tokeninitnormal_ cls_token_init_per deepspeedzeroGatheredParameters position_encnumelcopy_)rQrrr"rrRs&& r%r%PatchTSTPreTrainedModel._init_weights-s8 f% f8 9 9DKK.. 0H0HIDKKLdLdd))*,-.K{{((( V--48q !??4;; DL)++ ^^66v7J7JZ^6__**002Q6 6#6#6 E`_ 6.. =% :`__s ?AF// G cB\V\4'd W!nR#R#r)rPatchTSTEncodergradient_checkpointing)rQrrs&&&r%_set_gradient_checkpointing3PatchTSTPreTrainedModel._set_gradient_checkpointingGs f 0 0,1 ) 1r'c&<V^8dQh/S[;R&#ryr)r$rCs"r%r&r"s r'r)time)F)rmrnrorpbase_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attnr!no_gradrr)__annotate_func__rrrsrtrus@@r%rr"sY #O &+#N ]]_>>>22Kr'rcPaa]tRtRtoV3RlV3RlltV3RlRltRtVtV;t#)PatchTSTEmbeddingiLc <V^8dQhRS[/#ryr)r$rCs"r%r&PatchTSTEmbedding.__annotate__Ms \ \~ \r'c<\SV`4VPVnVPVnVP'd3\P !VP VP4VnR#\P!4Vn\VP4FGpVPP\P !VP VP44KI R#r) rHrIr share_embeddingrrLrrinput_embedding ModuleListranger)rQr@rrRs&& r%rIPatchTSTEmbedding.__init__Ms "(";";%55    #%99V-@-@&..#QD #%==?D 6445$$++BIIf6I6I6>>,Z[6r'c4<V^8dQhRS[P/#rr)r$rCs"r%r&r9Ys5<<r'c  ~VP^,pW P8wd\RVP RV R24hVP'dVP V4pV#\ V4Uu.uF&q@PV,!VRVRR3,4NK( pp\ P!V^R7pV#uupi)z Parameters: patch_input (`torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)`, *required*): Patch input for embedding return: `torch.Tensor` of shape `(batch_size, num_channels, num_patches, d_model)` z&The defined number of input channels (zQ) in the config has to be the same as the number of channels in the batch input (rrr))r\r rKr;r<r>r!stack)rQrr  embeddingsis&& r%rjPatchTSTEmbedding.forwardYs)..q1 !8!8 889P9P8QRTTfSgghj     --k:JUZZlTmnTmq..q1+aAqj2IJTmJnZQ7Jos2,B:)r<r r; rmrnrorprIrjrrrsrtrus@@r%r7r7Ls  \ \r'r7cpaa]tRtRtoRtV3RlV3Rllt]V3RlRl4tV3RlRltR t Vt V;t #) ripz Class for positional encoding c&<V^8dQhRS[RS[/#rr@rrrA)r$rCs"r%r&'PatchTSTPositionalEncoding.__annotate__us   ~  C  r'c<\SV`4VPVnVPVnVP'dF\P !\ P!^^^VP44Vn V^, pVPW4Vn VP^8d(\P!VP4Vn R#\P!4Vn R#)N)rHrIrr r Parameterr!rrrrr"positional_dropoutrrrQr@rrRs&&&r%rI#PatchTSTPositionalEncoding.__init__us #11"(";";    \\%++aAv~~*NODN 1 K MM&>6<5N5NQR5RBJJv00 1 XZXcXcXe r'c@<V^8dQhRS[RS[RS[P/#)rr@rrX)rrArrN)r$rCs"r%r&rKs&cbllr'cPVPR8Xd9\P!\P!WP 4RR7pV#VPR8XEd5\P !WP 4p\P!^V4P^4p\P!\P!^VP ^4\P!R4VP , ),4p\P!W4,4VR&\P!W4,4VR&W"P4, pW"P4^ ,, p\P!VRR7pV#\!VP R24h) rT requires_gradsincosg@FzN is not a valid positional encoder. Available types are 'random' and 'sincos'.)r:rNr)r:rMNr)positional_encoding_typerrNr!randnrrarangerexpmathlogsincosmeanrrK)r@rr"positiondiv_terms&& r%r#PatchTSTPositionalEncoding._init_pesD  * *h 6<< K(P`deL , , 8 ;;{NNCL||A{3==a@Hyya!CQXHY\b\j\jHjFk!klH$)IIh.A$BL !$)IIh.A$BL !'*;*;*==L'+;+;+=+BCL<< EJL 2234BC r'c4<V^8dQhRS[P/#rr)r$rCs"r%r&rKs  5<< r'cVP'dVPWPR,,4pVPVPR,,pVP VP ^,VP RR4p\P!W13^R7pV#VPWP,4pV#)rMNNr))rer):NrMNrr-) rrOr"rexpandr\r r!cat)rQrr cls_tokensr s&& r%rj"PatchTSTPositionalEncoding.forwards    11+@Q@QRW@X2XYK):):5)AAI"))+*;*;A*>@W@WY[]_`J 99j%>AFL 22;ARAR3RSLr')rr r"rOr) rmrnrorprqrI staticmethodrrjrrrsrtrus@@r%rrps7   &   r'rcXaa]tRtRtoRtV3RlV3RlltRV3RlRlltRtVtV;t #) r'iz PatchTST Encoder c&<V^8dQhRS[RS[/#rIrJ)r$rCs"r%r&PatchTSTEncoder.__annotate__s  ~ C r'c,<\SV`V4RVn\V4Vn\ W4Vn\P!\VP4Uu.uFp\V4NK up4Vn VP4R#uupi)FN)rHrIr(r7embedderrpositional_encoderrr=r>num_hidden_layersrlayers post_init)rQr@rrDrRs&&& r%rIPatchTSTEncoder.__init__sx  &+#*&1 "