+ j|^RIt^RIHt^RIt^RIt^RIHt^RIHt^RI H t ^RI H t ^RIHt^RIHt^R IHt^R IHt^R IHt^R IHt^R IHtHtHt^RIHtH t H!t!^RI"H#t#^RI$H%t%H&t&H't'^RI(H)t)H*t*^RI+H,t,H-t-^RI.H/t/]'P`!]14t2!RR]4t3!RR]4t4!RR]4t5!RR]Pl4t7!RR]Pl4t8!RR]Pl4t9!R R!]Pl4t:R9R"R#llt;!R$R%]Pl4t<!R&R']Pl4t=!R(R)]4t>!R*R+]Pl4t?]&!R,R-] 44t@R:R.R/lltA]&!R0R1]@44tB]&!R2R37!R4R5]@44tC]&!R6R37!R7R8]@44tD.R;OtER#)<N)Callable)nn)CrossEntropyLoss)initialization)ACT2FN)filter_output_hidden_states)is_deepspeed_zero3_enabled)is_fsdp_managed_module)create_bidirectional_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputCausalLMOutputSequenceClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel*get_torch_context_manager_or_global_device)Unpack)TransformersKwargsauto_docstringlogging)can_return_tuplemerge_with_config_defaults)OutputRecordercapture_outputs SEWConfigc<aa]tRt^1toRV3RlltRtRtVtV;t#)SEWNoLayerNormConvLayerc<\SV`4V^8dVPV^, ,M^VnVPV,Vn\ P !VPVPVPV,VPV,VPR7Vn \VP,Vn R#)r kernel_sizestridebiasN)super__init__conv_dim in_conv_dim out_conv_dimrConv1d conv_kernel conv_stride conv_biasconvrfeat_extract_activation activationselfconfiglayer_id __class__s&&&e/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/sew/modeling_sew.pyr& SEWNoLayerNormConvLayer.__init__2s !#Car8cZVP^8dVRRRVP)13,pV#)rNNNNrr;s&&r6r=SEWSamePadLayer.forwards6    ")!Q0F43F3F2F0F*FGMr8rr@rHs@@r6r|r|sKr8r|c8aa]tRt^toV3RltRtRtVtV;t#) SEWUpsamplingc<\SV`4\P!VPVPVP ,4Vn\VP,Vn VP VnR#r:) r%r&rLinearrnrq projectionrr/r0r2r3r5s&&r6r&SEWUpsampling.__init__sW ))F$6$68J8JVMbMb8bc !?!?@$33r8c8VPV4pVPV4pVP^8dgVP4wr#pW0P,pW@P,pVP W#VPV4pVP W%V4pV#))rr0rqsizereshape)r2r<bszsrc_len src_embed_dimtgt_len tgt_embed_dims&& r6r=SEWUpsampling.forwards 6  6    "*7*<*<*> 'C- 3 33G)-@-@@M)11#@S@SUbcM)11# NMr8)r0rrqr@rHs@@r6rrs4  r8rcBaa]tRt^toRtV3RltRtRtRtVt V;t #)SEWFeatureEncoderz.Construct the features from raw audio waveformc <\SV`4VPR8XdR\V^R7.\ VP ^, 4Uu.uFp\ W^,R7NK up,pMYVPR8Xd0\ VP 4Uu.uFp\WR7NK ppM\RVP R24h\P!V4Vn RVn RVn R#uupiuupi)group)r4layerz`config.feat_extract_norm` is z), but has to be one of ['group', 'layer']FTN)r%r&feat_extract_normrYrangenum_feat_extract_layersrrK ValueErrorr ModuleList conv_layersgradient_checkpointing_requires_grad)r2r3irr5s&& r6r&SEWFeatureEncoder.__init__s   # #w .0!DEINvOmOmpqOqIrIIrA'Q?IrIK % % 0NSTZTrTrNstNs0DNsKtK01I1I0JJst ==5&+#"Ius C6C;cPVP4F pRVnK RVnR#)FN) parameters requires_gradrr2params& r6_freeze_parameters$SEWFeatureEncoder._freeze_parameterss#__&E"'E '#r8cVR,pVP'dVP'dRVnVPF pV!V4pK V#)rTrN)rtrainingrr)r2 input_valuesr< conv_layers&& r6r=SEWFeatureEncoder.forwardsL$W-     4===*.M '**J&}5M+r8)rrr) rArBrCrD__doc__r&rr=rErFrGrHs@@r6rrs8#"$  r8rcV^8dQhR\PR\PR\PR\PR\PR,R\R,R\R \ \ ,/#) rRmodulequerykeyvalueattention_maskNscalingdropoutkwargs)rModuletorchTensorfloatrr)formats"r6 __annotate__rs{%% II% <<% % << % LL4' % T\ %%'(%r8c VfVPR4R,p\P!WP^^44V,pVe W,p\P P VRR7p\P PWVPR7p\P!W4p V P^^4P4p W3#)Nrl)prrT) rrmatmulrUr functionalsoftmaxrr contiguous) rrrrrrrr attn_weights attn_outputs &&&&&&&, r6eager_attention_forwardrs**R.D(<<}}Q':;gEL!#4 ==((2(>L==((6??([L,,|3K''1-88:K  $$r8c\aa]tRtRtoRtRV3RlV3RllltR V3RlRlltRtVtV;t #) SEWAttentioniz=Multi-headed attention from 'Attention Is All You Need' papercR<V^8dQhRS[RS[RS[RS[RS[RS[RS[R,/#) rR embed_dim num_headsr is_decoderr$ is_causalr3N)intrboolr)rrIs"r6rSEWAttention.__annotate__ s_CCCC C  C  CCD Cr8c<\SV`4WnW nW0nW,VnWpnVP V,VP8wd\RVP RV R24hVP R,VnW@n W`n \P!WVR7Vn \P!WVR7Vn\P!WVR7Vn\P!WVR7VnR#)z;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).)r$Nr)r%r&rrrhead_dimr3rrrrrrk_projv_projq_projout_proj) r2rrrrr$rr3r5s &&&&&&&&r6r&SEWAttention.__init__ s "" !.  MMI %$.. 8MdnnM]$YKr3 }}d* $"ii 4@ ii 4@ ii 4@  )TB r8c<V^8dQhRS[PRS[PR,RS[PR,RS[R,RS[S[,RS[S[PS[PR,3,/#)rRr<key_value_statesNroutput_attentionsrreturn)rrrrr tuple)rrIs"r6rr)s~0)0)||0) ,,-0) t+ 0)  $; 0)-.0) u||U\\D00 10)r8c VRJpVPRRp.VORNVPN5pVPV4PV4P ^^4p V'dTMTp .V PRRORNVPN5p VP V 4PV 4P ^^4p VP V 4PV 4P ^^4p \P!VPP\4pT!VV V V V3RVP'gRM VPRVPRV/VBwppVP!.VORN5!P!4pVP#V4pVV3#)z#Input shape: Batch x Time x ChannelNrrrrT)shaperrviewrUrrr get_interfacer3_attn_implementationrrrrrrr)r2r<rrrris_cross_attention input_shape hidden_shape query_statescurrent_stateskv_shape key_states value_statesattention_interfacerrs&&&&&, r6r=SEWAttention.forward)s.T9$))#2. 88b8$--8 {{=166|DNNqRST -?)]B^))#2.BBDMMB[[055h?II!QO {{>277AKKAqQ (?(M(M KK , ,.E) %8      % $}}}C$,, % LL % 0 %  % ! \"));;;;FFH mmK0 L((r8) r3rrrrrrrrrrr)rFTFN)NNF) rArBrCrDrr&r=rErFrGrHs@@r6rrs#GCC>0)0)0)r8rc8aa]tRtRtoV3RltRtRtVtV;t#)SEWFeedForwardi\c<\SV`4\P!VP4Vn\P !VPVP4Vn \VP\4'd\VP,VnMVPVn\P !VPVP4Vn\P!VP 4VnR#r:)r%r&rDropoutactivation_dropoutintermediate_dropoutrrnintermediate_sizeintermediate_dense isinstance hidden_actstrrintermediate_act_fn output_densehidden_dropoutoutput_dropoutrs&&r6r&SEWFeedForward.__init__]s $&JJv/H/H$I!"$))F,>,>@X@X"Y f'' - -'-f.?.?'@D $'-'8'8D $IIf&>&>@R@RS jj)>)>?r8cVPV4pVPV4pVPV4pVPV4pVP V4pV#r:)rrrrrr;s&&r6r=SEWFeedForward.forwardjsX// > 00? 11-@ ))-8 ++M: r8)rrrrrr@rHs@@r6rr\s @r8rcHaa]tRtRtoV3RltRV3RlRlltRtVtV;t#)SEWEncoderLayeritc<\SV`4\VPVPVP RVR7Vn\P!VP4Vn \P!VPVPR7Vn \V4Vn\P!VPVPR7VnR#)F)rrrrr3epsN)r%r&rrnnum_attention_headsattention_dropout attentionrrrrrNlayer_norm_epsrOr feed_forwardfinal_layer_normrs&&r6r&SEWEncoderLayer.__init__us %((00,,  zz&"7"78 ,,v'9'9v?T?TU*62 " V-?-?VEZEZ [r8cr<V^8dQhRS[PRS[PR,RS[S[,/#)rRr<rNr)rrrr)rrIs"r6rSEWEncoderLayer.__annotate__s<|| t++, r8c TpVP!V3RV/VBwrVPV4pWA,pVPV4pWPV4,pVP V4pV#)r)rrrOrr)r2r<rr attn_residual_s&&&, r6r=SEWEncoderLayer.forwardsr & >>-aaZ`a  ]3 %5  6 %(9(9-(HH --m< r8)rrrrrOr:r@rHs@@r6r r ts \r8r cHaa]tRtRtoV3RltRV3RlRlltRtVtV;t#) SEWEncoderic"<\SV`4Wn\V4Vn\ P !VPVP4Vn\ P!VPVPR7Vn \ P!VP4Vn\ P !\#VP$4Uu.uFp\'V4NK up4Vn\+V4VnRVnR#uupi)r FN)r%r&r3rcpos_conv_embedr AvgPool1drqpoolrNrnrrOrrrrrnum_hidden_layersr layersrupsampler)r2r3rr5s&& r6r&SEWEncoder.__init__s  8@LL!6!68M8MN ,,v'9'9v?T?TUzz&"7"78 mmeFLdLdFe$fFe_V%ATUXZe[eZeUeAff %//15 2;;m  6  ]3 02RR6LT6R [[E"'**R. !]]ZZ/B[[EZEZ/ZN![[ %m ]N ]V\ ] ! m4   q !$5 5MM--maAGX[n[nop[qGq=rsM??r8)r3rrrOr!rrr"r:r@rHs@@r6rrs ,7@7@7@r8rc aa]tRtRtoRtRtRtRtRtRt Rt R] R] !] ^RR 7/t]P !4V3R l4tV3R lR ltV3R lRltV3RltRtVtV;t#)SEWPreTrainedModelisewraudioTr< attentionsencoder)index layer_namec  |<\SV`V4\V\4'd\P !VP P^^\P!^VP P^,VP P,, 4,R7\P!VP P^4R#\V\P4'Ed\!4'd^RIp\%VR4'dy\%VR4'dgVP&P)VP*VP,.^R7;_uu_4\P.!VP4RRR4R#VP&P)VP^R7;_uu_4\P.!VP4RRR4R#\P.!VP4R#R# +'giR#;i +'giR#;i)zInitialize the weights)meanstdNrzryrh)r% _init_weightsrrcinitnormal_r.rjmathsqrtr" in_channels constant_r$rr*r rtrsrurvrzrykaiming_normal_)r2rrtr5s&& r6rG SEWPreTrainedModel._init_weightssr f% f8 9 9 LL "" !v{{'>'>q'AFKKD[D['["\]]  NN6;;++Q /  * *)++ 6:..76:3N3N"::FOOV__;]mn:oo,,V]];po#::6==XY:ZZ,,V]];[Z$$V]]3+ poo[ZZs-!H!H* H' * H; cD<V^8dQhRS[PS[,/#)rR input_lengths)r LongTensorr)rrIs"r6rSEWPreTrainedModel.__annotate__s  e>N>NQT>T r8c Rp\VPPVPP4Fwr4V!WV4pK V#)z8 Computes the output length of the convolutional layers cN\P!W, VRR7^,#)floor) rounding_mode)rdiv) input_lengthr"r#s&&&r6_conv_out_lengthMSEWPreTrainedModel._get_feat_extract_output_lengths.._conv_out_lengths 99\7wWZ[[ [r8)zipr3r+r,)r2rQrZr"r#s&& r6 _get_feat_extract_output_lengths3SEWPreTrainedModel._get_feat_extract_output_lengthssG  \ $'t{{'>'> @W@W#X K,]PM$Yr8c:<V^8dQhRS[RS[P/#)rRfeature_vector_lengthr)rrrR)rrIs"r6rrSs&HHH]b]m]mHr8cVPVPR44P\P4p\P !WP R7pVP^4VP^48#)r)devicerT)r]sumtorr.arangerbr*)r2r`routput_lengths attention_idss&&& r6"_get_feature_vector_attention_mask5SEWPreTrainedModel._get_feature_vector_attention_masksh>>~?Q?QRT?UVYYZ_ZdZde %:CXCXY &&q)N,D,DQ,GGGr8c&<V^8dQh/S[;R&#rRr3r)rrIs"r6rrSs r8)rArBrCrDbase_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attnr rr_can_record_outputsrno_gradrGr]rh__annotate_func__rErFrGrHs@@r6r=r=s$O&*#Nn\yQ  ]]_44.  HHmr8r=c V^8dQhR\\\3,R\R\R\PR,R\R\ P /#)rRr mask_prob mask_lengthrN min_masksr)rrrrrRnpndarray)rs"r6rrsctt c?ttt$$t+ t  t ZZ tr8c ^aaaaaVwpoS^8d \R4hSS8d\RS RS R24h\PP^4P 4oVVVVV3RlpVe.VP 4P R 4P4M\V4Uu.uFpSNK upp\P!VS3\R7p .p V!S4p V ^8XdV #VFp V!V 4p \PP\P!V S^, , 4V RR7p\V4^8Xd S^, pM V^,p\P!V\P!W, \P R7V,.4pV P#V4K \P$!V 4p \P&!V R ,W[S34p V P)W[S,4p \P!S4R ,p\P&!VW[S34P)W[S,4pV V,p V P+4S^, 8dS^, WS^, 8&\P,!W^R 4V #uupi) a2 Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://huggingface.co/papers/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. z&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `c<\SV,S, S,4p\VS4pVS,S8d SS,pVS^, , V8d\VS^, , ^4pV#)z;Given input length, compute how many spans should be masked)rmax)rYnum_masked_spanepsilonryrxrzsequence_lengths& r6compute_num_masked_span6_compute_mask_indices..compute_num_masked_spanAsoi,6DwNOoy9 [ (? :-G[G[chcmcm n /3/E/E/H/HI\I\/]M , ;; ( (1 ,#8)++77 KK;;++<< $ $)<<0DMaMainisis#t #7#@#G#GO]_#` 23M /r8F)tie_last_hidden_statesc <V^8dQhRS[PR,RS[PR,RS[PR,RS[S[,RS[S[,/#)rRrNrrrr)rrrrrrr)rrIs"r6rrsl"@"@llT)"@ t+"@!,,t3 "@ +, "@  "@r8c VPV4pVP^^4pVPV4pVP'dVP V4pVP V4pVe$VP VP^,V4pVPWcR7pVP!V3RV/VBpVPp\VR7#)a mask_time_indices (`torch.BoolTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices to mask extracted features for contrastive loss. When in training mode, model learns to predict masked extracted features in *config.proj_codevector_dim* space. )rrr() rrUrOrrrrhrrrAr)r)r2rrrrextract_featuresr<encoder_outputss&&&&, r6r=SEWModel.forwards 11,?+55a;??+;<  #667GH ,,-=>  %!DD]EXEXYZE[]klN000d ,,  )   (99 ??r8)r3rArrrrOrrNN) rArBrCrDr&rrrrr=rErFrGrHs@@r6rrsQ(,,\ E2"@"@3 "@"@r8rzk SEW Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC). ) custom_introc~aa]tRtRtoR V3RlV3RllltRtRtRt]] R V3RlRll44t R t Vt V;t #) SEWForCTCic.<V^8dQhRS[R,/#)rR target_langN)r)rrIs"r6rSEWForCTC.__annotate__sC$Jr8c <\SV`V4\V4Vn\P !VP 4VnW nVPf\RVP R24h\VR4'dVP'd VPM VPp\P !W1P4VnVP%4R#)a target_lang (`str`, *optional*): Language id of adapter weights. Adapter weights are stored in the format adapter..safetensors or adapter..bin. Only relevant when using an instance of [`SEWForCTC`] with adapters. Uses 'eng' by default. NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `SEWForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration. add_adapter)r%r&rr>rr final_dropoutrr vocab_sizerr5rsroutput_hidden_sizernrlm_headr)r2r3rrr5s&&& r6r&SEWForCTC.__init__s  F#zz&"6"67 &    $00@AHH *1)G)GFL^L^L^F % %djdvdv yy!35F5FG  r8c Z\4\P!R48XdR#VPpVe)\ VP RR4f\ RV R24hVf1\ VP RR4e\PR4R#VeVPVRR7R#R#) a This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when passing `target_lang=...` to `from_pretrained(...)`. This method is **not** supposed to be called by the user and is prone to be changed in the future. metaNadapter_attn_dimzCannot pass `target_lang`: z- if `config.adapter_attn_dim` is not defined.z)By default `target_lang` is set to 'eng'.T) force_load) rrrbrrr3rloggerinfo load_adapter)r2rrs&, r6 tie_weightsSEWForCTC.tie_weights!s 6 75<<;O O  &&  "wt{{rrr2s&r6freeze_feature_encoder SEWForCTC.freeze_feature_encoder9 ""557r8c VVPP4F pRVnK R#z Calling this function will disable the gradient computation for the base model so that its parameters will not be updated during training. Only the classification head will be updated. FNr>rrrs& r6freeze_base_modelSEWForCTC.freeze_base_model@# XX((*E"'E +r8c <V^8dQhRS[PR,RS[PR,RS[PR,RS[S[,RS[S[,/#rRrNrlabelsrr)rrrrrr)rrIs"r6rrJsc: : llT):  t+:  t# : +, :   : r8c xVeLVP4VPP8d#\RVPP 24hVP!V3RV/VBpVP pVP V4pVPV4pRpVEenVeTM%\P!V\PR7pVPVPR 44P\P4p V^8p V PR 4p VPV 4p \P P#VR \P$R7P'^^4p \P(P*P-RR7;_uu_4\P P/V V V V VPP0VPP2VPP4R7pRRR4\7WVP8VP:R 7# +'giL2;i) a labels (`torch.LongTensor` of shape `(batch_size, target_length)`, *optional*): Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to the sequence length of the output logits. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size - 1]`. Nz$Label values must be <= vocab_size: rr)rlrF)enabled)blank reduction zero_infinitylosslogitsr<r@rT)rr3rrr>r)rrr ones_liker.r]rcrd masked_selectrr log_softmaxfloat32rUbackendscudnnflagsctc_loss pad_token_idctc_loss_reductionctc_zero_infinityrr<r@)r2rrrroutputsr<rrrQ labels_masktarget_lengthsflattened_targets log_probss&&&&, r6r=SEWForCTC.forwardHs  &**,$++2H2H"HCDKKDZDZC[\] ]((  )    11  ]3 m,  #1"<%//R^fkfpfpBq !AA.BTBTUWBXY\\]b]g]ghM!A+K(__R0N & 4 4[ A  11&b 1V``abdefI%%++E+::}}--%!"++22"kk<<"&++"?"?.;G4I4IV]VhVh  ;:s A#H)) H9 )rrr>rr:r)rArBrCrDr&rrrrrr=rErFrGrHs@@r6rrsD :<08(: : : : r8rz SEW Model with a sequence classification head on top (a linear layer over the pooled output) for tasks like SUPERB Keyword Spotting. craa]tRtRtoV3RltRtRt]]] RV3RlRll444t Rt Vt V;t #) SEWForSequenceClassificationic:<\SV`V4\VR4'dVP'd \ R4h\ V4VnVP^,pVP'd7\P!\P!V4V, 4Vn \P!VPVP 4Vn\P!VP VP$4VnVP)4R#)rzZSequence classification does not support the use of SEW adapters (config.add_adapter=True)N)r%r&rsrrrr>r use_weighted_layer_sumrrrr layer_weightsrrnclassifier_proj_size projector num_labels classifierr)r2r3 num_layersr5s&& r6r&%SEWForSequenceClassification.__init__s  6= ) )f.@.@.@l F#--1  ( ( (!#ejj.Dz.Q!RD 6#5#5v7R7RS))F$?$?ARARS r8c NVPPP4R#rrrs&r6r3SEWForSequenceClassification.freeze_feature_encoderrr8c VVPP4F pRVnK R#rrrs& r6r.SEWForSequenceClassification.freeze_base_modelrr8c <V^8dQhRS[PR,RS[PR,RS[PR,RS[S[,RS[S[,/#r)rrrrrr)rrIs"r6r)SEWForSequenceClassification.__annotate__sd9 9 llT)9  t+9  t# 9 +, 9 ) ) 9 r8c (VPP'dRVR&VP!V3RV/VBpVPP'dzVPp\P !V^R7p\ PPVPRR7pWgPR^^4,P^R7pM VPpVPV4pVfVP^R7pMVPVP ^,V4p V P#R4P%^^VP ^,4p RWj(&VP^R7V P^R7PR^4, pVP'V4p Rp VeG\)4p V !V PRVPP*4VPR44p \-V V VPVP.R7#) a input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoProcessor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`SEWProcessor.__call__`] for details. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Toutput_hidden_statesrrNrrrT)r3rr>r<rstackrrrr rrcr)r rErhrr*r+r rr rr@)r2rrrrrr< norm_weights pooled_output padding_maskexpand_padding_maskrrloss_fcts&&&&, r6r=$SEWForSequenceClassification.forwards, ;; - - --1F) *((  )   ;; - - -#11M!KK 1=M==001C1C0LL*->->r1a-HHMMRSMTM#55M}5  !)..1.5MBB=CVCVWXCY[ijL"."8"8"<"C"CAq-J]J]^_J`"a 25M. /)--!-4|7G7GA7G7N7S7STVXY7ZZM/  ')HFKKDKK,B,BCV[[QS_UD'!//))   r8)r r r r>r)rArBrCrDr&rrrrrr=rErFrGrHs@@r6rrsB"8( 9 9 !9 9 r8r)Nr)Nr)rrrr=)FrJcollections.abcrnumpyr{rrtorch.nnrrrH activationsrbackbone_utilsrintegrations.deepspeedr integrations.fsdpr masking_utilsr modeling_flash_attention_utilsr modeling_layersr modeling_outputsrrrmodeling_utilsrrrprocessing_utilsrrrrrr utils.genericrrutils.output_capturingrrconfiguration_sewr get_loggerrArrrKrYrrcr|rrrrrr rr=rrrr__all__rlr8r6r2s* $ %&!9@76B9YYrr&@@IE(   H %8*6660((VbiiBII,# #L%8R)299R)jRYY00DC@C@L;H;H;H|tnh@!h@h@V A "A  A H ] #5]  ] @ Zr8