+ jXRt^RIt^RIt^RIHt^RIHtHtHt^RIH t ^RI H t ^RI Ht^RIHtHtHtHtHtHtHt^RIHt^R IHtHt^R IHt]P<!]4t !R R ]PB4t"!R R]PB4t#!RR]PH4t%!RR]PB4t&!RR]PB4t'!RR]PB4t(!RR]PB4t)!RR]PB4t*!RR]PB4t+!RR]PB4t,!RR ]PB4t-!R!R"]PB4t.]!R#R$]44t/]!R%R&]/44t0]!R'R(]/44t1]!R)R*7!R+R,]/44t2]!R-R.]/44t3]!R/R0]/44t4]!R1R2]/44t5.R3Ot6R#)4zPyTorch SqueezeBert model.N)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)create_bidirectional_mask)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)auto_docstringloggingSqueezeBertConfigc@aa]tRt^-toRtV3RltRRltRtVtV;t #)SqueezeBertEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.c<\SV`4\P!VPVP VP R7Vn\P!VPVP 4Vn \P!VPVP 4Vn \P!VPVPR7Vn \P!VP 4Vn\P$!\&P(!VP4P+R4RR7VnR#)) padding_idxepsF) persistentN)super__init__r Embedding vocab_sizeembedding_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNorm hidden_sizelayer_norm_epsDropouthidden_dropout_probdropoutBuffertorcharangeexpand position_idsselfconfig __class__s&&u/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/squeezebert/modeling_squeezebert.pyr SqueezeBertEmbeddings.__init__0s !||F,=,=v?T?Tbhbubuv#%<<0N0NPVPePe#f %'\\&2H2H&J_J_%`"f&8&8f>S>STzz&"<"<= IIell63Q3Q&R&Y&YZa&botucVeVP4pMVP4RRpV^,pVfVPRRV13,pVf<\P!V\PVPP R7pVfVP V4pVPV4pVPV4pWG,V,p VPV 4p VPV 4p V #)NNNNdtypedevicer) sizer4r1zeroslongr@r%r'r)r*r/) r6 input_idstoken_type_idsr4 inputs_embeds input_shape seq_lengthr'r) embeddingss &&&&& r9forwardSqueezeBertEmbeddings.forward<s  #..*K',,.s3K ^  ,,Q  ^ J"8;PP ^^J/ \\*- r;)r*r/r'r4r)r%)NNNN __name__ __module__ __qualname____firstlineno____doc__r rJ__static_attributes____classdictcell__ __classcell__r8 __classdict__s@@r9rr-sQ vr;rc<aa]tRt^UtoRtV3RltRtRtVtV;t #) MatMulWrapperz Wrapper for torch.matmul(). This makes flop-counting easier to implement. Note that if you directly call torch.matmul() in your code, the flop counter will typically ignore the flops of the matmul. c$<\SV`4R#N)rr )r6r8s&r9r MatMulWrapper.__init__[s r;c .\P!W4#)a :param inputs: two torch tensors :return: matmul of these tensors Here are the typical dimensions found in BERT (the B is optional) mat1.shape: [B, , M, K] mat2.shape: [B, , K, N] output shape: [B, , M, N] )r1matmul)r6mat1mat2s&&&r9rJMatMulWrapper.forward^s||D''r;rLrUs@@r9rXrXUs ((r;rXc4a]tRt^itoRtRRltRtRtVtR#)SqueezeBertLayerNormz This is a nn.LayerNorm subclass that accepts NCW data layout and performs normalization in the C dimension. N = batch C = channels W = sequence length cH\PPWVR7R#))normalized_shaperN)rr*r )r6r+rs&&&r9r SqueezeBertLayerNorm.__init__ps dcJr;cVP^^^4p\PPW4pVP^^^4#))permuterr*rJ)r6xs&&r9rJSqueezeBertLayerNorm.forwardss; IIaA  LL  )yyAq!!r;raN)g-q=) rMrNrOrPrQr rJrRrS)rVs@r9rcrcis K""r;rcc<aa]tRt^ytoRtV3RltRtRtVtV;t #)ConvDropoutLayerNormz0 ConvDropoutLayerNorm: Conv, Dropout, LayerNorm c<\SV`4\P!W^VR7Vn\ V4Vn\P!V4VnR#r in_channels out_channels kernel_sizegroupsN) rr rConv1dconv1drc layernormr-r/)r6cincoutrt dropout_probr8s&&&&&r9r ConvDropoutLayerNorm.__init__~s@ iiCPQZ`a -d3zz,/ r;c|VPV4pVPV4pW2,pVPV4pV#rZrvr/rw)r6 hidden_states input_tensorrjs&&& r9rJConvDropoutLayerNorm.forwards8 KK & LLO  NN1 r;r}rLrUs@@r9rmrmys0r;rmc<aa]tRt^toRtV3RltRtRtVtV;t #)ConvActivationz" ConvActivation: Conv, Activation c<\SV`4\P!W^VR7Vn\ V,VnR#ro)rr rrurvract)r6rxryrtrr8s&&&&&r9r ConvActivation.__init__s/ iiCPQZ`a #;r;cFVPV4pVPV4#rZ)rvr)r6rjoutputs&& r9rJConvActivation.forwardsQxxr;)rrvrLrUs@@r9rrs   r;rcNaa]tRt^toRV3RlltRtRtRtRtRt Vt V;t #)SqueezeBertSelfAttentionc <\SV`4W!P,^8wd\RV RVP R24hVPVn\ W!P, 4VnVPVP ,Vn\P!W"^VR7Vn \P!W"^VR7Vn \P!W"^VR7Vn \P!VP4Vn\P!RR7Vn\#4Vn\#4VnR#)z config = used for some things; ignored for others (work in progress...) cin = input channels = output channels groups = number of groups to use in conv1d layers zcin (z6) is not a multiple of the number of attention heads ()rpdimNr)rr num_attention_heads ValueErrorintattention_head_size all_head_sizerruquerykeyvaluer-attention_probs_dropout_probr/SoftmaxsoftmaxrX matmul_qk matmul_qkv)r6r7rxq_groupsk_groupsv_groupsr8s&&&&&&r9r !SqueezeBertSelfAttention.__init__s  ++ +q 0uRSYSmSmRnnop $*#=#= #&s-G-G'G#H !558P8PPYY3aX`a 99AV^_YY3aX`a zz&"E"EF zzb) &'/r;c VP4^,VPVPVP4R,3pVP!V!pVP ^^^^4#)zg - input: [N, C, W] - output: [N, C1, W, C2] where C1 is the head index, and C2 is one head's contents r)rArrviewrir6rj new_x_shapes&& r9transpose_for_scores-SqueezeBertSelfAttention.transpose_for_scoress[ vvx{D$<$V>VXYX^X^X`acXde FFK yyAq!$$r;c VP4^,VPVPVP4R,3pVP!V!pV#)zg - input: [N, C, W] - output: [N, C1, C2, W] where C1 is the head index, and C2 is one head's contents r)rArrrrs&& r9transpose_key_for_scores1SqueezeBertSelfAttention.transpose_key_for_scoressK vvx{D$<$V>VXYX^X^X`acXde FFK r;c VP^^^^4P4pVP4^,VPVP4^,3pVP!V!pV#)z- - input: [N, C1, W, C2] - output: [N, C, W] )ri contiguousrArrrs&& r9transpose_output)SqueezeBertSelfAttention.transpose_outputsZ IIaAq ! , , .vvx{D$6$6 D FFK r;c VPV4pVPV4pVPV4pVPV4pVP V4pVPV4p VP Wx4p V \ P!VP4, p Ve W,p VPV 4p VPV 4p VPW4p VPV 4p RV /p V'dWR&V #)z expects hidden_states in [N, C, W] data layout. The attention_mask data layout is [N, W], and it does not need to be transposed. context_layerattention_score) rrrrrrmathsqrtrrr/rr)r6r~attention_maskoutput_attentionsmixed_query_layermixed_key_layermixed_value_layer query_layer key_layer value_layerrattention_probsrresults&&&& r9rJ SqueezeBertSelfAttention.forwards !JJ}5((=1 JJ}5//0AB 11/B //0AB ..@)DIId6N6N,OO  %->O,,7,,7E --m< !=1 (7$ % r;) rrr/rrrrrrr)rrr) rMrNrOrPr rrrrJrRrSrTrUs@@r9rrs#*0%""r;rc8aa]tRt^toV3RltRtRtVtV;t#)SqueezeBertModulec <\SV`4VPpVPpVPpVPp\ WVP VP VPR7Vn\W#VPVPR7Vn \W4VPVPR7Vn\WEVP"VPR7VnR#)aP - hidden_size = input chans = output chans for Q, K, V (they are all the same ... for now) = output chans for the module - intermediate_size = output chans for intermediate layer - groups = number of groups for all layers in the BertModule. (eventually we could change the interface to allow different groups for different layers) )r7rxrrr)rxryrtrz)rxryrtrN)rr r+intermediate_sizerrrr attentionrmpost_attention_groupsr.post_attentionrintermediate_groups hidden_act intermediate output_groupsr)r6r7c0c1c2c3r8s&& r9r SqueezeBertModule.__init__s         % %   1FOOfoo`f`o`o 3F$@$@vOiOi +r6C]C]cictctu*F$8$8vGaGa  r;cVPWV4pVR,pVPWQ4pVPV4pVPWv4pRV/p V'd VR,V R&V #)r feature_mapr)rrrr) r6r~rrattattention_outputpost_attention_outputintermediate_output layer_output output_dicts &&&& r9rJSqueezeBertModule.forwardsunn].SqueezeBertEncoder.__init__..+s#gGf!$5f$=$=GfsN) rr r#r+r ModuleListrangenum_hidden_layerslayersr5s&fr9r SqueezeBertEncoder.__init__"sW $$(:(::  2 : mm#guVMeMeGf#gg r;c(VP^^^4pV'dRMRpV'dRMRpVPFopV'd0VP^^^4pWa3, pVP^^^4pVPWV4p V R,pV'gK_WyR,3, pKq VP^^^4pV'd Wa3, pV'g1\;QJd.RWV34FNK 5#!RWV344#\ WVR7#)rhNrrc30"TF qfKVxK R#5irZra)rvs& r9r-SqueezeBertEncoder.forward..Osh$Vq$Vs )last_hidden_stater~ attentionsra)rirrJtupler ) r6r~rroutput_hidden_states return_dictall_hidden_statesall_attentionslayerrs &&&&&& r9rJSqueezeBertEncoder.forward-s &--aA6 "6BD0d[[E# - 5 5aA > !%55! - 5 5aA >  ==HYZL(7M  0A#B"DD!&--aA6  !1 1 5h]~$Vh5 h5h]~$Vhh h+Yg  r;)r)NFFTrrUs@@r9rr!s h% % r;rc8aa]tRtRtoV3RltRtRtVtV;t#)SqueezeBertPooleriUc<\SV`4\P!VPVP4Vn\P !4VnR#rZ)rr rLinearr+denseTanh activationr5s&&r9r SqueezeBertPooler.__init__Vs9 YYv1163E3EF '')r;c\VR,pVPV4pVPV4pV#)r=)r=rh)rr)r6r~first_token_tensor pooled_outputs&& r9rJSqueezeBertPooler.forward[s2+40 #56  6 r;)rrrrUs@@r9rrUs$ r;rc8aa]tRtRtoV3RltRtRtVtV;t#)"SqueezeBertPredictionHeadTransformidc<\SV`4\P!VPVP4Vn\ VP\4'd\VP,Vn MVPVn \P!VPVPR7Vn R#)rN) rr rrr+r isinstancerstrrtransform_act_fnr*r,r5s&&r9r +SqueezeBertPredictionHeadTransform.__init__es~ YYv1163E3EF f'' - -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STr;clVPV4pVPV4pVPV4pV#rZ)rrr*r6r~s&&r9rJ*SqueezeBertPredictionHeadTransform.forwardns4 =1 --m< }5 r;)r*rrrrUs@@r9rrdsUr;rc8aa]tRtRtoV3RltRtRtVtV;t#)SqueezeBertLMPredictionHeadiuc<\SV`4\V4Vn\P !VP VPRR7Vn\P!\P!VP44Vn R#)T)biasN) rr r transformrrr+r"decoder Parameterr1rBr r5s&&r9r $SqueezeBertLMPredictionHead.__init__vs[ ;FCyy!3!3V5F5FTR LLV->->!?@ r;cJVPV4pVPV4pV#rZ)r r rs&&r9rJ#SqueezeBertLMPredictionHead.forwards$}5  ]3 r;)r r r rrUs@@r9rrusAr;rc8aa]tRtRtoV3RltRtRtVtV;t#)SqueezeBertOnlyMLMHeadicD<\SV`4\V4VnR#rZ)rr r predictionsr5s&&r9r SqueezeBertOnlyMLMHead.__init__s 6v>r;c(VPV4pV#rZr)r6sequence_outputprediction_scoress&& r9rJSqueezeBertOnlyMLMHead.forwards ,,_=  r;rrrUs@@r9rrs?!!r;rcjaa]tRtRtoRt]P !4V3Rl4tV3RltRt Vt V;t #)SqueezeBertPreTrainedModeli transformerc <\SV`V4\V\4'd#\P !VP 4R#\V\4'db\P!VP\P!VPPR,4PR44R#R#)zInitialize the weightsNrr)r _init_weightsrrinitzeros_r rcopy_r4r1r2shaper3)r6moduler8s&&r9r(SqueezeBertPreTrainedModel._init_weightss~ f% f9 : : KK $  5 6 6 JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h7r;c&<V^8dQh/S[;R&#)r7r)formatrVs"r9 __annotate__'SqueezeBertPreTrainedModel.__annotate__s r;ra) rMrNrOrPbase_model_prefixr1no_gradr__annotate_func__rRrSrTrUs@@r9rrs.& ]]_ii r;rc^aa]tRtRtoV3RltRtRt]RV3RlRll4tRt Vt V;t #) SqueezeBertModelic<\SV`V4\V4Vn\ V4Vn\ V4VnVP4R#rZ) rr rrIrencoderrpooler post_initr5s&&r9r SqueezeBertModel.__init__s@  /7)&1 '/  r;c.VPP#rZrIr%r6s&r9get_input_embeddings%SqueezeBertModel.get_input_embeddingss...r;c&WPnR#rZr6r6new_embeddingss&&r9set_input_embeddings%SqueezeBertModel.set_input_embeddingss *8'r;c4<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[R,RS[R,R S[R,R S[S[,/ #) r'rDNrrEr4rFrrrreturn)r1Tensor FloatTensorboolrr )r(rVs"r9r)SqueezeBertModel.__annotate__s? ? <<$&?  t+?  t+ ? llT) ? ((4/ ?  $;? #Tk? D[?  + +? r;c  TVeTMVPPpVeTMVPPpVeTMVPPpVeVe \ R4hVe#VP W4VP 4p M#VeVP 4RR p M \ R4hVe VPM VPp Vf\P!WR7pVf(\P!V \PV R7pVPWW5R7p \VPV VR7pVPV VVVVR7p V ^,pVPV4pV'gW3V R,,#\!VVV P"V P$R 7#) NzDYou cannot specify both input_ids and inputs_embeds at the same timez5You have to specify either input_ids or inputs_embeds)r@r>)rDr4rErF)r7rFr)r~rrrr:rNN)r pooler_outputr~rr)r7rrrr%warn_if_padding_and_no_attention_maskrAr@r1onesrBrCrIrr1r2r r~r)r6rDrrEr4rFrrrkwargsrGr@embedding_outputencoder_outputsrrs&&&&&&&&&, r9rJSqueezeBertModel.forwards2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y Q#..*K  &',,.s3KTU U%.%:!! @T@T  !"ZZ CN  !"[[EJJvVN??>+ 3;;*)  ,,*)/!5# ' *!, O4 #3ob6II I)-')77&11   r;)rIr1r2)NNNNNNNN) rMrNrOrPr r8r=rrJrRrSrTrUs@@r9r/r/s2/9? ? ? ? r;r/cjaa]tRtRtoRRRR/tV3RltRtRt]R V3R lR ll4t R t Vt V;t #) SqueezeBertForMaskedLMizcls.predictions.decoder.biaszcls.predictions.biaszcls.predictions.decoder.weightz-transformer.embeddings.word_embeddings.weightc<\SV`V4\V4Vn\ V4VnVP 4R#rZ)rr r/rrclsr3r5s&&r9r SqueezeBertForMaskedLM.__init__s5  +F3)&1 r;cBVPPP#rZ)rPrr r7s&r9get_output_embeddings,SqueezeBertForMaskedLM.get_output_embeddingssxx##+++r;cWPPnVPVPPnR#rZ)rPrr r r;s&&r9set_output_embeddings,SqueezeBertForMaskedLM.set_output_embeddings s*'5$$2$7$7!r;c\<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[R,R S[R,R S[R,R S[S[,/ # r'rDNrrEr4rFlabelsrrrr@)r1rArCrr )r(rVs"r9r)#SqueezeBertForMaskedLM.__annotate__s1 1 <<$&1  t+1  t+ 1 llT) 1 ||d* 1  t#1  $;1 #Tk1 D[1   1 r;c V eT MVPPp VPVVVVVVVV R7p V ^,p VPV 4p RpVeG\ 4pV!V P RVPP 4VP R44pV 'g"V 3V R,,pVe V3V,#T#\VV V PV PR7#)az labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` NrrEr4rFrrrr'NNlosslogitsr~rr) r7rrrPrrr"r r~r)r6rDrrEr4rFrZrrrrIoutputsrrmasked_lm_lossloss_fctrs&&&&&&&&&&, r9rJSqueezeBertForMaskedLM.forward s(&1%(*Y ,81 1 1 1 r;rNz SqueezeBERT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ) custom_introcRaa]tRtRtoV3Rlt]RV3RlRll4tRtVtV;t #)$SqueezeBertForSequenceClassificationiBcR<\SV`V4VPVnWn\ V4Vn\ P!VP4Vn \ P!VPVPP4Vn VP4R#rZ)rr num_labelsr7r/rrr-r.r/rr+ classifierr3r5s&&r9r -SqueezeBertForSequenceClassification.__init__Iss   ++ +F3zz&"<"<= ))F$6$6 8N8NO r;c\<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[R,R S[R,R S[R,R S[S[,/ #rY)r1rArCrr)r(rVs"r9r)1SqueezeBertForSequenceClassification.__annotate__VsE E <<$&E  t+E  t+ E llT) E ||d* E  t#E  $;E #TkE D[E  ) )E r;c V eT MVPPp VPVVVVVVVV R7p V ^,p VPV 4p VP V 4p RpVEeVPP fVP ^8XdRVPnMrVP ^8dQVP\P8Xg VP\P8XdRVPnMRVPnVPP R8XdK\4pVP ^8Xd'V!V P4VP44pMV!W4pMVPP R8Xd>\4pV!V PRVP 4VPR44pM-VPP R8Xd\4pV!W4pV 'g"V 3V R,,pVe V3V,#T#\!VV V P"V P$R7#) ae labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nr] regressionsingle_label_classificationmulti_label_classificationr^r_r)r7rrr/rm problem_typerlr?r1rCrrsqueezerrrrr~r)r6rDrrEr4rFrZrrrrIrbrrar`rdrs&&&&&&&&&&, r9rJ,SqueezeBertForSequenceClassification.forwardUs(&1%G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  "" ))%'/!5##     ]3 / ++b+6  ')HO4D%''"+5F)-)9TGf$ Ev E("!//))   r;)rmr/rrfrxrUs@@r9rzrzs,W W W W r;rzcRaa]tRtRtoV3Rlt]RV3RlRll4tRtVtV;t #)!SqueezeBertForTokenClassificationic2<\SV`V4VPVn\V4Vn\ P !VP4Vn\ P!VPVP4Vn VP4R#rZ) rr rlr/rrr-r.r/rr+rmr3r5s&&r9r *SqueezeBertForTokenClassification.__init__sj   +++F3zz&"<"<= ))F$6$68I8IJ r;c\<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[R,R S[R,R S[R,R S[S[,/ #rY)r1rArCrr)r(rVs"r9r).SqueezeBertForTokenClassification.__annotate__s1 1 <<$&1  t+1  t+ 1 llT) 1 ||d* 1  t#1  $;1 #Tk1 D[1  & &1 r;c V eT MVPPp VPVVVVVVVV R7p V ^,p VPV 4p VP V 4p RpVe=\ 4pV!V P RVP4VP R44pV 'g"V 3V R,,pVe V3V,#T#\VV V PV PR7#)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nr]r^r_r) r7rrr/rmrrrlrr~r)r6rDrrEr4rFrZrrrrIrbrrar`rdrs&&&&&&&&&&, r9rJ)SqueezeBertForTokenClassification.forwards$&1%rs! AA&!6.9   H %%BII%P(BII(( "2<< " 299(  RYY  XryyXv' 'T1 1 h  ""))&!RYY! i i iQ 1Q Q hH 7H H V S +ES  S lc #=c c L> (B> > BI &@I I X r;