+ j Rt^RIHt^RIt^RIt^RIHt^RIHtH t H t ^RI H t ^RIHt^RIHt^RIHt^R IHt^R IHt^R IHtHtHtHtHtHt^R IH t H!t!^R I"H#t#^RI$H%t%^RI&H't'H(t(H)t)^RI*H+t+H,t,^RI-H.t.^RI/H0t0])Pb!]24t3RRlt4RRlt5!RR]Pl4t7R4RRllt8!RR]Pl4t9!RR]Pl4t:!RR ]4t;!R!R"]Pl4t<](!R#R$]!44t=](!R%R&]=44t>](!R'R(7!R)R*]=44t?](!R+R(7!R,R-]=44t@](!R.R/]=44tA](!R0R1]=44tB](!R2R3]=44tC.R5OtDR#)6z PyTorch DistilBERT model adapted in part from Facebook, Inc XLM model (https://github.com/facebookresearch/XLM) and in part from HuggingFace PyTorch version of Google AI Bert model (https://github.com/google-research/bert) )CallableN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)get_activationPreTrainedConfig)is_deepspeed_zero3_enabled)create_bidirectional_mask)GradientCheckpointingLayer)BaseModelOutputMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward)TransformersKwargsauto_docstringlogging)can_return_tuplemerge_with_config_defaults)capture_outputsDistilBertConfigcPV^8dQhR\R\R\P/#n_posdimoutinttorchTensor)formats"s/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/distilbert/modeling_distilbert.py __annotate__r,>s*LLL#LELLLcB\4'dp^RIpVPPV^R7;_uu_4\P P 4^8Xd\WVR7uuRRR4#RRR4R#\WVR7# +'giR#;i)N) modifier_rankr#r$r%)r deepspeedzeroGatheredParametersr( distributedget_rank_create_sinusoidal_embeddings)r#r$r%r2s&&& r+create_sinusoidal_embeddingsr8>sz!## ^^ . .s! . D D  ))+q045sSE D0E D-5sKK E D Ds 0B  B cPV^8dQhR\R\R\P/#r!r&)r*s"r+r,r,Is%3U\\r-c\P!\V4UUu.uFNp\V4Uu.uF5qC\P!R^V^,,V, 4, NK7 upNKP upp4pRVn\ P !\P!VR,44VR&\ P !\P!VR,44VR&VP4V#uupiuuppi)i'F)NNN:r/Nr")r;:Nr") nparrayrangepower requires_gradr( FloatTensorsincosdetach_)r#r$r%posj position_encs&&& r+r7r7Is88hmnshtuhtadQVWZQ[\Q[ABHHUAaL34F$GGGQ[\htuvLC$$RVVL,A%BCCL$$RVVL,A%BCCLKKM J ]usD ;C<,D <D cTaa]tRt^RtoV3RlV3RlltRV3RlRlltRtVtV;t#) Embeddingsc <V^8dQhRS[/#r"configr )r* __classdict__s"r+r,Embeddings.__annotate__Ssvv/vr-c(<\SV`4\P!VPVP VP R7Vn\P!VPVP 4Vn \P!VP RR7Vn \P!VP4Vn \P!\P!VP4P!R4RR7VnR#)) padding_idx-q=epsF) persistentNr<)super__init__r Embedding vocab_sizer$ pad_token_idword_embeddingsmax_position_embeddingsposition_embeddings LayerNormDropoutdropoutBufferr(arangeexpand position_idsselfrM __class__s&&r+rYEmbeddings.__init__Ss !||F,=,=vzzW]WjWjk#%<<0N0NPVPZPZ#[ fjje<zz&..1 IIell63Q3Q&R&Y&YZa&botur-c<V^8dQhRS[PRS[PR,RS[PR,RS[P/#)r" input_ids inputs_embedsNrfreturn)r(r) LongTensor)r*rNs"r+r,rO\sO<<||d*&&-   r-cVeVPV4pVP^4pVf|\VR4'dVPRRV13,pMQ\P !V\P VPR7pVP^4PV4pVPV4pW%,pVPV4pVPV4pV#)Nrfr;)dtypedevice) r]sizehasattrrfr(rdlongrr unsqueeze expand_asr_r`rb)rhrlrmrf seq_lengthr_ embeddingss&&&& r+forwardEmbeddings.forward\s   00;M"''*  t^,,#00KZK@ $||JejjQZQaQab +55a8BB9M "66|D"8 ^^J/ \\*- r-)r`rbr_rfr])NN __name__ __module__ __qualname____firstlineno__rYrz__static_attributes____classdictcell__ __classcell__rirNs@@r+rJrJRs vvr-rJcV^8dQhR\PR\PR\PR\PR\PR,R\R,R\R \ \ ,/#) r"modulequerykeyvalueattention_maskNscalingrbkwargs)rModuler(r)floatrr)r*s"r+r,r,zs{%% II% <<% % << % LL4' % T\ %%'(%r-c VfVPR4R,p\P!WP^^44V,pVe W,p\P P VRR7p\P PWVPR7p\P!W4p V P^^4P4p W3#)Nr$)ptrainingrW) rsr(matmul transposer functionalsoftmaxrbr contiguous) rrrrrrrbr attn_weights attn_outputs &&&&&&&, r+eager_attention_forwardrzs**R.D(<<}}Q':;gEL!#4 ==((2(>L==((6??([L,,|3K''1-88:K  $$r-cTaa]tRt^toV3RlV3RlltRV3RlRlltRtVtV;t#)DistilBertSelfAttentionc <V^8dQhRS[/#rLr )r*rNs"r+r,$DistilBertSelfAttention.__annotate__s/r-cZ<\SV`4WnVPVnVPVnVPVP,VnVP R,VnVPVP,^8wd'\RVP RVP R24h\P!VPVPR7Vn \P!VPVPR7Vn \P!VPVPR7Vn \P!VPVPR7Vn \P!VPR7VnRVnR#) g?zself.n_heads: z must divide self.dim:  evenly in_features out_featuresrFNr)rXrYrMn_headsr$attention_head_sizer ValueErrorrLinearq_link_linv_linout_linraattention_dropoutrb is_causalrgs&&r+rY DistilBertSelfAttention.__init__s  ~~ ::#'88t||#; //5  88dll "a '~dll^;RSWS[S[R\\cde eYY6::FJJO YY6::FJJO YY6::FJJO yyVZZfjjQ zzF$<$<= r-c <V^8dQhRS[PRS[PR,RS[S[,RS[S[P,/#r" hidden_statesrNrrn)r(r)rBrrtuple)r*rNs"r+r,rsQ))||)))D0)+, ) u||  )r-c VPRRp.VORNVPN5pVPV4P!V!P ^^4pVP V4P!V!P ^^4pVP V4P!V!P ^^4p\P!VPP\4p T !VVVVV3RVP'gRMVPPRVP/VBwrV P !.VORN5!P#4p VP%V 4p W3#)NrbrrW)shaperrviewrrrr get_interfacerM_attn_implementationrrrbrrreshaperr) rhrrr input_shape hidden_shape query_layer key_layer value_layerattention_interfacerrs &&&, r+rzDistilBertSelfAttention.forwardsS $))#2. CCbC$*B*BC jj/44lCMMaQRS JJ}-22LAKKAqQ jj/44lCMMaQRS (?(M(M KK , ,.E) %8      % $}}}C$,,.. % LL %  % ! "));;;;FFH ll;/ ((r-) rrMr$rbrrrrrrrNr|rs@@r+rrs,)))r-rcbaa]tRt^toV3RlV3RlltV3RlRltV3RlRltRtVtV;t #)FFNc <V^8dQhRS[/#rLr )r*rNs"r+r,FFN.__annotate__s<</>VW]WaWaVbbijk k08\\6::5Qv;!#vzzu!Ur-c<V^8dQhRS[PRS[PR,RS[S[,RS[P/#r)r(r)rr)r*rNs"r+r,rsJ|| t++,   r-c VP!V3RV/VBwrEVPWA,4pVPV4pVPWd,4pV#)r)rrrr)rhrrrattention_output_ ffn_outputs&&&, r+rzTransformerBlock.forwardsl#nn  )    --.>.NOXX./ ++J,IJ r-)rrrrrr|rs@@r+rrs  V Vr-rcTaa]tRtRtoV3RlV3RlltRV3RlRlltRtVtV;t#) Transformeric <V^8dQhRS[/#rLr )r*rNs"r+r,Transformer.__annotate__s,,/,r-c<\SV`4VPVn\P!\ VP4Uu.uFp\ V4NK up4VnRVnR#uupi)FN) rXrYn_layersr ModuleListr?rlayergradient_checkpointing)rhrMrris&& r+rYTransformer.__init__sW  ]]eFOOF\#]F\$4V$?Z>[[_`a.I +)-)L)L)S)S)Y)Y)[&.0ll4;;;^;^`d`k`k`o`o.p+ ;; + + + (kk99t{{TXTlTlTsTs +a/]_]i]i6^DOO77>>?YAY@YZBD67P8PQBDOO77> ! ++..t{{;!s A3G;; H c4<V^8dQhRS[P/#rr )r*rNs"r+r,rxs//bll/r-c.VPP#rryr]r#s&r+get_input_embeddings$DistilBertModel.get_input_embeddingsxs...r-c4<V^8dQhRS[P/#r"new_embeddingsr )r*rNs"r+r,r{s992<<9r-c&WPnR#rr7rhr<s&&r+set_input_embeddings$DistilBertModel.set_input_embeddings{s *8'r-c<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[S[,RS[S[S[PR3,,/#) r"rlNrrmrfrrn.)r(r)rrrr)r*rNs"r+r,rs$ $ <<$&$  t+$ ||d* $ llT) $ +, $  5s!23 3$ r-c VRJVRJ, 'd \R4hVPWV4p\VPVVR7pVP!RRVRV/VB#) input_ids (`torch.LongTensor` of shape `(batch_size, num_choices)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Nz:You must specify exactly one of input_ids or inputs_embeds)rMrmrrrr )rryr rMr)rhrlrrmrfrrys&&&&&, r+rzDistilBertModel.forward~s{0  -t"; < <YZ Z__Y|L 2;;$)   $ )   r-)ryr)NNNN)r}r~rrrYr$r3r8r?rrrrzrrrrs@@r+rr=sf33 (<(s&&r+set_output_embeddings+DistilBertForMaskedLM.set_output_embeddingss-r-c6<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[S[,RS[S[S[PR 3,,/# r"rlNrrmlabelsrfrrn.)r(r)rorrrr)r*rNs"r+r,rIs1 1 <<$&1  t+1 ||d* 1   4' 1 llT) 1 +,1  % c 12 21 r-c VP!R RVRVRVRVRR/VBpV^,pVPV4p VPV 4p VPV 4p VP V 4p Rp VeAVP V P R V PR 44VP R 44p \V V VPVPR7#) a* input_ids (`torch.LongTensor` of shape `(batch_size, num_choices)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. rlrrmrf return_dictTNlosslogitsrrr rW) rrKrrLrMrNrrsrrr) rhrlrrmrirfr dlbrt_outputrprediction_logitsmlm_losss &&&&&&, r+rzDistilBertForMaskedLM.forwards8  ) ( &      %Q  00? OO,=> 112CD 001BC  (():)?)?DUDZDZ[]D^)_agalalmoapqH$&44#..   r-)rrrNrLrMrKNNNNN)r}r~rr_tied_weights_keysrYr$r3rarerrrzrrrrs@@r+rGrGsm 34bc 2 299 P P$$..1 1 1 1 r-rGz DistilBert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. caa]tRtRtoV3RlV3RlltV3RlRltV3RlRlt]]R V3RlR ll44t R t Vt V;t #) #DistilBertForSequenceClassificationic <V^8dQhRS[/#rLr )r*rNs"r+r,0DistilBertForSequenceClassification.__annotate__  / r-c<\SV`V4VPVnWn\ V4Vn\ P!VPVP4Vn \ P!VPVP4Vn \ P!VP4Vn VP4R#r)rXrY num_labelsrMrrrrr$pre_classifier classifierraseq_classif_dropoutrbrrgs&&r+rY,DistilBertForSequenceClassification.__init__s   ++ )&1 ii FJJ?))FJJ0A0ABzz&"<"<=  r-c4<V^8dQhRS[P/#rr )r*rNs"r+r,rx#rQr-c 6VPP4#r"rSr#s&r+r$;DistilBertForSequenceClassification.get_position_embeddings#rUr-c <V^8dQhRS[/#r'r))r*rNs"r+r,rx)rWr-c <VPPV4R#rYrZr[s&&r+r3>DistilBertForSequenceClassification.resize_position_embeddings)r]r-c6<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[S[,RS[S[S[PR 3,,/#rh)r(r)rorrrr)r*rNs"r+r,rx9s: : <<$&:  t+: ||d* :   4' : llT) : +,:  "E%,,*;$< <: r-c |VP!R RVRVRVRVRR/VBpV^,pVR ,p VPV 4p \P!4!V 4p VP V 4p VP V 4p Rp VEeVP PfVP^8XdRVP nMrVP^8dQVP\P8Xg VP\P8XdR VP nMR VP nVP PR8XdK\4p VP^8Xd'V !V P4VP44p MV !W4p MVP PR 8Xd>\4p V !V P!RVP4VP!R44p M-VP PR 8Xd\#4p V !W4p \%V V VP&VP(R 7#)ae labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). rlrrmrfrkTN regressionsingle_label_classificationmulti_label_classificationrlr r;r/rW)rr|rReLUrbr}rM problem_typer{rqr(rur'rsqueezerrrrrr) rhrlrrmrirfrdistilbert_output hidden_state pooled_outputrnrmloss_fcts &&&&&&, r+rz+DistilBertForSequenceClassification.forward7s"!OO  ) ( &      )+ $T* ++M:  -0  ]3 /  {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#FNN$4fnn6FGD#F3D))-JJ+- B @&++b/R))-II,./'+99(33   r-)r}rMrrbr{r|rs r}r~rrrYr$r3rrrzrrrrs@@r+rvrvsK  99 P P: : : : r-rvcaa]tRtRtoV3RlV3RlltV3RlRltV3RlRlt]]R V3RlR ll44t R t Vt V;t #) DistilBertForQuestionAnsweringivc <V^8dQhRS[/#rLr )r*rNs"r+r,+DistilBertForQuestionAnswering.__annotate__xryr-cb<\SV`V4\V4Vn\P !VP VP4VnVP^8wd\RVP 24h\P!VP4Vn VP4R#)r"z)config.num_labels should be 2, but it is N)rXrYrrrrr$r{ qa_outputsrra qa_dropoutrbrrgs&&r+rY'DistilBertForQuestionAnswering.__init__xs  )&1))FJJ0A0AB    !HIZIZH[\] ]zz&"3"34  r-c4<V^8dQhRS[P/#rr )r*rNs"r+r,rrQr-c 6VPP4#r"rSr#s&r+r$6DistilBertForQuestionAnswering.get_position_embeddingsrUr-c <V^8dQhRS[/#r'r))r*rNs"r+r,rrWr-c <VPPV4R#rYrZr[s&&r+r39DistilBertForQuestionAnswering.resize_position_embeddingsr]r-c^<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[S[,R S[S[S[PR 3,,/#) r"rlNrrmstart_positions end_positionsrfrrn.)r(r)rrrr)r*rNs"r+r,rs> > <<$&>  t+> ||d* > , > ||d* > llT)> +,>  &ellC.?(@ @> r-c &VP!R RVRVRVRVRR/VBpV^,p VPV 4p VPV 4p V P^R R7wrV P R 4P 4p V P R 4P 4p Rp VeVe\ VP44^8dVP R 4p\ VP44^8dVP R 4pV P^4pVP^V4pVP^V4p\P!VR 7pV!W4pV!W4pVV,^, p \V V V VPVPR 7#) rCrlrrmrfrkTrN) ignore_index)rm start_logits end_logitsrrr rW)rrbrsplitrrlenrsclamprrrrr)rhrlrrmrrrfrrrrnrr total_loss ignored_indexr start_lossend_losss&&&&&&&, r+rz&DistilBertForQuestionAnswering.forwards2!OO  ) ( &      *!,  ]3 /#)<<r<#: #++B/::< ''+668   &=+D?'')*Q."1"9"9""==%%'(1, - 5 5b 9 (--a0M-33A}EO)//=AM** FH!,@J :H$x/14J+%!+99(33   r-)rrbr)NNNNNNrrs@@r+rrvsK  99 P P> > > > r-rcaa]tRtRtoV3RlV3RlltV3RlRltV3RlRlt]]R V3RlR ll44t R t Vt V;t #) DistilBertForTokenClassificationic <V^8dQhRS[/#rLr )r*rNs"r+r,-DistilBertForTokenClassification.__annotate__  / r-c2<\SV`V4VPVn\V4Vn\ P !VP4Vn\ P!VPVP4Vn VP4R#r) rXrYr{rrrrarbr hidden_sizer}rrgs&&r+rY)DistilBertForTokenClassification.__init__sg   ++)&1zz&..1 ))F$6$68I8IJ r-c4<V^8dQhRS[P/#rr )r*rNs"r+r,rrQr-c 6VPP4#r"rSr#s&r+r$8DistilBertForTokenClassification.get_position_embeddingsrUr-c <V^8dQhRS[/#r'r))r*rNs"r+r,rrWr-c <VPPV4R#rYrZr[s&&r+r3;DistilBertForTokenClassification.resize_position_embeddingsr]r-c6<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[S[,RS[S[S[PR 3,,/#rh)r(r)rorrrr)r*rNs"r+r,rs% % <<$&%  t+% ||d* %   4' % llT) % +,%  u||S'8!9 9% r-c XVP!V3RVRVRVRR/VBpV^,pVPV4pVPV4p Rp Ve=\4p V !V P RVP 4VP R44p \ V V VPVPR7#) z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. rrmrfrkTNrlrW) rrbr}rrr{rrr) rhrlrrmrirfroutputssequence_outputrnrmrs &&&&&&, r+rz(DistilBertForTokenClassification.forwards//  ) ( &      "!*,,71  ')HFKKDOO>> from transformers import AutoTokenizer, DistilBertForMultipleChoice >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("distilbert-base-cased") >>> model = DistilBertForMultipleChoice.from_pretrained("distilbert-base-cased") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> choice0 = "It is eaten with a fork and a knife." >>> choice1 = "It is eaten while held in the hand." >>> labels = torch.tensor(0).unsqueeze(0) # choice0 is correct (according to Wikipedia ;)), batch size 1 >>> encoding = tokenizer([[prompt, choice0], [prompt, choice1]], return_tensors="pt", padding=True) >>> outputs = model(**{k: v.unsqueeze(0) for k, v in encoding.items()}, labels=labels) # batch size is 1 >>> # the linear classifier still needs to be trained >>> loss = outputs.loss >>> logits = outputs.logits ```NrrmrfrkTrlrWr) rrrsrr|rrrbr}rrrr)rhrlrrmrirfr num_choicesrrrrnreshaped_logitsrmrs&&&&&&, r+rz#DistilBertForMultipleChoice.forwardHsb-6,Aiooa(}GZGZ[\G] >G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImq(   r=#5#5b#9=;M;Mb;Q R  //  ) ( &      qz $T* ++M:  -0  ]3 / ++b+6  ')HO4D("!//))   r-)r}rrbr|rsrrs@@r+rr'sO  99 P PU U U U r-r)Nr)rGrrrvrrr)E__doc__collections.abcrnumpyr=r(rtorch.nnrrrrr activationsrconfiguration_utilsr integrations.deepspeedr masking_utilsr modeling_layersr modeling_outputsrrrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsrrr utils.genericrrutils.output_capturingrconfiguration_distilbertr get_loggerr}r,r8r7rrJrrrrrrrrGrvrrr__all__r r-r+rs % AA&)3@69G& J56   H % L$$P%85)bii5)p"))* 1 F@"))@0iii<g /g g T _ 5_  _ D ^ *C^  ^ Bb %>b b JG '@G G Tw ";w w t r-