+ ju{Rt^RIt^RIHt^RIHtHtHt^RIHt ^RI H t ^RI H t ^RIHtHtHtHtHtHtHt^RIHt^RIHt^R IHtHtHt^R IHt^R I H!t!H"t"H#t#H$t$H%t%^R I&H't']PP!])4t*!R R]"4t+!RR]%4t,!RR]!4t-!RR]#4t.]!RR]44t/!RR]$4t0]!RR7!RR]/] 44t1]!RR]/44t2!RR ]Pf4t4]!R!R7!R"R#]/44t5]!R$R%]/44t6]!R&R']/44t7!R(R)]Pf4t8]!R*R+]/44t9.R,Ot:R#)-zPyTorch RoBERTa model.N)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)gelu)GenerationMixin),BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging)can_return_tuple)BertCrossAttentionBertEmbeddings BertLayer BertModelBertSelfAttention) RobertaConfigclaa]tRt^,toV3RltRV3RlRllt]R4t]RRl4tRt Vt V;t #) RobertaEmbeddingsc<\SV`V4V=V=VPVn\ P !VPVPVPR7VnR#)) padding_idxN) super__init__ pad_token_idposition_embeddingsrnn Embeddingmax_position_embeddings hidden_sizeselfconfig __class__s&&l/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/roberta/modular_roberta.pyr RobertaEmbeddings.__init__-sT     $!..#%<<  * *F,>,>DL\L\$  c <V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[/#) input_idsNtoken_type_ids position_ids inputs_embedspast_key_values_length)torch LongTensor FloatTensorint)format __classdict__s"r+ __annotate__RobertaEmbeddings.__annotate__8si..##d*.((4/.&&- . ((4/ . !$ .r-c4Vf=VeVPWPV4pMVPW@P4pVeVP4pMVP4RRpVwrxVf\ VR4'drVP P VP4PVP^,R4p \P!V ^VR7p V PWx4pM;\P!V\PVPPR7pVfVPV4pVP!V4p WJ,p VP#V4p W,p VP%V 4p VP'V 4p V #)Nr1)dimindexdtypedevice)"create_position_ids_from_input_idsr&create_position_ids_from_inputs_embedssizehasattrr1torBexpandshaper5gatherzeroslongr2word_embeddingstoken_type_embeddingsr" LayerNormdropout) r(r0r1r2r3r4 input_shape batch_size seq_lengthbuffered_token_type_idsrO embeddingsr"s &&&&&& r+forwardRobertaEmbeddings.forward8st  $#FF//1G   $JJ=ZjZjk  #..*K',,.s3K!,  !t-..*.*=*=*@*@ATAT*U*\*\]i]o]opq]rtv*w'*/,,7NTU]i*j'!8!?!? !W!&[ SWSdSdSkSk!l   00;M $ : :> J": "66|D5 ^^J/ \\*- r-c VP4RRpV^,p\P!V^,W1,^,\PVPR7pVP ^4P V4#)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr@rC)rFr5arangerMrB unsqueezerI)r3rrRsequence_lengthr2s&& r+rE8RobertaEmbeddings.create_position_ids_from_inputs_embedshsn$((*3B/ %a.|| !O_:Q>ejjYfYmYm %%a(// <)ner8r5cumsumtype_asrM)r0rr4maskincremental_indicess&&& r+rD4RobertaEmbeddings.create_position_ids_from_input_idszsW||K(,,.$||Da8@@FI__cgg"'')K77r-)rr")NNNN)rf) __name__ __module__ __qualname____firstlineno__r rW staticmethodrErD__static_attributes____classdictcell__ __classcell__r*r:s@@r+rr,sB  ..`==" 8 8 8r-rc]tRt^tRtR#)RobertaSelfAttentionNrgrhrirjrlrrr-r+rqrqr-rqc]tRt^tRtR#)RobertaCrossAttentionrrNrsrrr-r+rvrvrtr-rvc]tRt^tRtR#) RobertaLayerrrNrsrrr-r+rxrxrtr-rxcaa]tRt^to]tRtRtRtRt Rt Rt R] R] R]/t]P"!4V3Rl4tRtVtV;t#)RobertaPreTrainedModelrobertaT hidden_states attentionscross_attentionsc <\SV`V4\V\4'd#\P !VP 4R#\V\4'd\P!VP\P!VPPR,4PR44\P !VP4R#R#)zInitialize the weightsNrC)rC)r _init_weights isinstance RobertaLMHeadinitzeros_biasrcopy_r2r5rZrJrIr1)r(moduler*s&&r+r$RobertaPreTrainedModel._init_weightss f% fm , , KK $  1 2 2 JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h KK-- .3r-rr)rgrhrirjr config_classbase_model_prefixsupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrxrqrv_can_record_outputsr5no_gradrrlrmrnros@@r+rzrzse L!&*#N"&*1  ]]_///r-rzc6aa]tRt^toRV3RlltRtVtV;t#) RobertaModelc&<\SV`W4R#N)rr )r(r)add_pooling_layerr*s&&&r+r RobertaModel.__init__s &r-rr)T)rgrhrirjr rlrmrnros@@r+rrs''r-rzS RoBERTa Model with a `language modeling` head on top for CLM fine-tuning. ) custom_introctaa]tRt^toRRRR/tV3RltRtRt]] R V3RlR ll44t R t Vt V;t #) RobertaForCausalLMlm_head.decoder.weight)roberta.embeddings.word_embeddings.weightlm_head.decoder.bias lm_head.biasc<\SV`V4VP'g\P R4\ VRR7Vn\V4VnVP4R#)zOIf you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`FrN rr is_decoderloggerwarningrr{rlm_head post_initr's&&r+r RobertaForCausalLM.__init__sO     NNl m#FeD $V,  r-c.VPP#rrdecoderr(s&r+get_output_embeddings(RobertaForCausalLM.get_output_embeddings||###r-c&WPnR#rrr(new_embeddingss&&r+set_output_embeddings(RobertaForCausalLM.set_output_embeddings - r-c0<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[PR,R S[PR,R S[S[S[P,,R,R S[R,R S[S[P ,R S[S[,RS[S[P ,S[ ,/ #)r/r0Nattention_maskr1r2r3encoder_hidden_statesencoder_attention_masklabelspast_key_values use_cachelogits_to_keepkwargsreturn) r5r6r7tupleboolr8Tensorrrr )r9r:s"r+r;RobertaForCausalLM.__annotate__s+O O ##d*O ))D0O ((4/ O &&- O ((4/ O  %0047O !& 1 1D 8O   4'O uU%6%6784?O $;O ell*O +,O  u|| @ @O r-c  VeRp VP!V3RVRVRVRVRVRVR V R V R R / V Bp V Pp\V \4'd\ V )R4MT pVP VR VR 3,4pRpVe-VP !RRVRVRVPP/V Bp\VVV PV PV PV PR7#)a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Example: ```python >>> from transformers import AutoTokenizer, RobertaForCausalLM, AutoConfig >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base") >>> config = AutoConfig.from_pretrained("FacebookAI/roberta-base") >>> config.is_decoder = True >>> model = RobertaForCausalLM.from_pretrained("FacebookAI/roberta-base", config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits ```NFrr1r2r3rrrr return_dictTNNNlogitsr vocab_size)lossrrr|r}r~rr)r{last_hidden_staterr8slicer loss_functionr)rr rr|r}r~)r(r0rr1r2r3rrrrrrroutputsr| slice_indicesrrs&&&&&&&&&&&&, r+rWRobertaForCausalLM.forwardsB`  I@D  A ) A * A & A ( A #8 A $: A , A   A  A  A  11 8B>SV8W8W~ot4]k mA}a,?@A  %%pVpFpt{{OeOepiopD0#33!//))$55   r-rr{) NNNNNNNNNNrfrgrhrirj_tied_weights_keysr rrrrrWrlrmrnros@@r+rrsS !"M $.O O O O r-rctaa]tRtRtoRRRR/tV3RltRtRt]] R V3R lR ll44t R t Vt V;t #) RobertaForMaskedLMi'rrrrc<\SV`V4VP'd\P R4\ VRR7Vn\V4VnVP4R#)znIf you want to use `RobertaForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.FrNrr's&&r+r RobertaForMaskedLM.__init__.sU      NN1  $FeD $V,  r-c.VPP#rrrs&r+r(RobertaForMaskedLM.get_output_embeddings=rr-c&WPnR#rrrs&&r+r(RobertaForMaskedLM.set_output_embeddings@rr-c<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[PR,R S[PR,R S[S[,R S[S[P ,S[,/ #) r/r0Nrr1r2r3rrrrr)r5r6r7rrrrr )r9r:s"r+r;RobertaForMaskedLM.__annotate__Es5 5 ##d*5 ))D05 ((4/ 5 &&- 5 ((4/ 5  %00475 !& 1 1D 85   4'5 +,5  u|| ~ -5 r-c  VP!V3RVRVRVRVRVRVRR/V Bp V ^,p VPV 4p R p VebVPV P4p\ 4pV!V P R VP P4VP R 44p \V V V PV PR 7#) a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` rr1r2r3rrrTNrrr|r}rC) r{rrHrBrviewr)rr r|r})r(r0rr1r2r3rrrrrsequence_outputprediction_scoresmasked_lm_lossloss_fcts&&&&&&&&&, r+rWRobertaForMaskedLM.forwardCs:,,   )  *  &  (  #8  $:      "!* LL9  YY0778F')H%&7&<&->!?@ r-c VPV4p\V4pVPV4pVPV4pV#r)rrrrr(featuresrxs&&, r+rWRobertaLMHead.forwards; JJx  G OOA  LLOr-)rrrr rgrhrirj__doc__r rWrlrmrnros@@r+rr}s4Ar-rz RoBERTa Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c\aa]tRtRtoV3Rlt]]RV3RlRll44tRtVt V;t #) RobertaForSequenceClassificationic<\SV`V4VPVnWn\ VRR7Vn\ V4VnVP4R#FrN) rr num_labelsr)rr{RobertaClassificationHead classifierrr's&&r+r )RobertaForSequenceClassification.__init__sH   ++ #FeD 3F; r-cZ<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[S[,R S[S[P ,S[,/# r/r0Nrr1r2r3rrr)r5r6r7rrrrr )r9r:s"r+r;-RobertaForSequenceClassification.__annotate__sC C ##d*C ))D0C ((4/ C &&- C ((4/ C   4'C +,C  u|| 7 7C r-c (VP!V3RVRVRVRVRR/VBpV^,p VPV 4p Rp VEeVPV P4pVPP fVP ^8XdRVPnMrVP ^8dQVP\P8Xg VP\P8XdR VPnMR VPnVPP R8XdK\4p VP ^8Xd'V !V P4VP44p MV !W4p MVPP R 8Xd>\4p V !V PR VP 4VPR 44p M-VPP R 8Xd\4p V !W4p \!V V VP"VP$R 7#) a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). rr1r2r3rTN regressionsingle_label_classificationmulti_label_classificationrrC)r{rrHrBr) problem_typerrAr5rMr8rsqueezerrrr r|r} r(r0rr1r2r3rrrrrrrs &&&&&&&, r+rW(RobertaForSequenceClassification.forwards6,,  ) * &  (     "!*1  YYv}}-F{{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#FNN$4fnn6FGD#F3D))-JJ+- B @&++b/R))-II,./'!//))   r-)rr)rr{NNNNNN rgrhrirjr rrrWrlrmrnros@@r+rrs4 C C C C r-rc\aa]tRtRtoV3Rlt]]RV3RlRll44tRtVt V;t #)RobertaForMultipleChoiceic<\SV`V4\V4Vn\P !VP 4Vn\P!VP^4Vn VP4R#)rN) rr rr{r#Dropouthidden_dropout_probrQrr&rrr's&&r+r !RobertaForMultipleChoice.__init__sV  #F+ zz&"<"<= ))F$6$6: r-cZ<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[S[,R S[S[P ,S[,/#) r/r0Nr1rrr2r3rr)r5r6r7rrrrr )r9r:s"r+r;%RobertaForMultipleChoice.__annotate__sP P ##d*P ((4/P ))D0 P   4' P &&- P ((4/P +,P  u|| 8 8P r-c ^VeVP^,MVP^,pVe"VPR VPR 44MRp Ve"VPR VPR 44MRp Ve"VPR VPR 44MRp Ve"VPR VPR 44MRp Ve2VPR VPR 4VPR 44MRp VP!V 3RV RV RV RV RR/VBpV^,pVP V4pVP V4pVPR V4pRpVe/VP VP4p\4pV!VV4p\VVVPVPR7#) aO input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Nr2r1rr3rTrrC) rJrrFr{rQrrHrBrr r|r})r(r0r1rrr2r3r num_choicesflat_input_idsflat_position_idsflat_token_type_idsflat_attention_maskflat_inputs_embedsr pooled_outputrreshaped_logitsrrs&&&&&&&, r+rW RobertaForMultipleChoice.forwardsV-6,Aiooa(}GZGZ[\G] CLCXINN2,>?^bLXLdL--b,2C2CB2GHjnR`Rln11"n6I6I"6MNrvR`Rln11"n6I6I"6MNrv(   r=#5#5b#9=;M;Mb;Q R  ,,  * / /  -        ]3 / ++b+6  YY556F')HOV4D("!//))   r-)rrQr{rr ros@@r+r r s4P P P P r-r c\aa]tRtRtoV3Rlt]]RV3RlRll44tRtVt V;t #)RobertaForTokenClassificationiNcn<\SV`V4VPVn\VRR7VnVP e VP M VP p\P!V4Vn \P!VPVP4Vn VP4R#r)rr rrr{classifier_dropoutrr#r rQrr&rrr(r)r r*s&& r+r &RobertaForTokenClassification.__init__Ps   ++#FeD )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ r-cZ<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[S[,R S[S[P ,S[,/#r)r5r6r7rrrrr)r9r:s"r+r;*RobertaForTokenClassification.__annotate__`s2 2 ##d*2 ))D02 ((4/ 2 &&- 2 ((4/ 2   4'2 +,2  u|| 4 42 r-c VP!V3RVRVRVRVRR/VBpV^,p VPV 4p VPV 4p Rp VeXVPV P4p\ 4p V !V P R VP4VP R 44p \V V VPVPR7#) a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. rr1r2r3rTNrrC) r{rQrrHrBrrrrr|r}rs &&&&&&&, r+rW%RobertaForTokenClassification.forward^s2,,  ) * &  (     "!*,,71  YYv}}-F')HFKKDOO > ##d*> ))D0> ((4/ > &&- > ((4/ > ))D0> ''$.> +,>  u|| ; ;> r-c VP!V3RVRVRVRVRR/VBp V ^,p VPV 4p V P^R R7wrV PR 4P 4p V PR 4P 4p RpVeVe\ VP 44^8dVPR 4p\ VP 44^8dVPR 4pV P ^4pVP^V4pVP^V4p\VR 7pV!W4pV!W4pVV,^, p\VV V V PV PR 7#) a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) rr1r2r3rTr_N) ignore_index)r start_logits end_logitsr|r}rC) r{r1splitr contiguouslenrFclamprr r|r})r(r0rr1r2r3r4r5rrrrr9r: total_loss ignored_indexr start_lossend_losss&&&&&&&&, r+rW#RobertaForQuestionAnswering.forwards0,,  ) * &  (     "!*1#)<<r<#: #++B/::< ''+668   &=+D?'')*Q."1"9"9""==%%'(1, - 5 5b 9 (--a0M-33A}EO)//=AM']CH!,@J :H$x/14J+%!!//))   r-)rr1r{)NNNNNNNr ros@@r+r/r/s0> > > > r-r/)rrr r/rrrrz);rr5torch.nnr#rrrrr activationsr generationrmodeling_outputsrr r r r r rmodeling_utilsrprocessing_utilsrutilsrrr utils.genericrbert.modeling_bertrrrrrconfiguration_robertar get_loggerrgrrrqrvrxrzrrrModulerrr rrr/__all__rrr-r+rRs AA&).&@@-ll0   H %\8\8~ ,  .  9 /_//2'9'  i /i  i XR /R R jBII, Q '=Q  Q h] 5] ] @C $:C C L ,K "8K K \ r-