+ j+Rt^RIt^RIHt^RIHtHtHt^RIHt ^RI H t ^RI H t ^RIHtHtHtHtHt^RIHt^R IHtHt^R IHt]P6!]4t!R R ]P<4t!R R]P<4t ]PBPDR4t#]PBPDR4t$]PBPDR4t%]PBPDR4t&]PBPDRRl4t']PBPDRRl4t(]PBPDRRl4t)]PBPDRRl4t*!RR]P<4t+!RR]P<4t,!RR ]P<4t-!R!R"]P<4t.!R#R$]P<4t/!R%R&] 4t0!R'R(]P<4t1]!R)R*]44t2]!R+R,]244t3!R-R.]P<4t4!R/R0]P<4t5!R1R2]P<4t6!R3R4]P<4t7!R5R6]P<4t8]!R7R8]244t9!R9R:]P<4t:]!R;R<7!R=R>]244t;]!R?R@]244t<]!RARB]244t=.RCOt>R#)DzPyTorch DeBERTa model.N)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)GradientCheckpointingLayer)BaseModelOutputMaskedLMOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)auto_docstringlogging DebertaConfigc@aa]tRt^&toRtRV3RlltRtRtVtV;t #)DebertaLayerNormz2LayerNorm module (epsilon inside the square root).c<\SV`4\P!\P !V44Vn\P!\P!V44VnW n R#N) super__init__r Parametertorchonesweightzerosbiasvariance_epsilon)selfsizeeps __class__s&&&m/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/deberta/modeling_deberta.pyrDebertaLayerNorm.__init__)sF ll5::d#34 LLT!23 #cVPpVP4pVPRRR7pW, P^4PRRR7pW, \P !W@P ,4, pVPV4pVPV,VP,pV#)T)keepdim) dtypefloatmeanpowrsqrtrtorr)r hidden_states input_typer-varianceys&& r$forwardDebertaLayerNorm.forward/s"(( %++- !!"d!3!(--a055b$5G&-HG\G\<\1]] %((4 KK- '$)) 3r&)rrr)g-q= __name__ __module__ __qualname____firstlineno____doc__rr5__static_attributes____classdictcell__ __classcell__r# __classdict__s@@r$rr&s<$ r&rc8aa]tRt^:toV3RltRtRtVtV;t#)DebertaSelfOutputc<\SV`4\P!VPVP4Vn\ VPVP4Vn\P!VP4Vn R#r) rrrLinear hidden_sizedenserlayer_norm_eps LayerNormDropouthidden_dropout_probdropoutr configr#s&&r$rDebertaSelfOutput.__init__;s\ YYv1163E3EF )&*<*S>STzz&"<"<= r&cxVPV4pVPV4pVPW,4pV#rrGrLrIr r1 input_tensors&&&r$r5DebertaSelfOutput.forwardA5 =1  ]3 }'CD r&)rIrGrL r8r9r:r;rr5r=r>r?r@s@@r$rCrC:s> r&rCcVPR4pVPR4p\P!V\PVPR7p\P!V\PVPR7pVR,VP ^R4P V^4, pVRV1R3,pVP^4pV#)a Build relative position according to the query and key We assume the absolute position of query \(P_q\) is range from (0, query_size) and the absolute position of key \(P_k\) is range from (0, key_size), The relative positions from query to key is \(R_{q \rightarrow k} = P_q - P_k\) Args: query_size (int): the length of query key_size (int): the length of key Return: `torch.LongTensor`: A tensor with shape [1, query_size, key_size] r+deviceNNNN)rZNr*)r!rarangelongrYviewrepeat unsqueeze) query_layer key_layer query_sizekey_sizeq_idsk_ids rel_pos_idss&& r$build_relative_positionrhHs$!!"%J~~b!H LL5::k>P>P QE LLI>;++A. 0@0@0C[EUEUVWEXZfZkZklnZop qqr&cVPVP^4VP^4VPR4VPR4.4#)rjr[rk)rmrarbs&&&r$p2c_dynamic_expandrqjsG >>;++A. 0@0@0CY^^TVEWYbYgYghjYkl mmr&cVPVP4R,VPR4VPR43,4#):NNr[rk) pos_indexp2c_attrbs&&&r$pos_dynamic_expandrvos;   GLLN2.)..2DinnUWFX1YY ZZr&cDV^8dQhR\PR\/#)rsra scale_factorrTensorint)formats"r$ __annotate__r}xs"\\%,,\c\r&c\P!\P!VPR4\PR7V,4#)r(r+r*)rr/tensorr!r,)rarxs&&r$scaled_size_sqrtrws0 ::ell;#3#3B#7u{{KlZ [[r&cXV^8dQhR\PR\P/#rsrarbrrz)r|s"r$r}r}}s"ELLU\\r&cfVPR4VPR48wd \W4#V#rsr[)r!rh)rarbrns&&&r$ build_rposr|s/y~~b11&{>>r&cdV^8dQhR\PR\PR\/#)rsrarbmax_relative_positionsry)r|s"r$r}r}s/dd ddgjdr&c \P!\\VP R4VP R44V44#r)rrminmaxr!)rarbrs&&&r$compute_attention_spanrs4 <<C 0 0 4innR6HIKab ccr&cXV^8dQhR\PR\P/#rr)r|s"r$r}r}s" r&c VPR4VPR48wd<VR,PR4p\P!V^\ W@V4R7#V#)rsdimindexr[)rZrZrZrjr*)r!r`rgatherrv)rurarbrnrts&&&& r$uneven_size_correctedrsTy~~b11 ,66r: ||G2DYYb2cddr&cdaa]tRt^toRtV3RltRtR V3RlRlltV3RlRltRt Vt V;t #) DisentangledSelfAttentionz Disentangled self-attention module Parameters: config (`str`): A model config class instance with the configuration to build a new model. The schema is similar to *BertConfig*, for more details, please refer [`DebertaConfig`] c<\SV`4VPVP,^8wd'\ RVP RVP R24hVPVn\ VPVP, 4VnVPVP ,Vn\P!VPVP^,RR7Vn \P!\P!VP\PR74Vn\P!\P!VP\PR74VnVP"e VP"M.Vn\%VRR4Vn\%VR R4VnVP('df\P!VPVPRR7Vn\P!VPVPRR7VnMRVnRVnVP&'d\%VR R 4VnVP.^8dVP0Vn\P2!VP44VnR VP"9d3\P!VPVPRR7VnR VP"9d1\P!VPVP4Vn\P2!VP<4VnR#)rjzThe hidden size (z6) is not a multiple of the number of attention heads ()FrrNrelative_attention talking_headrc2pp2cr*) rrrFnum_attention_heads ValueErrorr{attention_head_size all_head_sizerrEin_projrrrr,q_biasv_bias pos_att_typegetattrrrhead_logits_projhead_weights_projrmax_position_embeddingsrJrK pos_dropoutpos_proj pos_q_projattention_probs_dropout_probrLrMs&&r$r"DisentangledSelfAttention.__init__ss     : : :a ?#F$6$6#78 445Q8 $*#=#= #&v'9'9FP>PW\ ] )))"$))F,>,>@R@R"Szz&"E"EF r&cVP4RRVPR3,pVPV4pVP^^^^4#)Nr*)r!rr^permute)r x new_x_shapes&& r$transpose_for_scores.DisentangledSelfAttention.transpose_for_scoressFffhsmt'?'?&DD FF; yyAq!$$r&c<V^8dQhRS[PRS[PRS[RS[PR,RS[PR,RS[PR,RS[S[PS[PR,3,/#) rsr1attention_maskoutput_attentions query_statesNrnrel_embeddingsreturn)rrzbooltuple)r|rAs"r$r}&DisentangledSelfAttention.__annotate__sU0U0||U0 U0 U0 llT) U0 llT) U0 t+U0 u||U\\D00 1U0r&c  Vf9VPV4pVPV4P^RR7wrp EMVPPPVP^,^R7p \ ^4U U u.uFRp \ P!\ VP4U u.uFqV ^,V ,,NK up ^R7NKT pp p \ P!V^,VP4PV^,PR74p\ P!V^,VP4PV^,PR74p \ P!V^,VP4PV^,PR74pWV3Uu.uFpVPV4NK upwrp WPVPR,4,pWPVPR,4,p ^p^\VP4,p\!VV4pVVPVPR7, p\ P!WP#RR44pVP$'d-Ve)Ve%VP'V4pVP)WWVV4pVe VV,pVP*e6VP+VP-^^^^44P-^^^^4pVP/4pVP1V(\ P2!VP4P44p\6P8P;VRR7pVP=V4pVP>e6VP?VP-^^^^44P-^^^^4p\ P!VV 4pVP-^^^^4PA4pVPC4RRR,pVPEV4pV'gVR3#VV3#uup iuup p iuupi)a1 Call the module Args: hidden_states (`torch.FloatTensor`): Input states to the module usually the output from previous layer, it will be the Q,K and V in *Attention(Q,K,V)* attention_mask (`torch.BoolTensor`): An attention mask matrix of shape [*B*, *N*, *N*] where *B* is the batch size, *N* is the maximum sequence length in which element [i,j] = *1* means the *i* th token in the input can attend to the *j* th token. output_attentions (`bool`, *optional*): Whether return the attention matrix. query_states (`torch.FloatTensor`, *optional*): The *Q* state in *Attention(Q,K,V)*. relative_pos (`torch.LongTensor`): The relative position encoding between the tokens in the sequence. It's of shape [*B*, *N*, *N*] with values ranging in [*-max_relative_positions*, *max_relative_positions*]. rel_embeddings (`torch.FloatTensor`): The embedding of relative distances. It's a tensor of shape [\(2 \times \text{max_relative_positions}\), *hidden_size*]. Nrrr*)NNrZr[)r*)#rrchunkrrrangercatmatmultr0r+rrlenrr transposerrdisentangled_att_biasrrr masked_fillfinforr functionalsoftmaxrLr contiguousr!r^)r r1rrrrnrqprarb value_layerwskiqkvwqvrrel_attrxscaleattention_scoresattention_probs context_layernew_context_layer_shapes&&&&&&& r$r5!DisentangledSelfAttention.forwardsL  m,B262K2KB2O2U2UVW]_2U2` /KK$$**4+C+Ca+GQ*OBhmnohpqhpcdEIIeDX$YY !$=$=dkk->X$YY 3t0011  l;!EHH;3D3DH$EE  << 5H5HR5PQ  " " "~'AlF^!--n=N00gstG  /'9   ,#445E5M5MaQRTUWX5YZbbcdfgijlmn ',,.+77.8I5;;WbWhWhKiKmKmn--//0@b/I,,7  ! ! -"44_5L5LQPQSTVW5XYaabcefhiklmO _kB %--aAq9DDF "/"4"4"6s";e"C%**+BC  !4( (//UWq3ds,Q5/Q0 Q5 Q;0Q5c <V^8dQhRS[PRS[PRS[PRS[PRS[/#)rsrarbrnrrxry)r|rAs"r$r}r"sM66\\6<<6ll 6  6  6r&c Vf\WVP4pVP4^8Xd"VP^4P^4pMXVP4^8XdVP^4pM1VP4^8wd\ RVP4 24h\ WVP 4pVP4pVVP V, VP V,1R3,P^4p^pRVP9dVPV4pVPV4p\P!WPRR44p \P!W6,^V^,^, 4p \P!V R\!WV4R7p Wy, pRVP9dVP#V4p VPV 4p V \%W4,p \'VVV4p \P!V )V,^V^,^, 4p \P!W+PRR4P)VP*R74p\P!VR\-WV4R7PRR4p\/WW#4pW~, pV#) Nz2Relative position ids must be of dim 2 or 3 or 4. rZrrrrr*r[)rhrYrr`rrrr]rrrrrrclamprrorrrr0r+rqr)r rarbrnrrxatt_spanscore pos_key_layerc2p_attrmpos_query_layerr_posp2c_posrus&&&&&& r$r/DisentangledSelfAttention.disentangled_att_bias"sq  2;;K]K]^L     "'11!4>>qAL    1 $'11!4L    1 $QR^RbRbRdQefg g)+$B]B]^#((* '  ' '( 2T5P5PS[5[ []^ ^ )A,  D%% % MM.9M 55mDMll;0G0GB0OPGkk,"91hlQ>NOGll7:LWco:pqG  E D%% %"oon=O"77HO /N NOE kk5&8"3Q1 q8HIGll9.G.GB.O.R.RYbYhYh.R.ijGllR'9'PY'ZiB ,G)ZG  E r&)rrrLrrrrrrrrrrrrrFNNN) r8r9r:r;r<rrr5rr=r>r?r@s@@r$rrs2$GL% U0U0n666r&rc@aa]tRtRtoRtV3RltRRltRtVtV;t #)DebertaEmbeddingsi[zGConstruct the embeddings from word, position and token_type embeddings.c<\SV`4\VR^4p\VRVP4Vn\ P !VPVPVR7Vn\VRR4Vn VP'g RVn M0\ P !VPVP4Vn VP^8d2\ P !VPVP4Vn MRVn VPVP8wd4\ P!VPVPRR7VnMRVn\!VPVP"4Vn\ P&!VP(4VnWn\ P.!\0P2!VP4P5R 4RR7VnR#) pad_token_idembedding_size) padding_idxposition_biased_inputTNFr) persistentr(r*)rrrrFrr Embedding vocab_sizeword_embeddingsrposition_embeddingsrtype_vocab_sizetoken_type_embeddingsrE embed_projrrHrIrJrKrLrNBufferrr\rl position_ids)r rNrr#s&& r$rDebertaEmbeddings.__init__^ss v~q9 %f.>@R@RS!||F,=,=t?R?R`lm%,V5Ld%S")))'+D $')||F4R4RTXTgTg'hD $  ! !A %)+f6L6LdNaNa)bD &)-D &   &"4"4 4 ii(;(;V=O=OV[\DO"DO)&*<*S>STzz&"<"<=  IIell63Q3Q&R&Y&YZa&botur&cVeVP4pMVP4RRpV^,pVfVPRRV13,pVf<\P!V\PVPP R7pVfVP V4pVPe!VPVP 44pM\P!V4pTp VP'd W,p VPeVPV4p W,p VPeVPV 4p VPV 4p VeVP4V P48wdGVP4^8Xd!VP^4P^4pVP^4pVP!V P"4pW,p VP%V 4p V #)NrZrXr*)r!rrrr]rYrr zeros_likerrrrIrsqueezer`r0r+rL) r input_idstoken_type_idsrmask inputs_embeds input_shape seq_lengthr embeddingsrs &&&&&& r$r5DebertaEmbeddings.forward{s  #..*K',,.s3K ^  ,,Q  ^$>~$N !#;J ?? &4J^^J/  xxzZ^^--88:?<<?2215D~~a(77:++,D#*J\\*- r&) rIrNrLrrrrrrr)NNNNNr7r@s@@r$rr[sQv:,,r&rcHaa]tRtRtoV3RltRV3RlRlltRtVtV;t#)DebertaAttentionicp<\SV`4\V4Vn\ V4VnWnR#r)rrrr rCoutputrNrMs&&r$rDebertaAttention.__init__s+ -f5 '/  r&cr<V^8dQhRS[RS[S[PS[PR,3,/#rsrrNrrrrz)r|rAs"r$r}DebertaAttention.__annotate__s8,, , u||U\\D00 1,r&c VPVVVVVVR7wrxVfTpVPWt4p V'dW3#V R3#))rrnrN)r r ) r r1rrrrnr self_output att_matrixattention_outputs &&&&&&& r$r5DebertaAttention.forwards_#'))   %%) #,#    (L;;{A $1 1$d+ +r&)rNr r rrVr@s@@r$rrs ,,,r&rcDaa]tRtRtoV3RltV3RlRltRtVtV;t#)DebertaIntermediateic"<\SV`4\P!VPVP 4Vn\VP\4'd\VP,Vn R#VPVn R#r) rrrrErFintermediate_sizerG isinstance hidden_actstrrintermediate_act_fnrMs&&r$rDebertaIntermediate.__init__s` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $r&cN<V^8dQhRS[PRS[P/#)rsr1rr)r|rAs"r$r} DebertaIntermediate.__annotate__s#U\\ellr&cJVPV4pVPV4pV#rrGrr r1s&&r$r5DebertaIntermediate.forwards& =1 00? r&r rVr@s@@r$rrs9r&rc8aa]tRtRtoV3RltRtRtVtV;t#) DebertaOutputic$<\SV`4\P!VPVP 4Vn\VP VP4Vn \P!VP4Vn Wn R#r)rrrrErrFrGrrHrIrJrKrLrNrMs&&r$rDebertaOutput.__init__sa YYv779K9KL )&*<*S>STzz&"<"<=  r&cxVPV4pVPV4pVPW,4pV#rrQrRs&&&r$r5DebertaOutput.forwardrUr&)rIrNrGrLrVr@s@@r$r$r$sr&r$cHaa]tRtRtoV3RltRV3RlRlltRtVtV;t#) DebertaLayeric<\SV`4\V4Vn\ V4Vn\ V4VnR#r)rrr attentionr intermediater$r rMs&&r$rDebertaLayer.__init__s3 )&1/7#F+ r&cr<V^8dQhRS[RS[S[PS[PR,3,/#r r )r|rAs"r$r}DebertaLayer.__annotate__s8(( ( u||U\\D00 1(r&c VPVVVVVVR7wrxVPV4p VPW4p V'dW3#V R3#)rrrnrNr,r-r ) r r1rrrnrrrrintermediate_output layer_outputs &&&&&&& r$r5DebertaLayer.forwardsg(,~~  /%%) (6( $#//0@A{{#6I  - - $' 'r&r3)NNNFrVr@s@@r$r*r*s, (((r&r*cbaa]tRtRtoRtV3RltRtRtR RltR V3RlRllt R t Vt V;t #) DebertaEncoderi z8Modified BertEncoder with relative position bias supportc<\SV`4\P!\ VP 4Uu.uFp\ V4NK up4Vn\VRR4Vn VP'dl\VRR4Vn VP^8dVPVn \P!VP^,VP4VnRVnR#uupi)rFrNr*)rrr ModuleListrnum_hidden_layersr*layerrrrrrrFrgradient_checkpointing)r rN_r#s&& r$rDebertaEncoder.__init__s ]]%H`H`Ba#bBaQL$8Ba#bc ")&2F"N  " " "*1&:RTV*WD '**Q..4.L.L+"$,,t/J/JQ/NPVPbPb"cD &+#$csC/c^VP'dVPPpV#RpV#r)rrr)r rs& r$get_rel_embedding DebertaEncoder.get_rel_embeddings37;7N7N7N,,33UYr&c VP4^8:dIVP^4P^4pW"PR4PR4,pV#VP4^8XdVP^4pV#)rsr[r*)rr`r)r rextended_attention_masks&& r$get_attention_mask!DebertaEncoder.get_attention_masks    1 $&4&>&>q&A&K&KA&N #47V7VWY7Z7d7deg7hhN   !Q &+55a8Nr&cjVP'd!VfVe\W!4pV#\W4pV#r)rrh)r r1rrns&&&&r$ get_rel_posDebertaEncoder.get_rel_pos's=  " " "|';'6|S  7}T r&c `<V^8dQhRS[PRS[PRS[RS[RS[/#)rsr1routput_hidden_statesr return_dict)rrzr)r|rAs"r$r}DebertaEncoder.__annotate__/sE, , ||,  , # ,  , , r&c VPV4pVPWV4pV'dV3MRpV'dRMRp Tp VP4p \VP4FBwrV !V VVVV VR7wrV'd W3,pVeTpMTp V'gK9W3,p KD V'g1\ ;QJd.RWV 34FNK 5#!RWV 344#\ WV R7#)N)rrnrrc30"TF qfKVxK R#5ir).0rs& r$ )DebertaEncoder.forward..Xsh$Vq$Vs last_hidden_stater1 attentionsrP)rErHrA enumerater<rr )r r1rrKrrrnrLall_hidden_statesall_attentionsnext_kvrr layer_moduleatt_ms&&&&&&&& r$r5DebertaEncoder.forward/s00@'' \R L`8Hfj0d//1(4OA#/))-"3 $ M$$58H$H!', '  !/(!:' 5*5h]~$Vh5 h5h]~$Vhh h+Yg  r&)r=r<rrr)NN)TFNNT) r8r9r:r;r<rrArErHr5r=r>r?r@s@@r$r8r8 s+B ,, , , r&r8ctaa]tRtRtoRtR.tRt]P!4V3Rl4t V3Rlt Rt Vt V;t #)DebertaPreTrainedModeli^debertarTc @<\SV`V4\V\4'dC\P !VP 4\P !VP4R#\V\\34'd#\P !VP4R#\V\4'db\P!VP\P!VPP R,4P#R44R#R#)zInitialize the weights.Nr*r)r _init_weightsrrinitzeros_rrLegacyDebertaLMPredictionHeadDebertaLMPredictionHeadrrcopy_rrr\shaperl)r moduler#s&&r$rb$DebertaPreTrainedModel._init_weightses f% f7 8 8 KK & KK & !>@W X Y Y KK $  1 2 2 JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h3r&c&<V^8dQh/S[;R&#)rsrNr)r|rAs"r$r}#DebertaPreTrainedModel.__annotate__^s r&rP)r8r9r:r;base_model_prefix"_keys_to_ignore_on_load_unexpectedsupports_gradient_checkpointingrno_gradrb__annotate_func__r=r>r?r@s@@r$r_r_^s="*?)@&&*# ]]_ i ir&r_c^aa]tRtRtoV3RltRtRt]RV3RlRll4tRt Vt V;t #) DebertaModelirc<\SV`V4\V4Vn\ V4Vn^VnWnVP4R#)rjN) rrrrr8encoderz_stepsrN post_initrMs&&r$rDebertaModel.__init__ts>  +F3%f-    r&c.VPP#rrrr s&r$get_input_embeddings!DebertaModel.get_input_embeddings~s...r&c&WPnR#rrzr new_embeddingss&&r$set_input_embeddings!DebertaModel.set_input_embeddingss *8'r&c4<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[R,RS[R,R S[R,R S[S[,/ #) rsrNrrrrrrKrLr)rrzrrr )r|rAs"r$r}DebertaModel.__annotate__sO O <<$&O  t+O  t+ O llT) O ||d* O  $;O #TkO D[O   O r&c VeTMVPPpVeTMVPPpVeTMVPPpVeVe \ R4hVe#VP W4VP 4p M#VeVP 4RR p M \ R4hVe VPM VPp Vf\P!WR7pVf(\P!V \PV R7pVPVVVVVR7p VPV VRVVR7p V ^,pVP^8dVR ,p\VP4Uu.uF pVPP R ,NK" ppVR ,pVPP#4pVPP%V4pVPP'V 4pVR,F"pV!VVR VVVR 7pVP)V4K$ VR ,pV'gV3Y'd ^R,#^R,#\+TV'd V P,MRV P.R 7#uupi)NzDYou cannot specify both input_ids and inputs_embeds at the same timez5You have to specify either input_ids or inputs_embeds)rYrX)rrrrrT)rKrrLr(NNFr2rTr*r[)rNrrKrLr%warn_if_padding_and_no_attention_maskr!rYrrrr]rrurvrr<rArErHappendr r1rV)r rrrrrrrKrLkwargsrrYembedding_outputencoder_outputsencoded_layersr1r>layersrrrel_posr<sequence_outputs&&&&&&&&&, r$r5DebertaModel.forwardsw2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y Q#..*K  &',,.s3KTU U%.%:!! @T@T  !"ZZ CN  !"[[EJJvVN??)%' + ,,  !%/# ' )+ <R8\(]] ]XY8\(]] ]-;O/77UY&11  +Ks9&J)rNrrurv)NNNNNNNN r8r9r:r;rr|rrr5r=r>r?r@s@@r$rsrsrs6/9O O O O r&rsc8aa]tRtRtoV3RltRtRtVtV;t#)$LegacyDebertaPredictionHeadTransformic<\SV`4\VRVP4Vn\ P !VPVP4Vn\VP\4'd\VP,Vn MVPVn \ P!VPVPR7Vn R#)r)r"N)rrrrFrrrErGrrrrtransform_act_fnrIrHrMs&&r$r-LegacyDebertaPredictionHeadTransform.__init__s %f.>@R@RSYYv1143F3FG f'' - -$*6+<+<$=D !$*$5$5D !d&9&9v?T?TUr&clVPV4pVPV4pVPV4pV#r)rGrrIr!s&&r$r5,LegacyDebertaPredictionHeadTransform.forwards4 =1 --m< }5 r&)rIrGrrrVr@s@@r$rrs Vr&rc8aa]tRtRtoV3RltRtRtVtV;t#)reicR<\SV`4\V4Vn\ VRVP 4Vn\P!VP VPRR7Vn \P!\P!VP44VnR#)rTrN)rrr transformrrFrrrErdecoderrrrrrMs&&r$r&LegacyDebertaLMPredictionHead.__init__ss =fE%f.>@R@RSyy!4!4f6G6GdS LLV->->!?@ r&cJVPV4pVPV4pV#r)rrr!s&&r$r5%LegacyDebertaLMPredictionHead.forwards$}5  ]3 r&)rrrrrVr@s@@r$reres Ar&recDaa]tRtRtoV3RltV3RlRltRtVtV;t#)LegacyDebertaOnlyMLMHeadicD<\SV`4\V4VnR#r)rrre predictionsrMs&&r$r!LegacyDebertaOnlyMLMHead.__init__s 8@r&cN<V^8dQhRS[PRS[P/#)rsrrr)r|rAs"r$r}%LegacyDebertaOnlyMLMHead.__annotate__s#!!u||! !r&c(VPV4pV#rr)r rprediction_scoress&& r$r5 LegacyDebertaOnlyMLMHead.forwards ,,_=  r&rrVr@s@@r$rrsA!!!r&rc<aa]tRtRtoRtV3RltRtRtVtV;t #)rfizMhttps://github.com/microsoft/DeBERTa/blob/master/DeBERTa/deberta/bert.py#L270c<\SV`4\P!VPVP4Vn\ VP\4'd\VP,Vn MVPVn \P!VPVPRR7Vn \P!\P!VP 44VnR#)T)r"elementwise_affineN)rrrrErFrGrrrrrrIrHrrrrrrMs&&r$r DebertaLMPredictionHead.__init__ s YYv1163E3EF f'' - -$*6+<+<$=D !$*$5$5D !f&8&8f>S>ShlmLLV->->!?@ r&cVPV4pVPV4pVPV4p\P!WP P 44VP,pV#r)rGrrIrrrrr)r r1rs&&&r$r5DebertaLMPredictionHead.forwardsb =1 --m<    ]4J4J4L4L4NORVR[R[[ r&)rIrrGrr7r@s@@r$rfrfsW Ar&rfc8aa]tRtRtoV3RltRtRtVtV;t#)DebertaOnlyMLMHeadi"cD<\SV`4\V4VnR#r)rrrflm_headrMs&&r$rDebertaOnlyMLMHead.__init__#s .v6 r&c(VPW4pV#rr)r rrrs&&& r$r5DebertaOnlyMLMHead.forward(s LLJ  r&rrVr@s@@r$rr"s7 !!r&rcjaa]tRtRtoRRRR/tV3RltRtRt]R V3R lR ll4t R t Vt V;t #) DebertaForMaskedLMi-zcls.predictions.decoder.biaszcls.predictions.biaszcls.predictions.decoder.weight)deberta.embeddings.word_embeddings.weightc<\SV`V4VPVn\V4VnVP'd\ V4VnMRR/Vn\V4Vn VP4R#)zlm_predictions.lm_head.weightrN) rrlegacyrsr`rcls_tied_weights_keysrlm_predictionsrwrMs&&r$rDebertaForMaskedLM.__init__4si  mm #F+ ;;;/7DH01\'D ##5V"   ' ' -/=/B/BD   ' ' ,r&c\<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[R,R S[R,R S[R,R S[S[,/ # rsrNrrrrlabelsrrKrLr)rrzrrr )r|rAs"r$r}DebertaForMaskedLM.__annotate__Rs5 5 <<$&5  t+5  t+ 5 llT) 5 ||d* 5  t#5  $;5 #Tk5 D[5   5 r&c TV eT MVPPp VPVVVVVVVV R7p V ^,p VP'dVP V 4p M/VP WPP P4p RpVeG\4pV!V PRVPP4VPR44pV 'g"V 3V R,,pVe V3V,#T#\VV V PV PR7#)az labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` NrrrrrrKrLrlosslogitsr1rVr*)rNrLr`rrrrrrr^rr r1rV)r rrrrrrrrKrLroutputsrrmasked_lm_lossloss_fctr s&&&&&&&&&&, r$r5DebertaForMaskedLM.forwardQs*&1%r?r@s@@r$rr-sI '(>(*U 5 C5 5 5 5 r&rcHaa]tRtRtoV3RltRt]R4tRtVt V;t #) ContextPooleric<\SV`4\P!VPVP4Vn\P !VP4VnWn R#r) rrrrEpooler_hidden_sizerGrJpooler_dropoutrLrNrMs&&r$rContextPooler.__init__sG YYv88&:S:ST zz&"7"78  r&cVR,pVPV4pVPV4p\VPP,!V4pV#)rZ)rZrj)rLrGrrNpooler_hidden_act)r r1 context_token pooled_outputs&& r$r5ContextPooler.forwardsI&d+  ]3  =1 t{{<<=mL r&c.VPP#r)rNrFr{s&r$ output_dimContextPooler.output_dims{{&&&r&)rNrGrL) r8r9r:r;rr5propertyrr=r>r?r@s@@r$rrs( '''r&rz DeBERTa Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ) custom_introc^aa]tRtRtoV3RltRtRt]RV3RlRll4tRt Vt V;t #) DebertaForSequenceClassificationic<\SV`V4\VR^4pW n\ V4Vn\ V4VnVPPp\P!W24Vn \VRR4pVfVPPMTp\P!V4VnVP!4R#) num_labels cls_dropoutN)rrrrrsr`rpoolerrrrE classifierrNrKrJrLrw)r rNrrdrop_outr#s&& r$r)DebertaForSequenceClassification.__init__s  V\15 $#F+ #F+ [[++ ))J;6=$76>6F4;;22Hzz(+  r&c6VPP4#r)r`r|r{s&r$r|5DebertaForSequenceClassification.get_input_embeddingss||0022r&c<VPPV4R#r)r`rrs&&r$r5DebertaForSequenceClassification.set_input_embeddingss )).9r&c\<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[R,R S[R,R S[R,R S[S[,/ #r)rrzrrr )r|rAs"r$r}-DebertaForSequenceClassification.__annotate__sN N <<$&N  t+N  t+ N llT) N ||d* N  t#N  $;N #TkN D[N  ) )N r&c rV eT MVPPp VPVVVVVVVV R7p V ^,p VPV 4p VP V 4p VP V 4pRpVEeVPP EfVP^8XdY\P!4pVPR4PVP4pV!WPR44pEMsVP4^8XgVPR4^8XEdV^8P4pVP!4pVP^4^8d\"P$!V^VP'VP^4VP^444p\"P$!V^VPR44p\)4pV!VPRVP4P+4VPR44pEMU\"P,!^4PV4pEM.\P.!R4pV!V4V,P1R4P34)pMVPP R8XdK\4pVP^8Xd'V!VP54VP544pMV!W4pMVPP R8Xd>\)4pV!VPRVP4VPR44pM-VPP R8Xd\74pV!W4pV 'g"V3V R,,pVe V3V,#T#\9WV P:V P<R7#) ae labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N)rrrrrrKrL regressionsingle_label_classificationmulti_label_classificationrrr*)rNrLr`rrLr problem_typerrrr^r0r+rr!nonzeror]rrrlrr,r LogSoftmaxsumr-rrr r1rV)r rrrrrrrrKrLrr encoder_layerrrrloss_fn label_indexlabeled_logitsr log_softmaxr s&&&&&&&&&&, r$r5(DebertaForSequenceClassification.forwards(&1%#)Q;"7"7"9K#[[]F"''*Q.)."A{'9'9+:J:J1:Mv{{[\~'^*"'fa9I9I"9M!N#3#5'(;(;B(P(V(V(XZ`ZeZefhZij$||A11&9"$--"3K)&1F:??CIIKKD))\9"9??a'#FNN$4fnn6FGD#F3D))-JJ+- B @&++b/R))-II,./Y,F)-)9TGf$ Ev E'G4I4IV]VhVh  r&)rr`rLrrrrr@s@@r$rrs6$3:N N N N r&rcRaa]tRtRtoV3Rlt]RV3RlRll4tRtVtV;t #)DebertaForTokenClassificationic2<\SV`V4VPVn\V4Vn\ P !VP4Vn\ P!VPVP4Vn VP4R#r) rrrrsr`rrJrKrLrErFrrwrMs&&r$r&DebertaForTokenClassification.__init__si   ++#F+ zz&"<"<= ))F$6$68I8IJ r&c\<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[R,R S[R,R S[R,R S[S[,/ #r)rrzrrr )r|rAs"r$r}*DebertaForTokenClassification.__annotate__s. . <<$&.  t+.  t+ . llT) . ||d* .  t#.  $;. #Tk. D[.  & &. r&c V eT MVPPp VPVVVVVVVV R7p V ^,p VPV 4p VP V 4p RpVe=\ 4pV!V P RVP4VP R44pV 'g"V 3V R,,pVe V3V,#T#\WV PV PR7#)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrrr*) rNrLr`rLrrr^rr r1rV)r rrrrrrrrKrLrrrrrrr s&&&&&&&&&&, r$r5%DebertaForTokenClassification.forwards$&1%r?r@s@@r$rrs( . . . . r&rcRaa]tRtRtoV3Rlt]RV3RlRll4tRtVtV;t #)DebertaForQuestionAnsweringiPc<\SV`V4VPVn\V4Vn\ P !VPVP4VnVP4R#r) rrrrsr`rrErF qa_outputsrwrMs&&r$r$DebertaForQuestionAnswering.__init__RsS   ++#F+ ))F$6$68I8IJ r&c<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[PR,R S[R,R S[R,R S[R,R S[S[,/ #) rsrNrrrrstart_positions end_positionsrrKrLr)rrzrrr )r|rAs"r$r}(DebertaForQuestionAnswering.__annotate__]s= = <<$&=  t+=  t+ = llT) = ||d* = ,= ||d*=  $;= #Tk= D[=  - -= r&c xV eT MVPPp VPVVVVVVV V R7p V ^,p VPV 4pVP ^RR7wppVP R4P 4pVP R4P 4pRpVeVe\VP44^8dVP R4p\VP44^8dVP R4pVP^4pVP^V4pVP^V4p\VR7pV!W4pV!VV4pVV,^, pV 'g#VV3V R,,pVe V3V,#T#\VVVV PV PR7#)Nrr) ignore_indexr)r start_logits end_logitsr1rVr*)rNrLr`rsplitrrrr!rrr r1rV)r rrrrrrrrrKrLrrrrrr total_loss ignored_indexr start_lossend_lossr s&&&&&&&&&&&, r$r5#DebertaForQuestionAnswering.forward\s&1%r-s AA&!9.,0   H %ryy(  8rrnn[[\\ddC CLL L^,ryy,F "))  BII (-(BO RYYO di_ii&a )a a H299&BII&!ryy!bii6!!Y /Y Y x'BII', h '=h  h V; $:; ; |I "8I I X r&