+ j?Rt^RIt^RIt^RIHt^RIHtHtHt^RIH t ^RI H t ^RI Ht^RIHtHtHtHtHtHtHt^RIHt^R IHtHt^R IHt^R IHtH t H!t!H"t"H#t#H$t$]PJ!]&4t'!R R ]PP4t)!RR]PP4t*!RR]PP4t+!RR]PP4t,!RR]PP4t-!RR]PP4t.!RR]PP4t/!RR]PP4t0!RR]PP4t1]!RR]44t2]!R R!]244t3]!R"R#]244t4!R$R%]PP4t5]!R&R'7!R(R)]244t6]!R*R+]244t7]!R,R-]244t8!R.R/]PP4t9]!R0R1]244t:R3R2lt;.R4Ot(>4??['(>(>4??[%   %%)) ..  "*$,,4??!Szz&"<"<= cVfFVe1\WPV4PVP4pMVP V4pVeVP 4pMVP 4RRpVf<\ P!V\ PVPPR7pVfVPV4wrGMRpVPV4wrVPVVVV R7wrVPV4wrVPV V V V R7wrVPW4wrVPV 4p VP!W4wrW3#)Ndtypedeviceidentityidentity_scaling_factorr*)"create_position_ids_from_input_idsrtorQ&create_position_ids_from_inputs_embedssizer9zeroslongr=r5r7r?r>rBrFrC)rH input_idstoken_type_idsr= inputs_embedspast_key_values_length input_shapeinputs_embeds_scaling_factorr7$token_type_embeddings_scaling_factor embeddingsembeddings_scaling_factorr>"position_embeddings_scaling_factors&&&&&& rKforwardIBertEmbeddings.forwardbsq  $A//1G "Y%%& $JJ=Y  #..*K',,.s3K  !"[[EJJtO`O`OgOghN  :>:N:Ny:Y 7M7+/ (FJF`F`aoFpC040D0D  (*$H 1E1 - CGBZBZ[gBh?040D0D  %($F 1E1 - 15z0e- \\*- 040F0Fz0m- 44rMc .VP4RRpV^,p\P!VP^,W0P,^,\PVP R7pVP ^4PV4#)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor NrOr*)rXr9r:rrZrQ unsqueezer<)rHr]r_sequence_lengthr=s&& rKrW6IBertEmbeddings.create_position_ids_from_inputs_embedss~$((*3B/ %a.||   q /4D4D"Dq"HPUPZPZcpcwcw %%a(// <r=r!r7r5)NNNN) __name__ __module__ __qualname____firstlineno____doc__r,rerW__static_attributes____classdictcell__ __classcell__rJ __classdict__s@@rKrr.s ->^,5\==rMrc<aa]tRt^toV3RltRRltRtVtV;t#)IBertSelfAttentionc r<\SV`4VPVP,^8wd9\ VR4'g'\ RVP RVP R24hVP Vn^Vn^ Vn^Vn VPVn\VPVP, 4Vn VPVP,Vn \VPVPRVPVPVP RR7Vn\VPVPRVPVPVP RR7Vn\VPVPRVPVPVP RR7Vn\#VPVP R7Vn\#VPVP R7Vn\#VPVP R7Vn\#VPVP R7Vn\,P.!VP04Vn\5VPVP VP6R7VnR #) rkembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()Tbiasr bias_bitr! per_channelr#r!r'N)r+r,r3num_attention_headshasattr ValueErrorr!r r}r/intattention_head_size all_head_sizerquerykeyvaluerquery_activationkey_activationvalue_activationrCrrDattention_probs_dropout_probrFrr'softmaxrGs&&rKr,IBertSelfAttention.__init__s     : : :a ?PVXhHiHi#F$6$6#78 445Q8 !++  #)#=#= #&v'9'9Fu;:>((=:o7>Bjj>u;372G2G 3 / /3.A.A/.r+ 262G2G 3 / / $))#2. CCbC$*B*BC !&&|4>>q!D NN<0::1a@ !&&|4>>q!D !<< 5H5HR5PQ $223+e3 ???.H.cfk.k +.2 +  %/.@ ;?,, =; 77 ,,7 _kB ) 5+ILf+f (+/ (%--aAq9DDF "/"4"4"6s";t?Q?Q>S"S%**,CD 7;6L6L 77 3 37H=/2mM]!*+I J.0  ---rM)r/rrr}rFrrrrCr!rrrrrr NF rlrmrnror,rerqrrrsrts@@rKrwrws5pnH.H.rMrwc8aa]tRtRtoV3RltRtRtVtV;t#)IBertSelfOutputi%c <\SV`4VPVn^Vn^Vn^ Vn^Vn^ Vn\VPVPRVPVP VPRR7Vn \VP VPR7Vn \VPVPVPVPVPR7Vn\VPVPR7Vn\$P&!VP(4VnR#rTr{r#r$N)r+r,r!r/r r}r0r1rr3denser ln_input_actrrAr'rBrCrrDrErFrGs&&rKr,IBertSelfOutput.__init__&s  ++        ]]  %T%6%64??S%   %%)) ..  "*$,,4??!Szz&"<"<= rMcVPW4wrVPV4pVPVVVVR7wrVPW4wrVP W4wrW3#rRrrFrrBrCrHrr input_tensorinput_tensor_scaling_factors&&&&&rKreIBertSelfOutput.forwardCz6:jj6m3  ]3 6:6G6G  (!$? 7H7 3 7;nn]6q3 6:6L6L 7 3 ::rM rBr/r}rrFrr0r1rCr!r rrts@@rKrr%>:;;rMrc<aa]tRtRtoV3RltRRltRtVtV;t#)IBertAttentioniTc<\SV`4VPVn\V4Vn\ V4VnR#N)r+r,r!rwrHroutputrGs&&rKr,IBertAttention.__init__Us3  ++&v. %f- rMcVPVVVV4wrVVPV^,V^,W4wrxV3VR,,p V3VR,,p W3#)rkr)NN)rHr) rHrrrr self_outputsself_outputs_scaling_factorattention_outputattention_output_scaling_factorroutputs_scaling_factors &&&&& rKreIBertAttention.forward[sy59II  (   5 1 =AKK O8;]= 9$% R(88"A!CFabdFe!e..rM)rr!rHrrrts@@rKrrTs. //rMrc8aa]tRtRtoV3RltRtRtVtV;t#)IBertIntermediateipc <\SV`4VPVn^Vn^Vn^ Vn\ VPVPRVPVP VPRR7Vn VPR8wd \R4h\VPVPR7Vn\VPVPR7VnR#)rTr{rz3I-BERT only supports 'gelu' for `config.hidden_act`rr#N)r+r,r!r/r r}rr3intermediate_sizer hidden_actrrr'intermediate_act_fnrrCrGs&&rKr,IBertIntermediate.__init__qs  ++       $ $]]      &RS S#*dooU[UiUi#j !)$,,4??!SrMczVPW4wrVPW4wrVPW4wrW3#r)rrrC)rHrrs&&&rKreIBertIntermediate.forwardsL6:jj6m3 6:6N6N 7 3 7;6L6L 7 3 ::rM)r/r}rrrCr!r rrts@@rKrrpsT( ; ;rMrc8aa]tRtRtoV3RltRtRtVtV;t#) IBertOutputic <\SV`4VPVn^Vn^Vn^ Vn^Vn^ Vn\VPVPRVPVP VPRR7Vn \VP VPR7Vn \VPVPVPVPVP R7Vn\VPVPR7Vn\&P(!VP*4VnR#r)r+r,r!r/r r}r0r1rrr3rrrrrAr'rBrCrrDrErFrGs&&rKr,IBertOutput.__init__s  ++    $ $   ]]  %T%6%64??S%   %%)) ..  "*$,,4??!Szz&"<"<= rMcVPW4wrVPV4pVPVVVVR7wrVPW4wrVP W4wrW3#rrrs&&&&&rKreIBertOutput.forwardrrMrrrts@@rKrrrrMrcBaa]tRtRtoV3RltRRltRtRtVtV;t #) IBertLayericZ<\SV`4VPVn^Vn^Vn\ V4Vn\V4Vn\V4Vn \VPVPR7Vn \VPVPR7Vn R#)rr#N)r+r,r!r/ seq_len_dimr attentionr intermediaterrrpre_intermediate_actpre_output_actrGs&&rKr,IBertLayer.__init__s}  ++ '/-f5!&) $,T\\doo$V!&t||PrMcVPVVVVR7wrVV^,pV^,pVR,p VPWx4wrV 3V ,p V #))rr)rfeed_forward_chunk) rHrrrrself_attention_outputs%self_attention_outputs_scaling_factorrrr layer_outputlayer_output_scaling_factors &&&&& rKreIBertLayer.forwardsxIM  ( / IWI E 2!4*OPQ*R'(,484K4K 5 1  /G+rMcVPW4wrVPW4wr4VPW44wr4VPW4W4wrVWV3#r)rrrr)rHrrintermediate_output"intermediate_output_scaling_factorrrs&&& rKrIBertLayer.feed_forward_chunksu<@'IBertEncoder.forward.. s" A s )last_hidden_stater attentionscross_attentionsr) enumeratertupler ) rHrrrroutput_hidden_states return_dictall_hidden_statesall_self_attentionsall_cross_attentionsi layer_module layer_outputss &&&&&&& rKreIBertEncoder.forwards#7BD$5b4#(4OA#$58H$H!(,! M*!,M  &91=MR 4\P!VP@4R#R#)zInitialize the weightsg)meanstdNweight_integer bias_integer_is_hf_initializedFweight_scaling_factorshiftgh㈵>r*r(gh㈵)!r+ _init_weights isinstancerinitnormal_weightrIinitializer_ranger|zeros_getattrr)fc_scaling_factorr*rrr,rr- IBertLMHeadrcopy_r=r9r:rr<r constant_x_minx_maxact_scaling_factor)rHmodulerJs&&rKr."IBertPreTrainedModel._init_weightsGs2 f% fk * * LLSdkk6S6S T{{& FKK(v/6B F112 F445v~t4@ F//0A  / / LLSdkk6S6S T!!-gfmmMach6i6i FMM&*<*<=>v6=I F889 F112J - -vw-9 FLL):  , , KK $  0 0 JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h  ) ) NN6<< / NN6<< . KK11 2*rMc\R4h)z6`resize_token_embeddings` is not supported for I-BERT.)NotImplementedError)rHnew_num_tokenss&&rKresize_token_embeddings,IBertPreTrainedModel.resize_token_embeddingshs!"Z[[rMc&<V^8dQh/S[;R&#)rIr)formatrus"rK __annotate__!IBertPreTrainedModel.__annotate__Bs rMrr) rlrmrnrobase_model_prefixr9no_gradr.rB__annotate_func__rqrrrsrts@@rKr$r$Bs4  ]]_33@\MrMr$cfaa]tRtRtoRtR V3RlltRtRt]R V3RlRll4t Rt Vt V;t #) IBertModelila The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://huggingface.co/papers/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. c <\SV`V4WnVPVn\ V4Vn\ V4VnV'd \V4MRVn VP4R#)z^ add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) r+r,rIr!rrbrencoderrpooler post_init)rHrIadd_pooling_layerrJs&&&rKr,IBertModel.__init__wsX    ++)&1#F+ ->k&)D  rMc.VPP#rrbr5rHs&rKget_input_embeddingsIBertModel.get_input_embeddingss...rMc&WPnR#rrU)rHrs&&rKset_input_embeddingsIBertModel.set_input_embeddingss */'rMcX<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[R,RS[R,R S[R,R S[S[S[P,,/ #) rEr[Nrr\r=r]rrrreturn)r9 LongTensor FloatTensorboolr r )rFrus"rKrGIBertModel.__annotate__sE E ##d*E ))D0E ((4/ E &&- E ((4/ E  $;E #TkE D[E  6e>O>O8P PE rMc VeTMVPPpVeTMVPPpVeTMVPPpVeVe \ R4hVe#VP W4VP 4p M#VeVP 4RR p M \ R4hV wrVe VPM VPp Vf\P!W3V R7pVf(\P!V \PV R7pVPVVVVR7wr\VPVVR7pVPVVVVVVR7pV^,pVPeVPV4MRpV'gVV3VR,,#\!VVVP"VP$VP&R 7#) NzDYou cannot specify both input_ids and inputs_embeds at the same timez5You have to specify either input_ids or inputs_embeds)rQrO)r[r=r\r])rIr]r)rrrrr)r  pooler_outputrr r r*)rIrrrr%warn_if_padding_and_no_attention_maskrXrQr9onesrYrZrbrrOrPr rr r )rHr[rr\r=r]rrrkwargsr_ batch_size seq_lengthrQembedding_outputembedding_output_scaling_factorencoder_outputssequence_outputr!s&&&&&&&&&, rKreIBertModel.forwards2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y Q#..*K  &',,.s3KTU U!, %.%:!! @T@T  !"ZZ*)A6RN  !"[[EJJvVN<@OO%)' =L= 93;;*)  ,,  +)/!5# ' *!,8< 8O O4UY #]3ob6II I;-')77&11,==   rM)rIrbrOrPr!)T)NNNNNNNN) rlrmrnrorpr,rWrZrrerqrrrsrts@@rKrMrMls;"/0E E E E rMrMcjaa]tRtRtoRRRR/tV3RltRtRt]R V3R lR ll4t R t Vt V;t #) IBertForMaskedLMizlm_head.decoder.weightz(ibert.embeddings.word_embeddings.weight$zlm_head.decoder.biasz lm_head.biasc<\SV`V4\VRR7Vn\ V4VnVP 4R#F)rRN)r+r,rMr%r7lm_headrQrGs&&rKr,IBertForMaskedLM.__init__s6  %@ "6*  rMc.VPP#r)rrdecoderrVs&rKget_output_embeddings&IBertForMaskedLM.get_output_embeddingss||###rMc\WPnVPVPnR#r)rrrur|)rHnew_embeddingss&&rKset_output_embeddings&IBertForMaskedLM.set_output_embeddingss- *// rMc<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[R,R S[R,R S[R,R S[S[S[P,,/ # rEr[Nrr\r=r]labelsrrrr])r9r^r_r`r r )rFrus"rKrGIBertForMaskedLM.__annotate__s0 0 ##d*0 ))D00 ((4/ 0 &&- 0 ((4/ 0   4'0  $;0 #Tk0 D[0  % 1 12 20 rMc V eT MVPPp VPVVVVVVVV R7p V ^,p VPV 4p RpVeG\ 4pV!V P RVPP 4VP R44pV 'g"V 3V R,,pVe V3V,#T#\VV V PV PR7#)az labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Nrr\r=r]rrrrENNlosslogitsrr r*) rIrr%rrrrr2r rr )rHr[rr\r=r]r~rrrrfrrlprediction_scoresmasked_lm_lossloss_fctrs&&&&&&&&&&, rKreIBertForMaskedLM.forwards(&1%->!?@ rMc VPV4p\V4pVPV4pVPV4pV#r)rrrru)rHfeaturesrfxs&&, rKreIBertLMHead.forward-s; JJx  G OOA  LLOrM)r|rurr rlrmrnrorpr,rerqrrrsrts@@rKr7r7"s3ArMr7z I-BERT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ) custom_introcRaa]tRtRtoV3Rlt]RV3RlRll4tRtVtV;t #)IBertForSequenceClassificationi8c<\SV`V4VPVn\VRR7Vn\ V4VnVP4R#rq)r+r, num_labelsrMr%IBertClassificationHead classifierrQrGs&&rKr,'IBertForSequenceClassification.__init__?sC   ++%@ 1&9 rMc<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[R,R S[R,R S[R,R S[S[S[P,,/ #r})r9r^r_r`rr )rFrus"rKrG+IBertForSequenceClassification.__annotate__JsA A ##d*A ))D0A ((4/ A &&- A ((4/ A   4'A  $;A #TkA D[A  "E%*;*;$< <A rMc tV eT MVPPp VPVVVVVVVV R7p V ^,p VPV 4p RpVEeVPPfVP ^8XdRVPnMrVP ^8dQVP \P8Xg VP \P8XdRVPnMRVPnVPPR8XdK\4pVP ^8Xd'V!V P4VP44pMV!W4pMVPPR8Xd>\4pV!V PRVP 4VPR44pM-VPPR8Xd\4pV!W4pV 'g"V 3V R,,pVe V3V,#T#\VV V P V P"R7#) ae labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nr regressionsingle_label_classificationmulti_label_classificationrrr*)rIrr%r problem_typerrPr9rZrrsqueezerrrrrr rHr[rr\r=r]r~rrrrfrrlrrrrs&&&&&&&&&&, rKre&IBertForSequenceClassification.forwardIs(&1%?^bLXLdL--b,2C2CB2GHjnR`Rln11"n6I6I"6MNrvR`Rln11"n6I6I"6MNrv(   r=#5#5b#9=;M;Mb;Q R  ** *..,/!5#     ]3 / ++b+6  ')HOV4D%''"+5F)-)9TGf$ Ev E("!//))   rM)rrFr%rrrts@@rKrrs,V V V V rMrcRaa]tRtRtoV3Rlt]RV3RlRll4tRtVtV;t #)IBertForTokenClassificationic6<\SV`V4VPVn\VRR7Vn\ P !VP4Vn\ P!VPVP4Vn VP4R#rq) r+r,rrMr%rrDrErFrr3rrQrGs&&rKr,$IBertForTokenClassification.__init__sk   ++%@ zz&"<"<= ))F$6$68I8IJ rMc<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[R,R S[R,R S[R,R S[S[S[P,,/ #r})r9r^r_r`rr )rFrus"rKrG(IBertForTokenClassification.__annotate__s1 1 ##d*1 ))D01 ((4/ 1 &&- 1 ((4/ 1   4'1  $;1 #Tk1 D[1  u'8'8!9 91 rMc V eT MVPPp VPVVVVVVVV R7p V ^,p VPV 4p VP V 4p RpVe=\ 4pV!V P RVP4VP R44pV 'g"V 3V R,,pVe V3V,#T#\VV V PV PR7#)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrrr*) rIrr%rFrrrrrrr rs&&&&&&&&&&, rKre#IBertForTokenClassification.forwards$&1%rs" AA&6.,,cc   H %q=biiq=h@.@.F,;bii,;^/RYY/8; ;D,;")),;^5959p6 2996 r "))  &\?&\&\Rg %g g TG +G G T")), M %9M  M `b 1b b J> "6> > Bbii&I 4I I X4" rM