+ joRt^RIt^RIt^RIHt^RIt^RIHt^RIHtH t H t ^RI H t ^RIHt^RIHtHtHt^RIHt^R IHtHt^R IHt^R IHtHtHtHtH t H!t!H"t"^R I#H$t$H%t%^R I&H't'^RI(H)t)H*t*H+t+H,t,H-t-H.t.^RI/H0t0H1t1^RI2H3t3H4t4^RI5H6t6]-Pn!]84t9R9RRllt:!RR]Pv4t<!RR]Pv4t=!RR]Pv4t>!RR]Pv4t?!RR]Pv4t@!RR]Pv4tA!R R!]Pv4tB!R"R#]4tC!R$R%]Pv4tD],!R&R']%44tE!R(R)]E4tF],!R*R+]E44tG],!R,R-7!R.R/]E]44tH],!R0R1]E44tI],!R2R-7!R3R4]E44tJ],!R5R6]E44tK],!R7R8]E44tL.R:OtMR#);zPyTorch mT5 model.N)Callable)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) DUMMY_INPUTS DUMMY_MASKTransformersKwargsauto_docstringloggingtorch_compilable_check)can_return_tuplemerge_with_config_defaults)OutputRecordercapture_outputs MT5Configc<V^8dQhR\PR\PR\PR\PR\PR,R\R,R\R \PR,R \ \ ,/ #) modulequerykeyvalueattention_maskNscalingdropout position_biaskwargs)rModuletorchTensorfloatrr)formats"e/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/mt5/modeling_mt5.py __annotate__r73s%% II% <<% % << % LL4' % T\ %%<<$&%'(%c VfVPR4R,p\P!WP^^44V,p Ve W,p Ve W,p \P P V RR7p \P PWVPR7p \P!W4p V P^^4P4p W3#)Ndim)ptraining) sizer2matmul transposer functionalsoftmaxr.r= contiguous) r(r)r*r+r,r-r.r/r0 attn_weights attn_outputs &&&&&&&&, r6eager_attention_forwardrH3s**R.D(<<}}Q':;gEL #3 !#4 ==((2(>L==((6??([L,,|3K''1-88:K  $$r8c<aa]tRt^TtoRV3RlltRtRtVtV;t#) MT5LayerNormc <\SV`4\P!\P !V44VnW nR#)zT Construct a layernorm module in the MT5 style. No bias and no subtraction of mean. N)super__init__r Parameterr2onesweightvariance_epsilon)self hidden_sizeeps __class__s&&&r6rMMT5LayerNorm.__init__Us/ ll5::k#:; #r8cVP\P4P^4P RRR7pV\P !W P ,4,pVPP\P\P39d&VPVPP4pVPV,#)r'T)keepdimr>) tor2float32powmeanrsqrtrQrPdtypefloat16bfloat16)rR hidden_statesvariances&& r6forwardMT5LayerNorm.forward]s !##EMM266q9>>r4>P% H?T?T4T(UU  ;;   ? ?),,T[[->->?M{{]**r8)rQrP)gư> __name__ __module__ __qualname____firstlineno__rMrc__static_attributes____classdictcell__ __classcell__rU __classdict__s@@r6rJrJTs$ + +r8rJcDaa]tRt^ntoV3RlV3RlltRtRtVtV;t#)MT5DenseActDensec <V^8dQhRS[/#r'configr$)r5rns"r6r7MT5DenseActDense.__annotate__os//y/r8cn<\SV`4\P!VPVP RR7Vn\P!VP VPRR7Vn\P!VP4Vn \VP,Vn R#FbiasN)rLrMrLineard_modeld_ffwiwoDropout dropout_rater.r dense_act_fnactrRrsrUs&&r6rMMT5DenseActDense.__init__osn ))FNNFKKeD))FKKeDzz&"5"56 &--.r8cVPV4pVPV4pVPV4p\VPP \ P4'dVPVPP P8wdcVPP P\ P8wd0VPVPP P4pVP V4pV#N) r|rr. isinstancer}rPr2r3r^int8rYrRras&&r6rcMT5DenseActDense.forwardvs . /  ]3 tww~~u|| 4 4##tww~~';';;$$ 2),,TWW^^-A-ABM . r8)rr.r|r}rerms@@r6rprpns//  r8rpcDaa]tRt^toV3RlV3RlltRtRtVtV;t#)MT5DenseGatedActDensec <V^8dQhRS[/#rrr$)r5rns"r6r7"MT5DenseGatedActDense.__annotate__s//y/r8c<\SV`4\P!VPVP RR7Vn\P!VPVP RR7Vn\P!VP VPRR7Vn\P!VP4Vn \VP,VnR#rv)rLrMrryrzr{wi_0wi_1r}r~rr.rrrrs&&r6rMMT5DenseGatedActDense.__init__s IIfnnfkkF IIfnnfkkF ))FKKeDzz&"5"56 &--.r8cFVPVPV44pVPV4pW#,pVPV4p\ VP P \P4'dVPVP P P8wdcVP P P\P8wd0VPVP P P4pVP V4pV#r) rrrr.rr}rPr2r3r^rrY)rRra hidden_gelu hidden_linears&& r6rcMT5DenseGatedActDense.forwardshhtyy78  -0 #3  ]3 tww~~u|| 4 4##tww~~';';;$$ 2),,TWW^^-A-ABM . r8)rr.rrr}rerms@@r6rrs//r8rcDaa]tRt^toV3RlV3RlltRtRtVtV;t#) MT5LayerFFc <V^8dQhRS[/#rrr$)r5rns"r6r7MT5LayerFF.__annotate__s77y7r8c <\SV`4VP'd\V4VnM\ V4Vn\ VPVPR7Vn \P!VP4Vn R#)rTN)rLrM is_gated_actrDenseReluDenserprJrzlayer_norm_epsilon layer_normrr~rr.rs&&r6rMMT5LayerFF.__init__sb     "7"?D "26":D &v~~6;T;TUzz&"5"56 r8cxVPV4pVPV4pWPV4,pV#r)rrr.)rRraforwarded_statess&& r6rcMT5LayerFF.forwards;??=9../?@% 5E(FF r8)rr.rrerms@@r6rrs77r8rcvaa]tRt^toRV3RlV3Rlllt]R Rl4tR RltR V3RlRlltRt Vt V;t #) MT5Attentionc:<V^8dQhRS[RS[R,RS[/#)r'rs layer_idxN is_causal)r%intbool)r5rns"r6r7MT5Attention.__annotate__s-%,%,%,: %,  %,r8c<<\SV`4WnVPVnW@nW nVP VnVPVnVPVnVPVn VPVn VPVnVPVP,VnRVnW0nVf@VP'd.\$P'RVP(P* R24\,P.!VPVPRR7Vn\,P.!VPVPRR7Vn\,P.!VPVPRR7Vn\,P.!VPVPRR7VnVP 'd1\,P8!VP VP4VnRVnR#)?NzInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.Frw)rLrMrs is_decoderrhas_relative_attention_biasrelative_attention_num_bucketsrelative_attention_max_distancerzd_kvkey_value_proj_dim num_headsn_headsrr. inner_dimr-rlogger warning_oncerUrfrryqkvo Embeddingrelative_attention_biasgradient_checkpointing)rRrsrrrrUs&&&&&r6rMMT5Attention.__init__s{   ++"+F(.4.S.S+/5/U/U,~~ "(++'' ** (?(?? "     *4>>+B+B*CD,,  4<<eD4<<eD4<<eD4>>4<=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket. This should allow for more graceful generalization to longer sequences than the model has been trained on Args: relative_position: an int32 Tensor bidirectional: a boolean - whether the attention is bidirectional num_buckets: an integer max_distance: an integer Returns: a Tensor with the same shape as relative_position, containing int32 values in the range [0, num_buckets) ) rYr2longabsmin zeros_likelogr4math full_likewhere)relative_position bidirectional num_buckets max_distancerelative_buckets max_exactis_smallrelative_position_if_larges&&&& r6_relative_position_bucket&MT5Attention._relative_position_buckets,  A K Q!6 : :5:: F T T  % *; < !&+N>NO`>a!b b  1$ $0&/ II'--/); <hh|/0 1& ( "UZZ. &" &+YY &8RbcTc(d& " EKKE_``r8c Vf!VPPPp\P!V\P VR7R,V,p\P!V\P VR7R,pWe, pVP VVP'*VPVPR7pVPV4p V P.RO4P^4p V #)z%Compute binned relative position bias)r^device)rrr)NNNN)Nr)r') rrPrr2arangerrrrrpermute unsqueeze) rR query_length key_lengthrpast_seen_tokenscontext_positionmemory_positionrrelative_position_bucketvaluess &&&&& r6 compute_biasMT5Attention.compute_biass >1188??F << EJJvVW^_brr,,zFST[\+>#'#A#A #.;;== $B$  --.FG *44Q7 r8c0<V^8dQhRS[S[,/#r'r0rr)r5rns"r6r7rs X8X8+,X8r8c pVPRR p.VOR NVPN5pVeVPVP4M^p \ V \ P 4'dV P4MT p VRJp VPV4PV4P^^4p Rp \ V\4'dHVPPVP4p V 'dVPp MVPp MTp V 'dTMTpV 'd\VeXV 'dPV P VP,P"pV P VP,P$pM.VPRR OR NVPN5pVP'V4PV4P^^4pVP)V4PV4P^^4pVeXV P+VVVP4wppV 'd0\ V\4'dRVPVP&VfVPR ,pVP,'go\ P.!^VP0V^,V3V P2V P4R7pVP6'dVP8'dRVnM&VP=V^,VV P2V R7p\>P@!VPBPD\F4pT!VV VVV3RVP8'gRM VPHRVPJR V/VBwppVPL!.VOR N5!PO4pVPQV4pVVV3#) zp Self-attention (if key_value_states is None) or attention over source sentence (provided by key_value_states). NFT)rr^)rrr.r-r/r>))shaperget_seq_lengthrrr2r3clonerviewrBr is_updatedgetcross_attention_cacheself_attention_cachelayerskeysrrrupdaterzerosrrr^rr= requires_gradrr get_interfacers_attn_implementationrHr.r-reshaperEr)rRramaskkey_value_statesr/past_key_valuesr0 input_shape hidden_shaperis_cross_attention query_statesrcurr_past_key_valuescurrent_states key_states value_stateskv_shaperattention_interfacerGrFs&&&&&&, r6rcMT5Attention.forwardsY$))#2. BBbB$*A*AB M\Mh?99$..Ino7ABRTYT`T`7a7a+113gw.T9vvm,11,?II!QO  o': ; ;(3377GJ!'6'L'L$'6'K'K$#2 -?)] /"=*-44T^^DIIJ/66t~~FMMLP--cr2PBP8O8OPH/44X>HHANJ66.166x@JJ1aPL*+?+F+FzS_aeaoao+p( L%*_FY*Z*ZAEO..t~~>  #))"-J333 %  k!njA'..&,,! ...4===26M/ $ 1 1NJ|7J7J]m!2! )@(M(M KK , ,.E) %8      % $}}}C$,, % LL % ( %  % ! \"));;;;FFH ff[) M<77r8)rsrzr.rrrrrrrrrrrrrrr-r)FNF)T )NrNNNN) rfrgrhrirM staticmethodrrrcrjrkrlrms@@r6rrs<%,%,N- - ^"X8X8X8r8rcXaa]tRtRtoRV3RlV3RllltRV3RlRlltRtVtV;t#) MT5LayerSelfAttentioni{c.<V^8dQhRS[R,/#r'rNr)r5rns"r6r7"MT5LayerSelfAttention.__annotate__|s 7 7SSWZ 7r8c<\SV`4\VVVVPR7Vn\ VP VPR7Vn\P!VP4Vn R#)rrrrN) rLrMrr SelfAttentionrJrzrrrr~rr.rRrsrrrUs&&&&r6rMMT5LayerSelfAttention.__init__|s] ) (C''   'v~~6;T;TUzz&"5"56 r8c0<V^8dQhRS[S[,/#rr)r5rns"r6r7rs== +, =r8c VPV4pVP!V3RVRVRV/VBwrspWPV4,pWsV3#)rr/r)rrr.) rRrar,r/rr0normed_hidden_statesattention_outputrFs &&&&&, r6rcMT5LayerSelfAttention.forwardsr $}=8<8J8J 9 9 (9 , 9  9 5)<<8H+II <) r1r^r2r_risinfanyfinfor6clampr)rRrar,r/encoder_hidden_statesencoder_attention_maskencoder_decoder_position_biasrr0self_attn_position_bias_ clamp_valuecross_attn_position_biasdo_cross_attentions&&&&&&&&, r6rcMT5Block.forwardsP59JJqM 5 )5 (5 , 5  5 1    %-- /++ M*..0 M//044t; M//044K "KK <[YM#' !__RR1Fd1R 9=A:!6: 6:< : !0 :  : 6MQ""emm3#kkKK .224KK 3 34884?KK 3 3488 !& M|Q\ ]  2}5    %-- /++ M*..0 M//044t; M//044K "KK <[YM7OOOr8)rr1r)NNNNNNrerms@@r6r+r+s! . .=P=P=Pr8r+cTaa]tRtRtoRtV3RlV3RlltV3RlRltRtVtV;t #)MT5ClassificationHeadi z-Head for sentence-level classification tasks.c <V^8dQhRS[/#rrr$)r5rns"r6r7"MT5ClassificationHead.__annotate__ sEEyEr8c0<\SV`4\P!VPVP4Vn\P !VPR7Vn\P!VPVP4Vn R#))r<N) rLrMrryrzdenser~classifier_dropoutr. num_labelsout_projrs&&r6rMMT5ClassificationHead.__init__ sZ YYv~~v~~> zzF$=$=>  &..&2C2CD r8cN<V^8dQhRS[PRS[P/#)r'rareturnr2r3)r5rns"r6r7rGs#U\\ellr8cVPV4pVPV4p\P!V4pVPV4pVP V4pV#r)r.rIr2tanhrLrs&&r6rcMT5ClassificationHead.forwardsN ]3  =1  =1  ]3  m4 r8)rIr.rL) rfrgrhri__doc__rMrcrjrkrlrms@@r6rErE s#7EE r8rEc aa]tRtRtoRtRtR.tR.tRtRt Rt Rt R] !] ^R7R ] !]RR7R ] !]RR7/t]R 4t]P(!4V3R l4tR tV3RltRtVtV;t#)MT5PreTrainedModeli transformerTr+r}Fra)index attentionscross_attentionsc~\P!\4p\P!\4pRVRVRV/pV#)decoder_input_ids input_idsdecoder_attention_mask)r2tensorrr)rRr] input_mask dummy_inputss& r6raMT5PreTrainedModel.dummy_inputs2s<LL. \\*-   $j r8c <\SV`V4VPPp\ V\ 4'd+\ P!VPVR,4R#\ V\\\\34'Ed%\ P!VPPRVR,R7\VR4'dQVPP 'g5\ P!VP"PRVR,R7\VR4'd{\ P!VP$PRW PP&R ,,R7\ P(!VP$P*4R#R#\ V\,4'du\VR4'da\ P!VP.PRVR,R7\ P(!VP.P*4R#R#\ V\04'Ed_\ P!VP2PRW PP&R ,,R7\VP2R4'dCVP2P*e+\ P(!VP2P*4\ P!VP4PRW PP&R ,,R7\VP4R4'dGVP4P*e-\ P(!VP4P*4R#R#R#\ V\64'Ed_\ P!VP8PRW PP&R ,,R7\VP8R4'dCVP8P*e+\ P(!VP8P*4\ P!VP:PRW PP<R ,,R7\VP:R4'dGVP:P*e-\ P(!VP:P*4R#R#R#\ V\>4'Ed \ P!VP@PRW PP&R ,,R7\VP@R4'dCVP@P*e+\ P(!VP@P*4\ P!VPBPRW PP&R ,,R7\VPBR4'dCVPBP*e+\ P(!VPBP*4\ P!VP:PRW PP<R ,,R7\VP:R4'dGVP:P*e-\ P(!VP:P*4R#R#R#\ V\D4'EdVPP&pVPPFpVPPHp\ P!VPJPRW#V,R ,,R7\ P!VPLPRW#R ,,R7\ P!VPNPRW#R ,,R7\ P!VPPPRW%V,R ,,R7VPR'd=\ P!VPTPRW#R ,,R7R#R#R#) zInitialize the weightsrr)r\stdlm_head qa_outputs classifierrxNr?)+rL _init_weightsrsinitializer_factorrrJinit constant_rPMT5ModelMT5ForConditionalGenerationMT5EncoderModelMT5ForQuestionAnsweringnormal_sharedhasattrtie_word_embeddingsrerfrzzeros_rxMT5ForTokenClassificationrgrErIrLrpr|r}r{rrrrrrrrrrrr)rRr(factorrzrrrUs&& r6rh MT5PreTrainedModel._init_weights=sd f%// fl + + NN6==&3, 7   2OE\ ]   LL--CVc\ Jvy))$++2Q2Q2Q V^^22&3,Ov|,, V..55CVP[P[PcPchlOlEmn F--223- 9 : :v|,, V..55CVc\R F--223- 5 6 6 LL,,3F{{GZGZ_cFcVPPpVPPpVf \R4hVP VP 4pVRRR13,P 4VR&W$R&Vf \R4hVPVR8HV4V#)Nzself.model.config.decoder_start_token_id has to be defined. In MT5 it is usually set to the pad_token_id. See MT5 docs for more information..z1self.model.config.pad_token_id has to be defined.r>).:rNN).r)rsdecoder_start_token_id pad_token_id ValueError new_zerosrr masked_fill_)rRr]rzr{shifted_input_idss&& r6 _shift_rightMT5PreTrainedModel._shift_rightus!%!C!C{{// ! )5  &// @%.sCRCx%8%>%>%@'"$:&!  PQ Q&&'8D'@,O  r8c&<V^8dQh/S[;R&#rrr$)r5rns"r6r7MT5PreTrainedModel.__annotate__s r8r>)rfrgrhribase_model_prefixsupports_gradient_checkpointing_no_split_modules_keep_in_fp32_modules_supports_flash_attn_supports_flex_attn_supports_sdpa_can_compile_fullgraphr"r+rr _can_record_outputspropertyrar2no_gradrhr__annotate_func__rjrkrlrms@@r6rVrVs&&*## !F!N! :n%:"EN+AL  ]]_5p5pn!sr8rVclaa]tRtRtoV3RltRt]]]RV3RlRll444t Rt Vt V;t #)MT5Stackic "<\SV`V4\P!VPVP 4VnVPVn\P!\VP4Uu.uFp\V\V^8H4VR7NK up4Vn \VP VPR7Vn\P"!VP$4VnVP)4RVnR#uupi)rr/rFN)rLrMrr vocab_sizerz embed_tokensrr0range num_layersr+rblockrJrfinal_layer_normr~rr. post_initr)rRrsirUs&& r6rMMT5Stack.__init__s  LL):):FNNK ++]]^cdjdudu^v w^vYZXf$qAv,RS T^v w  !-V^^AZAZ [zz&"5"56  &+# xs9#D cWnR#r)rrRnew_embeddingss&&r6set_input_embeddingsMT5Stack.set_input_embeddingss*r8c0<V^8dQhRS[S[,/#rr)r5rns"r6r7MT5Stack.__annotate__s W W +,W r8c <VRJVRJ, 'd \R4hVfVPV4pVRJd"VP'g\RV R24hVP'duV'dlVfhVPP'd6\ \ VPR7\ VPR74pM+\ VPR7pMVP'gRpVPP'd=\VPTT\V\ 4'd VPMTR7pM\VPVVR7pRp VP'dVe\VPVVVR7p Rp Rp VPV4p VPF4p V !V VV VV V 3R V/VBwrpTp VP'gK,VfK2Tp K6 VPV 4p VPV 4p \V VR 7#) Nz:You must specify exactly one of input_ids or inputs_embedsTz)`use_cache` can only be set to `True` if z is used as a decoder)rs)rs inputs_embedsr,r)rsrr,)rsrr,r;r)last_hidden_stater)r|rrrsis_encoder_decoderr r rrrr r.rrr)rRr]r,r;r<rr use_cacher0encoder_extended_attention_maskr/r=ra layer_moduleself_attention_position_biascross_attention_position_biass&&&&&&&&, r6rcMT5Stack.forwards  -t"; < <YZ Z   --i8M  ??? #LTFRg!hii ???_4;;111&9$DKK8,dkk:Z'O'3$++&FO#O ;; ! ! !/{{+-o/BCC!0 D D$ N7{{+-N +/' ???4@.G{{+5&; / + (,% ]3  JJLYe%/- Z!0 Z Z VM9V9M#8#D0M-'"--m<  ]3 8++  r8)rr.rrrr)NNNNNNN) rfrgrhrirMrr!r#rrcrjrkrlrms@@r6rrsA, + W W  W W r8rcaa]tRtRtoRtRtR.tRRRR/tV3RlV3R lltR t R t ] ] RV3R lR ll44t V3RltRtVtV;t#)rliaG Examples: ```python >>> from transformers import MT5Model, AutoTokenizer >>> model = MT5Model.from_pretrained("google/mt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/mt5-small") >>> article = "UN Offizier sagt, dass weiter verhandelt werden muss in Syrien." >>> summary = "Weiter Verhandlung in Syrien." >>> inputs = tokenizer(article, return_tensors="pt") >>> labels = tokenizer(text_target=summary, return_tensors="pt") >>> outputs = model(input_ids=inputs["input_ids"], decoder_input_ids=labels["input_ids"]) >>> hidden_states = outputs.last_hidden_state ```mt5Fdecoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weightencoder.embed_tokens.weight shared.weightdecoder.embed_tokens.weightc <V^8dQhRS[/#rrr$)r5rns"r6r7MT5Model.__annotate__syr8c<\SV`V4\P!VPVP 4Vn\P!V4pRVn RVn \V4Vn \P!V4pRVn VPVn\V4VnVP!4R#FTN)rLrMrrrrzrqcopydeepcopyrrrencodernum_decoder_layersrdecoderrrRrsencoder_configdecoder_configrUs&& r6rMMT5Model.__init__s  ll6#4#4fnnE v.$)!#( / v.$(!$*$=$=!/  r8cVP#rrqrRs&r6get_input_embeddingsMT5Model.get_input_embeddings+ {{r8c~WnVPPV4VPPV4R#rrqrrrrs&&r6rMT5Model.set_input_embeddings/+$  )).9 )).9r8c<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[S[S[P,,R,RS[R,RS[P R,R S[P R,R S[R,R S[S[ ,R S[ / #) r'r]Nr,r\r^encoder_outputsrrdecoder_inputs_embedsrr0rO) r2 LongTensor FloatTensor BoolTensortupler r3rrrr)r5rns"r6r7r7sd d ##d*d ))D0d !++d2 d !& 0 04 7 d uU%6%6784? d d ||d*d  %||d2d $;d +,d  d r8c  VfVP!R RVRVRV/V BpM]\V\4'gH\V^,\V4^8d V^,MR\V4^8d V^,MRR7pV^,p VP!R RVRVRVRVRV RVR V /V Bp \ V P V PV PV PV PVP VPVPR 7#) aD input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. MT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [MT5 Training](./mt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) MT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [MT5 Training](./mt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Example: ```python >>> from transformers import AutoTokenizer, MT5Model >>> tokenizer = AutoTokenizer.from_pretrained("google/mt5-small") >>> model = MT5Model.from_pretrained("google/mt5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> decoder_input_ids = tokenizer("Studies show that", return_tensors="pt").input_ids # Batch size 1 >>> # preprocess: Prepend decoder_input_ids with start token which is pad token for MT5Model. >>> # This is not needed for torch's MT5ForConditionalGeneration as it does this internally using labels arg. >>> decoder_input_ids = model._shift_right(decoder_input_ids) >>> # forward pass >>> outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) >>> last_hidden_states = outputs.last_hidden_state ```Nr]r,rrrarYrr;r<r)rrdecoder_hidden_statesdecoder_attentionsrZencoder_last_hidden_stater;encoder_attentionsr) rrrlenrrrrrarYrZ) rRr]r,r\r^rrrrrr0radecoder_outputss &&&&&&&&&&, r6rcMT5Model.forward4sJD  ""ll#-, O O_==-"1!"4474H14Loa0RV14_1E1I?1-tO (* ,,  '  1  0  ,  #0  $2       "-??+;;"1"?"?.99,==&5&G&G"1"?"?.99  r8c&<V^8dQh/S[;R&#rrr$)r5rns"r6r7rs( )r8)rrrq NNNNNNNNN)rfrgrhrirT model_type"_keys_to_ignore_on_load_unexpected_tied_weights_keysrMrrr rrcrrjrkrlrms@@r6rlrlsl "J*r)s&%% $: d d d ur8rlz; MT5 Model with a `language modeling` head on top. ) custom_introcaa]tRtRtoRtRtR.tRRRRRR/tV3R lV3R lltR t R t ] ] RV3R lRll44t V3RlRltV3RltRtVtV;t#)rmia Examples: ```python >>> from transformers import MT5ForConditionalGeneration, AutoTokenizer >>> model = MT5ForConditionalGeneration.from_pretrained("google/mt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/mt5-small") >>> article = "UN Offizier sagt, dass weiter verhandelt werden muss in Syrien." >>> summary = "Weiter Verhandlung in Syrien." >>> inputs = tokenizer(article, text_target=summary, return_tensors="pt") >>> outputs = model(**inputs) >>> loss = outputs.loss ```rrrrrzlm_head.weightc <V^8dQhRS[/#rrr$)r5rns"r6r7(MT5ForConditionalGeneration.__annotate__syr8c<\SV`V4VPVn\P !VP VP4Vn\P!V4pRVn RVn \V4Vn \P!V4pRVn VPVn\V4Vn\P"!VPVP RR7VnVP'4R#)FTrwN)rLrMrz model_dimrrrrqrrrrrrrrrryrerrs&& r6rM$MT5ForConditionalGeneration.__init__s  ll6#4#4fnnE v.$)!#( / v.$(!$*$=$=!/ yy1B1BO  r8cVP#rrrs&r6r0MT5ForConditionalGeneration.get_input_embeddingsrr8c~WnVPPV4VPPV4R#rrrs&&r6r0MT5ForConditionalGeneration.set_input_embeddingsrr8c<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[S[S[P ,,R,RS[R,RS[PR,R S[PR,R S[PR,R S[R,R S[S[ ,R S[ / #)r'r]Nr,r\r^rrrrlabelsrr0rO) r2rrrrr3r rrrr)r5rns"r6r7rs} } ##d*} ))D0} !++d2 } !& 0 04 7 } uU\\23d: } } ((4/}  %0047}   4'} $;} +,}  } r8c  2VfVP!R RVRVRV/V BpM]\V\4'gH\V^,\V4^8d V^,MR\V4^8d V^,MRR7pV^,p V eVfVfVP V 4pVP !R RVRVRVRVRV RVR V /V Bp V ^,pVP V4pRpV e_\R R 7pV PVP4p V!VPRVPR44V PR44p\VVV PV PV PV P VP"VPVPR 7 #)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. MT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [MT5 Training](./mt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) MT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [MT5 Training](./mt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import AutoTokenizer, MT5ForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("google/mt5-small") >>> model = MT5ForConditionalGeneration.from_pretrained("google/mt5-small") >>> # training >>> input_ids = tokenizer("The walks in park", return_tensors="pt").input_ids >>> labels = tokenizer(" cute dog the ", return_tensors="pt").input_ids >>> outputs = model(input_ids=input_ids, labels=labels) >>> loss = outputs.loss >>> logits = outputs.logits >>> # inference >>> input_ids = tokenizer( ... "summarize: studies have shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> outputs = model.generate(input_ids) >>> print(tokenizer.decode(outputs[0], skip_special_tokens=True)) >>> # studies have shown that owning a dog is good for you. ```Nr]r,rrrr;r<r ignore_index losslogitsrrrrZrr;rrryr>)rrrrrrrerrYrrr@rrrarYrZr)rRr]r,r\r^rrrrrrr0rarsequence_output lm_logitsrloss_fcts&&&&&&&&&&&, r6rc#MT5ForConditionalGeneration.forwardsR  ""ll#-, O O_==-"1!"4474H14Loa0RV14_1E1I?1-tO (*  "3";@U@] $ 1 1& 9 ,,  '  1  0  ,  #0  $2       *!,LL1   'T:HYYy//0FINN2y~~b/ABFKKPROTD+;;"1"?"?.99,==&5&G&G"1"?"?.99  r8c4<V^8dQhRS[P/#)r'rrP)r5rns"r6r7r]s))ELL)r8c$VPV4#r)r)rRrs&&r6%prepare_decoder_input_ids_from_labelsAMT5ForConditionalGeneration.prepare_decoder_input_ids_from_labels]s  ((r8c&<V^8dQh/S[;R&#rrr$)r5rns"r6r7rs. /r8)rrrerrq NNNNNNNNNN)rfrgrhrirTrrrrMrrr rrcrrrjrkrlrms@@r6rmrms|  J*r)s&%%/*: } } } @))r8rmcaa]tRtRtoRtRtRR/tV3RlV3RlltRtR t ] ] RV3R lR ll44t V3R lt R tVtV;t#)rniaa Examples: ```python >>> from transformers import MT5EncoderModel, AutoTokenizer >>> model = MT5EncoderModel.from_pretrained("google/mt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/mt5-small") >>> article = "UN Offizier sagt, dass weiter verhandelt werden muss in Syrien." >>> input_ids = tokenizer(article, return_tensors="pt").input_ids >>> outputs = model(input_ids) >>> hidden_state = outputs.last_hidden_state ```rrrc <V^8dQhRS[/#rrr$)r5rns"r6r7MT5EncoderModel.__annotate__xs  y r8c<\SV`V4\P!VPVP 4VnTpRVnRVn\V4Vn VP4R#r) rLrMrrrrzrqrrrrr)rRrsrrUs&& r6rMMT5EncoderModel.__init__xsY  ll6#4#4fnnE #( ,1)/  r8cVP#rrrs&r6r$MT5EncoderModel.get_input_embeddingsrr8cHWnVPPV4R#r)rqrrrs&&r6r$MT5EncoderModel.set_input_embeddingss$  )).9r8c <V^8dQhRS[PR,RS[PR,RS[PR,RS[S[,RS[/#)r'r]Nr,rr0rO)r2rrrrr)r5rns"r6r7rsd%%##d*%))D0%((4/ % +, %  %r8c 6VP!RRVRVRV/VBpV#)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. MT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. To know more on how to prepare `input_ids` for pretraining take a look a [MT5 Training](./mt5#training). Example: ```python >>> from transformers import AutoTokenizer, MT5EncoderModel >>> tokenizer = AutoTokenizer.from_pretrained("google/mt5-small") >>> model = MT5EncoderModel.from_pretrained("google/mt5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> outputs = model(input_ids=input_ids) >>> last_hidden_states = outputs.last_hidden_state ```r]r,rr)r)rRr]r,rr0rs&&&&, r6rcMT5EncoderModel.forwards?B,0<<, , ), (,  , r8c&<V^8dQh/S[;R&#rrr$)r5rns"r6r7ras" #r8)rrqr)rfrgrhrirTrrrMrrr rrcrrjrkrlrms@@r6rnrnas]  J &  :%%%_r8rnz MT5 model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cnaa]tRtRtoR.tV3RlV3Rllt]]RV3RlRll44tRt Vt V;t #) MT5ForSequenceClassificationirc <V^8dQhRS[/#rrr$)r5rns"r6r7)MT5ForSequenceClassification.__annotate__syr8c<\SV`V4\V4Vn\ V4VnVP 4R#r)rLrMrlrWrEclassification_headrrs&&r6rM%MT5ForSequenceClassification.__init__s6  #F+#8#@  r8c<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[S[P,R,RS[PR,RS[PR,R S[PR,R S[R,R S[S[,R S[/ #) r'r]Nr,r\r^rrrrrr0rO) r2rr3listrrrrr)r5rns"r6r7r s} } ##d*}  t+} !++d2 } !& 0 04 7 } e//047 } ((4/}  %0047}   4'} $;} +,}  )} r8c  RVeRp Vf'Ve#\RVPP 24hVf%Vf!Vf \R4hVP V4pVP !V3RVRVRVRVR VR VR V /V Bp V ^,p VP VPP4PV P4p \\P!V P^44P4^8HR 4V P wrpWR 3,p\VP ^,V,^8R4VP#VRV4R,pVP%V4pRpVEeVPVP4pVPP&fVPP(^8XdRVPnM|VPP(^8dQVP*\P,8Xg VP*\P.8XdRVPnMRVPnVPP&R8XdV\14pVPP(^8Xd'V!VP34VP344pMV!VV4pMVPP&R8XdH\54pV!VP#RVPP(4VP#R44pM.VPP&R8Xd\74pV!VV4p\9VVV P:V P<V P>V P@V PBV PDV PFR7 #)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. MT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [MT5 Training](./mt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) MT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [MT5 Training](./mt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NFz8Passing input embeddings is currently not supported for If no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.r,r\r^rrrrz7All examples must have the same number of tokens.rz3Each example must contain at least one token. regressionsingle_label_classificationmulti_label_classificationrr>)rr>r)$NotImplementedErrorrUrfr|rrWeqrs eos_token_idrYrrr2unique_consecutivesumnumelrrr problem_typerKr^rrrsqueezerrrrrrrZrr;r)rRr]r,r\r^rrrrrr0outputsreos_mask batch_sizer?rSselectedsentence_representationrrrs&&&&&&&&&&, r6rc$MT5ForSequenceClassification.forwardsM^  I  !:%J4>>KbKbJcd   $)>)F  U !% 1 1) < &*&6&6  ' ) ' 0 ' $: ' , ' ( ' #8 '   '  ' "!*<< 8 89<<_=S=ST  $ $X\\!_ 5 ; ; = B E &5%:%:" {"Q;/ NN1  +q 0 A #+-- B "LX"V))*AB  YYv}}-F{{''/;;))Q./;DKK,[[++a/V\\UZZ5OSYS_S_chclclSl/LDKK,/KDKK,{{''<7"9;;))Q.#FNN$4fnn6FGD#FF3D))-JJ+- B 0F0F GUWY))-II,./.#33")"?"?&99$55&-&G&G")"?"?&99  r8)rrWr) rfrgrhrirrMr rrcrjrkrlrms@@r6r r sE+s)s&} } } } r8r chaa]tRtRtoV3RlV3Rllt]]RV3RlRll44tRtVt V;t #)ruiMc <V^8dQhRS[/#rrr$)r5rns"r6r7&MT5ForTokenClassification.__annotate__Ps  y r8c2<\SV`V4VPVn\V4Vn\ P !VP4Vn\ P!VPVP4Vn VP4R#r) rLrMrKrnrWrr~rJr.ryrSrgrrs&&r6rM"MT5ForTokenClassification.__init__Psj   ++*62zz&";";< ))F$6$68I8IJ r8c <V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[S[,RS[/#)r'r]Nr,rrr0rO)r2r3rrr)r5rns"r6r7r(^sq+ + <<$&+  t++ ||d* +  t# + +, +  + r8c PVP!V3RVRV/VBpV^,pVPV4pVPV4pRp Ve=\4p V !VP RVP 4VP R44p \ V VVPVPR7#)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. MT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [MT5 Training](./t5#training). labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. r,rN)rrrarYr>) rWr.rgrrrKrrarY) rRr]r,rrr0r rarrrs &&&&&, r6rc!MT5ForTokenClassification.forward[s2$(#3#3 $ )$ ($  $    ]3 /  ')HFKKDOO)r|rrrrrrrfsplitrrEr@rYrr:rrrrarYrZr)rRr]r,r\r^rr8r9rrrr0rarrrr;r< total_loss ignored_indexr start_lossend_losss&&&&&&&&&&&, r6rcMT5ForQuestionAnswering.forwardsZ  &=+DI  $)>)F  U !% 1 1) <   ""ll#-, O O_==-"1!"4474H14Loa0RV14_1E1I?1-tO (* ,,  '  1  0  !  #0  $2       *!,1#)<<r<#: j#++B/::< ''+668   &=+D?'')*Q."1"9"9""="@"@ATAT"U=%%'(1, - 5 5b 9 < rTs $ AA&!CC)J9G&rrIE(   H %%B+299+4ryy.BII<&A8299A8J=BII=B9RYY9@JP)JP\BII$j!j!j!\n !n b] !] ] @ {)"4o{)  {)|S(SSl L #5L  L ^; 2; ; |d 0d d N r8