+ jRt^RIt^RIt^RIHt^RIt^RIHt^RIHtH t H t ^RI H t ^RIHt^RIHtHtHt^RIHt^R IHtHt^R IHt^R IHtHtHtHtH t H!t!H"t"^R I#H$t$H%t%^R I&H't'^RI(H)t)H*t*H+t+H,t,H-t-H.t.^RI/H0t0H1t1^RI2H3t3H4t4^RI5H6t6]-Pn!]84t9!RR]Pt4t;!RR]Pt4t<!RR]Pt4t=!RR]Pt4t>R9RRllt?!RR]Pt4t@!RR]Pt4tA!R R!]Pt4tB!R"R#]4tC!R$R%]Pt4tD],!R&R']%44tE!R(R)]E4tF],!R*R+]E44tG],!R,R-7!R.R/]E]44tH],!R0R1]E44tI],!R2R-7!R3R4]E44tJ],!R5R6]E44tK],!R7R8]E44tL.R:OtMR#);zPyTorch T5 model.N)Callable)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) DUMMY_INPUTS DUMMY_MASKTransformersKwargsauto_docstringloggingtorch_compilable_check)can_return_tuplemerge_with_config_defaults)OutputRecordercapture_outputsT5Configc<aa]tRt^2toRV3RlltRtRtVtV;t#) T5LayerNormc <\SV`4\P!\P !V44VnW nR#)zS Construct a layernorm module in the T5 style. No bias and no subtraction of mean. N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s&&&c/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/t5/modeling_t5.pyr*T5LayerNorm.__init__3s/ ll5::k#:; #cVP\P4P^4P RRR7pV\P !W P ,4,pVPP\P\P39d&VPVPP4pVPV,#)T)keepdim) tor,float32powmeanrsqrtr/r.dtypefloat16bfloat16)r0 hidden_statesvariances&& r4forwardT5LayerNorm.forward;s !##EMM266q9>>r4>P% H?T?T4T(UU  ;;   ? ?),,T[[->->?M{{]**r6)r/r.)gư> __name__ __module__ __qualname____firstlineno__r*rE__static_attributes____classdictcell__ __classcell__r3 __classdict__s@@r4r'r'2s$ + +r6r'cDaa]tRt^KtoV3RlV3RlltRtRtVtV;t#)T5DenseActDensec <V^8dQhRS[/#r8configr$)formatrPs"r4 __annotate__T5DenseActDense.__annotate__Ls//x/r6cn<\SV`4\P!VPVP RR7Vn\P!VP VPRR7Vn\P!VP4Vn \VP,Vn R#FbiasN)r)r*rLineard_modeld_ffwiwoDropout dropout_ratedropoutr dense_act_fnactr0rUr3s&&r4r*T5DenseActDense.__init__Lsn ))FNNFKKeD))FKKeDzz&"5"56 &--.r6cVPV4pVPV4pVPV4p\VPP \ P4'dVPVPP P8wdcVPP P\ P8wd0VPVPP P4pVP V4pV#N) r`rfrd isinstancerar.r,Tensorr@int8r;r0rCs&&r4rET5DenseActDense.forwardSs . /  ]3 tww~~u|| 4 4##tww~~';';;$$ 2),,TWW^^-A-ABM . r6)rfrdr`rarGrOs@@r4rRrRKs//  r6rRcDaa]tRt^atoV3RlV3RlltRtRtVtV;t#)T5DenseGatedActDensec <V^8dQhRS[/#rTr$)rVrPs"r4rW!T5DenseGatedActDense.__annotate__bs//x/r6c<\SV`4\P!VPVP RR7Vn\P!VPVP RR7Vn\P!VP VPRR7Vn\P!VP4Vn \VP,VnR#rZ)r)r*rr]r^r_wi_0wi_1rarbrcrdrrerfrgs&&r4r*T5DenseGatedActDense.__init__bs IIfnnfkkF IIfnnfkkF ))FKKeDzz&"5"56 &--.r6cFVPVPV44pVPV4pW#,pVPV4p\ VP P \P4'dVPVP P P8wdcVP P P\P8wd0VPVP P P4pVP V4pV#rj) rfrurvrdrkrar.r,rlr@rmr;)r0rC hidden_gelu hidden_linears&& r4rET5DenseGatedActDense.forwardjshhtyy78  -0 #3  ]3 tww~~u|| 4 4##tww~~';';;$$ 2),,TWW^^-A-ABM . r6)rfrdrurvrarGrOs@@r4rqrqas//r6rqcDaa]tRt^~toV3RlV3RlltRtRtVtV;t#) T5LayerFFc <V^8dQhRS[/#rTr$)rVrPs"r4rWT5LayerFF.__annotate__s77x7r6c <\SV`4VP'd\V4VnM\ V4Vn\ VPVPR7Vn \P!VP4Vn R#)r2N)r)r* is_gated_actrqDenseReluDenserRr'r^layer_norm_epsilon layer_normrrbrcrdrgs&&r4r*T5LayerFF.__init__sb     "6v">D "1&"9D %fnn&:S:STzz&"5"56 r6cxVPV4pVPV4pWPV4,pV#rj)rrrd)r0rCforwarded_statess&& r4rET5LayerFF.forwards;??=9../?@% 5E(FF r6)rrdrrGrOs@@r4r}r}~s77r6r}c<V^8dQhR\PR\PR\PR\PR\PR,R\R,R\R \PR,R \ \ ,/ #) r8modulequerykeyvalueattention_maskNscalingrd position_biaskwargs)rModuler,rlfloatrr)rVs"r4rWrWs%% II% <<% % << % LL4' % T\ %%<<$&%'(%r6c VfVPR4R,p\P!WP^^44V,p Ve W,p Ve W,p \P P V RR7p \P PWVPR7p \P!W4p V P^^4P4p W3#)Ndim)ptrainingr:) sizer,matmul transposer functionalsoftmaxrdr contiguous) rrrrrrrdrr attn_weights attn_outputs &&&&&&&&, r4eager_attention_forwardrs**R.D(<<}}Q':;gEL #3 !#4 ==((2(>L==((6??([L,,|3K''1-88:K  $$r6cvaa]tRt^toRV3RlV3Rlllt]R Rl4tR RltR V3RlRlltRt Vt V;t #) T5Attentionc:<V^8dQhRS[RS[R,RS[/#)r8rU layer_idxN is_causal)r%intbool)rVrPs"r4rWT5Attention.__annotate__s-%,%,%,: %,  %,r6c<<\SV`4WnVPVnW@nW nVP VnVPVnVPVnVPVn VPVn VPVnVPVP,VnRVnW0nVf@VP'd.\$P'RVP(P* R24\,P.!VPVPRR7Vn\,P.!VPVPRR7Vn\,P.!VPVPRR7Vn\,P.!VPVPRR7VnVP 'd1\,P8!VP VP4VnRVnR#)?NzInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.Fr[)r)r*rU is_decoderrhas_relative_attention_biasrelative_attention_num_bucketsrelative_attention_max_distancer^d_kvkey_value_proj_dim num_headsn_headsrcrd inner_dimrrlogger warning_oncer3rHrr]qkvo Embeddingrelative_attention_biasgradient_checkpointing)r0rUrrrr3s&&&&&r4r*T5Attention.__init__s{   ++"+F(.4.S.S+/5/U/U,~~ "(++'' ** (?(?? "     *4>>+B+B*CD,,  4<<eD4<<eD4<<eD4>>4<=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket. This should allow for more graceful generalization to longer sequences than the model has been trained on Args: relative_position: an int32 Tensor bidirectional: a boolean - whether the attention is bidirectional num_buckets: an integer max_distance: an integer Returns: a Tensor with the same shape as relative_position, containing int32 values in the range [0, num_buckets) ) r;r,longabsmin zeros_likelogrmath full_likewhere)relative_position bidirectional num_buckets max_distancerelative_buckets max_exactis_smallrelative_position_if_larges&&&& r4_relative_position_bucket%T5Attention._relative_position_buckets,  A K Q!6 : :5:: F T T  % *; < !&+N>NO`>a!b b  1$ $0&/ II'--/); <hh|/0 1& ( "UZZ. &" &+YY &8RbcTc(d& " EKKE_``r6c Vf!VPPPp\P!V\P VR7R,V,p\P!V\P VR7R,pWe, pVP VVP'*VPVPR7pVPV4p V P.RO4P^4p V #)z%Compute binned relative position bias)r@device)rrr)NNNN)Nr)r8) rr.rr,arangerrrrrpermute unsqueeze) r0 query_length key_lengthrpast_seen_tokenscontext_positionmemory_positionrrelative_position_bucketvaluess &&&&& r4 compute_biasT5Attention.compute_biass >1188??F << EJJvVW^_brr,,zFST[\+>#'#A#A #.;;== $B$  --.FG *44Q7 r6c0<V^8dQhRS[S[,/#r8rrr)rVrPs"r4rWrs X8X8+,X8r6c pVPRR p.VOR NVPN5pVeVPVP4M^p \ V \ P 4'dV P4MT p VRJp VPV4PV4P^^4p Rp \ V\4'dHVPPVP4p V 'dVPp MVPp MTp V 'dTMTpV 'd\VeXV 'dPV P VP,P"pV P VP,P$pM.VPRR OR NVPN5pVP'V4PV4P^^4pVP)V4PV4P^^4pVeXV P+VVVP4wppV 'd0\ V\4'dRVPVP&VfVPR ,pVP,'go\ P.!^VP0V^,V3V P2V P4R7pVP6'dVP8'dRVnM&VP=V^,VV P2V R7p\>P@!VPBPD\F4pT!VV VVV3RVP8'gRM VPHRVPJR V/VBwppVPL!.VOR N5!PO4pVPQV4pVVV3#) zp Self-attention (if key_value_states is None) or attention over source sentence (provided by key_value_states). NFT)rr@)rrrdrrr:))shaperget_seq_lengthrrkr,rlclonerviewrr is_updatedgetcross_attention_cacheself_attention_cachelayerskeysrrrupdaterzerosrrr@rr requires_gradrr get_interfacerU_attn_implementationrrdrreshaperr)r0rCmaskkey_value_statesrpast_key_valuesr input_shape hidden_shaperis_cross_attention query_statesrcurr_past_key_valuescurrent_states key_states value_stateskv_shaperattention_interfacerrs&&&&&&, r4rET5Attention.forwardsY$))#2. BBbB$*A*AB M\Mh?99$..Ino7ABRTYT`T`7a7a+113gw.T9vvm,11,?II!QO  o': ; ;(3377GJ!'6'L'L$'6'K'K$#2 -?)] /"=*-44T^^DIIJ/66t~~FMMLP--cr2PBP8O8OPH/44X>HHANJ66.166x@JJ1aPL*+?+F+FzS_aeaoao+p( L%*_FY*Z*ZAEO..t~~>  #))"-J333 %  k!njA'..&,,! ...4===26M/ $ 1 1NJ|7J7J]m!2! )@(M(M KK , ,.E) %8      % $}}}C$,, % LL % ( %  % ! \"));;;;FFH ff[) M<77r6)rUr^rdrrrrrrrrrrrrrrrr)FNF)T )NrNNNN) rHrIrJrKr* staticmethodrrrErLrMrNrOs@@r4rrs<%,%,N- - ^"X8X8X8r6rcXaa]tRtRtoRV3RlV3RllltRV3RlRlltRtVtV;t#) T5LayerSelfAttentionitc.<V^8dQhRS[R,/#r8rNr)rVrPs"r4rW!T5LayerSelfAttention.__annotate__us 7 7SSWZ 7r6c<\SV`4\VVVVPR7Vn\ VP VPR7Vn\P!VP4Vn R#)rrrrN) r)r*rr SelfAttentionr'r^rrrrbrcrdr0rUrrr3s&&&&r4r*T5LayerSelfAttention.__init__us] ( (C''   &fnn&:S:STzz&"5"56 r6c0<V^8dQhRS[S[,/#rr)rVrPs"r4rWrs== +, =r6c VPV4pVP!V3RVRVRV/VBwrspWPV4,pWsV3#)rrr)rrrd) r0rCrrrrnormed_hidden_statesattention_outputrs &&&&&, r4rET5LayerSelfAttention.forwardsr $}=8<8J8J 9 9 (9 , 9  9 5)<<8H+II < zzF$=$=>  &..&2C2CD r6cN<V^8dQhRS[PRS[P/#)r8rCreturnr,rl)rVrPs"r4rWrG s#U\\ellr6cVPV4pVPV4p\P!V4pVPV4pVP V4pV#rj)rdrIr,tanhrLrns&&r4rET5ClassificationHead.forward sN ]3  =1  =1  ]3  m4 r6)rIrdrL) rHrIrJrK__doc__r*rErLrMrNrOs@@r4rErEs#7EE r6rEc aa]tRtRtoRtRtR.tR.tRtRt Rt Rt R] !] ^R7R ] !]RR7R ] !]RR7/t]R 4t]P(!4V3R l4tR tV3RltRtVtV;t#)T5PreTrainedModeli transformerTr+raFrC)index attentionscross_attentionsc~\P!\4p\P!\4pRVRVRV/pV#)decoder_input_ids input_idsdecoder_attention_mask)r,tensorrr)r0r] input_mask dummy_inputss& r4raT5PreTrainedModel.dummy_inputs's<LL. \\*-   $j r6c <\SV`V4VPPp\ V\ 4'd+\ P!VPVR,4R#\ V\\\\34'Ed%\ P!VPPRVR,R7\VR4'dQVPP 'g5\ P!VP"PRVR,R7\VR4'd{\ P!VP$PRW PP&R ,,R7\ P(!VP$P*4R#R#\ V\,4'du\VR4'da\ P!VP.PRVR,R7\ P(!VP.P*4R#R#\ V\04'Ed_\ P!VP2PRW PP&R ,,R7\VP2R4'dCVP2P*e+\ P(!VP2P*4\ P!VP4PRW PP&R ,,R7\VP4R4'dGVP4P*e-\ P(!VP4P*4R#R#R#\ V\64'Ed_\ P!VP8PRW PP&R ,,R7\VP8R4'dCVP8P*e+\ P(!VP8P*4\ P!VP:PRW PP<R ,,R7\VP:R4'dGVP:P*e-\ P(!VP:P*4R#R#R#\ V\>4'Ed \ P!VP@PRW PP&R ,,R7\VP@R4'dCVP@P*e+\ P(!VP@P*4\ P!VPBPRW PP&R ,,R7\VPBR4'dCVPBP*e+\ P(!VPBP*4\ P!VP:PRW PP<R ,,R7\VP:R4'dGVP:P*e-\ P(!VP:P*4R#R#R#\ V\D4'EdVPP&pVPPFpVPPHp\ P!VPJPRW#V,R ,,R7\ P!VPLPRW#R ,,R7\ P!VPNPRW#R ,,R7\ P!VPPPRW%V,R ,,R7VPR'd=\ P!VPTPRW#R ,,R7R#R#R#) zInitialize the weightsrr)r>stdlm_head qa_outputs classifierr\Nr)+r) _init_weightsrUinitializer_factorrkr'init constant_r.T5ModelT5ForConditionalGenerationT5EncoderModelT5ForQuestionAnsweringnormal_sharedhasattrtie_word_embeddingsrerfr^zeros_r\T5ForTokenClassificationrgrErIrLrRr`rar_rqrurvrrrrrrrrr)r0rfactorr^rrr3s&& r4rhT5PreTrainedModel._init_weights2sc f%// fk * * NN6==&3, 7   0.BX Y   LL--CVc\ Jvy))$++2Q2Q2Q V^^22&3,Ov|,, V..55CVP[P[PcPchlOlEmn F--223- 8 9 9v|,, V..55CVc\R F--223- 4 5 5 LL,,3F{{GZGZ_cFcVPPpVPPpVf \R4hVP VP 4pVRRR13,P 4VR&W$R&Vf \R4hVPVR8HV4V#)Nzself.model.config.decoder_start_token_id has to be defined. In T5 it is usually set to the pad_token_id. See T5 docs for more information..z1self.model.config.pad_token_id has to be defined.r:).:rNN).r)rUdecoder_start_token_id pad_token_id ValueError new_zerosrr masked_fill_)r0r]rzr{shifted_input_idss&& r4 _shift_rightT5PreTrainedModel._shift_rightjs!%!C!C{{// ! )4  &// @%.sCRCx%8%>%>%@'"$:&!  PQ Q&&'8D'@,O  r6c&<V^8dQh/S[;R&#rTr$)rVrPs"r4rWT5PreTrainedModel.__annotate__s r6r:)rHrIrJrKbase_model_prefixsupports_gradient_checkpointing_no_split_modules_keep_in_fp32_modules_supports_flash_attn_supports_flex_attn_supports_sdpa_can_compile_fullgraphr"r+rr _can_record_outputspropertyrar,no_gradrhr__annotate_func__rLrMrNrOs@@r4rVrVs&&*#" !F!N! q9n%9DN+@K  ]]_5p5pn!qr6rVclaa]tRtRtoV3RltRt]]]RV3RlRll444t Rt Vt V;t #)T5Stackic "<\SV`V4\P!VPVP 4VnVPVn\P!\VP4Uu.uFp\V\V^8H4VR7NK up4Vn \VP VPR7Vn\P"!VP$4VnVP)4RVnR#uupi)rr/rFN)r)r*rr vocab_sizer^ embed_tokensrr0range num_layersr+rblockr'rfinal_layer_normrbrcrd post_initr)r0rUir3s&& r4r*T5Stack.__init__s  LL):):FNNK ++]]]bcictct]u v]uXYWVa1fQR S]u v  !,FNN@Y@Y Zzz&"5"56  &+# ws9#D cWnR#rj)rr0new_embeddingss&&r4set_input_embeddingsT5Stack.set_input_embeddingss*r6c0<V^8dQhRS[S[,/#rr)rVrPs"r4rWT5Stack.__annotate__s W W +,W r6c <VRJVRJ, 'd \R4hVfVPV4pVRJd"VP'g\RV R24hVP'duV'dlVfhVPP'd6\ \ VPR7\ VPR74pM+\ VPR7pMVP'gRpVPP'd=\VPTT\V\ 4'd VPMTR7pM\VPVVR7pRp VP'dVe\VPVVVR7p Rp Rp VPV4p VPF4p V !V VV VV V 3R V/VBwrpTp VP'gK,VfK2Tp K6 VPV 4p VPV 4p \V VR 7#) Nz:You must specify exactly one of input_ids or inputs_embedsTz)`use_cache` can only be set to `True` if z is used as a decoder)rU)rU inputs_embedsrr)rUrr)rUrrr;r)last_hidden_stater)r|rrrUis_encoder_decoderr r rrkrr rdrrr)r0r]rr;r<rr use_cacherencoder_extended_attention_maskrr=rC layer_moduleself_attention_position_biascross_attention_position_biass&&&&&&&&, r4rET5Stack.forwards  -t"; < <YZ Z   --i8M  ??? #LTFRg!hii ???_4;;111&9$DKK8,dkk:Z'O'3$++&FO#O ;; ! ! !/{{+-o/BCC!0 D D$ N7{{+-N +/' ???4@.G{{+5&; / + (,% ]3  JJLYe%/- Z!0 Z Z VM9V9M#8#D0M-'"--m<  ]3 8++  r6)rrdrrrr)NNNNNNN) rHrIrJrKr*rr!r#rrErLrMrNrOs@@r4rrsA, + W W  W W r6rcaa]tRtRtoR.tRRRR/tV3RlV3RlltRtR t] ] R V3R lR ll44t R t Vt V;t#)rliFdecoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weightencoder.embed_tokens.weight shared.weightdecoder.embed_tokens.weightc <V^8dQhRS[/#rTr$)rVrPs"r4rWT5Model.__annotate__sxr6c<\SV`V4\P!VPVP 4Vn\P!V4pRVn RVn \V4Vn \P!V4pRVn VPVn\V4VnVP!4R#FTN)r)r*rrrr^rqcopydeepcopyrrrencodernum_decoder_layersrdecoderrr0rUencoder_configdecoder_configr3s&& r4r*T5Model.__init__s  ll6#4#4fnnE v.$)!#( ~. v.$(!$*$=$=!~.  r6cVP#rjrqr0s&r4get_input_embeddingsT5Model.get_input_embeddings  {{r6c~WnVPPV4VPPV4R#rjrqrrrrs&&r4rT5Model.set_input_embeddings+$  )).9 )).9r6c<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[S[S[P,,R,RS[R,RS[P R,R S[P R,R S[R,R S[S[ ,R S[ / #) r8r]Nrr\r^encoder_outputsrrdecoder_inputs_embedsrrrO) r, LongTensor FloatTensor BoolTensortupler rlrrrr)rVrPs"r4rWrsd d ##d*d ))D0d !++d2 d !& 0 04 7 d uU%6%6784? d d ||d*d  %||d2d $;d +,d  d r6c  VfVP!R RVRVRV/V BpM]\V\4'gH\V^,\V4^8d V^,MR\V4^8d V^,MRR7pV^,p VP!R RVRVRVRVRV RVR V /V Bp \ V P V PV PV PV PVP VPVPR 7#) a> input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) T5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [T5 Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Example: ```python >>> from transformers import AutoTokenizer, T5Model >>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small") >>> model = T5Model.from_pretrained("google-t5/t5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> decoder_input_ids = tokenizer("Studies show that", return_tensors="pt").input_ids # Batch size 1 >>> # preprocess: Prepend decoder_input_ids with start token which is pad token for T5Model. >>> # This is not needed for torch's T5ForConditionalGeneration as it does this internally using labels arg. >>> decoder_input_ids = model._shift_right(decoder_input_ids) >>> # forward pass >>> outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) >>> last_hidden_states = outputs.last_hidden_state ```Nr]rrrrCrYrr;r<r)rrdecoder_hidden_statesdecoder_attentionsrZencoder_last_hidden_stater;encoder_attentionsr) rrkrlenrrrrrCrYrZ) r0r]rr\r^rrrrrrrCdecoder_outputss &&&&&&&&&&, r4rET5Model.forwardsJB  ""ll#-, O O_==-"1!"4474H14Loa0RV14_1E1I?1-tO (* ,,  '  1  0  ,  #0  $2       "-??+;;"1"?"?.99,==&5&G&G"1"?"?.99  r6)rrrq NNNNNNNNNrHrIrJrK"_keys_to_ignore_on_load_unexpected_tied_weights_keysr*rrr rrErLrMrNrOs@@r4rlrlse Q*& &% ": d d d d r6rlz: T5 Model with a `language modeling` head on top. ) custom_introcaa]tRtRtoR.tRRRRRR/tV3RlV3RlltR tR t] ] RV3R lR ll44t V3R lRlt Rt VtV;t#)rmi}rzlm_head.weightrrrc <V^8dQhRS[/#rTr$)rVrPs"r4rW'T5ForConditionalGeneration.__annotate__sxr6c<\SV`V4VPVn\P !VP VP4Vn\P!V4pRVn RVn \V4Vn \P!V4pRVn VPVn\V4Vn\P"!VPVP RR7VnVP'4R#)FTr[N)r)r*r^ model_dimrrrrqrrrrrrrrrr]rerrs&& r4r*#T5ForConditionalGeneration.__init__s  ll6#4#4fnnE v.$)!#( ~. v.$(!$*$=$=!~. yy1B1BO  r6cVP#rjrrs&r4r/T5ForConditionalGeneration.get_input_embeddingsrr6c~WnVPPV4VPPV4R#rjrrs&&r4r/T5ForConditionalGeneration.set_input_embeddingsrr6c<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[S[S[P ,,R,RS[R,RS[PR,R S[PR,R S[PR,R S[R,R S[S[ ,R S[ / #)r8r]Nrr\r^rrrrlabelsrrrO) r,rrrrrlr rrrr)rVrPs"r4rWrs  ##d* ))D0 !++d2  !& 0 04 7  uU\\23d:   ((4/  %0047   4' $; +,   r6c  VfVP!R RVRVRV/V BpM]\V\4'gH\V^,\V4^8d V^,MR\V4^8d V^,MRR7pV^,p V eVfVfVP V 4pVP !R RVRVRVRVRV RVR V /V Bp V ^,pVP P'dWPR ,,pVPV4pRpV e_\RR 7pV PVP4p V!VPRVPR44V PR44p\VVV P V P"V P$V P&VP(VP"VP$R 7 #)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) T5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [T5 Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import AutoTokenizer, T5ForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small") >>> model = T5ForConditionalGeneration.from_pretrained("google-t5/t5-small") >>> # training >>> input_ids = tokenizer("The walks in park", return_tensors="pt").input_ids >>> labels = tokenizer(" cute dog the ", return_tensors="pt").input_ids >>> outputs = model(input_ids=input_ids, labels=labels) >>> loss = outputs.loss >>> logits = outputs.logits >>> # inference >>> input_ids = tokenizer( ... "summarize: studies have shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> outputs = model.generate(input_ids) >>> print(tokenizer.decode(outputs[0], skip_special_tokens=True)) >>> # studies have shown that owning a dog is good for you. ```Nr]rrrrr;r<r ignore_index losslogitsrrrrZrr;rrrryr:)rrkrrrrrUscale_decoder_outputsrrerr;rrrrrrCrYrZr)r0r]rr\r^rrrrrrrrCrsequence_output lm_logitsrloss_fcts&&&&&&&&&&&, r4rE"T5ForConditionalGeneration.forwardsP  ""ll#-, O O_==-"1!"4474H14Loa0RV14_1E1I?1-tO (*  "3";@U@] $ 1 1& 9 ,,  '  1  0  ,  #0  $2       *!, ;; , , ,-1EFOLL1   'T:HYYy//0FINN2y~~b/ABFKKPROTD+;;"1"?"?.99,==&5&G&G"1"?"?.99  r6c4<V^8dQhRS[P/#)r8rrP)rVrPs"r4rWr,s))ELL)r6c$VPV4#rj)r)r0rs&&r4%prepare_decoder_input_ids_from_labels@T5ForConditionalGeneration.prepare_decoder_input_ids_from_labels,s  ((r6)rrrerrq NNNNNNNNNN)rHrIrJrKrrr*rrr rrErrLrMrNrOs@@r4rmrm}st Q*& /%% *:    B)))r6rmcaa]tRtRtoRR/tR.tV3RlV3RlltRtRt] ] R V3R lR ll44t R t Vt V;t#) rni0rrrc <V^8dQhRS[/#rTr$)rVrPs"r4rWT5EncoderModel.__annotate__5s  x r6c<\SV`V4\P!VPVP 4VnTpRVnRVn\V4Vn VP4R#r) r)r*rrrr^rqrrrrr)r0rUrr3s&& r4r*T5EncoderModel.__init__5sY  ll6#4#4fnnE #( ,1)~.  r6cVP#rjrrs&r4r#T5EncoderModel.get_input_embeddingsArr6cHWnVPPV4R#rj)rqrrrs&&r4r#T5EncoderModel.set_input_embeddingsDs$  )).9r6c <V^8dQhRS[PR,RS[PR,RS[PR,RS[S[,RS[/#)r8r]NrrrrO)r,rrrrr)rVrPs"r4rWrJsd%%##d*%))D0%((4/ % +, %  %r6c 6VP!RRVRVRV/VBpV#)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). Example: ```python >>> from transformers import AutoTokenizer, T5EncoderModel >>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small") >>> model = T5EncoderModel.from_pretrained("google-t5/t5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> outputs = model(input_ids=input_ids) >>> last_hidden_states = outputs.last_hidden_state ```r]rrr)r)r0r]rrrrs&&&&, r4rET5EncoderModel.forwardHs?@,0<<, , ), (,  , r6)rrqr)rHrIrJrKrrr*rrr rrErLrMrNrOs@@r4rnrn0sR7I*4&  :%%%%r6rnz T5 model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cnaa]tRtRtoR.tV3RlV3Rllt]]RV3RlRll44tRt Vt V;t #) T5ForSequenceClassificationirrc <V^8dQhRS[/#rTr$)rVrPs"r4rW(T5ForSequenceClassification.__annotate__{sxr6c<\SV`V4\V4Vn\ V4VnVP 4R#rj)r)r*rlrWrEclassification_headrrgs&&r4r*$T5ForSequenceClassification.__init__{s5  "6?#7#?  r6c<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[S[P,R,RS[PR,RS[PR,R S[PR,R S[R,R S[S[,R S[/ #) r8r]Nrr\r^rrrrrrrO) r,rrllistrrrrr)rVrPs"r4rWr s} } ##d*}  t+} !++d2 } !& 0 04 7 } e//047 } ((4/}  %0047}   4'} $;} +,}  )} r6c  RVeRp Vf'Ve#\RVPP 24hVf%Vf!Vf \R4hVP V4pVP !V3RVRVRVRVR VR VR V /V Bp V ^,p VP VPP4PV P4p \\P!V P^44P4^8HR 4V P wrpWR 3,p\VP ^,V,^8R4VP#VRV4R,pVP%V4pRpVEeVPVP4pVPP&fVPP(^8XdRVPnM|VPP(^8dQVP*\P,8Xg VP*\P.8XdRVPnMRVPnVPP&R8XdV\14pVPP(^8Xd'V!VP34VP344pMV!VV4pMVPP&R8XdH\54pV!VP#RVPP(4VP#R44pM.VPP&R8Xd\74pV!VV4p\9VVV P:V P<V P>V P@V PBV PDV PFR7 #)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) T5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [T5 Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NFz8Passing input embeddings is currently not supported for If no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.rr\r^rrrrz7All examples must have the same number of tokens.rz3Each example must contain at least one token. regressionsingle_label_classificationmulti_label_classificationrr:)rr:r)$NotImplementedErrorr3rHr|rrWeqrU eos_token_idr;rrr,unique_consecutivesumnumelrrr  problem_typerKr@rrrsqueezerrrrrrrZrr;r)r0r]rr\r^rrrrrroutputsreos_mask batch_sizer?r1selectedsentence_representationrrrs&&&&&&&&&&, r4rE#T5ForSequenceClassification.forwardsM\  I  !:%J4>>KbKbJcd   $)>)F  U !% 1 1) < &*&6&6  ' ) ' 0 ' $: ' , ' ( ' #8 '   '  ' "!*<< 8 89<<_=S=ST  $ $X\\!_ 5 ; ; = B E &5%:%:" {"Q;/ NN1  +q 0 A #+-- B "LX"V))*AB  YYv}}-F{{''/;;))Q./;DKK,[[++a/V\\UZZ5OSYS_S_chclclSl/LDKK,/KDKK,{{''<7"9;;))Q.#FNN$4fnn6FGD#FF3D))-JJ+- B 0F0F GUWY))-II,./.#33")"?"?&99$55&-&G&G")"?"?&99  r6)r rWr) rHrIrJrKrr*r rrErLrMrNrOs@@r4rrrsE+s)s&} } } } r6rchaa]tRtRtoV3RlV3Rllt]]RV3RlRll44tRtVt V;t #)ruic <V^8dQhRS[/#rTr$)rVrPs"r4rW%T5ForTokenClassification.__annotate__s  x r6c2<\SV`V4VPVn\V4Vn\ P !VP4Vn\ P!VPVP4Vn VP4R#rj) r)r*rKrnrWrrbrJrdr]r1rgrrgs&&r4r*!T5ForTokenClassification.__init__sj   ++)&1zz&";";< ))F$6$68I8IJ r6c <V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[S[,RS[/#)r8r]NrrrrrO)r,rlrrr)rVrPs"r4rWr%sq+ + <<$&+  t++ ||d* +  t# + +, +  + r6c PVP!V3RVRV/VBpV^,pVPV4pVPV4pRp Ve=\4p V !VP RVP 4VP R44p \ V VVPVPR7#)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. rrN)rrrCrYr:) rWrdrgrrrKrrCrY) r0r]rrrrrrCrrrs &&&&&, r4rE T5ForTokenClassification.forwards0$(#3#3 $ )$ ($  $    ]3 /  ')HFKKDOO)F  U !% 1 1) <   ""ll#-, O O_==-"1!"4474H14Loa0RV14_1E1I?1-tO (* ,,  '  1  0  !  #0  $2       *!,1#)<<r<#: j#++B/::< ''+668   &=+D?'')*Q."1"9"9""="@"@ATAT"U=%%'(1, - 5 5b 9 < rQs $ AA&!CC)J9G&rrIE&   H %+"))+2bii,299: $%@A8"))A8H=299=@9BII9>JP(JPZ299$j!j!j!Zn n bH H H V k)!2Ok)  k)\>&>>B J "3J  J Z9 09 9 x` .` ` F r6