+ jq%Rt^RIt^RIt^RIt^RIt^RIHtHt^RIH t H t ^RI H t ^RI Ht^RIHt^RIt^RIHtHt^RIHt^R IHt^R IHtHt^R IHt^R IHt^R I H!t!H"t"H#t#^RI$H%t%^RI&H't'H(t(^RI)H*t*^RI+H,t,H-t-^RI.H/t/^RI0H1t1H2t2H3t3H4t4^RI5H6t6H7t7^RI8H9t9H:t:]Pv!]<4t=RMRRllt>]3!RR7]!RR]1444t?RNRlt@RORltARPRltB!RR ]P4tD!R!R"]P4tE!R#R$]P4tF!R%R&]P4tG!R'R(]P4tH!R)R*]P4tI!R+R,]P4tJ!R-R.]P4tK!R/R0]*4tL]3!R1R2]-44tM!R3R4]P4tN]IPtPRQtQ!R5R6]P]4tR!R7R8]R4tS!R9R:]R4tT!R;R<]R4tU!R=R>]P4tVR?]SR@]TRA]U/tWRBRCltX!RDRE]M4tY]3!RFRG]M44tZ]3!RHR7!RIRJ]M]%44t[]3!RKRL]M44t\.RROt]R#)SzPyTorch UDOP model.N)ABCabstractmethod)CallableSequence)deepcopy) dataclass)Any)Tensornn)CrossEntropyLoss UdopConfig)Seq2SeqLMOutputSeq2SeqModelOutput)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringis_torchdynamo_compiling)can_return_tuplemerge_with_config_defaults)OutputRecordercapture_outputsc<V^8dQhR\PR\PR\PR\PR\PR,R\R,R\R \PR,R \ \ ,/ #) modulequerykeyvalueattention_maskNscalingdropout position_biaskwargs)r Moduletorchr floatrr)formats"g/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/udop/modeling_udop.py __annotate__r4:s%% II% <<% % << % LL4' % T\ %%<<$&%'(%c VfVPR4R,p\P!WP^^44V,p Ve W,p Ve W,p \P P V RR7p \P PWVPR7p \P!W4p V P^^4P4p W3#)Ndim)ptraining) sizer0matmul transposer functionalsoftmaxr,r: contiguous) r&r'r(r)r*r+r,r-r. attn_weights attn_outputs &&&&&&&&, r3eager_attention_forwardrE:s**R.D(<<}}Q':;gEL #3 !#4 ==((2(>L==((6??([L,,|3K''1-88:K  $$r5z Class for the model's outputs that may also contain a past key/values (to speed up sequential decoding). Includes an additional attention mask. ) custom_introcHa]tRt^ZtoRtRtRtRtRtRt Rt V3Rlt Rt Vt R#) BaseModelOutputWithAttentionMaska last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. If `past_key_values` is used only the last hidden-state of the sequences of shape `(batch_size, 1, hidden_size)` is output. attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Attention mask used in the model's forward pass to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if `config.is_encoder_decoder=True` in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. cross_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` and `config.add_cross_attention=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the decoder's cross-attention layer, after the attention softmax, used to compute the weighted average in the cross-attention heads. NcJ<V^8dQh/S[PR,;R&S[PR,;R&S[R,;R&S[S[P,R,;R&S[S[P,R,;R&S[S[P,R,;R&#)r%Nlast_hidden_stater*past_key_values hidden_states attentionscross_attentions)r0 FloatTensorrtuple)r2 __classdict__s"r3r4-BaseModelOutputWithAttentionMask.__annotate__ZsJ((4/6KL%%,3MNT\(OP**+d29QRe''(4/6STE--.5q&A1 E #2  % %&>q&A1 E O OPQST U "  $ $%=a%@! D "  $ $%=a%@! D N NqRS T  *..r15 r5c`\V\P4'dVP^,pM!\ V4p\P !V4pW, pV^8dG\P !V.V,4PV4p\P!W.^R7pVRV#)r]r7N) isinstancer0r shapelentensorrbtocat)seq target_len pad_valuenmrets&&& r3 pad_sequencerys#u||$$ IIaL Hll3A1ukk9+/*--c2ii * { r5c  Tp \P!\P!VR ,VR ,,R, V ,4P4^V ^, 4p \P!\P!VR ,VR ,,R, V ,4P4^V ^, 4V ,p W,p VP \P 4pVP R 4R8HVP R 4R8H,p \P!V^V PR 4P^^VPR 444pRW&W, p\P!VR ,R4P4p\P!\P!\V 44R,P^V PR 44R,P V 4V R,.R R7pVP!^^4p\#V!wppRVVV3&\%\V44Uu.uFpVV,VV,,NK ppVfY\'WxR7pVP^4PVP^4^^4pVP VP(4p\%\V44Uu.uFpVV,VV,,NK ppVeLVUu.uF?p\P*!VP^4VP,VP(R7NKA ppV^8XdVP^4pMWaP^4, p\P.!VUu.uF+p\1VV\P2!VR,44NK- up4p\P.!VUu.uF+p\1VV\P2!VR,44NK- up4pVeM\P.!XUu.uF+p\1VV\P2!VR,44NK- up4p\P!VV.^4p\P!W#.^4pVe\P!VX.^4pWV3#uupiuupiuupiuupiuupiuupi)a Combine the image and text embeddings for the input to the encoder/decoder of UDOP. First, the image embeddings are created by checking for each visual patch if it is inside the bounding box of a token. If it is, the visual patch is combined with the token embedding. Then, the visual bounding boxes are combined with the text bounding boxes. Finally, the visual bounding boxes are combined with the text attention mask. g@r^Tr7F)rerfdtypedeviceNNNrr]rrr%rrr`rrr;rNrrN)r]r])r0clipfloorlongrqfloat64meangather unsqueezercr= full_likeboolrrraroflattenziprangerkr~onesr}rbry zeros_like)image_embeddings inputs_embedsbbox visual_bboxr* num_patchesmax_lenrerfsequence_length ocr_points_x ocr_points_y ocr_points target_segrepeated_vision_embeds patch_indsindrowscolsiinput_vision_patchesitemvisual_attention_maskinputs_vision_patchess&&&&&&&&& r3combine_image_text_embeddingsrs,&"O:: T']T']2c9OKLQQSUVXgjkXkL  5;;W W =DVW\\^`acruvcvw  ,J 775== !D))B-3&499R=C+?@J"\\!Z11"5<>H^_%S !++A.556F6K6KA6NPQSTU !nn%5%<%<= :?J:PQ:PQ;q>*Q-00:PKQ!kv! kvcgEJJtyy|>+?+?H]H] ^kv ! !|"''*..q11!KK[op[oSWdGU%5%56Ft6L%M N[op++epqep]a|D'5;K;KDQUJ;WXepqrK! % ]r s]rUY\$)9)9.:N)O P]r s! II}.CDaHM 99d(! ,D!N4I#JAN  ..?`R!  qq ts% S . SAS#1S/1S?1S"c<aa]tRt^toRtV3RltRtRtVtV;t #)UdopPatchEmbeddingsz2D Image to Patch Embeddingsc <\SV`4VPVPr2VPVP rT\ V\PP4'dTMW"3p\ V\PP4'dTMW33pV^,V^,,V^,V^,,,pW nW0nW@nW`n \P!WEW3R7Vn R#)r`) kernel_sizestrideN)super__init__rerf num_channels hidden_sizerm collectionsabcIterablerr Conv2dproj)selfconfigrerfrrr __class__s&& r3rUdopPatchEmbeddings.__init__s !'!2!2F4E4EJ$*$7$79K9Kk#-j+//:R:R#S#SZZdYq #-j+//:R:R#S#SZZdYq !!} 1 5*Q-:VW=:XY $$(&IIlZc r5c XVPwr#rEW@P^,8wgWPP^,8wd;\RV RV RVP^, RVP^, R2 4hVPV4pVP ^4P ^^4pV#)r]zInput image size (*z) doesn't match model (z).)rnre ValueErrorrrr?)r pixel_values batch_sizerheightwidth embeddingss&& r3forwardUdopPatchEmbeddings.forwards2>2D2D/ & __Q' '5OOA4F+F$VHAeW4KDOO\]L^K__`aeapapqras`ttvw YY|, ''*44Q: r5)rerrrfr) rTrUrVrWrXrrrZr[ __classcell__rrQs@@r3rrs& dr5rc<aa]tRtRtoRV3RlltRtRtVtV;t#) UdopLayerNormic <\SV`4\P!\P !V44VnW nR#)zU Construct a layernorm module in the Udop style. No bias and no subtraction of mean. N)rrr Parameterr0rweightvariance_epsilon)rrepsrs&&&r3rUdopLayerNorm.__init__s/ ll5::k#:; #r5cVP\P4P^4P RRR7pV\P !W P ,4,pVPP\P\P39d&VPVPP4pVPV,#)r%T)keepdimr;) rqr0float32powrrsqrtrrr}float16bfloat16)rrLvariances&& r3rUdopLayerNorm.forward%s !##EMM266q9>>r4>P% H?T?T4T(UU  ;;   ? ?),,T[[->->?M{{]**r5)rr)gư> rTrUrVrWrrrZr[rrs@@r3rrs$ + +r5rcDaa]tRtRtoV3RlV3RlltRtRtVtV;t#)UdopDenseActDensei6c <V^8dQhRS[/#r%rr )r2rQs"r3r4UdopDenseActDense.__annotate__7s//z/r5cn<\SV`4\P!VPVP RR7Vn\P!VP VPRR7Vn\P!VP4Vn \VP,Vn R#FbiasN)rrr Lineard_modeld_ffwiwoDropout dropout_rater,r dense_act_fnactrrrs&&r3rUdopDenseActDense.__init__7sn ))FNNFKKeD))FKKeDzz&"5"56 &--.r5cVPV4pVPV4pVPV4p\VPP \ P4'dVPVPP P8wdcVPP P\ P8wd0VPVPP P4pVP V4pV#N) rrr,rmrrr0r r}int8rq)rrLs&&r3rUdopDenseActDense.forward>s . /  ]3 tww~~u|| 4 4##tww~~';';;$$ 2),,TWW^^-A-ABM . r5)rr,rrrrs@@r3rr6s//  r5rcDaa]tRtRtoV3RlV3RlltRtRtVtV;t#)UdopDenseGatedActDenseiMc <V^8dQhRS[/#rr )r2rQs"r3r4#UdopDenseGatedActDense.__annotate__Ns//z/r5c<\SV`4\P!VPVP RR7Vn\P!VPVP RR7Vn\P!VP VPRR7Vn\P!VP4Vn \VP,VnR#r)rrr rrrwi_0wi_1rrrr,rrrrs&&r3rUdopDenseGatedActDense.__init__Ns IIfnnfkkF IIfnnfkkF ))FKKeDzz&"5"56 &--.r5cFVPVPV44pVPV4pW#,pVPV4p\ VP P \P4'dVPVP P P8wdcVP P P\P8wd0VPVP P P4pVP V4pV#r) rrrr,rmrrr0r r}rrq)rrL hidden_gelu hidden_linears&& r3rUdopDenseGatedActDense.forwardVshhtyy78  -0 #3  ]3 tww~~u|| 4 4##tww~~';';;$$ 2),,TWW^^-A-ABM . r5)rr,rrrrrs@@r3rrMs//r5rcDaa]tRtRtoV3RlV3RlltRtRtVtV;t#) UdopLayerFFikc <V^8dQhRS[/#rr )r2rQs"r3r4UdopLayerFF.__annotate__ls77z7r5c <\SV`4VP'd\V4VnM\ V4Vn\ VPVPR7Vn \P!VP4Vn R#)rN)rr is_gated_actrDenseReluDenserrrlayer_norm_epsilon layer_normr rrr,rs&&r3rUdopLayerFF.__init__lsb     "8"@D "3F";D 'F>+B+B*CD,,  4<<eD4<<eD4<<eD4>>4<=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket. This should allow for more graceful generalization to longer sequences than the model has been trained on Args: relative_position: an int32 Tensor bidirectional: a boolean - whether the attention is bidirectional num_buckets: an integer max_distance: an integer Returns: a Tensor with the same shape as relative_position, containing int32 values in the range [0, num_buckets) ) rqr0rabsminrlogr1mathrwhere)relative_position bidirectional num_buckets max_distancerelative_buckets max_exactis_smallrelative_position_if_larges&&&& r3_relative_position_bucket'UdopAttention._relative_position_buckets,  A K Q!6 : :5:: F T T  % *; < !&+N>NO`>a!b b  1$ $0&/ II'--/); <hh|/0 1& ( "UZZ. &" &+YY &8RbcTc(d& " EKKE_``r5c Vf!VPPPp\P!V\P VR7R,V,p\P!V\P VR7R,pWe, pVP VVP'*VPVPR7pVPV4p V P.RO4P^4p V #)z%Compute binned relative position biasr|r/r0r1rNr)r%r]r`) r%rr~r0rarr6rrrpermuter) r query_length key_lengthr~past_seen_tokenscontext_positionmemory_positionr.relative_position_bucketvaluess &&&&& r3 compute_biasUdopAttention.compute_biass >1188??F << EJJvVW^_brr,,zFST[\+>#'#A#A #.;;== $B$  --.FG *44Q7 r5c0<V^8dQhRS[S[,/#r%r.rr)r2rQs"r3r4rs X8X8+,X8r5c pVPRR p.VOR NVPN5pVeVPVP4M^p \ V \ P 4'dV P4MT p VRJp VPV4PV4P^^4p Rp \ V\4'dHVPPVP4p V 'dVPp MVPp MTp V 'dTMTpV 'd\VeXV 'dPV P VP,P"pV P VP,P$pM.VPRR OR NVPN5pVP'V4PV4P^^4pVP)V4PV4P^^4pVeXV P+VVVP4wppV 'd0\ V\4'dRVPVP&VfVPR ,pVP,'go\ P.!^VP0V^,V3V P2V P4R7pVP6'dVP8'dRVnM&VP=V^,VV P2V R7p\>P@!VPBPD\F4pT!VV VVV3RVP8'gRM VPHRVPJR V/VBwppVPL!.VOR N5!PO4pVPQV4pVVV3#) zp Self-attention (if key_value_states is None) or attention over source sentence (provided by key_value_states). NFTr~r})r~r>r,r{r+r-r;))rnrget_seq_lengthrrmr0r cloner rdr?r is_updatedgetcross_attention_cacheself_attention_cachelayerskeysrBr!r"updaterzerosrr~r}r&r: requires_gradrCr get_interfacer_attn_implementationrEr,r+reshaperBr#)rrLmaskkey_value_statesr-rKr. input_shape hidden_shaper>is_cross_attention query_statesrMcurr_past_key_valuescurrent_states key_states value_stateskv_shaper=attention_interfacerDrCs&&&&&&, r3rUdopAttention.forwardsY$))#2. BBbB$*A*AB M\Mh?99$..Ino7ABRTYT`T`7a7a+113gw.T9vvm,11,?II!QO  o': ; ;(3377GJ!'6'L'L$'6'K'K$#2 -?)] /"=*-44T^^DIIJ/66t~~FMMLP--cr2PBP8O8OPH/44X>HHANJ66.166x@JJ1aPL*+?+F+FzS_aeaoao+p( L%*_FY*Z*ZAEO..t~~>  #))"-J333 %  k!njA'..&,,! ...4===26M/ $ 1 1NJ|7J7J]m!2! )@(M(M KK , ,.E) %8      % $}}}C$,, % LL % ( %  % ! \"));;;;FFH ff[) M<77r5)rrr,r&rrrrr!rrrr#r r%rrr+r")FNF)T )Nr])NNNN) rTrUrVrWr staticmethodr6rCrrZr[rrs@@r3rr~s<%,%,N- - ^"X8X8X8r5rcXaa]tRtRtoRV3RlV3RllltRV3RlRlltRtVtV;t#) UdopLayerSelfAttentioniCc.<V^8dQhRS[R,/#r%rNr)r2rQs"r3r4#UdopLayerSelfAttention.__annotate__Ds 7 7SSWZ 7r5c<\SV`4\VVVVPR7Vn\ VP VPR7Vn\P!VP4Vn R#)rrrrN) rrrr SelfAttentionrrrrr rrr,rrrrrs&&&&r3rUdopLayerSelfAttention.__init__Ds] * (C''   (F4'EdVPPpVPP@pVPPBp\ P !VPDPRW#V,R,,R7\ P !VPFPRW#R,,R7\ P !VPHPRW#R,,R7\ P !VPJPRW%V,R,,R7VPL'd=\ P !VP"PRW#R,,R7R#R#R#)zInitialize the weightsr^r{)rstdNlm_headrr<)'r _init_weightsrinitializer_factorrmrinit constant_rr r trunc_normal_rzeros_RelativePositionBiasBasernormal_r% UdopModelsharedUdopForConditionalGenerationhasattrtie_word_embeddingsrrrrrrrrrrrr r!r"r#r)rr&factorrrrrs&& r3r!UdopPreTrainedModel._init_weightssk f%// fm , , NN6==&3, 7  * *   v}}3F C{{& FKK('  8 9 9[[33Fkk))G LL77>>SfeiXiNj k  * * LL--CVc\ J  < = =vy))$++2Q2Q2Q V^^22&3,O3R)  1 2 2 LL))KKDWDW\`C`9a bvyy&))fiinn.H FIINN+ LL))KKDTDTY]C]9^ _vyy&))fiinn.H FIINN+/I)  6 7 7 LL++#6kkFYFY^bEb;c dv{{F++ 0@0@0L FKK,,- LL++#6kkFYFY^bEb;c dv{{F++ 0@0@0L FKK,,- LL))KKDTDTY]C]9^ _vyy&))fiinn.H FIINN+/I)  . .kk))G!%!1!1 kk++G LLsM_C_dhBh8i j LLs4-8P Q LLs4-8P Q LLsM_C_dhBh8i j111 V;;BBRXim\mRno2/r5cVPPpVPPpVfQR4hVPVP4pVRRR13,P 4VR&W$R&VfQR4hVP VR8HV4\P!V^84P4'gQR4hV#) Nzself.model.config.decoder_start_token_id has to be defined. In Udop it is usually set to the pad_token_id. See Udop docs for more information.z1self.model.config.pad_token_id has to be defined.z8Verify that `shifted_input_ids` has only positive valuesr;).r_).r]) rdecoder_start_token_id pad_token_id new_zerosrnrL masked_fill_r0allr)r input_idsrrshifted_input_idss&& r3 _shift_right UdopPreTrainedModel._shift_rights!%!C!C{{// %1  @ 1 &// @%.sCRCx%8%>%>%@'"$:&!'\)\\'&&'8D'@,Oyy*a/05577s9ss7  r5c&<V^8dQh/S[;R&#rr )r2rQs"r3r4 UdopPreTrainedModel.__annotate__s r5rS)imagetextr;)rTrUrVrWbase_model_prefixinput_modalitiessupports_gradient_checkpointing_can_compile_fullgraph_keep_in_fp32_modules_supports_flash_attn_supports_flex_attn_supports_sdpar"rrjr|_can_record_outputsr0no_gradrrrYrZr[rrs@@r3rrs&(&*#"!F N  ;n%;2FN+B"M  ]]_-p-p`!Ir5rc<aa]tRtRtoRV3RlltRtRtVtV;t#)UdopCellEmbeddingsi.c<\SV`4Wn\P!W4Vn\P!W4VnR#r)rrmax_2d_position_embeddingsr r$x_position_embeddingsy_position_embeddings)rrrrs&&&r3rUdopCellEmbeddings.__init__/s8 *D'%'\\2L%Z"%'\\2L%Z"r5c\P!VRR4pWP^, ,P4pVP ^VP^, 4pVP VR,4pVP VR,4pVP VR,4pVP VR,4pVV,V,V,pV#)r{r^rrrr)r0rrrrrr)rrleft_position_embeddingsupper_position_embeddingsright_position_embeddingslower_position_embeddingsrs&& r3rUdopCellEmbeddings.forward6szz$S)77!;<BBDzz!T<$>tG}$M!$($>$>tG}$M!$($>$>tG}$M! %' (' (( ( r5)rrr)irrs@@r3rr.s[r5rcaa]tRtRtoRtR V3Rllt]R V3RlRll4tR V3RlRlltRt R V3R lR llt R t Vt V;t #)riQa Base class of relative biases. Args: num_heads (`int`): Number of attention heads in the model, it will create embeddings of size `num_heads`, which will be added to the scores of each token pair. relative_attention_num_buckets (`int`, *optional*, defaults to 32): Pair token metric (distance in the sequence, distance in pixels etc.) will be bucketed, parameter is defining number of such buckets. bidirectional (`bool`, *optional*, defaults to `True`): Whether the distance should be bidirectional for a pair of tokens. If `False`, then distance(tok1, tok2) == distance(tok2, tok1). scaling_factor (`int`, *optional*, defaults to 1): Defining factor which will be used to scale relative distance. max_distance (`int`, *optional*, defaults to 128): All distances above this value will end up in the one/same bucket. augmentation (`bool`, *optional*, defaults to `False`): Whether to multiply relative distances by a random scalar. expand (`bool`, *optional*, defaults to `False`): Whether to expand an existing pretrained model with subsequent additions of prefix_bucket. c :<\S V`4WnWpnW`nWPnW@nW0nWnWn W n V'dVP'g^M^p \P!VPV ,VP4Vn R#)r%N)rr prefix_bucket augmentationlevelr1scaling_factorr/rexpandrr r$r%) rrrr/rr1rrrr extra_headrs &&&&&&&&&& r3r!RelativePositionBiasBase.__init__gsz *( (,*" .L+' Q ')||D4W4WZd4dfjftft'u$r5c^<V^8dQhRS[R,RS[S[S[3,R,RS[/#r%r*Nrreturnr dictstrr)r2rQs"r3r4%RelativePositionBiasBase.__annotate__s:    38nt#    r5cR#rrS)rr*rs&&&r3 prepare_input&RelativePositionBiasBase.prepare_inputs r5c^<V^8dQhRS[R,RS[S[S[3,R,RS[/#rr)r2rQs"r3r4rs5$T#s(^VZEZflr5cVPW4p\VVPVPVPR7pV#)r9)rget_relative_position_bucketr/rr1)rr*rr. rp_buckets&&& r3 get_bucket#RelativePositionBiasBase.get_bucketsC ..~D8 ,,;;**  r5c8VR,pVR,pW2, P4pVP'd1VP'dV\P!\ ,pW@P ,pVP\P4#)rr)rNr) r1rr:randomuniformAUGMENTATION_RANGErrqr0r)r positionsr?r@r.s&& r3get_relative_position.RelativePositionBiasBase.get_relative_positionsv$Z0#J/,?FFH     1C!D D 000 ##EJJ//r5c^<V^8dQhRS[R,RS[S[S[3,R,RS[/#rr)r2rQs"r3r4rs5ftm$sCx.SWBWcir5cTVP'dVP'd\P!VP^,VP 4pVP PPVPPRVP%RVPPVPR%W0nRVnVPW4pVP'dVP^4^8Xd9VP^4^8d#VPVP^4^^4pVR,^8pVPR4p\VP4P44F6wrxVPWGRV1VR13&VP^,WGVR1RV13&K8 VP!^VP P"^, 4pVP V4p V P%4^8wd \'R4hV P).RO4p V #)r%Ng?Fz Wrong dimension of values tensorrr;)r]rr`r%)rrr r$rrr%rdatarr=rcsum enumeratecpunumpyrnum_embeddingsr8rr;) rr*rnew_biasr is_prefix num_prefixidxnum_prefix_rowrBs &&& r3r RelativePositionBiasBase.forwards ;;;4---||D$G$G!$KT^^\HJNJfJfJmJmJrJrHOO !F4#F#F GJMHOO !D!D!F G+3 (DKOON9    ~~a A%.*=*=a*@1*D%,,^-@-@-CQJ W )I"r*J'01A1G1G1I'J#CGCfCf ?@CGCfCfijCj ~?@(K OOAt'C'C'R'RUV'VW 55i@ ::<1 ?@ @ - r5) rr/rrr1rrr%rr) NrfTr`rgtokensFFFNN)rTrUrVrWrXrrrrrrrZr[rrs@@r3rrQsG*v2    0r5rcLaa]tRtRtoRV3RlltRV3RlRlltRtVtV;t#)RelativePositionBias1Dic 0<\SV`!RRVRV/VBR#)z Reimplementation of T5 relative position bias. Distance between given tokens is their distance in the sequence. Parameters are the same as in base class rr1NrSrrrrr1r.rs&&&,r3rRelativePositionBias1D.__init__ \\\\U[\r5c^<V^8dQhRS[R,RS[S[S[3,R,RS[/#rr)r2rQs"r3r4#RelativePositionBias1D.__annotate__s6!!FTM!SRUXY]H]!io!r5cVP^8wd \R4hVP\P!VP ^4\P VPR7R,4pV#)r`zNo need to scale 1d featuresr|r:)rrrr0rar=rr~)rr*rr.s&&& r3r$RelativePositionBias1D.prepare_inputs`   ! #;< < 66 LL,,Q/uzz.J_J_ `ah i ! r5rS)r`rgr  rTrUrVrWrrrZr[rrs@@r3rrs]!!!r5rcLaa]tRtRtoRV3RlltRV3RlRlltRtVtV;t#)RelativePositionBiasHorizontalic 0<\SV`!RRVRV/VBR#)zv Represents in the bucket embeddings horizontal distance between two tokens. Parameters are the same as in base class rr1NrSrrs&&&,r3r'RelativePositionBiasHorizontal.__init__rr5c^<V^8dQhRS[R,RS[S[S[3,R,RS[/#rr)r2rQs"r3r4+RelativePositionBiasHorizontal.__annotate__s6??FTM?SRUXY]H]?io?r5cVPR8g \R4hVf \R4hVRR^^.3,PRR7pVPV4#)r^ENeed to scale the values of bboxes, as there are in small (0,1) rangez6Bbox is required for horizontal relative position biasrr7r;rrrr)rr*rhorizontal_positions&&& r3r,RelativePositionBiasHorizontal.prepare_inputsb""S(de e <UV V&*1a!Q<&8&=&="&=&E))*=>>r5rSdr&r rrs@@r3rrs]???r5rcLaa]tRtRtoRV3RlltRV3RlRlltRtVtV;t#)RelativePositionBiasVerticalic 0<\SV`!RRVRV/VBR#)zt Represents in the bucket embeddings vertical distance between two tokens. Parameters are the same as in base class rr1NrSrrs&&&,r3r%RelativePositionBiasVertical.__init__rr5c^<V^8dQhRS[R,RS[S[S[3,R,RS[/#rr)r2rQs"r3r4)RelativePositionBiasVertical.__annotate__s6==FTM=SRUXY]H]=io=r5cVPR8g \R4hVf \R4hVRR^^.3,PRR7pVPV4#)r^r!z4Bbox is required for vertical relative position biasrr7r;r")rr*rvertical_positions&&& r3r*RelativePositionBiasVertical.prepare_inputsb""S(de e <ST T$(A1v$6$;$;$;$C))*;<$%[\\?,;K(   \)4C{C D 9 r5caa]tRtRtoRtV3Rlt]V3RlRl4tRtRt ] ] R V3RlR ll44t R t VtV;t#) UdopStacki%z{ This class is based on `T5Stack`, but modified to take into account the image modality as well as 2D position embeddings. c <\SV`V4\P!VPVP 4Vn\V4VnVPVn VPVn \P!\VP4Uu.uFp\V\V^8H4VR7NK up4Vn\!VP VP"R7Vn\P&!VP(4VnVP'g&\-VP.VP04VnVP5V4VnVP94R#uupi)r]rrN)rrr r$ vocab_sizer embed_tokensr embed_patchesr num_layersrrrrblockrrfinal_layer_normrrr,rrrcell_2d_embedding_get_relative_bias relative_bias post_init)rrrrs&& r3rUdopStack.__init__+s  LL):):FNNK08 ++ ++]]_deietet_u v_uZ[Yv4Q2B-P^O__lm   U[[%;q%@ 9dkk.F.FyO_O_gpgvgvwI"[[9;K;KS\SbSbcN;;|I4D4DIOO\D!,,^>> from transformers import AutoProcessor, AutoModel >>> from datasets import load_dataset >>> import torch >>> # load model and processor >>> # in this case, we already have performed OCR ourselves >>> # so we initialize the processor with `apply_ocr=False` >>> processor = AutoProcessor.from_pretrained("microsoft/udop-large", apply_ocr=False) >>> model = AutoModel.from_pretrained("microsoft/udop-large") >>> # load an example image, along with the words and coordinates >>> # which were extracted using an OCR engine >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> image = example["image"] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> inputs = processor(image, words, boxes=boxes, return_tensors="pt") >>> decoder_input_ids = torch.tensor([[model.config.decoder_start_token_id]]) >>> # forward pass >>> outputs = model(**inputs, decoder_input_ids=decoder_input_ids) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 1, 1024] ```rr*rrrrrKrrrt)rJrKdecoder_hidden_statesdecoder_attentionsrNencoder_last_hidden_staterencoder_attentionsrS) rr*rrrJrKrLrMrN)rrr*rrrrrrrrKrrtr.rLrdecoder_outputss&&&&&&&&&&&&&, r3rUdopModel.forwards&P  ""ll#-*  (  , O(* !0!?!?,,  '  1  0  ,  #0  $:       "-??+;;"1"?"?.99,==&5&G&G"1"?"?.99  r5)rrrrrrTrUrVrW_tied_weights_keysrrrir rrrZr[rrs@@r3rrs^ &%+-F)+B (: i i i i r5ra The UDOP encoder-decoder Transformer with a language modeling head on top, enabling to generate text given document images and an optional prompt. This class is based on [`T5ForConditionalGeneration`], extended to deal with images and layout (2D) data. caa]tRtRtoRRRRRRRRR R R R R R/tV3R ltRtRt]] RV3RlRll44t Rt Vt V;t #)ritrrrrrrr=encoder.relative_bias.biases.0.relative_attention_bias.weightDencoder.block.0.layer.0.SelfAttention.relative_attention_bias.weightz=decoder.relative_bias.biases.0.relative_attention_bias.weightzlm_head.weightc<\SV`V4\P!VPVP 4Vn\V4Vn\V4pRVn RVn \V4Vn \V4pRVn VPVn\V4Vn\P"!VP VPRR7VnVP'4R#)FTrN)rrr r$rQrrrrrrrtrOrrrTrrrrZrs&& r3r%UdopForConditionalGeneration.__init__s  ll6#4#4fnnE .v6!&)$)!#(  0 !&)$(!$*$=$=! 0 yy1B1BO  r5cVP#rrrcs&r3r1UdopForConditionalGeneration.get_input_embeddingsrr5c~WnVPPV4VPPV4R#rrrgs&&r3ri1UdopForConditionalGeneration.set_input_embeddingsrr5cf<V^8dQhRS[R,RS[R,RS[S[S[3,R,RS[R,RS[S[S[3,R,RS[R,RS[R,R S[R,R S[R,R S[R,R S[R,R S[R,RS[R,RS[S[,RS[/#)r%rNr*rrrrrrrrKrrtlabelsr.r) r rrrrrrrr)r2rQs"r3r4)UdopForConditionalGeneration.__annotate__s~ ~ D=~  ~ 38nt# ~ tm ~ #s(^d* ~ "D=~ !' ~ }~  $~ ~  &}~ $;~  ~ +,~ !~ r5c VfV eVPV 4pV fVP!RRVRVRVRVRVRV/VBp V ^,pV PpVP!RRVRVRV RV R VR VR V /VBpV^,pVPP 'd%VVPP R,,pVPV4pRpV eD\RR 7pV!VPRVPR44V PR44p\VVVPVPVPVPV P V PV PR 7 #)a bbox (`torch.LongTensor` of shape `({0}, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. Note that `sequence_length = token_sequence_length + patch_sequence_length + 1` where `1` is for [CLS] token. See `pixel_values` for `patch_sequence_length`. visual_bbox (`torch.LongTensor` of shape `(batch_size, patch_sequence_length, 4)`, *optional*): Bounding boxes of each patch in the image. If not provided, bounding boxes are created in the model. decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) T5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [T5 Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]`. Examples: ```python >>> from transformers import AutoProcessor, UdopForConditionalGeneration >>> from datasets import load_dataset >>> # load model and processor >>> # in this case, we already have performed OCR ourselves >>> # so we initialize the processor with `apply_ocr=False` >>> processor = AutoProcessor.from_pretrained("microsoft/udop-large", apply_ocr=False) >>> model = UdopForConditionalGeneration.from_pretrained("microsoft/udop-large") >>> # load an example image, along with the words and coordinates >>> # which were extracted using an OCR engine >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> image = example["image"] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> # one can use the various task prefixes (prompts) used during pre-training >>> # e.g. the task prefix for DocVQA is "Question answering. " >>> question = "Question answering. What is the date on the form?" >>> encoding = processor(image, question, text_pair=words, boxes=boxes, return_tensors="pt") >>> # autoregressive generation >>> predicted_ids = model.generate(**encoding) >>> print(processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]) 9/30/92 ```Nrrrrr*rrKrrrt) ignore_index) losslogitsrKrrrNrrrrSr<rr;)rrr*rrrrrr rdr=rrKrLrMrNrJ)rrr*rrrrrrrrKrrtrr.rLrrsequence_output lm_logitsrloss_fcts&&&&&&&&&&&&&&, r3r$UdopForConditionalGeneration.forwardsX  $); $ 1 1& 9   ""ll#(*  .  , O(* !0!?!?,,  '  1  0  ,  #0  $:       *!, ;; * * *-1D1Dd1JKOLL1   'T:HINN2y~~b/ABFKKPROTD+;;"1"?"?.99,==&5&G&G"1"?"?.99  r5)rrrrr) NNNNNNNNNNNNNrrs@@r3rrtsw &%+-F)+BGJPGJP/.: ~ ~ ~ ~ r5rcaa]tRtRtoRRRRRRRR /tV3R lV3R lltR tR t]] RV3RlRll44t Rt Vt V;t #)UdopEncoderModeli)rrrrrrrrc <V^8dQhRS[/#rr )r2rQs"r3r4UdopEncoderModel.__annotate__2szr5c&<\SV`V4\P!VPVP 4Vn\V4Vn\V4pRVn RVn RVn \V4VnVP4R#ry)rrr r$rQrrrrrrrtrsrOrrZ)rrrrs&& r3rUdopEncoderModel.__init__2su  ll6#4#4fnnE .v6!&)$)!#( ,1) 0  r5cVP#rrrcs&r3r%UdopEncoderModel.get_input_embeddingsBrr5cHWnVPPV4R#r)rrrirgs&&r3ri%UdopEncoderModel.set_input_embeddingsEs$  )).9r5c<V^8dQhRS[R,RS[S[S[3,R,RS[R,RS[R,RS[S[S[3,R,RS[R,RS[S[,R S[/#) r%rNrr*rrrr.r)r rrrrrrH)r2rQs"r3r4rKsAAD=A38nt#A A tm A #s(^d* A}A+,A *Ar5c BVP!RRVRVRVRVRVRV/VBpV#)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). bbox (`torch.LongTensor` of shape `({0}, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. Note that `sequence_length = token_sequence_length + patch_sequence_length + 1` where `1` is for [CLS] token. See `pixel_values` for `patch_sequence_length`. visual_bbox (`torch.LongTensor` of shape `(batch_size, patch_sequence_length, 4)`, *optional*): Bounding boxes of each patch in the image. If not provided, bounding boxes are created in the model. Example: ```python >>> from transformers import AutoProcessor, UdopEncoderModel >>> from huggingface_hub import hf_hub_download >>> from datasets import load_dataset >>> # load model and processor >>> # in this case, we already have performed OCR ourselves >>> # so we initialize the processor with `apply_ocr=False` >>> processor = AutoProcessor.from_pretrained("microsoft/udop-large", apply_ocr=False) >>> model = UdopEncoderModel.from_pretrained("microsoft/udop-large") >>> # load an example image, along with the words and coordinates >>> # which were extracted using an OCR engine >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> image = example["image"] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = processor(image, words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> last_hidden_states = outputs.last_hidden_state ```rrrrr*rrS)r) rrrr*rrrr.rs &&&&&&&, r3rUdopEncoderModel.forwardIs]r,,   $ &  *  (   r5)rrrrrrs@@r3rr)sh &+-F)+BGJP  :AAAAr5r)Nr{N))r])Nr]rr)g?g?)rrrr)^rXrloggingr,rrrrcollections.abcrrcopyr dataclassesrtypingrr0r r torch.nnr transformersr transformers.modeling_outputsrrrnrr activationsr cache_utilsrrr generationr masking_utilsrrmodeling_layersrmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericr r!utils.output_capturingr"r# getLoggerrTrrErHrkryrr/rrrrrrrjr|rrrr6rrrrrr(r1rGrMrOrrr__all__rSr5r3rs #.! %# '!CC)J9F& JE   8 $%@  #={#= #=L. P/f"))<+BII+4 .RYY<"))&A8BIIA8J=RYY=B9bii9@LP*LP^W!/W!W!t> -FF!hryy#hV!5!$?%=?&=#;=&RYY* 0, ,y #y xO #O O dj #6j j Zb*bbJ cr5