+ jK^RIHtHt^RIHt^RIt^RIHtHt^RIH t ^RI H t ^RI HtHt^RIHt^R IHt^R IHtHtHtHtHt^R IHtHt^R IHt^R IH t ^RI!H"t"H#t#H$t$^RI%H&t&H't'^RI(H)t)^RI*H+t+]#!RR7]!RR]444t,!RR]PZ4t.!RR]PZ4t/!RR]PZ4t0RERRllt1!RR]PZ4t2!R R!]PZ4t3!R"R#]PZ4t4!R$R%]4t5]#!R&R']44t6]#!R(R)]644t7!R*R+]PZ4t8]#!R,R7!R-R.]644t9]#!R/R7!R0R1]644t:!R2R3]PZ4t;!R4R5]PZ4t<!R6R7]PZ4t=!R8R9]PZ4t>!R:R;]PZ4t?!R<R=]PZ4t@!R>R?]PZ4tA]#!R@RA]644tB]#!RBR7!RCRD]]644tC.RFOtDR#)G)CallableIterable) dataclassN)Tensornn)initialization)ACT2FN) BackboneMixinfilter_output_hidden_states)create_bidirectional_mask)GradientCheckpointingLayer)BackboneOutputBaseModelOutputWithPoolingImageClassifierOutputMaskedLMOutputSemanticSegmenterOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)#compile_compatible_method_lru_cache)TransformersKwargsauto_docstring torch_int)can_return_tuplemerge_with_config_defaults)capture_outputs BeitConfigz- Class for outputs of [`BeitModel`]. ) custom_introc]tRt^0tRtRtR#)BeitModelOutputWithPoolinga2 pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`): Average of the last layer hidden states of the patch tokens (excluding the *[CLS]* token) if *config.use_mean_pooling* is set to True. If set to False, then the final hidden state of the *[CLS]* token will be returned. N)__name__ __module__ __qualname____firstlineno____doc____static_attributes__r"g/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/beit/modeling_beit.pyr!r!0sr)r!cTaa]tRt^?toRtV3RlV3RlltV3RlRltRtVtV;t #)BeitPatchEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. c <V^8dQhRS[/#configr)format __classdict__s"r* __annotate__ BeitPatchEmbeddings.__annotate__Fs x xz xr)c<\SV`4VPpVPp\ V\ 4'dTMW"3p\ V\ 4'dTMW33pV^,V^,,V^,V^,,,VnW nW0nVPVn\P!VPVPW3R7Vn R#) kernel_sizestrideN) super__init__ image_size patch_size isinstancer num_patches num_channelsrConv2d hidden_size projection)selfr0r<r= __class__s&& r*r;BeitPatchEmbeddings.__init__Fs && && #-j(#C#CZ*Ia #-j(#C#CZ*Ia &qMZ]:z!}PZ[\P]?]^$$"//))F$7$79K9KYcwr)cN<V^8dQhRS[PRS[P/#)r/ pixel_valuesreturntorchr)r1r2s"r*r3r4Ss'HHELLHU\\Hr)cVP^,pW P8wd\RVP RV R24hVPV4P ^4P ^^4#)r6zoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .)shaper@ ValueErrorrCflatten transpose)rDrHr@s&& r*forwardBeitPatchEmbeddings.forwardSss#))!, ,, ,!../yaI |,44Q7AA!QGGr))r<r@r?r=rC r#r$r%r&r'r;rRr(__classdictcell__ __classcell__rEr2s@@r*r,r,?s( x xHHHr)r,cjaa]tRt^]toRtV3RlV3RlltV3RlRltR V3RlRlltRtVt V;t #) BeitEmbeddingszZ Construct the CLS token, position and patch embeddings. Optionally, also the mask token. c$<V^8dQhRS[RR/#r/r0rINr)r1r2s"r*r3BeitEmbeddings.__annotate__bs > >z >d >r)c<\SV`4\P!\P !^^VP 44VnVP'd7\P!\P !^^VP 44MRVn \V4Vn VPVn VPPpVP'd>\P!\P !^V^,VP 44MRVn\P !VP"4VnR#)r6N)r:r;r ParameterrKzerosrB cls_tokenuse_mask_token mask_tokenr,patch_embeddingsr=r? use_absolute_position_embeddingsposition_embeddingsDropouthidden_dropout_probdropout)rDr0r?rEs&& r*r;BeitEmbeddings.__init__bs ekk!Q8J8J&KLQWQfQfQf",,u{{1a9K9K'LMlp 3F ; ++++77 666 LLQ a9K9KL M  zz&"<"<= r)cZ<V^8dQhRS[PRS[RS[RS[P/#)r/ embeddingsheightwidthrI)rKrint)r1r2s"r*r3r\qs;&D&D5<<&D&DUX&D]b]i]i&Dr)c VP^,^, pVPP^,^, p\PP 4'gWE8XdW#8Xd VP#VPR,pVPR,pVPR,pW P ,p W0P ,p \ VR,4p VP^WV4pVP^^^^4p\PPVW3RRR7pVP^^^^4P^RV4p\P!Wg3^R7#) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 g?bicubicFsizemode align_cornersdim)NNN:Nr6Nrwr6NN)rNrerKjit is_tracingr=rreshapepermuter functional interpolateviewcat) rDrkrlrmr? num_positionsclass_pos_embedpatch_pos_embedrv new_height new_widthsqrt_num_positionss &&&& r*interpolate_pos_encoding'BeitEmbeddings.interpolate_pos_encodingqsE!&&q)A- 0066q9A= yy##%%+*F6?++ +22592259r". __, &}c'9:)11!5G]`a)11!Q1=--33 ( 4 *11!Q1=BB1b#Nyy/;CCr)cv<V^8dQhRS[PRS[PR,RS[P/#)r/rHbool_masked_posNrI)rKr BoolTensor)r1r2s"r*r3r\s<ll))D0  r)cVPwr4pVPV4pVP4wrxpVeZVPP WxR4p VP R4P V 4p V^V , ,W,,pVPP VRR4p \P!W3^R7pVPeW`PWdV4,pVPV4pV#)Nrurz) rNrcrrrbexpand unsqueezetype_asr`rKrrerrh) rDrHr_rlrmrk batch_sizeseq_len mask_tokensmask cls_tokenss &&& r*rRBeitEmbeddings.forwards +001e**<8 !+!2 Q  &//00bIK",,R088ED#q4x0;3EEJ^^**:r2> YY 7Q?  # # /#&C&CJX]&^^J\\*- r))r`rhrbrcr=reN) r#r$r%r&r'r;rrRr(rUrVrWs@@r*rYrY]s0 > >&D&DPr)rYcaa]tRt^toV3RlV3Rllt]]!^ R7V3RlRl44tR V3RlRlltRt Vt V;t #) BeitRelativePositionBiasc$<V^8dQhRS[RR/#r[r)r1r2s"r*r3%BeitRelativePositionBias.__annotate__s   z  d  r)c<\SV`4VPp\V\\ 34'gW"3pV^,VP ,V^,VP ,3Vn^VP^,,^, ^VP^,,^, ,^,Vn\P!\P!VPVP44VnR#)rN)r:r;r<r>tuplelistr= window_sizenum_relative_distancerr^rKr_num_attention_headsrelative_position_bias_table)rDr0r<rEs&& r*r;!BeitRelativePositionBias.__init__s && *udm44$1J&qMV->->> 1 QWQbQb@bc&'$*:*:1*=&=&Aa$JZJZ[\J]F]`aFa%bef%f",.LL KK22F4N4N O- )r))maxsizecP<V^8dQhRS[S[S[3,RS[P/#)r/rrI)rrnrKr)r1r2s"r*r3rs'''eCHo'%,,'r)c  ^V^,,^, ^V^,,^, ,^,pV^,V^,,p\P!\P!\P!\P!V^,4\P!V^,4RR74^R7pVR,VR,, P ^^^4P 4pVR;;,V^,^, , uu&VR;;,V^,^, , uu&VR;;,^V^,,^, ,uu&\P!V^,3^,VPR7pVPR 4VR &V^, VR &V^, VR &V^, VR &V#)z This method creates the relative position index, modified to support arbitrary window sizes, as introduced in [MiDaS v3.1](https://huggingface.co/papers/2307.14460). ij)indexing) start_dim)rrdtype)rwrwN)rwNrw)rwrwr)rwrwr6rz)ryry)rrNN)rr)rr) rKrPstackmeshgridaranger~ contiguousr_rsum)rr window_areacoords_flattenrelative_coordsrelative_position_indexs& r* generate_relative_position_index9BeitRelativePositionBias.generate_relative_position_indexs"#[^!3a!7A A)B&)>)B&(=(A%&&r)c:<V^8dQhRS[RS[P/#)r/rrI)boolrKr)r1r2s"r*r3rs"-3-3T-3]b]i]i-3r)c t^VP^,,^, p^VP^,,^, p^V^,,^, p^V^,,^, pVPpVPp Wg,^,p VRV ^, p V P^WTR4P ^^^^4p \ P PV \V4\V43RR7p V P ^^^^4PV ^, R4p \P!WV ^, R.4p VPV4pWPR4,pVPV^,V^,,^,V^,V^,,^,R4pVP ^^^4P4pV'dC\ P PVP^4W33RRR7P^4pVP^4#)ze Modification of timm.models.beit.py: Attention._get_rel_pos_bias to support arbitrary window sizes. Nbilinear)rrrsFrqrz)rrrr}r~rrrrrKrrrrrsqueeze)rDrrdim_size old_height old_widthrr old_relative_position_bias_tableold_num_relative_distancenew_num_relative_distance old_sub_table new_sub_table new_relative_position_bias_tablerrelative_position_biass&&&& r*rR BeitRelativePositionBias.forwards!))!,,q0 ((++a/ Q'!+  A&* +/+L+L($($>$>!$.$:Q$>!89X;TWX;XY %--aKSSTUWXZ[]^_  11 :!6 )8L MT^2 &--aAq9AAB[^_B_acd +099 =VYZ=Z=\] ^, (#'"G"G "T!AB^B^_aBb!c"8!%>&003)# &?& gaj #&//22r))rrr)FN) r#r$r%r&r; staticmethodrrrRr(rUrVrWs@@r*rrs?   (4'5'4-3-3-3r)rcV^8dQhR\PR\PR\PR\PR\PR,R\R,R\R \ \ ,/#) r/modulequerykeyvalueattention_maskNscalingrhkwargs)rModulerKrfloatrr)r1s"r*r3r3 s{%% II% <<% % << % LL4' % T\ %%'(%r)c VfVPR4R,p\P!WP^^44V,pVe W,p\P P VR\PR7PVP4p\P PWVPR7p\P!W4p V P^^4P4p W3#)N)rvr)ptrainingrz) rrrKmatmulrQrrsoftmaxfloat32torrhrr) rrrrrrrhr attn_weights attn_outputs &&&&&&&, r*eager_attention_forwardr s**R.D(<<}}Q':;gEL!#4 ==((2U]](SVVW\WbWbcL==((6??([L,,|3K''1-88:K  $$r)cTaa]tRtRtoV3RlV3RlltRV3RlRlltRtVtV;t#) BeitAttentioni(c <V^8dQhRS[/#r.r)r1r2s"r*r3BeitAttention.__annotate__)s ` `z `r)c<\SV`4WnVPVn\ VRVP VP,4VnVPVnVP R,Vn RVn \P!VP VPVP ,4Vn \P!VP VPVP ,RR7Vn\P!VP VPVP ,4Vn\P!VPVP ,VP 4VnR#)head_dimF)biasNr)r:r;r0rgetattrrBrattention_probs_dropout_probattention_dropoutr is_causalrLinearq_projk_projv_projo_projrDr0rEs&&r*r;BeitAttention.__init__)s  #)#=#=  F4F4F&JdJd4de !'!D!D}}d* ii 2 2F4N4NQUQ^Q^4^_ ii 2 2F4N4NQUQ^Q^4^ejk ii 2 2F4N4NQUQ^Q^4^_ ii : :T]] JFL^L^_ r)c <V^8dQhRS[PRS[PR,RS[S[,RS[S[PS[P3,/#)r/ hidden_statesrNrrI)rKrrrr)r1r2s"r*r3r6sX))||) t+)+, ) u||U\\) * )r)c VPRRp.VORNVPN5pVPV4PV4P ^^4pVP V4PV4P ^^4pVP V4PV4P ^^4p\P!VPP\4p T !VVVVV3RVP'gRM VPRVP/VBwrV P!.VORN5!P!4p VP#V 4p W3#)Nrhrrz)rNrrrrQrrr get_interfacer0_attn_implementationrrrrr}rr) rDrrr input_shape hidden_shape query_states key_states value_statesattention_interfacerrs &&&, r*rRBeitAttention.forward6sN $))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST (?(M(M KK , ,.E) %8      % $}}}C$2H2H % LL %  % ! "));;;;FFH kk+. ((r)) rr0rrrrrrrrr r#r$r%r&r;rRr(rUrVrWs@@r*rr(s  ` `)))r)rcPaa]tRtRtoV3RlV3RlltV3RlRltRtVtV;t#)BeitMLPiXc <V^8dQhRS[/#r.r)r1r2s"r*r3BeitMLP.__annotate__YsKKzKr)c(<\SV`4Wn\VP,Vn\ P!VPVP4Vn \ P!VPVP4Vn R#r) r:r;r0r hidden_act activation_fnrrrBintermediate_sizefc1fc2rs&&r*r;BeitMLP.__init__Ysb  #F$5$5699V//1I1IJ99V55v7I7IJr)cN<V^8dQhRS[PRS[P/#r/rrIrJ)r1r2s"r*r3r`#U\\ellr)clVPV4pVPV4pVPV4pV#r)rrrrDrs&&r*rRBeitMLP.forward`s4/ **=9 / r))rr0rrrrWs@@r*rrXs KKr)rcjaa]tRtRtoRtR V3RlV3RllltV3RlRltV3RlRltR tVt V;t #) BeitDropPathihzStochastic depth (DropPath) per sample, for residual blocks. Identity when ``drop_prob`` is 0 or outside training. See `Deep Networks with Stochastic Depth `_. c$<V^8dQhRS[RR/#)r/ drop_probrIN)r)r1r2s"r*r3BeitDropPath.__annotate__os##%#$#r)c0<\SV`4WnR#r)r:r;r)rDrrEs&&r*r;BeitDropPath.__init__os "r)cN<V^8dQhRS[PRS[P/#r rJ)r1r2s"r*r3rss#<>S   & $$Q')DM4F4F4J,KK 50C0CML`L`a  M$=>   +m;;r)c <V^8dQhRS[/#)r/rI)str)r1r2s"r*r3r|s%%C%r)c RVP 2#)zp=rrDs&r* extra_reprBeitDropPath.extra_repr|sDNN#$$r)r$r) r#r$r%r&r'r;rRr&r(rUrVrWs@@r*rrhs- ##<<%%%r)rc\aa]tRtRtoRtRV3RlV3RllltR V3RlRlltRtVtV;t #) BeitLayeriz?This corresponds to the Block class in the timm implementation.c&<V^8dQhRS[RS[/#)r/r0drop_path_rate)rr)r1r2s"r*r3BeitLayer.__annotate__svvzv5vr)c<\SV`4\V4Vn\P !VP VPR7Vn\P !VP VPR7Vn \V4Vn \P!VP4VnVPVnVR8d \!V4M\P"!4VnVP&pV^8d>\P(!V\*P,!VP 4,RR7MRVnV^8d>\P(!V\*P,!VP 4,RR7MRVnVP2'd\5V4VnR#RVnR#)epsrT) requires_gradg?N)r:r;r attentionr LayerNormrBlayer_norm_epslayernorm_beforelayernorm_afterrmlprfrgrhr=rIdentity drop_pathlayer_scale_init_valuer^rKoneslambda_1lambda_2use_relative_position_biasrr)rDr0r, init_valuesrEs&&& r*r;BeitLayer.__init__sC &v. " V-?-?VEZEZ [!||F,>,>FDYDYZ6?zz&"<"<=  ++9G#9Mn5SUS^S^S`33 ^ilm^mBLLuzz&2D2D'EEUY Zsv _jlm^mBLLuzz&2D2D'EEUY Zsv KQJkJkJk&>v&F#qu#r)c <V^8dQhRS[PRS[PR,RS[RS[S[S[3,R,RS[S[,RS[P/#)r/rrNr resolutionrrI)rKrrrrnrr)r1r2s"r*r3r-sk&&||& t+&#' & #s(Od* & +, & &r)c bVPe[VwrgW`P,WpP,3pVPWVP^,R7p Ve W,MT pTp VPV4pVP!V3RV/VBwrVP V4pVP V,pVPV4V ,pTp VPV4pVPV4pVP V4pVPV,pVPV4V ,pV#)N)rr) rr=rNr5r2rhr<r9r6r7r=) rDrrrrBrrlrmrrresidualrs &&&&&, r*rRBeitLayer.forwards6  & & 2&MF!__4e6NOK%)%@%@ @S@STU@V&A& ">  )    ]3  5 }5@ !,,]; /  ]3  5 }5@ r)) r2r9rhr<r=r6r5r7r=rr(NFNrTrWs@@r*r*r*s#Ivv&&&&r)r*caa]tRtRtoRtRtRtRtR.tRt Rt Rt Rt Rt R]R]/tR tR .t]P(!4V3R l4tV3R ltR tVtV;t#)BeitPreTrainedModelibeitrHTr*Fr attentionsrcz.*relative_position_index.*c <\SV`V4\V\4'd\P !VP 4VPe!\P !VP4VPe#\P !VP4R#R#\V\4'd#\P !VP4R#\V\4'd\VP\P4'dm\P!VPVP P"4\P!VP$VP P"4R#R#R#)zInitialize the weightsN)r: _init_weightsr>rYinitzeros_r`rbrerrr*r<rr^ constant_r0r:r=)rDrrEs&&r*rL!BeitPreTrainedModel._init_weightss f% fn - - KK(( )  , F--.))5 F6676  8 9 9 KK;; <  * *&//2<<88v 0R0RSv 0R0RS9+r)c&<V^8dQh/S[;R&#r.r)r1r2s"r*r3 BeitPreTrainedModel.__annotate__s r)r")image)r#r$r%r&base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backend_can_compile_fullgraphr*r_can_record_outputs_input_embed_layer"_keys_to_ignore_on_load_unexpectedrKno_gradrL__annotate_func__r(rUrVrWs@@r*rHrHs$O!&*#$ N "&!m,*H)I& ]]_TT+r)rHcaa]tRtRtoR V3RlV3Rlllt]]!RR7]R V3RlRll444tRt Vt V;t #) BeitModelic*<V^8dQhRS[RS[RR/#)r/r0add_pooling_layerrIN)rr)r1r2s"r*r3BeitModel.__annotate__s"zddr)c  <\SV`V4Wn\V4VnVP 'd \ V4MRVn\VP4Uu.uF7q1PV,\VP^, ^4, NK9 pp\P!VUu.uFp\WR7NK up4VnVP 'd\P"!4M+\P$!VP&VP(R7VnV'd \-V4MRVnVP14R#uupiuupi)z^ add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N)r,r/)r:r;r0rYrk!use_shared_relative_position_biasrshared_position_biasrangenum_hidden_layersr,maxr ModuleListr*layersuse_mean_poolingr8r3rBr4 layernorm BeitPoolerpooler post_init)rDr0rfidrop_path_ratesrrEs&&& r*r;BeitModel.__init__s   (0060X0X0X $V ,^b !W\\b\t\tVu VuQR ! !A %F,D,Dq,H!(L L LVu  mmRa$bRaQYv%HRa$bc $444BKKM",,vGYGY_e_t_t:u ->j(4   %cs "=E5EF)tie_last_hidden_statesc <V^8dQhRS[PRS[PR,RS[RS[PR,RS[S[,RS[/#)r/rHrNrrrrI)rKrrrrrr!)r1r2s"r*r3rgsm-j-jll-j))D0-j#' -j  t+ -j +, -j $-jr)c 8VPWR7pVPR,p\VPVVR7pVPeoVwrWPP ,WPP ,3p VP WVP^,R7p Ve W,MT pTp VP Fp V !V 3RVRVRV/VBp K VPV 4pVPeVPV4MRp\WR 7#) z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). )rr/NN)r0 inputs_embedsrN)rrrrrB)last_hidden_state pooler_output) rkrNr r0rjr=rorqrsr!)rDrHrrrrembedding_outputrBrlrmrshared_relative_position_biasrlayersequence_output pooled_outputs&&&&&, r*rRBeitModel.forwards4 ??2  ) [[E!-*B&   M!..78< 8O O4UY )Oiir))r0rkrqrorsrj)TrF) r#r$r%r&r;rrrrRr(rUrVrWs@@r*rdrdsF2 E2-j-j3 -j-jr)rdcPaa]tRtRtoV3RlV3RlltV3RlRltRtVtV;t#)rri3c$<V^8dQhRS[RR/#r[r)r1r2s"r*r3BeitPooler.__annotate__4s  z d r)c<\SV`4VP'd4\P!VP VP R7VnR#RVnR#)r/N)r:r;rprr3rBr4rqrs&&r*r;BeitPooler.__init__4sD KQKbKbKbBLL++1F1F G hl r)cN<V^8dQhRS[PRS[P/#r rJ)r1r2s"r*r3r:s'vvU\\vellvr)c~VPe(VPVR,P^44#VR,#)Nrwryrw)rwr)rqmeanrs&&r*rRBeitPooler.forward:s6BF..B\t~~mH5::1=>uboptbuur))rqrrWs@@r*rrrr3s!  vvvr)rra Beit Model transformer with a 'language' modeling head on top. BEiT does masked image modeling by predicting visual tokens of a Vector-Quantize Variational Autoencoder (VQ-VAE), whereas other vision models like ViT and DeiT predict RGB pixel values. As a result, this class is incompatible with [`AutoModelForMaskedImageModeling`], so you will need to use [`BeitForMaskedImageModeling`] directly if you wish to do masked image modeling with BEiT. cnaa]tRtRtoV3RlV3RlltRt]]RV3RlRll44tRt Vt V;t #) BeitForMaskedImageModelingi?c$<V^8dQhRS[RR/#r[r)r1r2s"r*r3'BeitForMaskedImageModeling.__annotate__Hs  z d r)cN<\SV`V4VPVn\VRR7Vn\ P !VPVPR7Vn \ P!VPVP4Vn VP4R#)Frfr/N)r:r; num_labelsrdrIrr3rBr4rqr vocab_sizelm_headrtrs&&r*r;#BeitForMaskedImageModeling.__init__Hsu   ++f> f&8&8f>S>STyy!3!3V5F5FG  r)cR#rr"r%s&r*get_output_embeddings0BeitForMaskedImageModeling.get_output_embeddingsUsr)c<V^8dQhRS[PR,RS[PR,RS[PR,RS[RS[PR,RS[S[,RS[S[,/#) r/rHNrlabelsrrrrI)rKrrrrrrr)r1r2s"r*r3rZs@ @ llT)@ ))D0@  t# @ #' @  t+ @ +,@   @ r)c 8VP!V3RVRVRV/VBpVPpVPV4pVPVR,4p Rp Ve%\P !4p V !W,V4p \ V V VPVPR7#)a bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, BeitForMaskedImageModeling >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("microsoft/beit-base-patch16-224-pt22k") >>> model = BeitForMaskedImageModeling.from_pretrained("microsoft/beit-base-patch16-224-pt22k") >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2 >>> pixel_values = image_processor(images=image, return_tensors="pt").pixel_values >>> # create random boolean mask of shape (batch_size, num_patches) >>> bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss, logits = outputs.loss, outputs.logits >>> list(logits.shape) [1, 196, 8192] ```rrrNlosslogitsrrJrx) rIr~rqrrCrossEntropyLossrrrJ) rDrHrrrrroutputsrprediction_scoresmasked_lm_lossloss_fcts &&&&&&, r*rR"BeitForMaskedImageModeling.forwardXsX))  + &> *    "33..9 LL)?@  **,H%&7&H&QN$!//))   r))rIrqrr)NNNFN) r#r$r%r&r;rrrrRr(rUrVrWs@@r*rr?s>  @ @ @ @ r)rz Beit Model transformer with an image classification head on top (a linear layer on top of the average of the final hidden states of the patch tokens) e.g. for ImageNet. chaa]tRtRtoV3RlV3Rllt]]RV3RlRll44tRtVt V;t #)BeitForImageClassificationic$<V^8dQhRS[RR/#r[r)r1r2s"r*r3'BeitForImageClassification.__annotate__s  z d r)c8<\SV`V4VPVn\VRR7VnVP^8d,\ P !VPVP4M\ P!4Vn VP4R#)TrN) r:r;rrdrIrrrBr8 classifierrtrs&&r*r;#BeitForImageClassification.__init__ss   ++f= OUN_N_bcNc"))F$6$68I8IJikititiv r)c <V^8dQhRS[PR,RS[PR,RS[RS[S[,RS[S[,/#r/rHNrrrrI)rKrrrrrr)r1r2s"r*r3rs[   llT)   t#  #'  +,  & &  r)c VP!V3RV/VBpVPpVPV4pRpVeVPW'VP4p\ VVVP VPR7#)ab labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). rNr)rIrr loss_functionr0rrrJ) rDrHrrrrrrrs &&&&, r*rR"BeitForImageClassification.forwards))  %=    -- /  %%fdkkBD$!//))   r))rIrrNNF) r#r$r%r&r;rrrRr(rUrVrWs@@r*rrs5        r)rcTaa]tRtRtoRV3RlV3RllltV3RlRltRtVtV;t#) BeitConvLayeric<V^8dQhRS[RS[RS[S[S[S[3,,RS[RS[S[S[S[3,,S[,RS[RS[S[S[S[3,,RS[R S[/ #) r/ in_channels out_channelsr8r9paddingrdilationgroups activation)rnrr"r)r1r2s"r*r3BeitConvLayer.__annotate__sZZZZ5c?* Z  Z uS#X&, ZZc3h'ZZZr)c <\S V`4\P!VVVVVVVVR7Vn\P !V4VnV e\V ,Vn R#\P!4Vn R#))rrr8r9rrrrN) r:r;rrA convolution BatchNorm2d normalizationr r8r) rDrrr8r9rrrrrrEs &&&&&&&&&&r*r;BeitConvLayer.__init__si 99#%#   ^^L90:0F&,BKKMr)cN<V^8dQhRS[PRS[P/#r rJ)r1r2s"r*r3rs#U\\ellr)clVPV4pVPV4pVPV4pV#r)rrrrs&&r*rRBeitConvLayer.forwards6((7 **=9  6 r))rrr)r6rFr6r6relurrWs@@r*rrs ZZ4r)rcPaa]tRtRtoV3RlV3RlltV3RlRltRtVtV;t#)BeitPyramidPoolingBlockic0<V^8dQhRS[RS[RS[RR/#)r/ pool_scalerchannelsrINrn)r1r2s"r*r3$BeitPyramidPoolingBlock.__annotate__s/HH3HSHCHDHr)c~<\SV`4\P!V4Vn\ W#^R7VnR#)r6r8N)r:r;rAdaptiveAvgPool2dpoolingrconv)rDrrrrEs&&&&r*r; BeitPyramidPoolingBlock.__init__s. ++J7 !+QG r)cj<V^8dQhRS[PRS[S[S[3,RS[P/#)r/inputrrrI)rKrrrn)r1r2s"r*r3rs2U\\sCxU\\r)cVPV4pVPV4p\PP W2RRR7pV#)rFrq)rrrrr)rDrrr hidden_states&&& r*rRBeitPyramidPoolingBlock.forwards@||E* yy. }}00zin0o r))rrrrWs@@r*rrs HH r)rcTaa]tRtRtoRtV3RlV3RlltV3RlRltRtVtV;t #)BeitPyramidPoolingModuleiaK Pyramid Pooling Module (PPM) used in PSPNet. Args: pool_scales (tuple[int]): Pooling scales used in Pooling Pyramid Module. in_channels (int): Input channels. channels (int): Channels after modules, before conv_seg. Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. cD<V^8dQhRS[S[R3,RS[RS[RR/#)r/ pool_scales.rrrIN)rrn)r1r2s"r*r3%BeitPyramidPoolingModule.__annotate__s3   E#s(O  #  QT  Y]  r)c <\SV`4WnW nW0n\ P !VUu.uFp\WBVR7NK up4VnR#uupi))rrrN) r:r;rrrrrnrblocks)rDrrrrrEs&&&& r*r;!BeitPyramidPoolingModule.__init__sY && mm#. "-J(:aij"-    sAc^<V^8dQhRS[PRS[S[P,/#r )rKrr)r1r2s"r*r3rs,SSU\\Sd5<<6HSr)c~VP4R,pVPUu.uF q3!WR7NK up#uupi)r|)rr)rrr)rDr original_sizeblocks&& r*rR BeitPyramidPoolingModule.forwards6%**,R0 FJkkRkUm8kRRRs:)rrrrrTrWs@@r*rrs&    SSSr)rcfaa]tRtRtoRtV3RlV3RlltV3RlRltV3RlRltR tVt V;t #) BeitUperHeadi"z Unified Perceptual Parsing for Scene Understanding. This head is the implementation of [UPerNet](https://huggingface.co/papers/1807.10221). Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. c$<V^8dQhRS[RR/#r[r)r1r2s"r*r3BeitUperHead.__annotate__*s   z  d  r)c <\SV`4VPVnVP.^,VnVPVn\ P!VP VP^R7Vn \VPVPR,VP 4Vn \VPR,\VP4VP ,,VP ^^R7Vn\ P!4Vn\ P!4VnVPRRFopVP P%\W P ^R74VP"P%\VP VP ^^R74Kq \\VP4VP ,VP ^^R7VnR#)rr8rNrz)r:r;rrBrrrrArrr psp_modulesrlenpsp_bottleneckrn lateral_convs fpn_convsappendfpn_bottleneck)rDr0rrEs&& r*r;BeitUperHead.__init__*s| !--"../!3** ))DMM63D3DRST4      R MM  ,   R 3t'7'7#84==#H H MM   ]]_++CR0K    % %mK\]&^ _ NN ! !- t}}Z[ef"g h1,   !DMM 1 MM  r)c^<V^8dQhRS[S[P,RS[P/#r rrKr)r1r2s"r*r3rLs(11ell);1 1r)cVR,p\P!V.VPV4O^R7pVPV4#)r6rurz)rKrrr)rDrrs&& r* psp_forwardBeitUperHead.psp_forwardLsA$R( yy,!P1A1A,1O!PVWX ""<00r)c^<V^8dQhRS[S[P,RS[P/#r/encoder_hidden_statesrIr)r1r2s"r*r3rQs(T%,,-?ELLr)c .p\VPV4Fwr4VPV!V44K VPVPV44\ V4p\ V^, ^R4FipW&^, ,P R,pW&^, ,\PPW&,VRRR7,W&^, &Kk .p\ V^, 4F1pVPVPV,!W&,44K3 VPVR,4\ V^, ^R4FFp\PPW,V^,P R,RRR7W&KH \P!V^R7pVPV4p VPV 4p V #)r6r|rFrqrurz)ziprrrrrkrNrrrrrKrrr) rDrlaterals lateral_convrused_backbone_levelsru prev_shapefpn_outsoutputs && r*rRBeitUperHead.forwardQs*-d.@.@BW*X &L OOL6 7+Y (()>?@ #8}+a/B7A!a%..r2J&1uo 0I0I *:U1J1HUO8+a/0A OODNN1-hk: ;1  %+a/B7A--33 (1+"3"3B"7jX]4HK899X1-$$X.( r)) rrrrrrrrr) r#r$r%r&r'r;rrRr(rUrVrWs@@r*rr"s.   D11 r)rcXaa]tRtRtoRtRV3RlV3RllltV3RlRltRtVtV;t #) BeitFCNHeadisa Fully Convolution Networks for Semantic Segmentation. This head is implemented of [FCNNet](https://huggingface.co/papers/1411.4038>). Args: config (BeitConfig): Configuration. in_channels kernel_size (int): The kernel size for convs in the head. Default: 3. dilation (int): The dilation rate for convs in the head. Default: 1. Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. c \<V^8dQhRS[RS[RS[RS[S[S[S[3,,RR/#)r/r0in_indexr8rrIN)rrnr)r1r2s"r*r3BeitFCNHead.__annotate__sN!U!U !U,/!UBE!UUX[`adfiai[jUj!U !Ur)c l<\SV`4VPVnVPVnVP VnVPVn W n V^,V,p\P!4Vn VP^8dVPP\VPVP W5VR74\!VP^, 4F@pVPP\VP VP VVVR74KB VP'd@\VPVP ,VP W3^,R7Vn\P$!VP VP&^R7VnR#)r/)r8rrrrN)r:r;rBrauxiliary_channelsrauxiliary_num_convs num_convsauxiliary_concat_input concat_inputr rrnconvsrrrkconv_catrArr)rDr0r r8r conv_paddingrrEs&&&&& r*r;BeitFCNHead.__init__sG !--11 33"99 #q(H4 ]]_ >>A  JJ  $$dmmmu  4>>A-. !!!  $/ ,!) /    )  4==0$--[qrbrDM))DMM63D3DRSTr)c^<V^8dQhRS[S[P,RS[P/#rr)r1r2s"r*r3r s(T%,,-?ELLr)cWP,pTpVPF pV!V4pK VP'd)VP\P !W#.^R74pVP V4pV#)r6ru)r rrrrKrr)rDrrDrrs&& r*rRBeitFCNHead.forwardsh(7 JJD /M     MM%))X4MST*UVM 6 r))rrrrrrr r)r/rr6rTrWs@@r*r r ss& !U!UFr)r cXaa]tRtRtoRtRV3RlV3RllltV3RlRltRtVtV;t #) BeitFPNUpBlockiuE4x upsampling block: ConvTranspose → BN → GELU → ConvTranspose.c0<V^8dQhRS[RS[RS[RR/#)r/rBr8r9rINr)r1r2s"r*r3BeitFPNUpBlock.__annotate__s0ttCtctstSWtr)c<\SV`4\P!WW#R7Vn\P !V4Vn\P!4Vn\P!WW#R7Vn R#)r7N) r:r;rConvTranspose2dconv_transpose1rrGELUrconv_transpose2)rDrBr8r9rEs&&&&r*r;BeitFPNUpBlock.__init__sX !11+Xcs^^K8'')!11+Xcsr)cN<V^8dQhRS[PRS[P/#r rJ)r1r2s"r*r3rr r)cVPV4pVPV4pVPV4pVPV4pV#r)r!rrr#rs&&r*rRBeitFPNUpBlock.forwardsF,,]; **=9  6 ,,]; r))rr!r#r)r/r/rTrWs@@r*rrs#Ottr)rcTaa]tRtRtoRtV3RlV3RlltV3RlRltRtVtV;t #) BeitFPNNeckiz 4-level feature pyramid neck for BeiT. Produces x4 upsample, x2 upsample, identity, and x2 downsample outputs from the four selected ViT feature maps. c <V^8dQhRS[/#r.r)r1r2s"r*r3BeitFPNNeck.__annotate__s::z:r)c<\SV`4\VP4Vn\ P !VPVP^^R7Vn\ P!^^R7Vn R#)r/r7N) r:r;rrBfpn1rr fpn2 MaxPool2dfpn4rs&&r*r;BeitFPNNeck.__init__sX "6#5#56 &&v'9'96;M;M[\efg LLQq9 r)cv<V^8dQhRS[S[PR3,RS[S[PR3,/#)r/ feature_maps.rI)rrKr)r1r2s"r*r3r+s8  E%,,*;$< u||UXGXAY r)cVPV^,4VPV^,4V^,VPV^,43#)rr-r.r0)rDr3s&&r*rRBeitFPNNeck.forwardsC IIl1o & IIl1o & O IIl1o &   r)r5rTrWs@@r*r)r)s# ::    r)r)craa]tRtRtoV3RlV3Rllt]]]RV3RlRll444tRt Vt V;t #)BeitForSemanticSegmentationic$<V^8dQhRS[RR/#r[r)r1r2s"r*r3(BeitForSemanticSegmentation.__annotate__szdr)cr<\SV`V4VPVn\VRR7Vn\ VP P4^8wd \R4h\V4Vn \V4Vn VP'd \V4MRVnVP!4R#)FrzBeitForSemanticSegmentation requires config.out_indices to be a list of 4 integers, specifying which features to use from the backbone. One can use [3, 5, 7, 11] in case of a base-sized architecture.N)r:r;rrdrIrr0 out_indicesrOr)fpnr decode_headuse_auxiliary_headr auxiliary_headrtrs&&r*r;$BeitForSemanticSegmentation.__init__s   ++f> t{{&& '1 ,-  v&(/5;5N5N5Nk&1TX r)c <V^8dQhRS[PR,RS[PR,RS[RS[S[,RS[S[,/#r)rKrrrrrr)r1r2s"r*r3r:sbG G llT)G  t#G #' G +, G ( ( G r)c Da aaaVe'VPP^8Xd \R4hVP!V3RV/VBpVPoVP wo rgpWpPP ,oWPP ,o\;QJd..V VVV3RlVPP4FNK 5M'!V VVV3RlVPP44p VPV 4p VPV 4p Rp VPeVPV 4p Rp Ve?VPV VVPPV VPPR7p \V V VPVP R7#)a labels (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*): Ground truth semantic segmentation maps for computing the loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1`, a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, BeitForSemanticSegmentation >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("microsoft/beit-base-finetuned-ade-640-640") >>> model = BeitForSemanticSegmentation.from_pretrained("microsoft/beit-base-finetuned-ade-640-640") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> # logits are of shape (batch_size, num_labels, height, width) >>> logits = outputs.logits ```Nz/The number of labels should be greater than onerc3<"TF>pSV^, ,R,P^^4PSRSS4xK@ R#5i)r6Nrxrz)rQr}).0rurr patch_height patch_widths& r* 6BeitForSemanticSegmentation.forward..sJ , "!a% ( / 9 9!Q ? G G TVXdfq r r,sAA ) ignore_indexauxiliary_logitsauxiliary_loss_weightr)r0rrOrIrrNr=rr<r=r>r@rsemantic_loss_ignore_indexrLrrJ)rDrHrrrrrrlrmr3rrKrrrrFrGs&&&&, @@@@r*rR#BeitForSemanticSegmentation.forwards}B  $++"8"8A"=NO O))  %=   !( 5 5'3'9'9$ Au!7!77 {{555 u [[,, uu [[,,  xx - !!,/    *#22<@   %%![[CC!1&*kk&G&G &D'!//))   r))r@rIr>r=rr r#r$r%r&r;rr rrRr(rUrVrWs@@r*r8r8sA* G G !G G r)r8zM BEiT backbone, to be used with frameworks like DETR and MaskFormer. cbaa]tRtRtoV3Rlt]]]V3RlRl444tRt Vt V;t #) BeitBackbonei;cV<\SV`V4\VP^,4Uu.uFq!PNK upVn\ VRR7VnVP'd \V4M\P!4Vn VP4R#uupi)r6FrN)r:r;rkrlrB num_featuresrdrIadd_fpnr)rr8r=rt)rDr0rrEs&& r*r;BeitBackbone.__init__Asz  9>v?W?WZ[?[9\]9\A//9\]f> *0...;v&bkkm  ^sB&c<<V^8dQhRS[RS[S[,RS[/#)r/rHrrI)rrrr)r1r2s"r*r3BeitBackbone.__annotate__Ns/3 3 3 +,3   3 r)c @VPwr4rVWPPP,pW`PP,pVP!V3/VBp V Pp Rp \ VP V 4FjwrWP9gKVPP'd/V R,p V P^^4p V PVRWx4p W3, p Kl VPV 4p \V V PV PR7#)a Examples: ```python >>> from transformers import AutoImageProcessor, AutoBackbone >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> processor = AutoImageProcessor.from_pretrained("microsoft/beit-base-patch16-224") >>> model = AutoBackbone.from_pretrained( ... "microsoft/beit-base-patch16-224", out_features=["stage1", "stage2", "stage3", "stage4"] ... ) >>> inputs = processor(image, return_tensors="pt") >>> outputs = model(**inputs) >>> feature_maps = outputs.feature_maps >>> list(feature_maps[-1].shape) [1, 768, 14, 14] ```)r3rrJr"rrz)rNr0r=rIrr stage_names out_featuresreshape_hidden_statesrQr}r=rrJ)rDrHrrrrlrmrFrGrrr3stagers&&, r*rRBeitBackbone.forwardKs@(4'9'9$ v!7!77 {{555 ))L3F3--  #&t'7'7#G E)));;444#/#9L#/#9#9!Q#?L#/#7#7 B #bL/ $Hxx - %!//))  r))rIr=rSrOrWs@@r*rQrQ;s3  3 !3 3 r)rQ)Nr)rrr8rdrHrQ)Ecollections.abcrr dataclassesrrKrrrrM activationsr backbone_utilsr r masking_utilsr modeling_layersr modeling_outputsrrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsrrr utils.genericrrutils.output_capturingrconfiguration_beitrr!rr,rYrrrrrr*rHrdrrrrrrrrr rr)r8rQ__all__r"r)r*rnsb*/! &!H69G&@BBI5*  !;  H"))H