+ j<^RIHt^RIt^RIHt^RIHt^RIHt ^RI H t ^RI H t Ht^RIHtHt^RIHt^R IHtHtHt^R IHtHtHtHtHtHt^R IH t ^R I!H"t"H#t#]PH!]%4t&]!R R7]!RR] 444t'!RR]4t(!RR]"4t)!RR]4t*!RR]4t+!RR]4t,!RR]4t-]!RR]44t.!RR ]4t/.R!Ot0R#)")CallableN)strict)initialization)PreTrainedConfig)BaseModelOutputBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging)CLIPMLP CLIPAttention CLIPEncoderCLIPEncoderLayerCLIPVisionEmbeddingsCLIPVisionModel)eager_attention_forward)Qwen2_5_VLVisionRotaryEmbeddingapply_rotary_pos_emb_visionz&DeepGlint-AI/mlcd-vit-bigG-patch14-336) checkpointcta]tRt^)toRtRtRtRtRtRt ^0t ^t ^t ^t Rt^tRtR tR tR tR tR tV3RltRtVtR #)MLCDVisionConfigaF num_key_value_groups (`int`, *optional*, defaults to 1): Number of key-value groups used in Attention. Example: ```python >>> from transformers import MLCDVisionConfig, MLCDVisionModel >>> # Initializing a MLCDVisionConfig with DeepGlint-AI/mlcd-vit-bigG-patch14-336 style configuration >>> configuration = MLCDVisionConfig() >>> # Initializing a MLCDVisionModel (with random weights) from the DeepGlint-AI/mlcd-vit-bigG-patch14-336 style configuration >>> model = MLCDVisionModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```mlcd_vision_model vision_configaxialii iPgelugh㈵>g{Gz?g?Ncl<V^8dQh/S[;R&S[;R&S[;R&S[;R&S[;R&S[;R&S[S[S[,,S[S[S[3,,;R&S[S[S[,,S[S[S[3,,;R&S[;R &S[;R &S[S[,;R &S[;R &S[;R &S[R,;R&#) hidden_sizeintermediate_sizenum_hidden_layersnum_attention_headsnum_key_value_groups num_channels image_size patch_size hidden_actlayer_norm_epsattention_dropoutinitializer_rangeinitializer_factorNrope_parameters)intlisttuplestrfloatdict)format __classdict__s"f/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/mlcd/modular_mlcd.py __annotate__MLCDVisionConfig.__annotate__)s456!789:!;<!=>?@d3i%S/17ABd3i%S/16CDEF GHs{(IJ#KL#MND['O)__name__ __module__ __qualname____firstlineno____doc__ model_typebase_config_keydefault_rope_typer"r#r$r%r&r'r(r)r*r+r,r-r.r/__annotate_func____static_attributes____classdictcell__)r7s@r8rr)sq &%J%OK!! !L47J46JJ N%(# ##'OOr;rc]tRt^StRtR#)MLCDMLPr<Nr=r>r?r@rFr<r;r8rIrISr;rIc]tRt^WtRtR#)MLCDRotaryEmbeddingr<NrJr<r;r8rMrMWrKr;rMcPaa]tRt^[toV3RlV3RlltV3RlRltRtVtV;t#)MLCDVisionEmbeddingsc <V^8dQhRS[/#r!configr)r6r7s"r8r9!MLCDVisionEmbeddings.__annotate__\s$$/$r;c*<\SV`V4V=R#N)super__init__position_embeddingselfrR __class__s&&r8rXMLCDVisionEmbeddings.__init__\s   #r;cN<V^8dQhRS[PRS[P/#)r! pixel_valuesreturn)torch FloatTensorTensor)r6r7s"r8r9rT`s%  E$5$5 %,, r;c\VP^,pVPPPpVPVP VR74pVP ^4P ^^4pVPPV^R4p\P!WT.^R7pV#)r)dtypedim) shapepatch_embeddingweightretoflatten transposeclass_embeddingexpandracat)r[r_ batch_size target_dtype patch_embeds class_embeds embeddingss&& r8forwardMLCDVisionEmbeddings.forward`s!''* ++2288 ++LOO,O,OP #++A.88A> ++22:q"E YY ;C r;r< r=r>r?r@rXrwrFrG __classcell__r\r7s@@r8rOrO[s$$   r;rOcXaa]tRt^mtoRtV3RlV3RlltRV3RlRlltRtVtV;t #) MLCDAttentionzMulti-headed attention with RoPE. Refer to papers: - Attention is all you need: https://huggingface.co/papers/1706.03762 - RoFormer: Enhanced Transformer with Rotary Position Embedding: https://huggingface.co/papers/2104.09864 c <V^8dQhRS[/#rQrS)r6r7s"r8r9MLCDAttention.__annotate__us/r;cV<\SV`V4VPVnRVnR#)FN)rWrXr& is_causalrZs&&r8rXMLCDAttention.__init__us%  $*$?$?!r;c <V^8dQhRS[PRS[S[PS[P3,RS[PR,RS[S[,RS[S[PS[PR,3,/#r! hidden_statesposition_embeddingsattention_maskNkwargsr`)rarcr2r r )r6r7s"r8r9rzsy,),)||,)#5<<#=>,) t+ ,) +, ,) u||U\\D00 1 ,)r;c VPRRwrVVPV4PWVVPVP34pVP V4PWVVPVP34pVP V4PWVVPVP34p V^,P^4P4p V^,P^4P4p \WxW4wrxVP^^^^4P4pVP^^^^4P4pV P^^^^4P4p \P!VPP\ 4p T !VVVV V3RVP"'gRM VP$RVP&RVP(/VBwrV P^^^^4P4p V P+WeR4p VP-V 4p V P^^^4P4p W3#)Ndropoutrscalingrrh)riq_projreshape num_headshead_dimk_projv_proj unsqueezer4rpermute contiguousr get_interfacerR_attn_implementationrtrainingrscalerviewout_proj)r[rrrrrr seq_length query_states key_states value_statescossinattention_interface attn_output attn_weightss&&&&, r8rwMLCDAttention.forwardzs6"/!4!4Sb!9 {{=199:SWSaSacgcpcp:qr [[/77QUQ_Q_aeanan8op {{=199:SWSaSacgcpcp:qr "!$..q1779!!$..q1779#>|Y\#b $++Aq!Q7BBD ''1a3>>@ #++Aq!Q7BBD (?(M(M KK , ,.E) %8      % $}}}C$,, % JJ % nn %  % ! "))!Q15@@B !&&zrB mmK0 !))!Q2==? ((r;)rr&rV r=r>r?r@rArXrwrFrGrzr{s@@r8r}r}ms# ,),),)r;r}cTaa]tRt^toV3RlV3RlltRV3RlRlltRtVtV;t#)MLCDEncoderLayerc <V^8dQhRS[/#rQrS)r6r7s"r8r9MLCDEncoderLayer.__annotate__s////r;cF<\SV`V4\V4VnR#rV)rWrXr} self_attnrZs&&r8rXMLCDEncoderLayer.__init__s  &v.r;c <V^8dQhRS[PRS[S[PS[P3,RS[PR,RS[S[,RS[S[P ,/#r)rarcr2r r rb)r6r7s"r8r9rsm""||"#5<<#=>" t+ " +, " u  ! "r;c TpVPV4pVP!RRVRVRV/VBwrWQ,pTpVPV4pVPV4pWQ,pV#)ah Args: hidden_states (`torch.FloatTensor`): Input to the layer of shape `(batch, seq_len, embed_dim)`. Represents the hidden states from the previous layer or the input embeddings. position_embeddings (`tuple[torch.Tensor, torch.Tensor]`): A tuple of two tensors, each of shape `(batch, seq_len, embed_dim)`. Represents absolute positional embeddings for the query and key in the attention mechanism. attention_mask (`torch.FloatTensor`): Attention mask of shape `(batch, 1, q_len, k_v_seq_len)` where padding elements are indicated by very large negative values. rrrr<) layer_norm1r layer_norm2mlp)r[rrrrresidual_s&&&&, r8rwMLCDEncoderLayer.forwards$!((7 >> ' 3 *    !0  ((7 /  0 r;)rrVryr{s@@r8rrs//"""r;rcXaa]tRt^toRtV3RlV3RlltRV3RlRlltRtVtV;t #) MLCDEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`MLCDEncoderLayer`]. Args: config: MLCDVisionConfig c <V^8dQhRS[/#rQrS)r6r7s"r8r9MLCDEncoder.__annotate__s!!/!r;c &<\SV`V4R#)z3Overwrite dummy `MLCDConfig` to `MLCDVisionConfig`.N)rWrXrZs&&r8rXMLCDEncoder.__init__s  r;c <V^8dQhRS[PRS[S[PS[P3,RS[PR,RS[S[,RS[S[,/#)r! inputs_embedsrrNrr`)rarbr2rcr r r)r6r7s"r8r9rsi! ! ((! #5<<#=>!  t+ ! +, !  ! r;c ZTpVPFpV!VVV3/VBpK \VR7#)a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. position_embeddings (`tuple[torch.Tensor, torch.Tensor]`): A tuple of two tensors, each of shape `(batch, seq_len, embed_dim)`. Represents absolute positional embeddings for the query and key in the attention mechanism. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) )last_hidden_state)layersr)r[rrrrr encoder_layers&&&&, r8rwMLCDEncoder.forwardsF,& ![[M)# M)+  r;r<rVrr{s@@r8rrs#!!! ! ! r;rcaa]tRtRtoRtR.tRtRtRtRt Rt Rt R] R] /t]P !4V3Rl4tV3R ltR tVtV;t#) MLCDPreTrainedModeli vision_modelrTFr attentionsc <\SV`V4VPPp\ V\ 4'dVPPp\ P!VPRVPR,V,R7\ P!VPPVPPV,R7\ P!VP\P !VPP"R,4P%R44R#\ V\&4'Ed&VPPpVPR,^VPP(,R,,V,pVPR,V,p\ P!VP*PVR7\ P!VP,PVR7\ P!VP.PVR7\ P!VP0PVR7R#\ V\24'dVPPpVPP4R,^VPP(,R,,V,p^VPP4,R,V,p\ P!VP6PVR7\ P!VP8PVR7R#\ V\:4'dVPPpVPP4VPP<,^,R,V,p\ P!VP>RVR7R#R#)zInitialize the weightsr)meanstd)rNgrh)rh) rW _init_weightsrRr. isinstancerOinitnormal_ro embed_dimrjrkr-copy_ position_idsraarangerirpr}r$rrrrrIr"fc1fc2MLCDVisionModelr% class_pos_emb)r[modulefactor in_proj_std out_proj_stdfc_std pos_emb_stdr\s&& r8r!MLCDPreTrainedModel._init_weightss f%// f2 3 3[[33F LL//cv?O?OQU?UX^?^ _ LL//66FMM<[<[^dr?r@base_model_prefix_no_split_modulessupports_gradient_checkpointingaccepts_loss_kwargs_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrr}_can_record_outputsrano_gradrrErFrGrzr{s@@r8rrso'+,&*#N"&)m  ]]_JJ#r;rcTaa]tRtRtoV3RlV3RlltRV3RlRlltRtVtV;t#)ri2c <V^8dQhRS[/#rQrS)r6r7s"r8r9MLCDVisionModel.__annotate__3sqq/qr;c<\SV`V4\V4Vn\P !\ P!^VPVP,^,44Vn R#)rN) rWrXrMvision_rotary_embeddingnn Parameterrarandnr"r%rrZs&&r8rXMLCDVisionModel.__init__3sP  ':6'B$\\%++a9K9KvOiOi9imn9n*opr;cn<V^8dQhRS[PR,RS[S[,RS[S[,/#)r!r_Nrr`)rarbr r r2r)r6r7s"r8r9r8sE@ @ ''$.@ +,@  + + @ r;c NVf \R4hVPV4pVPV4pVPR,VPP ,pVPR,VPP ,p\ P!WAPR7P^4PRV4p\ P!WQPR7P^4PVR4p\ P!VP4VP4.RR7pVPW84p \ P!VPVP.RR7p \ P!V P!4V ^,.^R7\ P!V P#4V ^,.^R7.p VP$!R RVRV /VBp V ^,p V R ,p VP'V 4p \)V V R7#) ac Example: ```python >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> from transformers import AutoProcessor, MLCDVisionModel >>> model = MLCDVisionModel.from_pretrained("DeepGlint-AI/mlcd-vit-bigG-patch14-448") >>> processor = AutoProcessor.from_pretrained("DeepGlint-AI/mlcd-vit-bigG-patch14-448") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs, output_attentions=True) >>> features = outputs.last_hidden_state >>> print(f"Extracted features shape: {features.shape}") >>> print(f"Number of attention layers: {len(outputs.attentions)}") >>> print(f"Attention shape: {outputs.attentions[0].shape}") ```z You have to specify pixel_values)devicerfrr)r pooler_outputrhr<)NNNrr) ValueErrorrv pre_layrnormrirRr)rarrrrpstackrmrrqrrrencoderpost_layernormr)r[r_rrnum_patches_heightnum_patches_widthhpos_idswpos_idspos_idsrrencoder_outputsr pooled_outputs&&, r8rwMLCDVisionModel.forward8s:  ?@ @ 5 ))-8 )//3t{{7M7MM(..r2dkk6L6LL LL+4G4G H R RST U \ \]_ar s  LL*3F3F G Q QRS T [ [\npr s ++x//183C3C3EFBO"::=R 4#5#5t7I7I"JPRS II}((*,?,BC K II}((*,?,BC K  ,, ' 3   ,A.)'2 ++M: )/'  r;)rrrVryr{s@@r8rr2s#qq @ @ @ r;r)rrr)1collections.abcrratorch.nnrhuggingface_hub.dataclassesrrrconfiguration_utilsrmodeling_outputsrrmodeling_utilsr r processing_utilsr utilsr r rclip.modeling_cliprrrrrrllama.modeling_llamarqwen2_5_vl.modeling_qwen2_5_vlrr get_loggerr=loggerrrIrMrOr}rrrr__all__r<r;r8rs% .&3KF&@@;i   H %CD%('%(E%(P g  9 /$9)M9)x'''T. +. b*J/*J*JZF oF R r;