+ j[Rt^RIt^RIt^RIHt^RIHt^RIHt^RIt^RIH t ^RI H t ^RI Ht^RIHt^R IHt^R IHtHt^R IHt^R IHt^R IHtHtHtHtHtH t ^RI!H"t"H#t#H$t$H%t%H&t&]PN!](4t)Rt*^t+Rt,]&]$,]%,t-]!RR7]!RR]444t.]!RR7]!RR]444t/]!RR7]!RR]444t0!RR] Pb4t2!RR] Pb4t3!RR ] Pb4t4!R!R"] Pb4t5!R#R$] Pb4t6!R%R&] Pb4t7!R'R(] Pb4t8!R)R*] Pb4t9!R+R,]4t:!R-R.] Pb4t;!R/R0] Pb4t<]!R1R2]44t=]!R3R4]=44t>]!R5R6]=44t?]!R7R8]=44t@]!R9R:]=44tA!R;R<] Pb4tB!R=R>] Pb4tC!R?R@] Pb4tD]!RAR7!RBRC]=44tE!RDRE] Pb4tF!RFRG] Pb4tG!RHRI] Pb4tH!RJRK] Pb4tI]!RLR7!RMRN]=44tJ.ROOtKR#)PzPyTorch FLAVA model.N) OrderedDict) dataclass)Any)nn)initialization)ACT2FN)create_bidirectional_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPooling)PreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging torch_int) FlavaConfigFlavaImageCodebookConfigFlavaImageConfigFlavaMultimodalConfigFlavaTextConfigzfacebook/flava-image-codebookg$(~k@a Output from FlavaModel containing embeddings and outputs from individual encoders. Note that `image_embeddings` and `text_embeddings` returned are similar to pooled output returned from a transformer. If you want embeddings for contrastive loss or retrieval use a FLAVA model's `image_projection` and `text_projection` layers on `image_embeddings` and `text_embeddings` respectively. ) custom_introcZa]tRt^4toRtRtRtRtRtRt Rt V3RlRlt V3Rlt Rt VtR#)FlavaModelOutputa image_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `pixel_values` are present): The image embeddings which are basically the pooled output of [`FlavaImageModel`]. image_output (`BaseModelOutputWithPooling`, *optional*, returned when `pixel_values` are present): The output of the [`FlavaImageModel`]. text_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` are present): The text embeddings which are basically the pooled output of [`FlavaTextModel`]. text_output (`BaseModelOutputWithPooling`, *optional*, returned when `input_ids` are present): The output of the [`FlavaTextModel`]. multimodal_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` and `pixel_values` are present and `skip_multimodal_encoder` is `None` or `False`): The multimodal embeddings which are basically the pooled output of [`FlavaTextModel`]. multimodal_output (`BaseModelOutputWithPooling`, returned when `input_ids` and `pixel_values` are present and `skip_multimodal_encoder` is `None` or `False`): The output of the [`FlavaMultimodalModel`]. Nc0<V^8dQhRS[S[,/#returntupler)format __classdict__s"i/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/flava/modeling_flava.py __annotate__FlavaModelOutput.__annotate__Us  %* ca\;QJd%.V3RlSP44FNK 5#!V3RlSP444#)c3x<"TF/pVR9d SV,M\SV4P4xK1 R#5i) text_outputN)r* image_outputmultimodal_outputgetattrto_tuple).0kselfs& r$ ,FlavaModelOutput.to_tuple..Vs<   TTDGZabfhiZjZsZsZu u s7:r!keysr2sfr$r/FlavaModelOutput.to_tupleUsAu YY[ u u YY[   r'c<V^8dQh/S[PR,;R&S[R,;R&S[PR,;R&S[R,;R&S[PR,;R&S[R,;R&#)rNimage_embeddingsr+text_embeddingsr*multimodal_embeddingsr,)torch FloatTensorr )r"r#s"r$r%r&4s4''$.556-t3:78&&-49:,d29;<!,,t3:=>2D8??r')__name__ __module__ __qualname____firstlineno____doc__r:r+r;r*r<r,r/__annotate_func____static_attributes____classdictcell__r#s@r$rr4s? 266:L04O59K6:;?  Cr'rz@ Class representing pretraining losses from FLAVA model cZa]tRt^\toRtRtRtRtRtRt Rt V3RlRlt V3Rlt Rt VtR#) FlavaLossesas mim (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mim_labels` and `pixel_values` are present, `input_ids_masked` is absent and `mim_weight` > 0.): Masked Image Modeling loss as used in BeIT calculated only for unimodal image data. mlm (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mlm_labels` and `input_ids_masked` are present, `pixel_values` is absent and `mlm_weight` > 0.): Masked Language Modeling loss as used in BERT calculated only for unimodal text data. itm (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `itm_labels`, `input_ids_masked`, `pixel_values` are present and `itm_weight` > 0.): Image Text Matching (ITM) loss calculated for paired image-text data. Note that ITM loss is calculated on masked pairs in FLAVA. global_contrastive (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `input_ids` and `pixel_values` are present and `global_contrastive_weight` > 0.): Contrastive loss for image-text similarity similar to CLIP but calculated globally for paired image-text data. This is calculated on unmasked images and texts. mmm_image (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mim_labels`, `pixel_values` and `input_ids_masked` are present and `mmm_image_weight` > 0.): Masked Multimodal Modeling loss's image component calculated on paired image-text data. mmm_text (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mlm_labels`, `pixel_values` and `input_ids_masked` are present and `mmm_text_weight` > 0.): Masked Multimodal Modeling loss's text component calculated on paired image-text data. Nc <V^8dQhRS[/#r)bool)r"r#s"r$r%FlavaLosses.__annotate__{s$r'cJRpVP4F pVfK RpV# V#)TF)values)r2all_nonevs& r$rPFlavaLosses.all_none{s0A}  r'c.<V^8dQh/S[PR,;R&S[PR,;R&S[PR,;R&S[PR,;R&S[PR,;R&S[PR,;R&#)rNmimmlmitmglobal_contrastive mmm_imagemmm_text)r=r>)r"r#s"r$r%rM\s0   T !(12   T !(34   T !(56))D0778  4'.9:$&-;r'r?)r@rArBrCrDrTrUrVrWrXrYrPrErFrGrHs@r$rJrJ\s<"%)C$(C$(C37*.I)-H?r'rJa Output from FlavaForPreTraining containing embeddings, and outputs from individual encoders. Note that `image_embeddings` and `text_embeddings` returned are similar to pooled output returned from a transformer. If you want embeddings for contrastive loss or retrieval use a FLAVA model's `image_projection` and `text_projection` layers on `image_embeddings` and `text_embeddings` respectively. ca]tRt^toRtRtRtRtRtRt Rt Rt Rt Rt RtRtRtRtRtRtRtRtRtRtRtRtV3RlRltV3RltRtVtR#)FlavaForPreTrainingOutputa loss (`torch.FloatTensor`, *optional*, returned when `return_loss` is True): Total loss calculated for this model. loss_info (`FlavaLosses`): Detailed info for FLAVA Pretraining losses. Check `FlavaLosses` class description for the information on the keys. image_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `pixel_values` are present): The image embeddings which are basically the pooled output of [`FlavaImageModel`]. image_output (`BaseModelOutputWithPooling`, *optional*, returned when `pixel_values` are present): The output of the [`FlavaImageModel`]. text_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` are present): The text embeddings which are basically the pooled output of [`FlavaTextModel`]. text_output (`BaseModelOutputWithPooling`, *optional*, returned when `input_ids` are present): The output of the [`FlavaTextModel`]. multimodal_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` and `pixel_values` are present and `skip_unmasked_multimodal_encoder` is `None` or `False`): The multimodal embeddings which are basically the pooled output of [`FlavaTextModel`]. multimodal_output (`BaseModelOutputWithPooling`, returned when `input_ids` and `pixel_values` are present and `skip_unmasked_multimodal_encoder` is `None` or `False`): The output of the [`FlavaMultimodalModel`]. image_masked_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `pixel_values` are present): The image embeddings which are basically the pooled output of [`FlavaImageModel`]. Uses `bool_masked_pos` to create masked images. image_masked_output (`BaseModelOutputWithPooling`, *optional*, returned when `pixel_values` are present): The output of the [`FlavaImageModel`]. Uses `bool_masked_pos` to create masked images. text_masked_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids_masked` are present): The text embeddings which are basically the pooled output of [`FlavaTextModel`]. text_masked_output (`BaseModelOutputWithPooling`, *optional*, returned when `input_ids_masked` are present): The output of the [`FlavaTextModel`]. multimodal_masked_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` and `pixel_values` are present): The multimodal embeddings which are basically the pooled output of [`FlavaTextModel`]. multimodal_masked_output (`BaseModelOutputWithPooling`, *optional*, returned when `input_ids_masked` and `pixel_values` are present): The output of the [`FlavaMultimodalModel`]. mim_logits (`torch.FloatTensor` of shape `(batch_size, num_image_patches, image_vocab_size)` or of shape `(total_masked_patches, image_vocab_size)` , *optional*, returned when `pixel_values` are present and `input_ids_masked` are not): The logits for MIM unimodal loss. Uses `bool_masked_pos` to get masked patches. The flattened output is returned when `bool_masked_pos` has some of the patches masked. mlm_logits (`torch.FloatTensor` of shape `(batch_size, text_seq_length, text_vocab_size)` or of shape `(total_masked_seq_length, text_vocab_size)`, *optional*, returned when `input_ids_masked` are present and `pixel_values` are not): The logits for MLM unimodal loss. The flattened output is returned when `input_ids_masked` has some of the tokens masked. itm_logits (`torch.FloatTensor` of shape `(batch_size, 2)`, *optional*, returned when `input_ids_masked` and `pixel_values` are present): The logits for ITM loss. Note that ITM loss is calculated on masked pairs in FLAVA. contrastive_logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeddings` and `text_embeddings` but passed through FLAVA's `image_projection` and `text_projection` layers respectively. This represents the image-text similarity scores. This is calculated on unmasked images and texts. contrastive_logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeddings` and `image_embeddings` but passed through FLAVA's `text_projection` and `image_projection` layers respectively. This is calculated on unmasked images and texts. mmm_image_logits (`torch.FloatTensor` of shape `(batch_size, num_image_patches, image_vocab_size)` or of shape`(total_masked_patches, image_vocab_size)`, *optional*, returned when `pixel_values` and `input_ids_masked` are present): The logits for MMM image multimodal loss. Uses `bool_masked_pos` to get masked patches. The flattened output is returned when `bool_masked_pos` has some of the patches masked. mmm_text_logits (`torch.FloatTensor` of shape `(batch_size, text_seq_length, text_vocab_size)` or of shape `(total_masked_seq_length, text_vocab_size)`, *optional*, returned when `pixel_values` and `input_ids_masked` are present): The logits for MMM text multimodal loss. The flattened output is returned when `input_ids_masked` has some of the tokens masked. Nc0<V^8dQhRS[S[,/#rr )r"r#s"r$r%&FlavaForPreTrainingOutput.__annotate__s t t%* tr'caa.ROo\;QJd&.VV3RlSP44FNK 5#!VV3RlSP444#)r*c3v<"TF.qS9d SV,M\SV4P4xK0 R#5iNr-)r0r1r2transformer_outputss& r$r35FlavaForPreTrainingOutput.to_tuple..s4sgrbc)< rJr )r"r#s"r$r%r]sD   d ")EF!GH''$.5IJ-t3:KL&&-4MN,d29OP!,,t3:QR2D8?ST#..5 YY 7Q?  $#&C&CJX]&^^J#&>&>>J\\*- r')rrwrrrr~rFNF) r@rArBrCrDrrrrFrG __classcell__rr#s@@r$rurus0&&D&DPr'ruc\aa]tRtRtoRtRV3RlV3RllltR V3RlRlltRtVtV;t #) riNz Image to Patch Embedding. c<V^8dQhRS[S[S[,,S[S[S[3,,RS[S[S[S[3,,RS[RS[/#)rr}r~rr)rlistr!)r"r#s"r$r%PatchEmbeddings.__annotate__Ss[hh$s)OeCHo5h%S/)h h  hr'c<\SV`4\V\PP 4'gW3p\V\PP 4'gW"3pV^,V^,,V^,V^,,,pWnW nWPn\P!W4W"R7Vn R#)r|) kernel_sizestrideN) rr isinstance collectionsabcIterabler}r~rrConv2d projection)r2r}r~rrrrs&&&&& r$rPatchEmbeddings.__init__Ss *koo&>&>??$1J*koo&>&>??$1J!!} 1 5*Q-:VW=:XY $$&))Lgr'cT<V^8dQhRS[PRS[RS[P/#)rrrr)r=rrL)r"r#s"r$r%rfs-  ELL D ]b]i]i r'c dVPwr4rVV'giWPP^,8wgW`P^,8wd;\RV RV RVP^, RVP^, R2 4hVPV4P ^4P ^^4pV#)zInput image size (*z) doesn't match model (z).)rr} ValueErrorrflatten transpose)r2rrrrrrxs&&& r$rPatchEmbeddings.forwardfs2>2D2D/ &'++u8J/J (%9+,Adooa.@-AE OOL ) 1 1! 4 > >q! Dr')r}rr~r)ir r@rArBrCrDrrrFrGrrs@@r$rrNs%hh&   r'rcLaa]tRtRtoRtV3RltRV3RlRlltRtVtV;t #)FlavaTextEmbeddingsirzGConstruct the embeddings from word, position and token_type embeddings.cN<\SV`4\P!VPVP VP R7Vn\P!VPVP 4Vn \P!VPVP 4Vn \P!VP VPR7Vn \P!VP4Vn\P"!\$P&!VP4P)R4RR7Vn\P"!\$P,!VP*P/4\$P0R7RR7VnR#)) padding_idxepsF) persistent)dtypeNr|r)rrr Embedding vocab_sizer pad_token_idword_embeddingsmax_position_embeddingsrtype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsrrrBufferr=aranger position_idsrrlongtoken_type_idsr2rwrs&&r$rFlavaTextEmbeddings.__init__us  !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<= IIell63Q3Q&R&Y&YZa&botu ii D4E4E4J4J4LTYT^T^(_lqrr'c<V^8dQhRS[PR,RS[PR,RS[PR,/#)r input_idsNrrr=r)r"r#s"r$r% FlavaTextEmbeddings.__annotate__sD  <<$&  t+ llT) r'c@VP4pV^,pVfVPRRV13,pVf\VR4'd4VPRRV13,pVP V^,V4pTpM;\ P !V\ PVPPR7pVPV4pVPV4p W,p VPV4p W, p VPV 4p VPV 4p V #)r|Nrr)rdevice)rrhasattrrrr=rrrrrrrr) r2rrr input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expanded inputs_embedsrrrs &&&& r$rFlavaTextEmbeddings.forwards   nn&  ^  ,,Q  ^ J": "66|D) ^^J/ \\*- r')rrrrrrrNNNrrs@@r$rrrsQ s   r'rcTaa]tRtRtoV3RlV3RlltRV3RlRlltRtVtV;t#)FlavaSelfAttentionic$<V^8dQhRS[RR/#rrwrNFlavaPossibleConfigs)r"r#s"r$r%FlavaSelfAttention.__annotate__s GG3GGr'cF<\SV`4VPVP,^8wd9\ VR4'g'\ RVP RVP R24hVPVn\ VPVP, 4VnVPVP,Vn\P!VPVPVPR7Vn \P!VPVPVPR7Vn \P!VPVPVPR7Vn\P!VP 4VnR#)rembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .biasN)rrrnum_attention_headsrrrattention_head_size all_head_sizerLinearqkv_biasquerykeyvaluerattention_probs_dropout_probrrs&&r$rFlavaSelfAttention.__init__s1     : : :a ?PVXhHiHi"6#5#5"67334A7  $*#=#= #&v'9'9FHHAN jj/44\BLLQPQR !<< 5H5HR5PQ+dii8P8P.QQ  %/@ --//0@b/I,,7 _B %--aAq9DDF "/"4"4"6s";t?Q?Q>S"S%**,CD 6G=2O\M]r')rrrrrrrr r@rArBrCrrrFrGrrs@@r$r r s GG$###r'r cTaa]tRtRtoRtV3RlV3RlltV3RlRltRtVtV;t #)FlavaSelfOutputiz The residual connection is defined in FlavaLayer (same as ViTLayer) instead of here (as is the case with other models), due to the layernorm applied before each block. c$<V^8dQhRS[RR/#r r )r"r#s"r$r%FlavaSelfOutput.__annotate__>>3>>r'c<\SV`4\P!VPVP4Vn\P !VP4VnR#r`) rrrrrdenserrrrs&&r$rFlavaSelfOutput.__init__sB YYv1163E3EF zz&"<"<= r'ch<V^8dQhRS[PRS[PRS[P/#rr  input_tensorrr)r"r#s"r$r%r9s1U\\RWR^R^r'cJVPV4pVPV4pV#r`r<rr2r r@s&&&r$rFlavaSelfOutput.forwards$ =1  ]3 r'rBrrs@@r$r7r7s# >> r'r7cTaa]tRtRtoV3RlV3RlltRV3RlRlltRtVtV;t#)FlavaAttentionic$<V^8dQhRS[RR/#r r )r"r#s"r$r%FlavaAttention.__annotate__s..3..r'cd<\SV`4\V4Vn\ V4VnR#r`)rrr  attentionr7outputrs&&r$rFlavaAttention.__init__s& +F3%f- r'c <V^8dQhRS[PRS[PR,RS[RS[S[PS[P3,S[S[P,,/#rr#)r"r#s"r$r%rHs`  ||  t+   u||U\\) *U5<<-@ @ r'cVPWVR7pVPV^,V4pV3VR,,pV#)r!r"rrJrK)r2r r!r" self_outputsattention_outputr3s&&&& r$rFlavaAttention.forwardsK ~~ L]&  ;;|A F#% R(88r'rPrr5rs@@r$rFrFs..   r'rFcPaa]tRtRtoV3RlV3RlltV3RlRltRtVtV;t#)FlavaIntermediateic$<V^8dQhRS[RR/#r r )r"r#s"r$r%FlavaIntermediate.__annotate__s99399r'c"<\SV`4\P!VPVP 4Vn\VP\4'd\VP,Vn R#VPVn R#r`) rrrrrintermediate_sizer<r hidden_actstrrintermediate_act_fnrs&&r$rFlavaIntermediate.__init__s` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $r'cN<V^8dQhRS[PRS[P/#)rr rr)r"r#s"r$r%rWs#U\\ellr'cJVPV4pVPV4pV#r`r<r\r2r s&&r$rFlavaIntermediate.forwards& =1 00? r'r`r5rs@@r$rUrUs99r'rUcPaa]tRtRtoV3RlV3RlltV3RlRltRtVtV;t#) FlavaOutputic$<V^8dQhRS[RR/#r r )r"r#s"r$r%FlavaOutput.__annotate__r:r'c<\SV`4\P!VPVP 4Vn\P!VP4Vn R#r`) rrrrrYrr<rrrrs&&r$rFlavaOutput.__init__sB YYv779K9KL zz&"<"<= r'ch<V^8dQhRS[PRS[PRS[P/#r?r)r"r#s"r$r%rfs1U\\RWR^R^r'cZVPV4pVPV4pW,pV#r`rBrCs&&&r$rFlavaOutput.forwards, =1  ]3 %4 r'rBr5rs@@r$rdrds>> r'rdcXaa]tRtRtoRtV3RlV3RlltRV3RlRlltRtVtV;t #) FlavaLayeri&z?This corresponds to the Block class in the timm implementation.c$<V^8dQhRS[RR/#r r )r"r#s"r$r%FlavaLayer.__annotate__)s [ [3 [ [r'cx<\SV`4VPVn^Vn\ V4Vn\ V4Vn\V4Vn \P!VPVPR7Vn\P!VPVPR7VnR#)r|rN)rrchunk_size_feed_forward seq_len_dimrFrJrU intermediaterdrKrrrrlayernorm_beforelayernorm_afterrs&&r$rFlavaLayer.__init__)s '-'E'E$'/-f5!&) !# V-?-?VEZEZ [!||F,>,>FDYDYZr'c <V^8dQhRS[PRS[PR,RS[RS[S[PS[P3,S[S[P,,/#rr#)r"r#s"r$r%ro5s`|| t+  u||U\\) *U5<<-@ @ r'cVPVPV4VVR7pV^,pVR,pWQ,pVPV4pVPV4pVP Wq4pV3V,pV#rO)rJrtrursrK)r2r r!r"self_attention_outputsrRr3 layer_outputs&&&& r$rFlavaLayer.forward5s "&  ! !- 0)/"0"  2!4(,)8 ++M: ((6 {{<? /G+r')rJrqrsrurtrKrrrrrs@@r$rmrm&s#I [ [r'rmcTaa]tRtRtoV3RlV3RlltRV3RlRlltRtVtV;t#) FlavaEncoderiRc$<V^8dQhRS[RR/#r r)r"r#s"r$r%FlavaEncoder.__annotate__Ss,,{,t,r'c<\SV`4Wn\P!\ VP 4Uu.uFp\V4NK up4VnRVn R#uupi)FN) rrrwr ModuleListrangenum_hidden_layersrmlayergradient_checkpointing)r2rwrrs&& r$rFlavaEncoder.__init__SsR  ]]fF^F^@_#`@_1Jv$6@_#`a &+#$asA&c <V^8dQhRS[PRS[PR,RS[RS[RS[RS[S[,/#rr r!Nr"output_hidden_states return_dictr)r=rrLr!r )r"r#s"r$r%rYsY  ||  t+   #       r'cV'dRMRpV'dRMRp\VP4FBwrV'd Wa3,pV !WV4p V ^,pV'gK2Wz^,3,pKD V'd Wa3,pV'g1\;QJd.RWV34FNK 5#!RWV344#\WVR7#)Nc30"TF qfKVxK R#5ir`r?)r0rQs& r$r3'FlavaEncoder.forward..ssm$[q$[ )last_hidden_stater  attentionsr?) enumeraterr!r ) r2r r!r"rrall_hidden_statesall_self_attentionsi layer_module layer_outputss &&&&&& r$rFlavaEncoder.forwardYs#7BD$5b4(4OA#$58H$H!(HYZM)!,M  &91=MS>ST->k&)D  r'c4<V^8dQhRS[P/#rrModule)r"r#s"r$r%rs00bii0r'c.VPP#r`rrr7s&r$get_input_embeddings$FlavaImageModel.get_input_embeddingss///r'c4<V^8dQhRS[P/#rrr)r"r#s"r$r%rs11"))1r'c&WPnR#r`rr2rs&&r$set_input_embeddings$FlavaImageModel.set_input_embeddingss +0(r'c<V^8dQhRS[PR,RS[PR,RS[R,RS[PR,RS[R,RS[R,RS[R,R S[S[,/#) rrNrrr!r"rrr)r=rrrLr!r )r"r#s"r$r%rs/ / llT)/ ))D0/ #'+ /  t+ /  $; / #Tk/ D[/  + +/ r'c VeTMVPPpVeTMVPPpVeTMVPPpVf \ R4hVP WVR7p VP V VVVVR7p V ^,p VPV 4p VPeVPV 4MRp V'gW3V R,,#\V V V PV PR7#)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, image_num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Nz You have to specify pixel_values)rrr!r"rrrr pooler_outputr r) rwr"rrrrrrrr r r) r2rrrr!r"rrkwargsembedding_outputencoder_outputssequence_outputrs &&&&&&&&, r$rFlavaImageModel.forwards 2C1N-TXT_T_TqTq$8$D $++JjJj &1%S>ST->k&)D  r'c <V^8dQhRS[/#r)r)r"r#s"r$r%r s//o/r'c.VPP#r`rrr7s&r$r#FlavaTextModel.get_input_embeddings s...r'c4<V^8dQhRS[P/#rr)r"r#s"r$r%rs00"))0r'c&WPnR#r`rrs&&r$r#FlavaTextModel.set_input_embeddingss */'r'c <V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[R,RS[R,RS[R,R S[S[,/#) rrNr!rrr"rrrr=rrLr!r )r"r#s"r$r%rs? ? <<$&?  t+?  t+ ? llT) ?  $; ? #Tk? D[?  + +? r'c :VeTMVPPpVeTMVPPpVeTMVPPpVf \ R4hVP VVVR7p \ VPV VR7pVPV VVVVR7p V ^,p VPV 4p VPeVPV 4MRp V'gW3V R,,#\V V V PV PR7#)a input_ids (`torch.LongTensor` of shape `(batch_size, text_seq_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, text_seq_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) NzYou have to specify input_ids)rrrrwrr!rrr) rwr"rrrrrrrrr r r) r2rr!rrr"rrrrrrrs &&&&&&&&, r$rFlavaTextModel.forwards-02C1N-TXT_T_TqTq$8$D $++JjJj &1%S>ST->k&)D  r'c <V^8dQhRS[PRS[PR,RS[R,RS[R,RS[R,RS[S[,/#rr)r"r#s"r$r%rpsf3 3 ||3  t+3  $; 3 #Tk 3 D[ 3  + +3 r'c VeTMVPPpVeTMVPPpVeTMVPPpVP 4wrxp VP 'd@VP PVRR4p \P!W3^R7pV^, p\VPVVR7pVPVVVVVR7p V ^,p VPV 4p VPeVPV 4MRp V'gW3V R,,#\V V V PV P R7#)z hidden_states (`torch.FloatTensor` of shape `(batch_size, image_num_patches + text_seq_len, hidden_size)`): The concatenated hidden states of unimodal encoders. Nrrrrrr)rwr"rrrrrrr=rrrrrr r r)r2r r!r"rrrrrrrrrrs&&&&&&, r$rFlavaMultimodalModel.forwardosO2C1N-TXT_T_TqTq$8$D $++JjJj &1%>++,-Q0  &--/?@@,,-.a1  &224IJJ_V%=%= >?qAB  (( ** "44$33 + 7 7!-!9!9/;;(5*<8 45F G " $*@*@$BUBU V!yy)>)>@S@ST<< T[[5W5W(XY&(ii0F0FH[H[&\#%'YYt/D/DdFYFY%Z" r'c <V^8dQhRS[PRS[PR,RS[PR,RS[PR,RS[S[,RS[S[,/#)rrr!Nrrrr)r=rr rr!r )r"r#s"r$r%r sr//<</ t+/ t+ / llT) / +, / + +/r'c VP!RRVRVRVRVRR/VBpVPpVPV4VnV#)aQ input_ids (`torch.LongTensor` of shape `(batch_size, text_seq_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, text_seq_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) Examples: ```python >>> import torch >>> from transformers import AutoProcessor, FlavaModel >>> model = FlavaModel.from_pretrained("{0}") >>> processor = AutoProcessor.from_pretrained("{0}") >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], max_length=77, padding="max_length", return_tensors="pt" ... ) >>> with torch.inference_mode(): ... text_features = model.get_text_features(**inputs) ``` rr!rrrTr?)rrrr)r2rr!rrr text_outputsrs&&&&&, r$get_text_featuresFlavaModel.get_text_featuresssL48??4 4 )4 *4 & 4  4  4 )::%)%9%9:K%L "r'c <V^8dQhRS[PRS[PR,RS[R,RS[PR,RS[S[,RS[S[,/#)rrrNrr!rr)r=rrrLr rr!r )r"r#s"r$r%r  so**ll*))D0*#'+ *  t+ * +, * + +*r'c VP!RRVRVRVRVRR/VBpVPpVPV4VnV#)a bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, image_num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Examples: ```python >>> import torch >>> from transformers import AutoProcessor, FlavaModel >>> from transformers.image_utils import load_image >>> model = FlavaModel.from_pretrained("{0}") >>> processor = AutoProcessor.from_pretrained("{0}") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = load_image(url) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.inference_mode(): ... image_features = model.get_image_features(**inputs) ``` rrr!rrTr?)rrrr)r2rrrr!r image_outputsrs&&&&&, r$get_image_featuresFlavaModel.get_image_featuressuB594D4D5 %5 +5 *5 &> 5  5  5 *;;&*&;&;rrLr!r)r"r#s"r$r%r 7sN N ##d*N ''$.N  t+ N  t+ N , N &&-N $llT1N "&N  $;N #N D[N  ! !N r'c ^V eT MVPPp V 'g \R4hRp RpRpRpVeAVPVVVV V V R7pV^,V^,rVP VR ,4pRpRpRpRpVeCVP VVVVV V V R7pV^,V^,ppVP VR ,4pRpRpVeVeV'gVetVPwpppVPP'd V^, p\P!VVVPR7p\P!VV.^R7pMRp\P!VV.^R7pVPVVV R7pV^,pV 'g V VVVVV3#\V VVVVVR7#) a input_ids (`torch.LongTensor` of shape `(batch_size, image_num_patches + text_seq_len)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, image_num_patches + text_seq_len)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, image_num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). image_attention_mask (`torch.Tensor` of shape `(batch_size, image_num_patches)`, *optional*): Mask to avoid performing attention on padding pixel values for image inputs. Mask values selected in `[0, 1]`: - 1 for pixel values that are real (i.e., **not masked**), - 0 for pixel values that are padding (i.e., **masked**). skip_multimodal_encoder (*bool*, *optional*): Skip any calculations for multimodal encoder. Useful if multimodal encoding is not going to be used. Examples: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, FlavaModel >>> model = FlavaModel.from_pretrained("facebook/flava-full") >>> processor = AutoProcessor.from_pretrained("facebook/flava-full") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(text=["a photo of a cat"], images=image, return_tensors="pt", padding=True) >>> outputs = model(**inputs) >>> image_embeddings = outputs.image_embeddings >>> text_embeddings = outputs.text_embeddings >>> multimodal_embeddings = outputs.multimodal_embeddings >>> outputs.image_embeddings.shape torch.Size([1, 197, 768]) >>> text_embeddings.shape torch.Size([1, 7, 768]) >>> multimodal_embeddings.shape torch.Size([1, 205, 768]) ``` NzRFLAVA model requires hidden states to work. Please set `output_hidden_states=True`)rrr!r"rr)rr!rrr"rrrr)r!r)r:r+r;r*r<r,r)rwrrrrrrrrrr=onesrrr)r2rrr!rrrr+r,r"rrrr: image_statesimage_mm_projectionr+r; text_statestext_mm_projectionr*r<r,rrrattention_mask_imageattention_multimodalmultimodal_inputs&&&&&&&&&&&&, r$rFlavaModel.forward6s L&1%;q>[O!%!;!;KO!L  $   */A/MVm))<)B)B& GQ((666qLG',zz*gNaNhNh'i$',yy2F1W]^'_$'+$$yy*=?Q)RXYZ  $ 5 5 1ES^!6! %6a$8 ! %!   -%+#"7/   r'c&<V^8dQh/S[;R&#rr)r"r#s"r$r%r rr') rrrrrrrrrrrrr) NNNNNNNNNTN)r@rArBrCrrrr#r(rrErFrGrrs@@r$rrsx))V///b***XN N N cr'rcPaa]tRtRtoV3RlV3RlltV3RlRltRtVtV;t#)FlavaImageCodebookResPathic&<V^8dQhRS[RS[/#)rin_sizeout_sizer)r"r#s"r$r%&FlavaImageCodebookResPath.__annotate__s(((s(r'c  <\SV`4V^,p\4p\P!4VR&\P !W^^R7VR&\P!4VR&\P !WD^^R7VR&\P!4VR&\P !WD^^R7VR&\P!4VR&\P !WB^^R7VR &\P !V4VnR #) relu_1rpaddingconv_1relu_2conv_2relu_3conv_3relu_4conv_4N)rrrrReLUr Sequentialpath)r2r=r>rhid_sizerOrs&&&, r$r"FlavaImageCodebookResPath.__init__s q=}X7!QOXX81aPXX81aPXX81aPXMM$' r'cN<V^8dQhRS[PRS[P/#rrrr)r"r#s"r$r%r@s#%,,r'c$VPV4#r`rOr2rs&&r$r!FlavaImageCodebookResPath.forwardsyy|r'rUr5rs@@r$r;r;s(( r'r;cPaa]tRtRtoV3RlV3RlltV3RlRltRtVtV;t#)FlavaImageCodebookBlockic,<V^8dQhRS[RS[RS[/#)rr=r> num_layersr?)r"r#s"r$r%$FlavaImageCodebookBlock.__annotate__s' E E Es E Er'c <\SV`4^V^,, VnW8wd \P!W^^R7VnM\P !4Vn\W4VnR#)r|rDN) rr post_gainrrid_pathIdentityr;res_path)r2r=r>r[rrs&&&&,r$r FlavaImageCodebookBlock.__init__sQ j!m,  99WAqQDL;;=DL1'D r'cN<V^8dQhRS[PRS[P/#rSr)r"r#s"r$r%r\s'CCC%,,Cr'crVPV4VPVPV4,,#r`r_r^rarVs&&r$rFlavaImageCodebookBlock.forwards'||A$--2B!BBBr'rer5rs@@r$rYrYs# E ECCCr'rYcTaa]tRtRtoRV3RlV3RllltV3RlRltRtVtV;t#)FlavaImageCodebookLayerGroupic 8<V^8dQhRS[RS[RS[RS[RS[/#)r num_blocksr[r=r>use_pool)rrL)r"r#s"r$r%)FlavaImageCodebookLayerGroup.__annotate__s2 + +3 +C +# +QT +`d +r'cL<\SV`4\4p\V4F>pV^8Xd\ W4V4VRV^, 2&K%\ WDV4VRV^, 2&K@ V'd\ P !^R7VR&\ P!V4VnR#)rblock_)rpoolN) rrrrrYr MaxPool2drNgroup) r2rjr[r=r>rkblocksrrs &&&&&& r$r%FlavaImageCodebookLayerGroup.__init__s z"AAv+B7V`+aAw'(+B8Wa+bAw'( # \\a8F6N]]6* r'cN<V^8dQhRS[PRS[P/#rSr)r"r#s"r$r%rls#%,,r'c$VPV4#r`rqrVs&&r$r$FlavaImageCodebookLayerGroup.forwardszz!}r'rvrr5rs@@r$rhrhs + +r'rha" The FLAVA's image codebook model inspired from DALL-E's original encoder. Outputs raw hidden states and can be used to generate image tokens for an image based on DALL-E's vocab. Used to generate labels for MIM. Use `get_codebook_indices` to get image tokens for an image. caa]tRtRtoRtRtRtRtV3RlV3RlltV3RlRlt V3R lR lt V3R lR lt V3R lt Rt VtV;t#)FlavaImageCodebookimodelrFc&<V^8dQhRS[RS[/#)rrwr)rr)r"r#s"r$r%FlavaImageCodebook.__annotate__s*,*,(*,*,r'c :<\SV`V4WnVPVnVPVnVP VnVP VnVPVnVPVP ,p\4p\P!4VR&\P!^VP ,VP^^R7VR&\4p\P!VP^VP ,^^R7VR&\VP V^VP ,^VP ,4VR&\VP V^VP ,^VP ,4VR&\VP V^VP ,^VP ,4VR&\VP V^VP ,^VP ,RR7VR &\P!V4VR &\P!V4VnVP4VPP 'd!VP#4F pRVnK R #R #) relurDconvinputgroup_1group_2group_3F)rkgroup_4rKN)rrrw num_groupsinput_channelsnum_blocks_per_grouprrrrrMrrhrNrrrfreeze parameters requires_grad)r2rwrr[ output_blocksrrparamrs&&, r$rFlavaImageCodebook.__init__s    ++$33$*$?$?!!-- ++__t'@'@@ # " f " !d.>.>*>]^hi j f))D$7$7T=M=M9M[\fghw8  % %z1t7G7G3GTM]M]I] y9  % %z1t7G7G3GTM]M]I] y9  % %z1t7G7G3GTM]M]I] y9  % %z1t7G7G3GTM]M]I]hm y==7xmmF+   ;;   *&+#+ r'cN<V^8dQhRS[PRS[P/#rrrr)r"r#s"r$r%r|<s#...%,,.r'cvR\ R\ R2VPV4p\P!V^R7#)aI Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Codebook pixel values can be obtained using [`AutoImageProcessor`] by passing `return_codebook_pixels=True`. See [`FlavaImageProcessor.__call__`] for details. Examples: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoImageProcessor, FlavaImageCodebook >>> model = FlavaImageCodebook.from_pretrained("E") >>> image_processor = AutoImageProcessor.from_pretrained("a") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = image_processor([image], return_codebook_pixels=True, return_tensors="pt") >>> inputs = dict(pixel_values=inputs.codebook_pixel_values) >>> outputs = model.get_codebook_indices(**inputs) ``` )axis)_CHECKPOINT_FOR_CODEBOOK_DOCrrr=argmaxr2rz_logitss&& r$get_codebook_indices'FlavaImageCodebook.get_codebook_indices<sI 9:V8VWCC_B` a  4;;|,||H1--r'cN<V^8dQhRS[PRS[P/#rr)r"r#s"r$r%r|Zs#++u||+ +r'c^VPV4p\P!^R7!V4#)r|r)rrrSoftmaxrs&& r$get_codebook_probs%FlavaImageCodebook.get_codebook_probsZs$;;|,zza **r'cN<V^8dQhRS[PRS[P/#r)r=r>r)r"r#s"r$r%r|^s%")")E$5$5")ELL")r'c HR\ R\ R2\VP4^8wd\RVP R24hVP^,VP8wd-\RVP^, RVP 24hVP V4#)aJ Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Codebook pixel values can be obtained using [`AutoImageProcessor`] by passing `return_codebook_pixels=True`. See [`FlavaImageProcessor.__call__`] for details. Examples: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoImageProcessor, FlavaImageCodebook >>> model = FlavaImageCodebook.from_pretrained("ra") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = image_processor([image], return_codebook_pixels=True, return_tensors="pt") >>> inputs = dict(pixel_values=inputs.codebook_pixel_values) >>> outputs = model(**inputs) >>> print(outputs.shape) (1, 196) ``` z input shape z is not 4dz input has z channels but model built for )rlenrrrrr)r2rrs&&,r$rFlavaImageCodebook.forward^s 9:V8VWCC_B` a  : |!! "a '|L,>,>+?zJK K   a D$7$7 7z,*<*S>STr'clVPV4pVPV4pVPV4pV#r`)r<rrras&&r$r$FlavaPredictionHeadTransform.forwards4 =1 --m< }5 r')rr<rr5rs@@r$rrsUr'rc<aa]tRtRtoRV3RlltRtRtVtV;t#)ricR<\SV`4Wn\V4Vn\ P !VPVPRR7Vn \ P!\P!VP44Vn VeW PnR#R#)TrN)rrrwr transformrrrrdecoderrr=rrweight)r2rwrrs&&&r$r"FlavaMaskedPredictionHead.__init__ss  5f=yy!3!3V5F5FTR LLV->->!?@  "(LL  r'cJVPV4pVPV4pV#r`)rrrVs&&r$r!FlavaMaskedPredictionHead.forwards" NN1  LLOr')rrwrrr`r5rs@@r$rrs)r'rc8aa]tRtRtoV3RltRtRtVtV;t#) FlavaITMHeadic<\SV`4Wn\V4Vn\ P !VP^4VnR#)rN) rrrwrrrrrseq_relationshiprs&&r$rFlavaITMHead.__init__s8  !&) " &*<*#ii(;< <<(8:W:WXY[\:]^all,,8V8VWXZ[8\]`kk&88(f&es J/;J4)rwrr5rs@@r$rrsN 99r'rzk The FLAVA model for pretraining which outputs losses, embeddings, logits and transformer outputs. caa]tRtRtoRRRRRRRR /tRV3R lV3R llltV3R lR lt]RV3RlRll4tRt Vt V;t #)FlavaForPreTrainingizmmm_text_head.biaszmmm_text_head.decoder.biasz mim_head.biaszmim_head.decoder.biasz mlm_head.biaszmlm_head.decoder.biaszmmm_image_head.biaszmmm_image_head.decoder.biascH<V^8dQhRS[RS[PR,/#)rrwimage_codebookN)rrr)r"r#s"r$r% FlavaForPreTraining.__annotate__s$!!{!BII >%11<< ++ ++)/)I)I&%55 ++ & 7 7%55060W0W- r'c4<V^8dQhRS[P/#)rrr)r"r#s"r$r%r su||r'crVP4^8d"VPVP^4R4pV#)rr)rrrrVs&&r$ _resize_to_2d!FlavaForPreTraining._resize_to_2d s, 557Q;qvvay"%Ar'c%<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[PR,R S[PR,R S[PR,R S[R,R S[PR,R S[PR,RS[PR,RS[R,RS[RS[R,RS[R,RS[S[P,S[,/#)rrNinput_ids_maskedrcodebook_pixel_valuesr!rrrr+r mlm_labels mim_labels itm_labelsr"rr return_lossr)r=r-r>rrLr!r[)r"r#s"r$r%rshp p ##d*p  **T1p ''$. p %0047 p  t+ p  t+p ,p &&-p $llT1p +/+p LL4'p LL4'p LL4'p  $;p #!p "D[#p $D[%p ( u|| 8 8)p r'c VeTMVPPpVeTMVPPpV eT M VPp VfVe\P R4TpVP VVVVVV V VVRR7 pVP VVVVV VVVRR7 pRpVPpVPpVPpVPpVPpR;p;p;p;p;p;p p!R;p";p#;p$p%R;p&;p'p(VfVePV fLV'dDVPf \R4hVf \R4hVPPV4p VP^8Ed%VEe VEfTp)V EeVPV 4p VPV4pVP WP#R4&V)RV P%^4)R1R3,p)V P#VP 4p*V V*,p+V)V*R3,p)VP'V)4p"V'd]\(P*P-V"P/R"VP04V+P/R"44pVVP,pMVP'V)4p"VP2^8dVeVfTp,V eVPV 4p V,RV P%^4)R1R3,p,V P#VP 4p*V V*,p-V,V*R3,p,VP5V,4p#V'd]\(P*P-V#P/R"VP64V-P/R"44pVVP2,pMVP5V,4p#VP8^8dVeVP;V4p&V eV P#^4p.V.V.P=4(,pV'd4\(P*P-V&V 4p!V!VP8,p!Ve VV,pV e V V,p V eV V,p VV,pVEe;VP>^8Ed)Tp)VP%^4^, p/V)R^^V/,1R3,p)V eVPV 4p VPV4pVP WP#R4&V P#VP 4p*V V*,p+V)V*R3,p)VPAV)4p%V'd]\(P*P-V%P/R"VP04V+P/R"44pVVP>,pMVPAV)4p%VeVPB^8dTp,V,RVP%^4)R1R3,p,V eVPV 4p V P#VP 4p*V V*,p-V,V*R3,p,VPEV,4p$V'd]\(P*P-V$P/R"VP64V-P/R"44pVVPB,pMVPEV,4p$VEeVEeVPF^8EdVP PIVR#,4p0\(P*PKV0R"R 7p0VP PMVR#,4p1\(P*PKV1R"R 7p1VPN'd9VP PPPRPU\V\X4VP[V1V0VP PP4wp'p(p2VeV'V,p'V(V,p(V2V,p2V'dd\(P*P-V'V24p3\(P*P-V(V24p4V3V4,^, p V VPF,p \]VVV!V VVR 7p5V'd7V5P_4'g!\aR V5Pc444pV'EgeTVPdeVPdPg4MRTVPheVPhPg4MRVPVPjeVPjPg4MRTVPdeVPdPg4MRTVPheVPhPg4MRTVPjeVPjPg4MRV"V#V&V'V(V%V$3p6V'd"V5P_4'g VV53V6,p6\l;QJd.R V64FNK 5#!R V644#\oR$/R VbRV5bRVbRVPdbRVbRVPhbRVPbRVPjbRVbRVPdbRVbRVPhbRVbRVPjbRV"bRV#bRV&bRV'bRV(bR V%bR!V$b#)%a input_ids (`torch.LongTensor` of shape `(batch_size, text_seq_len)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) input_ids_masked (`torch.LongTensor` of shape `(batch_size, text_seq_len)`): Indices of input sequence tokens in the vocabulary. These ones are the masked version of the original task to be used with MLM. Indices can be obtained using [`AutoTokenizer`] along with [`DataCollatorForMaskedLanguageModeling`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) codebook_pixel_values (`torch.FloatTensor` of shape `(batch_size, num_image_patches, patch_size, patch_size, 3)`, *optional*): Pixel values for image patches that are used to compute the image codebook labels for masked image modeling. token_type_ids (`torch.LongTensor` of shape `(batch_size, text_seq_len)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, image_num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). image_attention_mask (`torch.FloatTensor` of shape `(batch_size, image_num_patches)`, *optional*): Mask to avoid performing attention on padding token indices specifically for images. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) skip_unmasked_multimodal_encoder (*bool*, *optional*): Skip any calculations for multimodal encoder for unmasked inputs. FLAVA pretraining doesn't need unmasked multimodal embeddings or outputs as of now. mlm_labels (`torch.LongTensor` of shape `(batch_size, text_seq_len)`, *optional*): Labels for computing the left-to-right language and multimodal masked modeling loss (next word prediction). Indices should be in `[-100, 0, ..., text_config.vocab_size - 1]` (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., text_config.vocab_size - 1]`. mim_labels (`torch.LongTensor` of shape `(batch_size, image_num_patches)`, *optional*): Labels for computing the image and multimodal masked modeling loss. Indices should be in `[-100, 0, ..., image_config.vocab_size - 1]`. Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., image_config.vocab_size - 1]`. If not passed, they are generated automatically using the image codebook assigned to the model. By default, it uses [`FlavaImageCodebook`]. See [`FlavaImageCodebook`] to understand how to generate mim_labels. itm_labels (`torch.LongTensor` of shape `(batch_size, 1)`, *optional*): Labels for computing the image-text matching loss. 0 means the pairs don't match and 1 means they match. The pairs with 0 will be skipped for calculation of MMM and global contrastive losses as well. return_loss (`bool`, *optional*, default to None): Whether to return calculated loss or not. Examples: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import FlavaForPreTraining, AutoProcessor >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> model = FlavaForPreTraining.from_pretrained("facebook/flava-full") >>> processor = AutoProcessor.from_pretrained("facebook/flava-full") >>> text = ["a photo of a cat"] >>> inputs = processor( ... images=[image], ... text=text, ... return_masks=True, ... return_codebook_pixels=True, ... padding=True, ... max_length=77, ... return_tensors="pt", ... ) >>> output = model(**inputs) ``` Nz`input_ids_masked` isn't passed which means MLM loss won't be calculated correctlySetting it to `input_ids` so that model can work. Please pass it if this is unintentional. This is usually OKAY if you are doing inference on unmasked text...T) rrr!rrr+r,r"rr) rrr!rr+rr"rrz`return_loss` is set to True but the image codebook is not initialized and no `mim_labels` have been passed. Reinstantiate the model with `init_codebook` set to True or pass in your custom `mim_labels`z`codebook_pixel_value` are required to generate `mim_labels` if loss is expected. Call `AutoProcessor` with `return_codebook_pixels` set to Truerr)rTrUrVrWrXrYc30"TF qeTM^xK R#5ir`r?)r0rhs& r$r3.FlavaForPreTraining.forward..Gs_I^%5T1eGV^GKK KZK/4D:>> >% # .2N2Z!k&&.&; )0$Y"00EEF[\  ??Q #:#FKgKo!8 %!// ; "&"4"4_"E7;7K7K --d34%7JOOA !}}::"D,A,ABDWD\D\]_D` H/H!]]+=>  ??Q #9#EJfJn 6 %!// ; $5a*//!:L9L9NPQ6Q$R! * d.B.B C &0&?#$5mQ6F$G!!]]+<= !}}::"D,@,@ACVC[C[\^C_ H/H!]]+<=  ??Q #?#K'CDJ%&MM!, $ '77!}}:::zRH/H/;3OPX3Y0)!+H!5J)!+H!5J&5h&?O ( 38M8MPQ8Q!= /44Q7!;I!3Aq1y=7H!4K!L %!// ; "&"4"4_"E7;7K7K --d34 * d.B.B C &0&?#%7 q8H%I"#'#6#67I#J %']]%@%@(--b$2G2GHJ]JbJbceJf&N#d&;&;;N#'#6#67I#J  ( 38L8Lq8P <  1!6L6Q6QRS6T5T5VXY2Y Z %!// ; * d.B.B C &0&?#$5mQ6F$G!"&"4"45F"G$&MM$?$?',,R1E1EFH[H`H`acHd%M"T%9%99M"&"4"45F"G  'O,GDLjLjmnLn!ZZ778PQN]]44^4LN"jj99:J7:STO mm55o25NO}}} &&++223HJ_`;?;W;W1G1G< 8 oy ##3H#= "1(";%h/  " ; ;
  • Q>]>]>i#//88:os,&88D$55>>@  +F.<#8#8#:#:  5> *F &9%J%J " "! ""# $*:% &)8' (.) *,+  r')rrrrrrrrrrrrrrrrrrr`)NNNNNNNNNNNNNNTNN) r@rArBrC_tied_weights_keysrrrrrFrGrrs@@r$rrs^ :00< !!F p p p p r'r)rryrrrrr)LrDrr'r dataclassesrtypingrr=rrr activationsr masking_utilsrmodeling_layersr modeling_outputsr r modeling_utilsr processing_utilsr utilsrrrrrrconfiguration_flavarrrrr get_loggerr@rrrrr rrJr[rrurrr r7rFrUrdrmr}rrrrrrr;rYrhryrrrrr__all__r?r'r$r+s0 #! &!69K-&jj   H %>&)99