+ j)&Rt^RIHt^RIHt^RIHt^RIt^RIHt^RI H t ^RI H t ^RIHtHt^R IHtHtHt^R IHtHt^R IHt^R IHtHtHtHtHt^R I H!t!H"t"^RI#H$t$^RI%H&t&^RI'H(t(H)t)H*t*H+t+^RI,H-t-H.t.^RI/H0t0^RI1H2t2H3t3H4t4]*Pj!]64t7Rt8])!RR7]!RR]444t9])!RR7]!RR]444t:!RR]Pv4t<!RR]Pv4t=!R R!]Pv4t>!R"R#]Pv4t?!R$R%]Pv4t@!R&R']Pv4tA!R(R)]Pv4tB!R*R+]Pv4tC!R,R-]Pv4tDRYR.R/lltE!R0R1]Pv4tF!R2R3]Pv4tG!R4R5]Pv4tH!R6R7]Pv4tI!R8R9]4tJ!R:R;]Pv4tK!R<R=]Pv4tL])!R>R?]"44tM!R@RA]M4tN])!RBR7!RCRD]M44tO])!RER7!RFRG]M44tP!RHRI]Pv4tQ!RJRK]Pv4tR!RLRM]Pv4tS])!RNR7!RORP]M44tT])!RQR7!RRRS]M44tU!RTRU]Pv4tV])!RVR7!RWRX]M44tW.RZOtXR#)[zPyTorch BridgeTower Model) OrderedDict)Callable) dataclassN)nn)CrossEntropyLoss)initialization)ACT2FNQuickGELUActivation)Cache DynamicCacheEncoderDecoderCache)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentionsMaskedLMOutput ModelOutputSequenceClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward)TransformersKwargsauto_docstringlogging torch_int)can_return_tuplemerge_with_config_defaults)capture_outputs)BridgeTowerConfigBridgeTowerTextConfigBridgeTowerVisionConfigRobertaTokenizerz. Output type of [`BridgeTowerModel`]. ) custom_introcDa]tRt^2toRtRtRtRtRtRt V3Rlt Rt Vt R#)BridgeTowerModelOutputa text_features (`torch.FloatTensor` of shape `(batch_size, text_sequence_length, hidden_size)`): Sequence of hidden-states at the text output of the last layer of the model. image_features (`torch.FloatTensor` of shape `(batch_size, image_sequence_length, hidden_size)`): Sequence of hidden-states at the image output of the last layer of the model. pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size x 2)`): Concatenation of last layer hidden-state of the first token of the text and image sequence (classification token), respectively, after further processing through layers used for auxiliary pretraining tasks. Nc <V^8dQh/S[PR,;R&S[PR,;R&S[PR,;R&S[S[P,R,;R&S[S[P,R,;R&#)N text_featuresimage_features pooler_output hidden_states attentionstorch FloatTensortuple)format __classdict__s"u/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/bridgetower/modeling_bridgetower.py __annotate__#BridgeTowerModelOutput.__annotate__2s"$$t+2#$%%,3%&$$t+2'(**+d29)*e''(4/6+) __name__ __module__ __qualname____firstlineno____doc__r)r*r+r,r-__annotate_func____static_attributes____classdictcell__r3s@r4r&r&2s,/3M/3N.2M59M26J+r7r&z> Output type of ['BridgeTowerForContrastiveLearning'] cLa]tRt^JtoRtRtRtRtRtRt Rt Rt V3Rlt Rt VtR#)BridgeTowerContrastiveOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Image-text contrastive loss. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). text_embeds (`torch.FloatTensor)`, *optional*, returned when model is initialized with `with_projection=True`): The text embeddings obtained by applying the projection layer to the pooler_output. image_embeds (`torch.FloatTensor)`, *optional*, returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. cross_embeds (`torch.FloatTensor)`, *optional*, returned when model is initialized with `with_projection=True`): The text-image cross-modal embeddings obtained by applying the projection layer to the pooler_output. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Nc<V^8dQh/S[PR,;R&S[PR,;R&S[S[P,R,;R&S[S[P,R,;R&S[S[P,R,;R&S[S[P,R,;R&S[S[P,R,;R&#) r(Nlosslogits text_embeds image_embeds cross_embedsr,r-r.)r2r3s"r4r5)BridgeTowerContrastiveOutput.__annotate__Js.   d ")/0    $+12u(()D0734))*T1856))*T1878**+d299:e''(4/6;r7r8)r9r:r;r<r=rErFrGrHrIr,r-r>r?r@rAs@r4rCrCJs6 &*D'+F37K48L48L59M26J;r7rCcZaa]tRt^jtoV3RltV3RlRltRV3RlRlltRtVtV;t #)BridgeTowerResidualAttentionc <\SV`4\P!VPVP^@,4Vn\P !VPVPR7Vn\P!\R\P!VPVP^,43R\43R\P!VP^,VP43.44Vn \P !VPVPR7VnRVnR#)@epsc_fcgeluc_projN)super__init__rMultiheadAttention hidden_sizeattn LayerNormlayer_norm_epsln_1 ModuleDictrLinearr mlpln_2 attn_maskselfconfig __class__s&&r4rU%BridgeTowerResidualAttention.__init__ks ))&*<*P>PTV>VW LL!3!39N9NO == RYYv'9'96;M;MPQ;QRS023ryy););a)?ASASTU  LL!3!39N9NO r7cN<V^8dQhRS[PRS[P/#)r( hidden_stateattention_maskr/Tensor)r2r3s"r4r5)BridgeTowerResidualAttention.__annotate__|s#  ell ELL r7c <Ve,VP\PVPR7pVPe2VPPVP VPR7MRVnVP VVVRVPVR7^,#)NdtypedeviceF) need_weightsr`key_padding_mask)tor/boolror`rnrX)rbrgrhs&&&r4 attention&BridgeTowerResidualAttention.attention|s  %+..UZZ H[H[.\N~~) NN  L$6$6|?R?R  S  yy   nn+     r7c\<V^8dQhRS[PRS[PR,/#r(rgrhNri)r2r3s"r4r5rks)ELL%,,QUBUr7cWPVPV4V4,pVPV4pVPP 4F pV!V4pK W1,pV#N)rtr[r_r^values)rbrgrhresidual_statelayers&&& r4forward$BridgeTowerResidualAttention.forwardsZ%tyy7NP^(__yy0 XX__&E .L'%4 r7)rXr`r[r_r^ry) r9r:r;r<rUrtr}r?r@ __classcell__rdr3s@@r4rLrLjs#"  "r7rLcHaa]tRt^toV3RltRV3RlRlltRtVtV;t#)BridgeTowerTransformerc<\SV`4VPVnVPVnVP'dP\ P !\VP^, 4Uu.uFp\V4NK up4Vn MG\ P !\VP4Uu.uFp\V4NK up4Vn VPVn R#uupiuupi)N) rTrUrWnum_hidden_layersremove_last_layerr ModuleListrangerL resblocks stop_gradientrbrc_rds&& r4rUBridgeTowerTransformer.__init__s !--!'!9!9  # # #]]?DTE[E[^_E_?`a?`!-f5?`aDN ]]?DTE[E[?\]?\!-f5?\]DN$11 b^s 2C-:C2c\<V^8dQhRS[PRS[PR,/#rwri)r2r3s"r4r5#BridgeTowerTransformer.__annotate__s)ELL%,,QUBUr7c.pVPFOpV!W4pVP'd"VPVP44K>VPV4KQ V#ry)rrappenddetach)rbrgrhr,blocks&&& r4r}BridgeTowerTransformer.forwardsX ^^E >L!!!$$\%8%8%:;$$\2 $ r7)rWrrrry r9r:r;r<rUr}r?r@rrs@@r4rrs 2r7rcfaa]tRt^toV3RlV3RlltV3RlRltRV3RlRlltRtVtV;t #) BridgeTowerVisionEmbeddingsc <V^8dQhRS[/#r(rcr")r2r3s"r4r5(BridgeTowerVisionEmbeddings.__annotate__sjj6jr7c<\SV`4WnVPVnVP VnVP Vn\P!\P!VP44Vn \P!VPVPVP VP RR7VnVP VP ,^,VnVP^,Vn\P"!VP VP4Vn\P&!\P(!VP 4P+R4RR7VnR#)F) in_channels out_channels kernel_sizestridebias persistentNr)rTrUrcrW embed_dim image_size patch_sizer Parameterr/randnclass_embeddingConv2d num_channelspatch_embedding num_patches num_positions Embeddingposition_embeddingBufferarangeexpand position_idsras&&r4rU$BridgeTowerVisionEmbeddings.__init__s  ++ ++ ++!||EKK,GH!yy++??  !OOt>1D!--1"$,,t/A/A4>>"RIIell43E3E&F&M&Mg&Vchir7cZ<V^8dQhRS[PRS[RS[RS[P/#)r( embeddingsheightwidthreturnr/rjint)r2r3s"r4r5rs;'D'D5<<'D'DUX'D]b]i]i'Dr7c VP^,^, pVPPP^4pVP^,^, p\P P 4'g(WF8Xd"W#8XdVPVP4#VR,pVR,pVPR,p W P,p W0P,p \VR,4p VP^WV 4pVP^^^^4p\PPVW3RRR7pVP^^^^4P^RV 4p\P !Wx3^R7#) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 ?bicubicF)sizemode align_cornersdim)NNN:NrN)r:rNNr)shaperweight unsqueezer/jit is_tracingrrrreshapepermuter functional interpolateviewcat) rbrrrrrrclass_pos_embedpatch_pos_embedr new_height new_widthsqrt_num_positionss &&&& r4interpolate_pos_encoding4BridgeTowerVisionEmbeddings.interpolate_pos_encodings[!&&q)A- !44;;EEaH*003a7 yy##%%+*F6?**4+<+<= =,U3,U3r". __, &}c'9:)11!5G]`a)11!Q1=--33 ( 4 *11!Q1=BB1b#Nyy/;CCr7cN<V^8dQhRS[PRS[P/#)r( pixel_valuesr)r/r0rj)r2r3s"r4r5rs(E$5$5Z_ZfZfr7c VPwr4rVV'gMWPP8wgW`P8wd-\RV RV RVP RVP R2 4hVPPP pVPVP VR74pVP^4P^^4pVPPV^R4p \P!W.^R7p V'dWPWV4,p V #WPVP4,p V #)zInput image size (*z) doesn't match model (z).rnrr)rr ValueErrorrrrnrrflatten transposerrr/rrrr) rbrr batch_sizerrr target_dtype patch_embeds class_embedsrs &&& r4r}#BridgeTowerVisionEmbeddings.forwards$'3'9'9$ v'V-F%SbSbJb$VHAeW4KDOOK\\]^b^m^m]nnpq ++2288 ++LOO,O,OP #++A.88A> ++22:q"E YY ;C ##&C&CJX]&^^J$&=&=d>O>O&PPJr7) rrcrrrrrrrrF) r9r:r;r<rUrr}r?r@rrs@@r4rrs-jj,'D'DRr7rcpaa]tRtRtoV3RltR V3RlRlltR V3RlRlltV3RlRltR tVt V;t #) BridgeTowerVisionTransformeric L<\SV`4\V4Vn\P !VP VPR7Vn\V4Vn \P !VP VPR7Vn VPVn VP'gk\P!\VP4Uu.uF/p\P !VP VPR7NK1 up4VnR#R#uupirON)rTrUrrrrYrWrZln_prer transformerln_postshare_layernormrrr ln_separaters&& r4rU%BridgeTowerVisionTransformer.__init__s 5f=ll6#5#56;P;PQ 1&9||F$6$6Fc <\SV`4VPVnVPVnVPR 9dVPR8Xd1\P !\ P!R44VnM@VPR8Xd0\P !\ P!R44Vn \P!VPVPR7Vn R#\RVP R24h) add scaled_addrg?rrOlink_tower_type  is not implementedN)r r r) rTrUlink_tower_typerWrrr/tensor scaled_factorbetarYrZNotImplementedErrorras&&r4rUBridgeTowerLinkTower.__init__?s %55!--  ! !%I I%%5%'\\%,,s2C%D"''=8LLc):; \\$*:*:@U@UVDN%(89O9O8PPc&de er7cVPR8XdVPW,4#VPR8Xd)VPWP,V,4#VPR8XdAVPV^VP, ,W P,,4#\ RVP R24h)r r rr r )rrYrrr)rbr,cross_modal_hidden_statesrhs&&&&r4r}BridgeTowerLinkTower.forwardLs   5 (>>-"KL L  ! !\ 1>>-2D2D"DG`"`a a  ! !] 2>>-1tyy="AD]`i`iDi"ij j%(89M9M8NNa&bc cr7)rYrrWrrrrs@@r4rr>s fddr7rcDaa]tRtRtoV3RltV3RlRltRtVtV;t#)BridgeTowerSelfOutputiXc0<\SV`4\P!VPVP4Vn\P !VPVPR7Vn\P!VP4Vn R#r) rTrUrr]rWdenserYrZDropouthidden_dropout_probdropoutras&&r4rUBridgeTowerSelfOutput.__init__Ys` YYv1163E3EF f&8&8f>S>STzz&"<"<= r7ch<V^8dQhRS[PRS[PRS[P/#r(r, input_tensorrri)r2r3s"r4r5"BridgeTowerSelfOutput.__annotate___1U\\RWR^R^r7cxVPV4pVPV4pVPW,4pV#ryrrrYrbr,r!s&&&r4r}BridgeTowerSelfOutput.forward_5 =1  ]3 }'CD r7rYrrrrs@@r4rrX> r7rcDaa]tRtRtoV3RltV3RlRltRtVtV;t#)BridgeTowerIntermediateigc"<\SV`4\P!VPVP 4Vn\VP\4'd\VP,Vn R#VPVn R#ry) rTrUrr]rWintermediate_sizer isinstance hidden_actstrrintermediate_act_fnras&&r4rU BridgeTowerIntermediate.__init__hs` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $r7cN<V^8dQhRS[PRS[P/#r(r,rri)r2r3s"r4r5$BridgeTowerIntermediate.__annotate__ps#U\\ellr7cJVPV4pVPV4pV#ryrr2rbr,s&&r4r}BridgeTowerIntermediate.forwardps& =1 00? r7r8rrs@@r4r,r,gs9r7r,cDaa]tRtRtoV3RltV3RlRltRtVtV;t#)BridgeTowerOutputiwc0<\SV`4\P!VPVP 4Vn\P!VP VPR7Vn\P!VP4Vn R#r) rTrUrr]r.rWrrYrZrrrras&&r4rUBridgeTowerOutput.__init__xs` YYv779K9KL f&8&8f>S>STzz&"<"<= r7ch<V^8dQhRS[PRS[PRS[P/#r ri)r2r3s"r4r5BridgeTowerOutput.__annotate__~r#r7cxVPV4pVPV4pVPW,4pV#ryr%r&s&&&r4r}BridgeTowerOutput.forward~r(r7r)rrs@@r4r<r<wr*r7r<cDaa]tRtRtoV3RltV3RlRltRtVtV;t#)BridgeTowerPooleric<\SV`4\P!VPVP4Vn\P !4VnR#ry)rTrUrr]rWrTanh activationras&&r4rUBridgeTowerPooler.__init__s9 YYv1163E3EF '')r7cN<V^8dQhRS[PRS[P/#r5ri)r2r3s"r4r5BridgeTowerPooler.__annotate__s#U\\ellr7c\VR,pVPV4pVPV4pV#)r)r)rrG)rbr,first_token_tensor pooled_outputs&& r4r}BridgeTowerPooler.forwards2+40 #56  6 r7)rGrrrs@@r4rDrDs$ r7rDcV^8dQhR\PR\PR\PR\PR\PR,R\R,R\R \ \ ,/#) r(modulequerykeyvaluerhNscalingrkwargs)rModuler/rjfloatrr)r2s"r4r5r5s{%% II% <<% % << % LL4' % T\ %%'(%r7c VfVPR4R,p\P!WP^^44V,pVe W,p\P P VRR7p\P PWVPR7p\P!W4p V P^^4P4p W3#)Nr)ptrainingr) rr/matmulrrrsoftmaxrr[ contiguous) rQrRrSrTrhrUrrV attn_weights attn_outputs &&&&&&&, r4eager_attention_forwardrbs**R.D(<<}}Q':;gEL!#4 ==((2(>L==((6??([L,,|3K''1-88:K  $$r7cLaa]tRtRtoRV3RlltRV3RlRlltRtVtV;t#)BridgeTowerSelfAttentionict<\SV`4VPVP,^8wd9\ VR4'g'\ RVP RVP R24hWnVPVn\VPVP, 4VnVPVP,Vn VPR,Vn \P!VPVP4Vn \P!VPVP4Vn\P!VPVP4Vn\P !VP"4VnVP&VnW nW0nR#rLembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()Nr\)rTrUrWnum_attention_headshasattrrrcrattention_head_size all_head_sizerUrr]rRrSrTrattention_probs_dropout_probr is_decoder is_causal layer_idxrbrcrorprds&&&&r4rU!BridgeTowerSelfAttention.__init__sM     : : :a ?PVXhHiHi#F$6$6#78 445Q8  #)#=#= #&v'9'9F??*9*N*N'&=%C%CI\`\j\j%k "I(?(M(M KK , ,.E) %8      % $}}}C$,,.. % LL %  % ! "));;;;FFH ((r7) rlrkrcrrornrSrprirRrUrTFN)NNrrs@@r4rdrds#0')')')r7rdcLaa]tRtRtoRV3RlltRV3RlRlltRtVtV;t#)BridgeTowerCrossAttentionicR<\SV`4VPVP,^8wd9\ VR4'g'\ RVP RVP R24hWnVPVn\VPVP, 4VnVPVP,Vn VPR,Vn \P!VPVP4Vn \P!VPVP4Vn\P!VPVP4Vn\P !VP"4VnW nW0nR#rf)rTrUrWrirjrrcrrkrlrUrr]rRrSrTrrmrrorprqs&&&&r4rU"BridgeTowerCrossAttention.__init__s@     : : :a ?PVXhHiHi#F$6$6#78 445Q8  #)#=#= #&v'9'9FB**4>>:(?(M(M KK , ,.E) %8      % $}}}C$,,.. % LL %  % ! "));;;;FFH ((r7) rlrkrcrrorSrprirRrUrTrNNNrrs@@r4rrs#.1)1)1)r7rcLaa]tRtRtoRV3RlltRV3RlRlltRtVtV;t#)BridgeTowerAttentioniDc<\SV`4W@nV'd\M\pV!WVR7Vn\ V4VnR#)rorpN)rTrUis_cross_attentionrrdrbroutput)rbrcrorprattention_classrds&&&&& r4rUBridgeTowerAttention.__init__Es9 "47I3Og#F9U +F3 r7c<V^8dQhRS[PRS[PR,RS[PR,RS[PR,RS[R,RS[S[,RS[S[P,/# r(r,rhNrencoder_attention_maskrtrVrru)r2r3s"r4r5!BridgeTowerAttention.__annotate__Ls..||.))D0. %0047 . !& 1 1D 8 .  .+,. u|| .r7c VP'gTMTpVP!V3RVRVRV/VBwrxVPWq4pWx3#)rrhrt)rrbr) rbr,rhrrrtrVattention_outputr`s &&&&&&, r4r}BridgeTowerAttention.forwardLsm04/F/F/FLb)- * "7* ** , *  * & ;;'7G--r7)rrrb)FNFNNNNrrs@@r4rrDs4...r7rcRaa]tRtRtoRV3RlltRV3RlRlltRtRtVtV;t #) BridgeTowerBertCrossLayeriac&<\SV`4VPVn^Vn\ VRVR7VnVP VnVPVn\ VRVRR7Vn\V4Vn \V4Vn R#)rTrFrorprN) rTrUchunk_size_feed_forward seq_len_dimrrtrnadd_cross_attentioncrossattentionr, intermediater<rrbrcrprds&&&r4rU"BridgeTowerBertCrossLayer.__init__bs '-'E'E$-fPYZ ++#)#=#= 2 #   4F;'/ r7c0<V^8dQhRS[S[,/#)r(rV)rr)r2r3s"r4r5&BridgeTowerBertCrossLayer.__annotate__rs" " +," r7c VP!V3RVRR/VBwrxTp VP!V 3RVRVRVRV/VBwrT p \VPVPVP V 4p V VV 3#)rhrtNrr)rtrrfeed_forward_chunkrr) rbr,rrhrrtrVself_attention_outputself_attn_weightsrcross_attention_outputcross_attn_weights layer_outputs &&&&&&, r4r}!BridgeTowerBertCrossLayer.forwardrs48>> 4 )4 !4  4 0 1595H5H 6 )6 #86 $: 6 , 6  6 220  # #T%A%A4CSCSUe      r7cJVPV4pVPW!4pV#ryrrrbrintermediate_outputrs&& r4r,BridgeTowerBertCrossLayer.feed_forward_chunk)"//0@A{{#6I r7rrtrrrrnrrryr r9r:r;r<rUr}rr?r@rrs@@r4rras0 " " Hr7rcRaa]tRtRtoRV3RlltRV3RlRlltRtRtVtV;t #) BridgeTowerTextLayeric<\SV`4VPVn^Vn\ WP VR7VnVP VnVPVnVP'd5VP 'g\V R24h\ VRVRR7Vn \V4Vn \V4Vn R#)rrz> should be used as a decoder model if cross attention is addedFTrN)rTrUrrrrnrtrrrr,rr<rrs&&&r4rUBridgeTowerTextLayer.__init__s '-'E'E$-f@Q@Q]fg ++#)#=#=  # # #??? D6)g!hii"6##' #D  4F;'/ r7c<V^8dQhRS[PRS[PR,RS[PR,RS[PR,RS[R,RS[S[,RS[P/#r)r/rjr0r rr)r2r3s"r4r5!BridgeTowerTextLayer.__annotate__s%%||%))D0% %0047 % !& 1 1D 8 %  %+,% %r7c 8VP!VV3RV/VBwrxTp VP'dBVe>\VR4'g\RV R24hVP!VRVV3RV/VBwrT p \ VP VPVPV 4p V #)rtNrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`) rtrnrjrrrrrr) rbr,rhrrrtrVrrrrrs &&&&&&, r4r}BridgeTowerTextLayer.forwards$(>>  $ ,$  $   1 ???4@4!122 =dVDDD )-(;(;%%& ) !0 )  ) % " 6 0  # #T%A%A4CSCSUe r7cJVPV4pVPW!4pV#ryrrs&& r4r'BridgeTowerTextLayer.feed_forward_chunkrr7rryrrrs@@r4rrs0(%%Nr7rcHaa]tRtRtoV3RltRV3RlRlltRtVtV;t#)BridgeTowerTextEncoderic <\SV`4Wn\P!\ VP 4Uu.uFp\WR7NK up4VnR#uupi)rpN) rTrUrcrrrrrr|)rbrcirds&& r4rUBridgeTowerTextEncoder.__init__sK  ]]@EfF^F^@_ `@_1 !& 6@_ `  `sA c<V^8dQhRS[PRS[PR,RS[PR,RS[PR,RS[R,RS[R,RS[S[,R S[/#) r(r,rhNrrrt use_cacherVr)r/rjr0r rsrrr)r2r3s"r4r5#BridgeTowerTextEncoder.__annotate__s  || ))D0  %0047  !& 1 1D 8    $; +,  3 r7c ~VPFpV!VVV3RVRV/VBpK \TV'dVR7#RR7#)rrtN)last_hidden_statert)r|r) rbr,rhrrrtrrV layer_modules &&&&&&&, r4r}BridgeTowerTextEncoder.forwardsg!JJL(%(>  !0   M'9+/8O  >B  r7)rcr|)NNNNNrrs@@r4rrs    r7rcpaa]tRtRtoRtV3RltR V3RlRllt]R4t]R Rl4t Rt Vt V;t #) BridgeTowerTextEmbeddingsizGConstruct the embeddings from word, position and token_type embeddings.c<\SV`4\P!VPVP VP R7Vn\P!VPVP 4Vn \P!VP VPR7Vn \P!VP4Vn\P!\ P"!VP$4P'R4RR7Vn\P!\ P*!VP(P-4\ P.R7RR7VnVP Vn\P!VP$VP VP2R7VnR#)) padding_idxrOFrrNr)rTrUrr vocab_sizerW pad_token_idword_embeddingstype_vocab_sizetoken_type_embeddingsrYrZrrrrr/rmax_position_embeddingsrrzerosrlongtoken_type_idsrposition_embeddingsras&&r4rU"BridgeTowerTextEmbeddings.__init__s% !||F,=,=v?Q?Q_e_r_rs%'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<= IIell63Q3Q&R&Y&YZa&botu ii D4E4E4J4J4LTYT^T^(_lqr!..#%<<  * *F,>,>DL\L\$  r7c <V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[RS[P/#)r( input_idsNrr inputs_embedspast_key_values_lengthr)r/ LongTensorr0rrj)r2r3s"r4r5&BridgeTowerTextEmbeddings.__annotate__sw..##d*.((4/.&&- . ((4/ . !$ . .r7c4Vf=VeVPWPV4pMVPW@P4pVeVP4pMVP4RRpVwrxVf\ VR4'drVP P VP4PVP^,R4p \P!V ^VR7p V PWx4pM;\P!V\PVPPR7pVfVPV4pVP!V4p WJ,p VP#V4p W,p VP%V 4p VP'V 4p V #)Nr)rindexrmr)"create_position_ids_from_input_idsr&create_position_ids_from_inputs_embedsrrjrrrrorrr/gatherrrrrrrrYr) rbrrrrrr}r seq_lengthbuffered_token_type_idsrrrs &&&&&& r4r}!BridgeTowerTextEmbeddings.forwardst  $#FF//1G   $JJ=ZjZjk  #..*K',,.s3K!,  !t-..*.*=*=*@*@ATAT*U*\*\]i]o]opq]rtv*w'*/,,7NTU]i*j'!8!?!? !W!&[ SWSdSdSkSk!l   00;M $ : :> J": "66|D5 ^^J/ \\*- r7c VP4RRpV^,p\P!V^,W1,^,\PVPR7pVP ^4P V4#)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nrmr)rr/rrrorr)rrr}sequence_lengthrs&& r4r@BridgeTowerTextEmbeddings.create_position_ids_from_inputs_embedsEsn$((*3B/ %a.|| !O_:Q>ejjYfYmYm %%a(// <4M\ V\@4'dP\PB!VPD\FPH!VPJ4PMR44M\ V\N4'd\PB!VPD\FPH!VPDPPR,4PMR44\P!VPR4\ V\0P2\T34'd3VPVe#\P!VPV4R#R#R#) r)stdrxg?)meanr Nr\rr),rT _init_weightsrcinitializer_factorr/rrWrrrinitnormal_rXin_proj_weightzeros_ in_proj_biasout_projrr^rQrSrrrrr]rr!BridgeTowerForContrastiveLearning constant_ logit_scalelogit_scale_init_valuercopy_rr/rrrrrrBridgeTowerMLMHeadr)rbrQr proj_stdattn_stdfc_stdrrds&& r4r (BridgeTowerPreTrainedModel._init_weightsvs f%kk,, f: ; ; //51t{{?\?\;\ae:efH{{..4H$++111d:F++55 UZZ66HNK EJJ334 UZZ0077X^L UYY^^22 E UYY--44(.II 6 LL**:: O LL**==DD(. Y BIIr|| D E E LLSdSj A  A B B NN6--t{{/Q/Q R  ; < < JJv**ELL9M9M,N,U,UV],^ _  9 : : JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h KK-- . fryy*<= > >6;;CZ KK $D[ >r7c&<V^8dQh/S[;R&#r)r )r2r3s"r4r5rhs r7r8)imagetext)r9r:r;r<base_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modules_skip_keys_device_placementrrdr_can_record_outputsr/no_gradr r>r?r@rrs@@r4rrhsn&(&+#35ST#4"5-.5  ]]_%%%r7rc\aa]tRtRtoRtV3Rlt]R4tRRltV3Rlt Rt Vt V;t #) BridgeTowerVisionModelicf<\SV`V4\V4VnVP 4R#ry)rTrUrvisual post_initras&&r4rUBridgeTowerVisionModel.__init__s&  26:  r7cjVPPPPP#ry)r*rrrrnrbs&r4rnBridgeTowerVisionModel.dtypes${{%%55<<BBBr7c XVPVPVP4W#4#ry)r*typern)rbr image_maskrrVs&&&&,r4r}BridgeTowerVisionModel.forwards {{5::djj1:XXr7c&<V^8dQh/S[;R&#rr)r2r3s"r4r5#BridgeTowerVisionModel.__annotate__s ##r7)r*)r)NF) r9r:r;r<r!rUpropertyrnr}r>r?r@rrs@@r4r(r(s3! CCYr7r(a0 The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in *Attention is all you need*_ by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. .. _*Attention is all you need*: https://huggingface.co/papers/1706.03762 caa]tRtRtoR tR V3RlltRtRt]] ] R V3RlRll444t Rt V3Rlt R tVtV;t#) BridgeTowerTextModelic <\SV`V4WnRVn\ V4Vn\ V4VnV'd \V4MRVn VP4R#)z^ add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer FN) rTrUrcgradient_checkpointingrrrencoderrDpoolerr+)rbrcadd_pooling_layerrds&&&r4rUBridgeTowerTextModel.__init__sT   &+#3F;-f5 3D'/$  r7c.VPP#ryrrr.s&r4get_input_embeddings)BridgeTowerTextModel.get_input_embeddingss...r7c&WPnR#ryr@rbrTs&&r4set_input_embeddings)BridgeTowerTextModel.set_input_embeddingss */'r7cv<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[PR,R S[R,R S[R,R S[S[,R S[/ #) r(rNrhrrrrrrtrrVr)r/rjr rsrrr)r2r3s"r4r5!BridgeTowerTextModel.__annotate__s9 9 <<$&9  t+9  t+ 9 llT) 9 ||d* 9  %||d29 !& t 39 9 $;9 +,9  69 r7c  DVRJVRJ, 'd \R4hVPP'gRp V 'd9Vf5\\ VPR7\ VPR74pVeVP 4M^p VP VVVVV R7p VPVVV VVR7wr'VP!V 3RVRVRVR VR V R V/V Bp V ^,pVPeVPV4MRp\VVV PR 7#) Nz:You must specify exactly one of input_ids or inputs_embedsF)rc)rrrrr)rhrembedding_outputrrtrhrrrtrr)rr+rt) rrcrnr r get_seq_lengthr_create_attention_masksr;r<rrt)rbrrhrrrrrrtrrVrrJencoder_outputssequence_outputrNs&&&&&&&&&&, r4r}BridgeTowerTextModel.forwardsa&  -t"; < <YZ Z{{%%%I 01,dkk2RT`hlhshsTtuOETE`!?!?!Afg??%)'#9 + 261M1M)#9-"7+ 2N2 .,,   )  #8  $:  ,    &    *!,8< 8O O4UY ;-'+;;  r7cVPP'd\VPVVVR7pM\VPVVR7pVe\VPVVVR7pW3#))rcrrhrtrcrrh)rcrrhr)rcrnrr )rbrhrrJrrts&&&&&&r4rL,BridgeTowerTextModel._create_attention_maskssu ;; ! ! !/{{.- / N7{{.-N " -%>{{.5&; & "55r7c&<V^8dQh/S[;R&#r)r!)r2r3s"r4r5rHs "!r7)rcrr;r:r<)r)T) NNNNNNNNN)r9r:r;r<r!rUrArErrrr}rLr>r?r@rrs@@r4r8r8sS !"/0 9 9  9 x6Wr7r8zv The bare BridgeTower Model transformer outputting BridgeTowerModelOutput object without any specific head on caa]tRtRtoV3RltRtRtV3RlRltV3RlRlt] ] R V3R lR ll44t R t R t VtV;t#)BridgeTowerModeli2c , <\SV`V4WnVPpVPpVP 'db\ P!VPVP4Vn \ P!VPVP4Vn M\ P!\VP4Uu.uF.p\ P!VPVP4NK0 up4Vn \ P!\VP4Uu.uF.p\ P!VPVP4NK0 up4Vn \ P!^VP4Vn\!V4Vn\%V4VnVP('gVP*'dVP"P,P.FpVP"P,P0P2P4VP2nVP"P,P0P6P4VP6nK \ P!\VP4Uu.uFp\9V4NK up4Vn\ P!\VP4Uu.uFp\9V4NK up4Vn\?V4Vn \?V4Vn!\ PD!VPVPFR7Vn$\ PD!VPVPFR7Vn%VPL'd"\OV4Vn(\OV4Vn)M\ P!\VP^, 4Uu.uFp\OV4NK up4Vn(\ P!\VP^, 4Uu.uFp\OV4NK up4Vn)VPU4R#uupiuupiuupiuupiuupiuupi)r(rON)+rTrUrc vision_config text_config$share_cross_modal_transformer_layersrr]rWcross_modal_text_transformcross_modal_image_transformrrrrrr( vision_modelr8 text_modelr"init_layernorm_from_vision_encoderr*cross_modal_ln_separaterrdatarrcross_modal_image_layerscross_modal_text_layersrDcross_modal_image_poolercross_modal_text_poolerrYrZcross_modal_text_layernormcross_modal_image_layernormshare_link_tower_layersrcross_modal_text_link_towercross_modal_image_link_towerr+)rbrcrWrXrrrds&& r4rUBridgeTowerModel.__init__8sa   ,, ((  6 6 6.0ii 8O8OQWQcQc.dD +/1yy9R9RTZTfTf/gD ,.0mmQVW]WoWoQpqQpA;22F4F4FGQpq/D +02}}SXY_YqYqSrsSra=44f6H6HISrs0D ,&(\\!V5G5G%H"2=A.{;,,,1Z1Z1Z''..FF!%!2!2!9!9!A!A!H!H!M!M #0077??DDII G)+ =B6C[C[=\ ]=\ &{ 3=\ ]) %(*}}=B6C[C[=\ ]=\ &{ 3=\ ]( $ ):&(A%'8'@$+-,,v7I7IvOdOd*e'+-<<8J8JPVPePe+f(  ) ) )/CF/KD ,0DV0LD -/1}}7>> from transformers import BridgeTowerProcessor, BridgeTowerModel >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> # prepare image and text >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> text = "hello world" >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base") >>> model = BridgeTowerModel.from_pretrained("BridgeTower/bridgetower-base") >>> inputs = processor(image, text, return_tensors="pt") >>> outputs = model(**inputs) >>> outputs.keys() odict_keys(['text_features', 'image_features', 'pooler_output']) ```NzYBridgeTowerModel does not use `inputs_embeds`. Make sure to pass in `input_ids` instead.)rrmrQrr)rhr)r)r*r+r,r-)r:rLrNrr)(rrr]rrr/onesrror rclenr;r|rr\r*rr1rnrrrrrtrr expand_asrerxfullrfrbrarrhriget_cls_featuresr&r1)*rbrrhrrr{rrHr|r}rrVall_hidden_states_textall_hidden_states_imageall_hidden_states_crossall_self_attentionsr}rGextend_text_masks split_indexr|rimage_embeds_with_lncross_modal_texttext_token_type_embeddingsimage_token_type_embeddingscross_modal_imageextend_image_maskslayer_outputs_textcross_text_featureslayer_outputs_imagecross_image_featureslink_layer_indexrtext_link_towerimage_link_towertransformed_text_embedscross_text_features_cross_image_features_r)r* cls_featuress*&&&&&&&&&&&, r4r}BridgeTowerModel.forwards@\"$"$"$   $):%k  488qnn& oo0090E %%k2  !"ZZ 5::iN^N^_N5;;%i0) $//117784;;;X;XX[\\ __,,22?K " ) )+ 6A  ,,33??!!$"3"3"9"9:Um@L (//1a8L&&|4&&--99CCL[QE .L # * *< 8R $0077DD\EVEVW[WhWhWnWnEop 55kQ5O%)%?%? KKI4D4D E& )$ % # ::;KNh;hi#::;O[\:]&*&@&@ JJt1IL\L\ ]' )( ) $ 46QQ <<=QRZZ  # #A &(9(>(>q(A B**## 7;;+%  "99!<  ,#5   13";;A>  -#4   315&&(;=Q'RS""$6q$9;Nq;Q#RS{C(?(?(E(E$FGA//1177:;HYZK,,33??II!L\Z__!!''L++D,=,=,D,D,Q,QR^,_aqtuauv-. ! #>>?OPO#@@AQR '+&@&@N^abNb&c ##2'*DD#!$ %55IK_as$t !"&!=!=>NQR>R!S$%0'9 "  #5Q"7 "&"?"?@PST@T"U%$1'8 #  $7q#9  !  " ) )+ 6 # * *< 8 # * *,?AU+V W  & &(:1(=?RST?U'V W[H`)<=Q~ ,,]NK %')&,--.-. 01  r7cxVPV4pVPV4p\P!W4.RR7#)rrr)rdrcr/r)rbr)r*cls_features_textcls_features_images&&& r4r!BridgeTowerModel.get_cls_featuresRs9 88G!::>Jyy+@bIIr7)rcrfrarircr[rerbrhrdrZr]rr\) NNNNNNNNNF)r9r:r;r<rUrArErtrxrrr}rr?r@rrs@@r4rUrU2s^ 6p64II JJ N N N `JJr7rUc8aa]tRtRtoV3RltRtRtVtV;t#)"BridgeTowerPredictionHeadTransformiYc<\SV`4\P!VPVP4Vn\ VP\4'd\VP,Vn MVPVn \P!VPVPR7Vn R#r) rTrUrr]rWrr/r0r1rtransform_act_fnrYrZras&&r4rU+BridgeTowerPredictionHeadTransform.__init__Zs~ YYv1163E3EF f'' - -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STr7clVPV4pVPV4pVPV4pV#ry)rrrYr9s&&r4r}*BridgeTowerPredictionHeadTransform.forwardcs4 =1 --m< }5 r7)rYrrrrs@@r4rrYsUr7rc<aa]tRtRtoRV3RlltRtRtVtV;t#)rijcz<\SV`4Wn\V4Vn\ P !VPVPPRR7Vn \ P!\P!VPP44VnVeW PnR#R#)F)rN)rTrUrcr transformrr]rWrXrdecoderrr/rrr)rbrcrrds&&&r4rUBridgeTowerMLMHead.__init__ks  ;FCyy!3!3V5G5G5R5RY^_ LLV-?-?-J-J!KL  "(LL  r7clVPV4pVPV4VP,pV#ry)rrr)rbx mlm_scores&& r4r}BridgeTowerMLMHead.forwardts-NN1% LL+dii7 r7)rrcrrryrrs@@r4rrjs)r7rc8aa]tRtRtoV3RltRtRtVtV;t#)BridgeTowerITMHeadizc\<\SV`4\P!V^4VnR#r(NrTrUrr]fc)rbrWrds&&r4rUBridgeTowerITMHead.__init__{s  ))K+r7c(VPV4pV#ryr)rbr itm_scores&& r4r}BridgeTowerITMHead.forwardsGGAJ r7rrrs@@r4rrzs,r7rz\ BridgeTower Model with a language modeling head on top as done during pretraining. cpaa]tRtRtoRR/tV3RltRtRt]] R V3RlRll44t R t Vt V;t #) BridgeTowerForMaskedLMizmlm_score.decoder.weightz8bridgetower.text_model.embeddings.word_embeddings.weightc<\SV`V4\V4Vn\ V4VnVP 4R#ry)rTrUrUrrrr+ras&&r4rUBridgeTowerForMaskedLM.__init__s5  +F3+F3 r7c.VPP#ryrrr.s&r4get_output_embeddings,BridgeTowerForMaskedLM.get_output_embeddingss~~%%%r7c&WPnR#ryr)rbnew_embeddingss&&r4set_output_embeddings,BridgeTowerForMaskedLM.set_output_embeddingss !/r7cv<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[PR,R S[PR,R S[S[,R S[/ # r(rNrhrrr{rrHr}rVr)r/rr0rrr)r2r3s"r4r5#BridgeTowerForMaskedLM.__annotate__sH H ##d*H ))D0H ((4/ H ''$. H $$t+ H ((4/H ''$.H   4'H +,H  H r7c  VP!R RVRVRVRVRVRVRV/V Bp VPV P4p Rp Vel\4p VP V P 4pV !V P R VPPP4VP R 44p \V V V PV PR 7#) a image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import BridgeTowerProcessor, BridgeTowerForMaskedLM >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> url = "http://images.cocodataset.org/val2017/000000360943.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())).convert("RGB") >>> text = "a looking out of the window" >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") >>> model = BridgeTowerForMaskedLM.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> results = processor.decode(outputs.logits.argmax(dim=-1).squeeze(0).tolist()) >>> print(results) .a cat looking out of the window. ```rrhrrr{rrHNrErFr,r-r8r) rrr)rrrrorrcrXrrr,r-)rbrrhrrr{rrHr}rVoutputs mlm_logitsmasked_lm_lossloss_fcts&&&&&&&&&, r4r}BridgeTowerForMaskedLM.forwardsd""    )  *  &  "  (  &    ^^G$9$9:   ')HYYz001F%joob$++:Q:Q:\:\&]_e_j_jkm_noN!//))   r7)rrNNNNNNNN)r9r:r;r<_tied_weights_keysrUrrrrr}r?r@rrs@@r4rrsL 56pq&0H H H H r7rz BridgeTower Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for image-to-text matching. c\aa]tRtRtoV3Rlt]]RV3RlRll44tRtVt V;t #)#BridgeTowerForImageAndTextRetrievalic<\SV`V4\V4Vn\ VP ^,4VnVP4R#r)rTrUrUrrrWrr+ras&&r4rU,BridgeTowerForImageAndTextRetrieval.__init__s@  +F3+F,>,>,BC r7cv<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[PR,R S[PR,R S[S[,R S[/ #r)r/rr0rrr)r2r3s"r4r50BridgeTowerForImageAndTextRetrieval.__annotate__sG G ##d*G ))D0G ((4/ G ''$. G $$t+ G ((4/G ''$.G   4'G +,G  "G r7c  (VP!R RVRVRVRVRVRVRV/V Bp V Pp VPV 4p Rp Ve.\4pVP V P 4pV!W4p \ V V V PV PR 7#) a image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size, 1)`, *optional*): Labels for computing the image-text matching loss. 0 means the pairs don't match and 1 means they match. The pairs with 0 will be skipped for calculation. Examples: ```python >>> from transformers import BridgeTowerProcessor, BridgeTowerForImageAndTextRetrieval >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"] >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") >>> model = BridgeTowerForImageAndTextRetrieval.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") >>> # forward pass >>> scores = dict() >>> for text in texts: ... # prepare inputs ... encoding = processor(image, text, return_tensors="pt") ... outputs = model(**encoding) ... scores[text] = outputs.logits[0, 1].item() ```rrhrrr{rrHNrr8) rr+rrrrrorr,r-)rbrrhrrr{rrHr}rVrr+rFitm_lossrs&&&&&&&&&, r4r}+BridgeTowerForImageAndTextRetrieval.forwards\""    )  *  &  "  (  &     --  .  ')HYYv}}-F/H'!//))   r7)rrr r9r:r;r<rUrrr}r?r@rrs@@r4rrs4G G G G r7rc8aa]tRtRtoV3RltRtRtVtV;t#)BridgeTowerContrastiveHeadiEcZ<\SV`4\P!W4VnR#ryr)rbrW embed_sizerds&&&r4rU#BridgeTowerContrastiveHead.__init__Fs ))K4r7c(VPV4pV#ryr)rbrs&&r4r}"BridgeTowerContrastiveHead.forwardJs GGAJr7rrrs@@r4rrEs5r7rzl BridgeTower Model with a image-text contrastive head on top computing image-text contrastive loss. c\aa]tRtRtoV3Rlt]]RV3RlRll44tRtVt V;t #)riOc<\SV`V4\V4Vn\ VP VP 4Vn\ VP VP 4Vn\ VP ^,VP 4Vn \P!\P!VPP44VnVP#4R#r)rTrUrUrrrWcontrastive_hidden_size itc_text_headitc_image_headitc_cross_modal_headrrr/rrcrrr+ras&&r4rU*BridgeTowerForContrastiveLearning.__init__Us  +F378J8JFLjLjk89K9KVMkMkl$>v?Q?QTU?UW]WuWu$v!<< T[[5W5W(XY r7cb<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[PR,R S[R,R S[S[,R S[/ #) r(rNrhrrr{rrH return_lossrVr)r/rr0rsrrrC)r2r3s"r4r5.BridgeTowerForContrastiveLearning.__annotate__dss s ##d*s ))D0s ((4/ s ''$. s $$t+ s ((4/s ''$.s D[s +,s  &s r7c  BV PRR4VP!RRVRVRVRVRVRVR V/V Bp V Pp V PwrpV R,pV R,pVPPP P V4pVPP\P!R^\PVPPPPR 74PV4pVPPV4V,p\P P#VP%VR,4R^R 7p\P P#VP'VR,4R^R 7P)VPR 7p\P P#VP+V 4R^R 7P)VPR 7p\P,!WV.RR 7pVP.P14P)VPR 7p\P2!WP544V,p\P2!VVP544V,p\P2!VVP544V,pRpV'd\P6!\9V4VPR 7p\P P;VV4p\P P;VV4p\P P;VV4pVV,V,R, p\=VVVVVV PV P>R7#)a image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. Examples: ```python >>> from transformers import BridgeTowerProcessor, BridgeTowerForContrastiveLearning >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> import torch >>> image_urls = [ ... "https://farm4.staticflickr.com/3395/3428278415_81c3e27f15_z.jpg", ... "http://images.cocodataset.org/val2017/000000039769.jpg", ... ] >>> texts = ["two dogs in a car", "two cats sleeping on a couch"] >>> with httpx.stream("GET", urls[0]) as response: ... image1 = Image.open(BytesIO(response.read())) >>> with httpx.stream("GET", urls[1]) as response: ... image2 = Image.open(BytesIO(response.read())) >>> images = [image1, image2] >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-large-itm-mlm-itc") >>> model = BridgeTowerForContrastiveLearning.from_pretrained("BridgeTower/bridgetower-large-itm-mlm-itc") >>> inputs = processor(images, texts, padding=True, return_tensors="pt") >>> loss = model(**inputs, return_loss=True).loss >>> inputs = processor(images, texts[::-1], padding=True, return_tensors="pt") >>> loss_swapped = model(**inputs, return_loss=True).loss >>> print("Loss", round(loss.item(), 4)) Loss 0.0019 >>> print("Loss with swapped images", round(loss_swapped.item(), 4)) Loss with swapped images 2.126 ```output_hidden_statesTrrhrrr{rrHrm)rrZ)rorNg@)rErFrGrHrIr,r-r8rr)rrLr) setdefaultrr+r,r\r*rrr/rrrrorr[rr normalizerrrrrrrexpr]trr cross_entropyrCr-)rbrrhrrr{rrHrrVrr+hidden_states_txthidden_states_imghidden_states_cross_modalrGrrrIrFrlogits_text_to_imagelogits_text_to_crosslogits_image_to_crossitc_lossr}text_to_image_losstext_to_cross_lossimage_to_cross_losss&&&&&&&&&, r4r})BridgeTowerForContrastiveLearning.forwardbs9v 0$7""    )  *  &  "  (  &     -- JQJ_J_G.G'+ (, #//<<CCPPQ]^&*&6&6&L&L JJtQejj9I9I9_9_9f9f9m9m n' )( ) $''CCDXY\ww mm--d.@.@WAU.V\^bc-d }}..t/B/B>;K L{ Z \\#f+fmmDF!#!SU["\ *-??BUUY\\H+#%%!//))  r7)rrrrrrrrs@@r4rrOs4 s s s s r7r)Nrx)rrrrUr)Yr= collectionsrcollections.abcr dataclassesrr/rtorch.nnrrr  activationsrr cache_utilsr r r masking_utilsr rmodeling_layersrmodeling_outputsrrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsrrrr utils.genericrrutils.output_capturingrconfiguration_bridgetowerr r!r" get_loggerr9logger_TOKENIZER_FOR_DOCr&rCrWrLrrrrrr,r<rDrbrdrrrrrrrr(r8rUrrrrrrr__all__r8r7r4rs( #$! %&6CCJ9G&6KKI5hh   H %'  7[7  7$  7;7  74)299)XRYY6P"))Pf7"2997"td299d4 BII  bii      %:@)ryy@)HI) I)Z.299.:8 8v?5?F RYY Fc8 c8L)%)%)%XY7Y"  {65{6 {6| ^J1^J  ^JD "    \ 7\  \ ~ T *DT  T n C (BC  C L r7