+ j^RIt^RIHt^RIHt^RIt^RIHt^RIHuH t ^RI H t ^RI Ht^RIHt^RIHtHt^RIHt^R IHtHt^R IHt^R IHt^R IHtH t H!t!H"t"H#t#^R I$H%t%H&t&^RI'H(t(H)t)^RI*H+t+^RI,H-t-H.t.H/t/H0t0H1t1^RI2H3t3^RI4H5t5H6t6^RI7H8t8^RI9H:t:H;t;]0Px!]=4t>!RR]P~4t@!RR]P~4tA!RR]PP~4tB!RR]P~4tC!RR]P4tE!RR ]P~4tF!R!R"]P~4tGR#R$ltHR%R&ltIRTR'R(lltJRTR)R*lltK!R+R,]P~4tL!R-R.]4tM].!R/R0])44tN].!R1R2]N44tO!R3R4]N]4tP].!R5R67]!R7R8]#444tQ!R9R:]PP~4tR!R;R<]P~4tSR=tT!R>R?]P~4tUR@RAltVRBRCltW!RDRE]P~4tX!RFRG]P~4tY!RHRI]4tZ!RJRK]P~4t[!RLRM]P~4t\!RNRO]P~4t]!RPRQ]N4t^!RRRS]N]4t_.RUOt`R#)VN)Callable) dataclassLlama4VisionConfig)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)create_causal_maskcreate_chunked_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPastBaseModelOutputWithPoolingCausalLMOutputWithPast ModelOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleloggingtorch_compilable_check)deprecate_kwarg)maybe_autocastmerge_with_config_defaults)capture_outputs) Llama4ConfigLlama4TextConfigcPaa]tRt^7toV3RlV3RlltV3RlRltRtVtV;t#)Llama4TextExpertsc <V^8dQhRS[/#configr$)format __classdict__s"k/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/llama4/modeling_llama4.py __annotate__Llama4TextExperts.__annotate__8s00/0c0<\SV`4VPVnVPVnVP VnVPVn\P!\P!VPVP ^VP ,44Vn \P!\P!VPVP VP 344Vn \VP,VnR#r)N)super__init__num_local_experts num_expertsintermediate_size hidden_size expert_dimnn Parametertorchzeros gate_up_projempty down_projr hidden_actact_fnselfr* __class__s&&r.r5Llama4TextExperts.__init__8s !33!'!9!9!--00LLT5E5EtGWGWYZ]a]l]lYl)mnekk43C3CT__VZVfVf2g&hiV../ r1cN<V^8dQhRS[PRS[P/#r) hidden_statesreturnr=Tensor)r,r-s"r.r/r0Bs#U\\ellr1c VPVPP^,RVP4p\P !WP4pVP ^RR7wr4\P !W@PV4,VP4pVPRVP4pV#)a This should really not be run on a single machine, as we are reaching compute bound: - the inputs are expected to be "sorted" per expert already. - the weights are viewed with another dim, to match num_expert, 1, shape * num_tokens, shape Args: hidden_states (torch.Tensor): (batch_size * token_num, hidden_size) selected_experts (torch.Tensor): (batch_size * token_num, top_k) routing_weights (torch.Tensor): (batch_size * token_num, top_k) Returns: torch.Tensor dim) viewr?shaper9r=bmmchunkrCrA)rErJgate_upgateup next_statess&& r.forwardLlama4TextExperts.forwardBs&**4+<+<+B+B1+Er4K[K[\ ))M+<+<====+iikk$&7!7$..I !&&r4+;+;< r1)rCrAr:r?r9r8r7 __name__ __module__ __qualname____firstlineno__r5rZ__static_attributes____classdictcell__ __classcell__rFr-s@@r.r&r&7s00r1r&c<aa]tRt^XtoRV3RlltRtRtVtV;t#) Llama4TextMLPcv<\SV`4Vf VPpWn\P !VP VRR7Vn\P !VP VRR7Vn\P !W!P RR7Vn \VP,Vn R#)NFbias) r4r5r8r*r;Linearr9 gate_projup_projrArrB activation_fn)rEr*r8rFs&&&r.r5Llama4TextMLP.__init__Ys   $ & 8 8  6#5#57HuUyy!3!35FUS #46H6HuU#F$5$56r1cVPVPV44VPV4,pVPV4#N)rmrkrlrA)rExrAs&& r.rZLlama4TextMLP.forwardes7&&t~~a'89DLLOK ~~i((r1)rmr*rArkrlrpr\rds@@r.rfrfXs 7))r1rfcTaa]tRt^jtoRV3RlV3RllltRtRtRtRtVt V;t #)Llama4TextL2Normc <V^8dQhRS[/#)r)eps)float)r,r-s"r.r/Llama4TextL2Norm.__annotate__ksEr1c0<\SV`4WnR#rp)r4r5rv)rErvrFs&&r.r5Llama4TextL2Norm.__init__ks r1cV\P!VP^4PRRR7VP,4,#r)T)keepdimrQr=rsqrtpowmeanrvrErqs&&r._normLlama4TextL2Norm._normo45;;quuQx}}R}>IJJJr1c^VPVP44PV4#rp)rrwtype_asrs&&r.rZLlama4TextL2Norm.forwardrs"zz!'')$,,Q//r1c RVP 2#)zeps=rvrEs&r. extra_reprLlama4TextL2Norm.extra_reprusdhhZ  r1r)gư> r]r^r_r`r5rrZrrarbrcrds@@r.rtrtjs$K0!!r1rtcHaa]tRt^ytoRV3RlltRtRtRtRtVt V;t #)Llama4TextRMSNormc <\SV`4W n\P!\ P !V44VnR#)z, Llama4RMSNorm is equivalent to T5LayerNorm N)r4r5rvr;r<r=onesweight)rEr9rvrFs&&&r.r5Llama4TextRMSNorm.__init__zs. ll5::k#:; r1cV\P!VP^4PRRR7VP,4,#r|r~rs&&r.rLlama4TextRMSNorm._normrr1cVPVP44PV4pW P,#rp)rrwrr)rErqoutputs&& r.rZLlama4TextRMSNorm.forwards.AGGI&..q1 ##r1c^\VPP4 RVP 2#)z, eps=)tuplerrSrvrs&r.rLlama4TextRMSNorm.extra_reprs' ))*+6$((<rrds@@r.rrys<K$==r1rc>aa]tRt^toV3RltV3RltRtVtV;t#) Llama4Routerc<\SV`VPVPRR7VPVnVP VnR#FrhN)r4r5r9r6r7num_experts_per_toktop_krDs&&r.r5Llama4Router.__init__s> ++V-E-EER!33// r1cp<\SV`V4p\P!W P^R7wr4\P !V\ R44P^WC4p\PPPVP 44PVP4pWR3#)rOz-inf) r4rZr=topkr full_likerwscatter_r; functionalsigmoidtodtype)rErJ router_logitsrouter_top_valuerouter_indices router_scoresrFs&& r.rZLlama4Router.forwards 6 +0::mZZUV+W( uV}ENNqR`s ++33M4G4G4IJMMmNaNab ++r1)r7rr\rds@@r.rrs0 ,,r1rc8aa]tRt^toV3RltRtRtVtV;t#) Llama4TextMoec<\SV`4VPVnVPVnVP Vn\V4Vn \V4Vn \V4Vn R#rp)r4r5rrr9 hidden_dimr6r7r&expertsrrouterrf shared_expertrDs&&r.r5Llama4TextMoe.__init__s[ //  ,,!33(0 "6* *62r1cVPRVP4pVPV4wr#VPVP^,^4pWBP ^^4PR^4,pVP V4pVPV4pVPVPVP^,RVPR,4P^R74Wc3#)rrOrQ) reshaperrrepeatrS transposerradd_sum)rErJrr routed_in routed_outouts&& r.rZLlama4TextMoe.forwards%--b$//B '+{{='A$ !(()<)>Y &.t~~>L+7 V+L(( (u= !#8>>+;!Nr1cP<V^8dQhRS[RS[S[PS[3,/#)r)r*rK)r$rr=rMrw)r,r-s"r.r/rs/550@5\abgbnbnpubu\v5r1c 0VPR,p\VRR4;'gVPVP,pRpRV\P !^V^\P R7V, ,, pVPV4V3#)a~ Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimN?r)rgetattrr9num_attention_headsr=arangerwr)r*rkwargsbaserPattention_factorrs&&, r.r9Llama4TextRotaryEmbedding.compute_default_rope_parameterss~%%l3fj$/cc63E3EIcIc3c$5<<3#MPS#STU{{6"$444r1cVPR,P4PVP^,R^4pVR,P4p\ VP P \4'd2VP P R8wdVP P MRp\VRR7;_uu_4VPVP 4V,P^^4p\P!\P!V4V4pWpP,pRRR4V# +'giX#;i)NmpscpuF) device_typeenabled)NNNNNrQ)rNr)rrwexpandrS isinstancertypestrr rrr=polar ones_liker)rErq position_idsinv_freq_expandedposition_ids_expandedrfreqs freqs_ciss&&& r.rZ!Llama4TextRotaryEmbedding.forwards!MM-8>>@GG HZHZ[\H]_acde ,Z 8 > > @'1!((--'E'E!((--[`J`ahhmmfk  U C C&))!((36KKVVWXZ[\E EOOE$:EBI!$:$::ID  D C s A0E E )rr*rrrrrrp)r]r^r_r`rr5 staticmethodrr=no_gradrrZrarbrcrds@@r.rrsuXv.OO/O Xv.55/5& ]]_   r1rc V^8dQhR\PR\PR\PR\\P\P3,/#)r)xqxkrrKr=rMr)r,s"r.r/r/sN 2 2  2  2|| 2 5<< %& 2r1c&\P!VP4P!.VPRRORN^N5!4p\P!VP4P!.VPRRORN^N5!4p\P !W2R,,4P ^4p\P !WBR,,4P ^4pVPV4VPV43#)NrQ)rrNr)r=view_as_complexrwrrS view_as_realflattenr)rrrxq_xk_xq_outxk_outs&&& r.apply_rotary_embrs    2 2 IBHHSbM I2 Iq I JC    2 2 IBHHSbM I2 Iq I JC    &> > ? G G JF    &> > ? G G JF >>" v~~b1 11r1cdV^8dQhR\PR\R\P/#)r)rJn_reprK)r=rMint)r,s"r.r/r/s. U UU\\ U# U%,, Ur1cVPwr#rEV^8XdV#VR,PW#WV4pVPW#V,WE4#)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) )rrNrr)rSrr)rJr batchnum_key_value_headsslenrs&& r. repeat_kvrsU 2?1D1D.E z!"23::5W\dlmM  e(CT TTr1cV^8dQhR\PR\PR\PR\PR\PR,R\R\/# r)modulequerykeyvalueattention_maskNscalingdropoutr;Moduler=rMrw)r,s"r.r/r/g%% II% <<% % << % LL4' %  %%r1c \W P4p\W0P4p \P!WP ^^44V,p Ve W,p \ P PV RR7p \ P PWVPR7p \P!W4p V P ^^4P4p W3#)r)rOptrainingrQ) rnum_key_value_groupsr=matmulrr;rsoftmaxrr  contiguous rrrrrrrr key_states value_states attn_weights attn_outputs &&&&&&&, r.eager_attention_forwardr*s3 ; ;L==((6??([L,,|:K''1-88:K  $$r1cV^8dQhR\PR\PR\PR\PR\PR,R\R\/#rr)r,s"r.r/r/rr1c \W P4p\W0P4p \P!WP ^^44VP R,,p Ve W,p \ PPV RR7p \ PPWVPR7p \P!W4p V P ^^4P4p W3#)r)rOrrQ) rr!r=r"rrr;rr#rr r$r%s &&&&&&&, r.vision_eager_attention_forwardr.s3 ; ;L==((6??([L,,|:K''1-88:K  $$r1cXaa]tRtRtoRtV3RlV3RlltRV3RlRlltRtVtV;t #) Llama4TextAttentioni5z=Multi-headed attention from 'Attention Is All You Need' paperc <V^8dQhRS[/#r(r+)r,r-s"r.r/ Llama4TextAttention.__annotate__8sAA/Ar1c<\SV`4WnW n\ VRVP VP ,4VnVP VnVP VP,Vn VPVnVPR,Vn VPVn VPVn VPVn VPVnRVnVP V,Vn\$P&!VP VP VP,VP(R7Vn\$P&!VP VPVP,VP(R7Vn\$P&!VP VPVP,VP(R7Vn\$P&!VP VP,VP VP(R7VnVPP2'd1VP"'d\5VP64VnR#R#R#)rTrhNr-)r4r5r* layer_idxrr9rrrr!r attn_scale floor_scaleattn_temperature_tuningattention_dropout is_causalno_rope_layersuse_roper;rjattention_biasq_projk_projv_projo_proj use_qk_normrt rms_norm_epsqk_normrEr*r4rFs&&&r.r5Llama4TextAttention.__init__8s  " F4F4F&JdJd4de #)#=#= $*$>$>&B\B\$\!#)#=#= }}d*  ++!--'-'E'E$!'!9!9--i8 ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii  & & 68J8JQWQfQf   ;; " " "t}}}+F,?,?@DL(5 "r1cR<V^8dQhRS[PRS[S[PS[P3,RS[PR,RS[R,RS[S[,RS[S[PS[PR,S[S[P,R,3,/#)r)rJposition_embeddingsrNpast_key_valuesrrK)r=rMrr rr)r,r-s"r.r/r2Vs8)8)||8)#5<<#=>8) t+ 8)  8) -. 8) u||U\\D0% 2E2LL M8)r1c jVPRRp.VORNVPN5pVPV4PV4pVP V4P!.VORNVPN5!p VP V4PV4P ^^4p VP'd(\WVPVP44wr\VR4'd#VPV4pVPV 4p VP'Ed*VP'EgVeVPVP4M^p \ P"!VP^,VPR7V ,p \ P$!\ P&!V P)4R,VP*, 44VP,,R,p V P^VR,^^34P/.VO^N^N54p W,PVP04pVP ^^4pV P ^^4p VeVP3WVP4wr\4P6!VP8P:\<4pT!VVV V V3RVP>'gRM VP@RVPB/VBwppVPD!.VORN5!PG4pVPIV4pVV3#)NrCrrrrrQ)%rSrr=rRr>r?rr;rrrhasattrrCr7get_seq_lengthr4r=rlog1pfloorrwr6r5rrupdater get_interfacer*_attn_implementationr*r r8rrr$r@)rErJrGrrHr input_shape hidden_shape query_statesr&r'past_seen_tokens positions attn_scalesattention_interfacer)r(s&&&&&, r.rZLlama4TextAttention.forwardVs$))#2. 88b8$--8 {{=166|D [[/44UkU2Ut}}U {{=166|DNNqRST ==='7*=*@*@ATAT*U( $L 4 # #<< 5Lj1J  ' ' ' Q`Ql==dnnMrs  ]%8%8%;MDXDXY\llI EKK):S)@DDTDT(TUVY]YhYhhknn &**A{21+EFMMNbP[Nb]^Nb`aNbcK(6::<;M;MNL#--a3 ))!Q/  &'6'='=jX\XfXf'g $J(?(M(M KK , ,.E) %8      % $}}}C$2H2H % LL %  % ! \"));;;;FFH kk+. L((r1)r8r5r7r*r6rr9r>r4rr!rr@r=rCrr;r?rp r]r^r_r`__doc__r5rZrarbrcrds@@r.r0r05s#GAA<8)8)8)r1r0cHaa]tRtRtoV3RltRV3RlRlltRtVtV;t#)Llama4TextDecoderLayeric<\SV`4VPVnW n\ W4VnW!P 9VnVP'd\V4Vn M\WPR7Vn \VPVPR7Vn\VPVPR7VnR#))r8rN)r4r5r9r4r0 self_attn moe_layers is_moe_layerr feed_forwardrfintermediate_size_mlprrBinput_layernormpost_attention_layernormrDs&&&r.r5Llama4TextDecoderLayer.__init__s !--",V?%):)::     -f 5D  -fHdHd eD 01C1CI\I\](9&:L:LRXReRe(f%r1c<V^8dQhRS[PRS[PR,RS[PR,RS[R,RS[R,RS[S[PS[P3,R,RS[S[,R S[S[PS[S[PS[P3,R,3,/#) r)rJrNrrH use_cacherGrrK) r=rM LongTensorr boolrrr FloatTensor)r,r-s"r.r/#Llama4TextDecoderLayer.__annotate__s  ||  t+ &&-    $; #5<<#=>E -.  u  %(9(95;L;L(L"MPT"TT U r1c .TpVPV4pVP!RRVRVRVRVRV/VBwrW,pTpVPV4pVPV4pVP'dVwrWP VP 4,pV#)rJrGrrHri)rer`rfrcrbrRrS) rErJrrrHrirGrresidualattention_states_s &&&&&&&, r.rZLlama4TextDecoderLayer.forwards!,,]; #nn ' 3 * ,      !3 !55mD ))-8    , M #5#5hnn#EE r1)rcr9rerbr4rfr`)NNNFNr\rds@@r.r^r^s g   r1r^caa]tRtRtoRtRtR.tRtRtRt Rt Rt ] P!4V3Rl4tV3RltRtVtV;t#) Llama4PreTrainedModeliTrHFc <\SV`V4\VPR4'dVPPMVPP Pp\ V\4'dI\P!VPRVR7\P!VPRVR7R#\ V\4'd=\P!VPVPVP44R#\ V\ 4'd[\P!VP"VP$R7\P!VP&VP$R7R#R#)initializer_rangerK)rstd)rxN)r4 _init_weightsrLr*rw text_configrr&initnormal_r?rALlama4VisionRotaryEmbeddingcopy_freqs_ci_compute_freqs_ciLlama4VisionModelclass_embeddingscalepositional_embedding_vlm)rErrxrFs&& r.ry#Llama4PreTrainedModel._init_weightss f%t{{$788 KK ) )((:: f/ 0 0 LL,,3C @ LL)) =  ; < < JJv(@(@(O P  1 2 2 LL//V\\ B LL88fll K3r1c&<V^8dQh/S[;R&#r(r#)r,r-s"r.r/"Llama4PreTrainedModel.__annotate__s r1ro)imagetext)r]r^r_r`input_modalitiessupports_gradient_checkpointing_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendr=rry__annotate_func__rarbrcrds@@r.rurusZ)&*##4"5 N!"& ]]_LLr1rucaa]tRtRtoR.tRtR tR]R]R] /t V3RlV3Rllt ] ] ]RV3R lR ll444tV3R ltR tVtV;t#)Llama4TextModelir^model attentionsrJrc <V^8dQhRS[/#r(r+)r,r-s"r.r/Llama4TextModel.__annotate__s/r1c <\SV`V4VPVnVPVn\ P !VPVPVP4Vn\ P!\VP4Uu.uFp\W4NK up4Vn \VPVPR7Vn\#VR7VnRVnVP)4R#uupi)rr*FN)r4r5 pad_token_id padding_idx vocab_sizer; Embeddingr9 embed_tokens ModuleListrangenum_hidden_layersr^layersrrBnormr rotary_embgradient_checkpointing post_initrDs&& r.r5Llama4TextModel.__init__s  !.. ++LL):):FQ>QR 36B&+#  isDc<V^8dQhRS[PR,RS[PR,RS[PR,RS[R,RS[PR,RS[R,RS[S[,R S[S[ ,/#) r) input_idsNrrrH inputs_embedsrirrK) r=rjrMr rlrkrrrr)r,r-s"r.r/rs< < ##d*<  t+< &&- <  < ((4/ < $;< +,<  ( (< r1c VRJVRJ, 'd \R4hVf?VPVPVPPP44pV'dVf\ VP R7pVfbVeVP4M^p\P!VP^,VPR7V,pVP^4p\T;p \4'g/RVP RVRVRVRV/p R \R/V BR \R/V B/p Tp VP!W4p \#VP$RVP P&4F9wrV!V 3RWP P(V ,,RVRVR VR V /VBp K; VP+V 4p \-T V'dVR 7#RR 7#)N:You must specify exactly one of input_ids or inputs_embedsrrJr*rrrHrfull_attentionchunked_attentionrirG)last_hidden_staterHro) ValueErrorrrrrr r*rMr=rrS unsqueezerdictr r r enumeraterr layer_typesrr)rErrrrHrrirrVcausal_mask_mapping mask_kwargsrJfreq_cisi decoder_layers&&&&&&&, r.rZLlama4TextModel.forwards  -t"; < <YZ Z   --ill4;L;L;S;S;Z;Z.[\M 0*$++>O  CRC^==?de  << (;(;A(>}G[G[\_ooL'11!4L?-FF$++ .!? K!"4"C{"C#%?%N+%N#  & ??=? )$++6U 8U8U*V W A)2;;3J3J13MN*!0  $  %- M!X -0 &+/8O  >B  r1c&<V^8dQh/S[;R&#r(r+)r,r-s"r.r/rs  r1)rrrrrrr)r)NNNNNN)r]r^r_r`_no_split_modulesbase_model_prefixrr0r^r_can_record_outputsr5r!r"rrZrrarbrcrds@@r.rrso12  )/   < <  < ?r1rcaa]tRtRtoR.tRtRR/tRR/tRR/tV3R lV3R llt ] ] RV3R lR ll44t V3R lt RtVtV;t#)Llama4ForCausalLMi@r^language_modelzlm_head.weightzmodel.embed_tokens.weightlm_headkeep_full_weightcolwise_gather_outputc <V^8dQhRS[/#r(r+)r,r-s"r.r/Llama4ForCausalLM.__annotate__Hs/r1c<\SV`V4\V4VnVPVn\ P !VPVPRR7VnVP4R#r) r4r5rrrr;rjr9rrrDs&&r.r5Llama4ForCausalLM.__init__HsU  $V,  ++yy!3!3V5F5FUS  r1c`<V^8dQhRS[PR,RS[PR,RS[PR,RS[R,RS[PR,RS[PR,RS[R,R S[S[P,R S[S[,R S[ S[ ,/ #) r)rNrrrHrlabelsrilogits_to_keeprrK) r=rjrMr rlrkr rrrr)r,r-s"r.r/rSs: : ##d*:  t+: &&- :  : ((4/ :   4': $;: ell*: +,:  ' ': r1c  VP!R RVRVRVRVRVRV/V Bp V ^,p \V\4'd\V)R4MTp VP V RV R3,4p RpVe-VP !R R V R VR VP P/V Bp\VV V PV PV PR 7#)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, Llama4ForCausalLM >>> model = Llama4ForCausalLM.from_pretrained("meta-llama4/Llama4-2-7b-hf") >>> tokenizer = AutoTokenizer.from_pretrained("meta-llama4/Llama4-2-7b-hf") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```rrrrHrriNrlogitsrr)lossrrHrJrro) rrr slicer loss_functionr*rrrHrJr)rErrrrHrrrirroutputsrJ slice_indicesrrs&&&&&&&&&, r.rZLlama4ForCausalLM.forwardQsH**  ) & ,  (       8B>SV8W8W~ot4]k mA}a,?@A  %%pVpFpt{{OeOepiopD%#33!//))   r1c&<V^8dQh/S[;R&#r(r+)r,r-s"r.r/r@s  r1)rrr)NNNNNNNr)r]r^r_r`rr_tied_weights_keys _fsdp_plan_tp_planr5rrrZrrarbrcrds@@r.rr@sh12(*,GH/0J23H: : : 'r1rzQ Base class for Llava causal language model (or autoregressive) outputs.  custom_introcHa]tRtRtoRtRtRtRtRtRt Rt V3Rlt Rt Vt R#)Llama4CausalLMOutputWithPastia loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size (batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nc:<V^8dQh/S[PR,;R&S[PR,;R&S[R,;R&S[S[P,R,;R&S[S[P,R,;R&S[PR,;R&#)r)NrrrHrJrimage_hidden_states)r=rlr r)r,r-s"r.r/)Llama4CausalLMOutputWithPast.__annotate__s,   d ")-.    $+/0T\(12**+d2934e''(4/656**T187r1ro)r]r^r_r`r\rrrHrJrrrrarb)r-s@r.rrs2 &*D'+F$(O59M26J487r1rc8aa]tRtRtoV3RltRtRtVtV;t#)Llama4VisionMLP2ic<\SV`4VPVnVPVn\P !VPVP RR7Vn\P !VPVPRR7Vn \P!4Vn VPVn R#r)r4r5r9r8r;rjprojector_input_dimfc1projector_output_dimfc2GELUrmprojector_dropoutrrDs&&r.r5Llama4VisionMLP2.__init__s !--!'!9!999T33V5O5OV[\99V88&:U:U\abWWY// r1cVPV4pVPV4p\P!WPVPR7pVPVP V44#)r)rrmFrr rrErJs&&r.rZLlama4VisionMLP2.forwardsR/ **=9  -<<$--X !!$((="9::r1)rmrrrr9r8r\rds@@r.rrs0;;r1rc8aa]tRtRtoV3RltRtRtVtV;t#)Llama4MultiModalProjectoric<\SV`4\P!VPP VP PRR7VnR#r) r4r5r;rj vision_configvision_output_dimrzr9linear_1rDs&&r.r5"Llama4MultiModalProjector.__init__s?   2 2    * *  r1c(VPV4pV#rpr)rEimage_featuresrJs&& r.rZ!Llama4MultiModalProjector.forwards n5 r1rr\rds@@r.rrs r1rc pVPwr#p\\P!V44pVP W%VR4pVP 4wr&rtVP W&\Wq,4\WA, 44pVP ^^^^4P4pVP V\Wa,4\Wq,4\WA^,, 44pVP ^^^^4P4pVP VRVPR,4p V #)rrQ)rSr mathsqrtrRsizepermuter$) input_tensor shuffle_ratio batch_size num_patcheschannels patch_sizeheightwidthreshaped_tensor output_tensors && r. pixel_shuffler s(4(:(:%JXTYY{+,J$$ZZLL*6*;*;*='J"'' C@Ur?r@rDs&&r.r5Llama4VisionAttention.__init__s  ++33**f.H.HH $%!!'!9!9}}d* ii0NUYZ ii0NUYZ ii0NUYZ ii >UYZ r1c(<V^8dQhRS[PRS[PRS[PR,RS[R,RS[S[,RS[S[PS[PR,S[S[P,R,3,/#)r)rJrrNrHrrK)r=rMr rrr)r,r-s"r.r/r&s')')||'),,') t+ ')  ') -. ') u||U\\D0% 2E2LL M')r1c VPRRp.VORNVPN5pVPV4PV4pVP V4PV4p VP V4PV4p \ WVR7wrVP^^4pV P^^4p V P^^4p \P!VPP\4p T !VVV V R3RVP'gRM VPRRRR/VBwrV P!.VORN5!P!4p VP#V 4p W3#)N)rrrKrr9FrQ)rSrr=rRr>r?r"rrrQr*rRr.r r8rr$r@)rErJrrrHrrSrTrUr&r'rYr)r(s&&&&&, r.rZLlama4VisionAttention.forwardsp$))#2. 88b8$--8 {{=166|D [[/44\B {{=166|D #:<^f#g #--a3 ))!Q/ #--a3 (?(M(M KK , ,.L) %8      % $}}}C$2H2H %  %  %  % ! "));;;;FFH kk+. ((r1) r8r*r(rr>r)r!r@r=rr?NNr\rds@@r.r$r$s  [ [')')')r1r$cDaa]tRtRtoV3RltV3RlRltRtVtV;t#)Llama4VisionMLPi=c,<\SV`4Wn\P!4Vn\P !VPVPRR7Vn \P !VPVPRR7Vn R#)TrhN) r4r5r*r;rrmrjr9r8rrrDs&&r.r5Llama4VisionMLP.__init__>sc  WWY99V//1I1IPTU99V55v7I7IPTUr1cN<V^8dQhRS[PRS[P/#rIrL)r,r-s"r.r/Llama4VisionMLP.__annotate__E#U\\ellr1clVPV4pVPV4pVPV4pV#rp)rrmrrs&&r.rZLlama4VisionMLP.forwardEs4/ **=9 / r1)rmr*rrr\rds@@r.r0r0=sVr1r0cTaa]tRtRtoV3RlV3RlltRV3RlRlltRtVtV;t#)Llama4VisionEncoderLayeriLc <V^8dQhRS[/#r(r)r,r-s"r.r/%Llama4VisionEncoderLayer.__annotate__MsII1Ir1c<\SV`4VPVn\V4Vn\ V4Vn\P!VP4Vn \P!VP4Vn R#rp) r4r5r9r$r`r0rr; LayerNormrerfrDs&&r.r5!Llama4VisionEncoderLayer.__init__Msb !--.v6"6*!||F,>,>?(* V5G5G(H%r1c <V^8dQhRS[PRS[PRS[PR,RS[R,/#)r) hidden_staterrNoutput_attentions)r=rMrk)r,r-s"r.r/r;WsHll,, t+   $; r1cTpVPV4pVPVVVR7wrWQ,pTpVPV4pVPV4pWQ,pV3pV'd Wv3, pV#))rr)rer`rfr)rEr@rrrArpr(rs&&&&& r.rZ Llama4VisionEncoderLayer.forwardWs ++L9 %)^^ )&4& "  .  44\B xx - . /   &Gr1)r9rerrfr`r.r\rds@@r.r9r9Ls IIr1r9cXaa]tRtRtoRtV3RlV3RlltRV3RlRlltRtVtV;t #) Llama4VisionEncoderixz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`Llama4VisionEncoderLayer`]. Args: config: Llama4VisionConfig c <V^8dQhRS[/#r(r)r,r-s"r.r/ Llama4VisionEncoder.__annotate__s1r1c<\SV`4Wn\P!\ VP 4Uu.uFp\V4NK up4VnRVn WnR#uupi)FN) r4r5r*r;rrrr9rr)rEr*rrrFs&& r.r5Llama4VisionEncoder.__init__sY  mmuU[UmUmOn$oOn!%=f%EOn$op &+# %psA,c<V^8dQhRS[PRS[PRS[PR,RS[R,RS[R,RS[R,RS[S[,/#) r)rJrrNrAoutput_hidden_states return_dictrK)r=rMrkrr)r,r-s"r.r/rGss? ? ||? ,,?  t+ ?  $; ? #Tk ? D[?   ? r1c (VeTMVPPpVeTMVPPpVeTMVPPpV'dRMRpV'dRMRpVPFAp V'd Wq3,pV !VVVVR7p V'dW^,3,pV ^,pKC V'd Wq3,pV'g1\ ;QJd.RWV34FNK 5#!RWV344#\ WVR7#)a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. N)r@rrArc30"TF qfKVxK R#5irpro.0vs& r. .Llama4VisionEncoder.forward..se$Sq$S rrJrro)r*rArKrLrrr) rErJrrrArKrLencoder_statesall_attentions encoder_layer layer_outputss &&&&&&& r.rZLlama4VisionEncoder.forwards>2C1N-TXT_T_TqTq$8$D $++JjJj &1%>N5e]N$Se5 e5e]N$See e+Vd  r1)r*rrNNNNr[rds@@r.rErExs#? ? ? r1rEcDaa]tRtRtoV3RltV3RlRltRtVtV;t#)Llama4UnfoldConvolutionich<\SV`4VPp\V\4'dW"3p\ P PW!PR7Vn\ P!VPV^,,V^,,VPRR7Vn R#)) kernel_sizestrideFrhN) r4r5rrr r=r;Unfoldunfoldrj num_channelsr9linear)rEr*r_rFs&& r.r5 Llama4UnfoldConvolution.__init__s '' k3 ' '&4Khhoo+FWFWoX ii   +a. 0;q> A     r1cN<V^8dQhRS[PRS[P/#rIrL)r,r-s"r.r/$Llama4UnfoldConvolution.__annotate__r5r1cpVPV4pVP^^^4pVPV4pV#)r)rbrrdrs&&r.rZLlama4UnfoldConvolution.forwards8 M2 %--aA6  M2 r1)rdrbr\rds@@r.r]r]s  r1r]cTaa]tRtRtoV3RlV3Rllt]R4tRtRtVt V;t #)r}ic <V^8dQhRS[/#r(r)r,r-s"r.r/(Llama4VisionRotaryEmbedding.__annotate__sTT1Tr1c<\SV`4Wn\P!VP V4RR7VnR#)FrN)r4r5r*r;rrrrDs&&r.r5$Llama4VisionRotaryEmbedding.__init__s0   $"8"8"@US r1cRVPVP,p\P!V^,\PR7P V^,^4p\P !W"R,.^R7pRVR &W!,pW!,pVPVP,^,pRVPR,\P!^V^4RV^,P4V, ,, pV^,R ,VR ,,P^RR7pV^,R ,VR ,,P^RR7p\P !Wx.RR7P4P4R ,p V PVP R^^4^8^4p \P!\P!\P !V 4\P"!V 4.RR74p V #) r)r:NrNrOrrNrQ)rQrQ).N)NNr).:NNr)) image_sizerr=rint32rcatr9rrrwrepeat_interleaver$ masked_fillrstackcossin) r*idximg_idx frequencies_x frequencies_yfreq_dim rope_freqfreqs_xfreqs_yrrs & r.r-Llama4VisionRotaryEmbedding._compute_freqs_cis6#4#44,,sAvU[[9AA#q&!L))Wbk2:  %%)C)CCqH  " "< 0 Q!,->A ?EEG(R T "A%y1Im4LL__`agi_j!A%y1Im4LL__`agi_j 7,"5;;=HHJ8T!!'//"a";a"?C((eii6F RWHX5Y_a)bcr1cLVPPVP4#rp)rrrrs&&r.rZ#Llama4VisionRotaryEmbedding.forwards}} 4 455r1)r*r) r]r^r_r`r5rrrZrarbrcrds@@r.r}r}s/TT &66r1r}ctaa]tRtRtoRtR tR.tV3RlV3RlltRtR V3RlRllt V3R lt R t Vt V;t #) ri vision_modelr9c <V^8dQhRS[/#r(r)r,r-s"r.r/Llama4VisionModel.__annotate__s1r1c<\SV`V4VPVnVPVnVPVnVP VnVPVP,^,^,VnVPR,Vn\V4Vn \P!VP\P!VP4,4Vn\P!VP\P!VP VP4,4Vn\!V4Vn\P$!VP4Vn\P$!VP4Vn\+V4Vn\/V4VnVP34R#)r)Nr-)r4r5rqrr9rcrrr]patch_embeddingr;r<r=randnrrr}rotary_embeddingr= layernorm_prelayernorm_postrErr vision_adapterrrDs&&r.r5Llama4VisionModel.__init__sA   ++ ++!--"// OOt>1DqH''- 6v>!||DJJTEUEU9V,VW(* TZZ%++dN^N^`d`p`pBq5q(r% ;F C \\$*:*:; ll4+;+;<)0 9&A r1c VP#)zW This function is used to fetch the first embedding layer to activate grads on inputs. )rrs&r.get_input_embeddings&Llama4VisionModel.get_input_embeddingss###r1c<V^8dQhRS[PRS[PR,RS[R,RS[R,RS[R,RS[S[S[PR3,,/#) r) pixel_valuesrNrArKrLrK.)r=rMrkrr)r,r-s"r.r/r"s{b b llb  t+b  $; b #Tk b D[ b  $eELL#,=&> >b r1c VeTMVPPpVeTMVPPpVeTMVPPpVPwrxr^p ^p VP V4p V PwrpV P W{,V ,VV4p VPPV P^,^V PR,4p\P!V V.^R7p V^, pV P W{,WV4p VPPV PV PR7pV V,p VPV 4p V P!VRV4p VP#V4pVP%V RVVVR7pVP&p VP)V 4p V RRR1R3,p VP+V 4p V'd VP,MRpV'd V^,pMRpV'g3\.;QJd.RV VV34FNK 5#!RV VV344#\1V VVR7#) a Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, MllamaVisionModel >>> checkpoint = "meta-llama/Llama-3.2-11B-Vision" >>> model = MllamaVisionModel.from_pretrained(checkpoint) >>> processor = AutoProcessor.from_pretrained(checkpoint) >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, return_tensors="pt") >>> output = model(**inputs) >>> print(output.last_hidden_state.shape) torch.Size([1, 1, 4, 1025, 7680]) ``` NrOrr)rrKrArrc30"TF qfKVxK R#5irprorOs& r.rR,Llama4VisionModel.forward..~s_$Mq$MrTrUrQ)r*rArKrLrSrrrrr=rsrrrrrrRrrrrrrJrr)rErrrArKrLrbatch_size_times_num_tilesrcrrnum_concurrent_media num_chunksr@rrrrrpositional_embeddingrrrJrs&&&&&&, r.rZLlama4VisionModel.forward"s^D2C1N-TXT_T_TqTq$8$D $++JjJj &1%v%F"/0B0BC ,,77 4;; / / $ 8 8D  $ 7 7 D D J JD  r1c6VPP4#rp)rget_output_embeddingsrs&r.r4Llama4ForConditionalGeneration.get_output_embeddingss""88::r1c<VPPV4R#rp)rset_output_embeddings)rEnew_embeddingss&&r.r4Llama4ForConditionalGeneration.set_output_embeddingss 11.Ar1c<VPPV4R#rp)r set_decoder)rEdecoders&&r.r*Llama4ForConditionalGeneration.set_decoders ''0r1c6VPP4#rp)r get_decoderrs&r.r*Llama4ForConditionalGeneration.get_decoders""..00r1F)tie_last_hidden_stateszOObtains image last hidden states from the vision tower and apply al projection.rc f<V^8dQhRS[PRS[RS[S[,RS[S[,/#)r)rvision_feature_select_strategyrrK)r=rlrrrrr)r,r-s"r.r/rsE99''9),9+, 9 + + 9r1c VP4UUu/uF wrEVfK WEbK pppVP!V3/VB#uuppi)a9 pixel_values (`torch.FloatTensor` of shape `(batch_size, channels, height, width)`) The tensors corresponding to the input images. vision_feature_select_strategy (`str`): The feature selection strategy used to select the vision feature from the vision backbone. Can be one of `"default"` or `"full"` )itemsr)rErrrkrQs&&&, r.get_image_features1Llama4ForConditionalGeneration.get_image_featuressB $*<<>C>41Q$!$>C  888Ds==ch<V^8dQhRS[PRS[PRS[P/#)r)rrr)r=rjrl)r,r-s"r.r/rs7""))":?:K:K"]b]n]n"r1c  $VfoW P4!\P!RVPP\P VP R748HpVPR4pMWPP8HpVP4pVPR4PVP 4p\WRPR,,VP48HRV RVP^, 24V#)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. rz6Image features and image tokens do not match, tokens: z , features: rorQ)rr=fullr*image_token_idlongrallrrrrrSnumel)rErrrspecial_image_maskn_image_tokenss&&&& r.get_placeholder_mask3Llama4ForConditionalGeneration.get_placeholder_masks  !.2K2K2M 2t{{99TaThThi3" "4!7!7!; !*kk.H.H!H +//1/99"=@@AUAUV 004 48L8L8N ND^DTT`aoauauvwax`y z "!r1c<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[R,RS[PR,RS[R,R S[PR,R S[R,R S[R,R S[R,R S[R,RS[S[P,RS[S[ ,RS[ S[ ,/#)r)rNrrrrHrrrrirArKrLrrrK) r=rjrlrMr rrkr rrrr)r,r-s"r.r/rsp p ##d*p ''$.p  t+ p &&- p  p ((4/p ),d p   4'p $;p  $;p #Tkp D[p ell*p +,p - -!p r1c V eT MVPPp V eT MVPPp V eT MVPPp VRJVRJ, 'd \ R4hVeVe \ R4hVfVP 4!V4pVeVP VVRR7PpVPRVPR44pVPV4PVPVP4pVPWVR7pVPVV4pVP !RRVRVR VR VR V R V R V RV RV / VBpV^,pRpVe7VP"!RRVRVRVPP$P&/VBpV 'g"V3VR,,pVe V3V,#T#\)TTVP*VP,VP.VeXR7#RR7#)aA labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, LlavaForConditionalGeneration >>> model = LlavaForConditionalGeneration.from_pretrained("llava-hf/llava-1.5-7b-hf") >>> processor = AutoProcessor.from_pretrained("llava-hf/llava-1.5-7b-hf") >>> prompt = "USER: \nWhat's the content of the image? ASSISTANT:" >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, text=prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(**inputs, max_new_tokens=15) >>> processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "USER: \nWhat's the content of the image? ASSISTANT: The image features a busy city street with a stop sign prominently displayed" ```NrzdYou cannot specify both pixel_values and inputs_embeds at the same time, and must specify either oneT)rrrL)rrrrrHrrirArKrLrrrr:rNN)rrrHrJrrrQro)r*rArKrLrrrrrRrrrrrrmasked_scatterrrrzrrrHrJr)rErrrrrHrrrrirArKrLrrr vision_flatprojected_vision_flatrrrrrs&&&&&&&&&&&&&&, r.rZ&Llama4ForConditionalGeneration.forwardsd2C1N-TXT_T_TqTq$8$D $++JjJj &1%$>{$K$N$N$$m&9&9% !"&!:!:G\";" *889KMbcM%%  )  %  ,  (    0  "6  $  *      %%&,9=9P9P9[9[_eDY,F'+'7D7V# CV C+#33!//))2>2J   QU   r1c&<V^8dQh/S[;R&#r(r)r,r-s"r.r/rs  r1)rrrrr) NNNNNNNNNNNNr)r]r^r_r`rrrr5rrrrr!r"rrrrZrrarbrcrds@@r.rrs13MNH  ;B11 E2!rs9t3 9 "". E2p p 3 p [r1r)rK)rurrrr)arcollections.abcr dataclassesrr=torch.nnr;torch.nn.functionalrr/transformers.models.llama4.configuration_llama4rrr{ activationsr cache_utilsr r generationr masking_utilsr r modeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrrrutils.deprecationr utils.genericr r!utils.output_capturingr"configuration_llama4r#r$ get_loggerr]loggerrr&rfrtrrjrrrrrr*r.r0r^rurrrrrr r rr"r$r0r9rEr]r}rr__all__ror1r.rsi $! N&!.)KB9G&jj0G5@   H % B)BII)$ !uxx != =( ,299 ,"BII",4 4n 2 U%4%2Y)"))Y)x/7/dLOLL<Z +Z Z zM -M `  9;9  90;uxx;"  ( )")) )! :7)BII7)t bii )9)XO "))O dbii*6"))6