+ jls^RIt^RIHt^RIHt^RIt^RIHt^RIHt ^RI H t ^RI H t Ht^RIHtHt^R IHtHt^R IHtHt^R IHt^R IHtHtHtHt^R IH t ^RI!H"t"^RI#H$t$^RI%H&t&H't'^RI(H)t)H*t*H+t+H,t,H-t-H.t.H/t/H0t0H1t1^RI2H3t3]Ph!]54t6!RR]P Pn4t8!RR]04t9!RR]&4t:!RR])4t;!RR]$4t<!RR]Pn4t=!RR ]*4t>!R!R"].4t?!R#R$]-4t@]!R%R&]44tA!R'R(]/]A4tB!R)R*]+4tC!R+R,],4tD.R-OtER#).N)Callable)cycle)nn)initialization)ACT2FN)Cache DynamicCache)create_causal_maskcreate_recurrent_attention_mask)BaseModelOutputWithPast SequenceClassifierOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)merge_with_config_defaults)capture_outputs) BambaMixer)LlamaRotaryEmbeddingapply_rotary_pos_emb) ZambaAttentionZambaAttentionDecoderLayerZambaForCausalLMZambaForSequenceClassificationZambaHybridLayerZambaMambaDecoderLayer ZambaModel ZambaRMSNormeager_attention_forward Zamba2Configc@aa]tRt^3toRV3RlltRRltRtVtV;t#)Zamba2RMSNormGatedc<\SV`4\P!\P !V44VnW0nW nR#N) super__init__r Parametertorchonesweightvariance_epsilon group_size)self hidden_sizer0eps __class__s&&&&j/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/zamba2/modular_zamba2.pyr*Zamba2RMSNormGated.__init__4s2 ll5::k#:; #$cVPpVP\P4pVeDV\P P VP\P44,pVPEvrEWPP,pVP!.VOVNVPN5!pVP^4PRRR7pV\P!WP,4,pVP!.VOW`P,N5!pVPVPV4,#)NT)keepdim)dtypetor,float32r functionalsilushaper0viewpowmeanrsqrtr/r.) r1 hidden_statesgate input_dtype prefix_dimslast_dim group_counthidden_states_groupvariances &&& r5forwardZamba2RMSNormGated.forward:s#)) %((7  )BMM,>,>twwu}}?U,VVM!.!4!4//1 +00\+\{\DOO\&**1-222t2D1EKKK`K`@`4aa+00]+]{__?\] {{]--k:::r7)r0r/r.)gư>r( __name__ __module__ __qualname____firstlineno__r*rM__static_attributes____classdictcell__ __classcell__r4 __classdict__s@@r5r&r&3s% ; ;r7r&c]tRt^HtRtR#) Zamba2RMSNormNrPrQrRrSrTr[r7r5rZrZHr7rZc]tRt^LtRtR#)Zamba2RotaryEmbeddingr[Nr\r[r7r5r_r_Lr]r7r_c\aa]tRt^PtoRtRV3RlV3RllltRV3RlRlltRtVtV;t #)Zamba2Attentiona* Multi-headed attention from 'Attention Is All You Need' paper. Adapted from transformers.models.mistral.modeling_mistral.MistralAttention: The input dimension here is attention_hidden_size = 2 * hidden_size, and head_dim = attention_hidden_size // num_heads. The extra factor of 2 comes from the input being the concatenation of original_hidden_states with the output of the previous (mamba) layer (see fig. 2 in https://huggingface.co/papers/2405.16712). Additionally, replaced attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) / math.sqrt(self.head_dim) with attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) / math.sqrt(self.head_dim/2) Finally, this attention layer contributes to tied transformer blocks aimed to increasing compute without increasing model size. Because this layer is tied, un-tied adapters (formally the same as LoRA but used in the base model) modules are added to the q, k, v projectors to increase expressivity with a small memory overhead (see Fig. 2 of https://huggingface.co/papers/2411.15242). c \<V^8dQhRS[RS[R,RS[R,RS[R,/#)config layer_idxNnum_fwd_mem_blocksblock_idr$int)formatrXs"r5 __annotate__Zamba2Attention.__annotate__`sE'\'\'\:'\ $J '\ * '\r7c $<\S V`W4W0nVPVnW@nVP 'Ed\P!.4Vn \P!.4Vn \P!.4Vn \VP4EF0pWQP,V8XEd\P!\P!VP VP"P$RR7\P!VP"P$VP RR74p\P!\P!VP VP"P$RR7\P!VP"P$VP RR74p\P!\P!VP VP"P$RR7\P!VP"P$VP RR74pM?\P&!4p\P&!4p\P&!4pVPP)V4VPP)V4VPP)V4EK3 \+VP4U U u/uFwrWbK up p VnR#uup p i)FbiasN)r)r*rfhybrid_layer_idslayer_block_maprguse_shared_attention_adapterr ModuleListlinear_q_adapter_listlinear_k_adapter_listlinear_v_adapter_listrangenum_mem_blocks SequentialLinearattention_hidden_sizerd adapter_rankIdentityappend enumerate layer_dic) r1rdrerfrgilinear_q_adapterlinear_k_adapterlinear_v_adapterindexvaluer4s &&&&& r5r*Zamba2Attention.__init__`s +"4%66  . . .)+r):D &)+r):D &)+r):D &4223,,,8')}} $"<"V>V]bc $++":":DV>V]bc $++":":DV>V]bc $++":":DE 1)+,1) u||U\\D0% 2E2LL M1)r7c \VPRRp.VORNVPN5pVPV4p VPV4p VP V4p VP P 'dqVPV,p WPV ,!V4,p WPV ,!V4,p WPV ,!V4,p V PV4P^^4p V PV4P^^4p V PV4P^^4p VP P'dVwr\WW4wrVeVPWV4wr\ P"!VP P$\&4pT!VV V V V3RVP('gRM VP*RVP,/VBwppVP.!.VORN5!P14pVP3V4pVV3#)Ndropoutgscalingr:)r@head_dimq_projk_projv_projrdrrrrtrurvrA transpose use_mem_roperupdater get_interface_attn_implementationr"trainingattention_dropoutrreshape contiguouso_proj)r1rErerrrr input_shape hidden_shape query_states key_states value_statesadapter_layer_idxcossinattention_interface attn_output attn_weightss&&&&&&, r5rMZamba2Attention.forwards $))#2. 88b8$--8 {{=1 [[/ {{=1 ;; 3 3 3 $y 9 '*D*DEV*WXe*ffL#&@&@AR&STa&bbJ'*D*DEV*WXe*ffL#((6@@AF __\2<)r0r3N) rdr)r* use_conv_bias activationrSiLUact mamba_ngroupsn_groups mamba_headdimr n_mamba_heads num_heads chunk_size time_step_minfloattime_step_limitConv1dconv_dim mamba_d_convconv1dintermediate_sizerzr2add_bias_linearin_projr&normout_projlayer_norm_epsilonuse_bias)r1rdrerprojection_sizer4s&&&& r5r*Zamba2MambaMixer.__init__sk  ,DE#11 779,, ,, 22 ++ & 4 4eElCii ++==''!+   004==@4>>Qyy    ''  '  " "t/E/E/V\`   $"8"8$:J:JQWQgQgh  # Mr7) rrrrdrrrrrrrrr)NT) rPrQrRrSrr*rTrUrVrWs@@r5rrs"""r7rcLaa]tRt^toRV3RlV3RllltRRltRtVtV;t#) Zamba2MLPc4<V^8dQhRS[RS[R,/#)rcrdrgNrh)rjrXs"r5rkZamba2MLP.__annotate__s&WW|WPSVZPZWr7c  P<\S V`4WnVPVnVPVnW nW0n\P!VP^VP,VPR7Vn \P!VPVPVPR7Vn \VP,Vn\P!.4Vn\#VP 4FpWAP$,V8Xd\P&!\P!VPPVPP(RR7\P!VPP(^VP,RR74pM\P*!4pVP P-V4K VP.p\1V4UUu/uFwrxWbK uppVnR#uuppi)a9 This MLP layer contributes to tied transformer blocks aimed to increasing compute without increasing model size. Because this layer is tied, un-tied adapter modules (formally same as LoRA, but used in the base model) are added to the up and gate projectors to increase expressivity with a small memory overhead. rnFN)r)r*rdr2rrfrgrrzr gate_up_proj down_projr hidden_actact_fnrsgate_up_proj_adapter_listrwrxryr|r}r~rprr) r1rdrfrgrgate_up_proj_adapterrqrrr4s &&&& r5r*Zamba2MLP.__init__s  !--!'!9!9"4 IId&6&6D> '  * ,  !4     --m< ))-C r7)rgrrrrrOrWs@@r5rrs \\)))r7rc>aa]tRtRtoV3RlV3RlltRtVtV;t#)Zamba2MambaDecoderLayeriHc&<V^8dQhRS[RS[/#)rcrdrerh)rjrXs"r5rk$Zamba2MambaDecoderLayer.__annotate__IsZZ|ZZr7c<\SV`W4\WR7Vn\ VP VP R7VnR#))rdrer3N)r)r*rmambarZr2 rms_norm_epsr)r1rdrer4s&&&r5r* Zamba2MambaDecoderLayer.__init__Is7 +%VI ,V-?-?VEXEXYr7)rr)rPrQrRrSr*rTrUrVrWs@@r5rrHsZZZr7rcTaa]tRtRtoV3RlV3RlltRV3RlRlltRtVtV;t#)Zamba2HybridLayeriOc@<V^8dQhRS[RS[PRS[/#)rcshared_transformerlinearr)rrrzr)rjrXs"r5rkZamba2HybridLayer.__annotate__Ps)55"=5GIyy5Yp5r7c8<\SV`WV4V=WnR#r()r)r* shared_transfr )r1r r rr4s&&&&r5r*Zamba2HybridLayer.__init__Ps! +U;  "4r7c<V^8dQhRS[PRS[PR,RS[R,RS[PR,RS[PR,RS[R,RS[R,R S[P R,R S[P R,R S[S[,R S[S[PS[S[PS[P3,R,3,/ #) rcrErNrer causal_maskr use_cacher position_idsrr) r,rrirrrrrrr)rjrXs"r5rkrWs44||4!& t 34: 4  t+ 4 \\D( 44$;4#--44&&-4+,4 u  %(9(95;L;L(L"MPT"TT U4r7c  VP!V3RVRVRVRVRVRV /V Bp VPV 4p VP!V3RV RVRVRVRV/V BpV#) a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` original_hidden_states (`torch.FloatTensor`): word embedding output that will be concatenated with hidden activations to form the input of the shared transformer layer. layer_idx (`int`): layer number. attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_values (`Cache`, *optional*): cached past key and value projection states use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. rrerrrrtransformer_hidden_statesr)r r  mamba_decoder) r1rErrerrrrrrrrs &&&&&&&&&&, r5rMZamba2HybridLayer.forwardWs<%)$;$;  % #9 %   % ' % , % !4 % & %  % !%)KK0I$J!**  &? * ,    !4   r7)r )NNNNNFNNrOrWs@@r5r r Os55444r7r caa]tRtRtoRtRtRR.tR.tRtRt Rt Rt R] R] /t]P !4V3R l4tV3R ltR tVtV;t#) Zamba2PreTrainedModelimodelTr rrrE attentionsc<\SV`V4\V\4'Ed\P !\P !VPP4\P!VPP4\P!VPP4, ,\P!VPP4,4PVPPR7pV\P!\P!V)4)4,p\ P"!VP$V4\P&!^VP(^,4p\ P"!VP*\P!V44\ P,!VP.4R#R#))minN)r) _init_weights isinstancerr,exprandrdrmathlog time_step_maxrclamptime_step_floorexpm1initcopy_dt_biasarangerA_logones_D)r1moduledtinv_dtAr4s&& r5r #Zamba2PreTrainedModel._init_weightss+ f% f. / / 4;;44588DKK556$++B[B[9\\^((4;;4456e 33e4  %))U[["%5$566F JJv~~v . Q 0 01 45A JJv||UYYq\ 2 JJvxx  0r7c&<V^8dQh/S[;R&#rcrdr#)rjrXs"r5rk"Zamba2PreTrainedModel.__annotate__s r7r[)rPrQrRrSbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_flex_attn_supports_sdpa _is_statefulrra_can_record_outputsr,no_gradr __annotate_func__rTrUrVrWs@@r5rrsr &*#,.GH#4"5NL0o  ]]_!!#r7rcpa]tRtRtoRtV3RlRltRt]]] R V3RlRll444t R t Vt R#) Zamba2ModelizX Model consisting of *config.num_hidden_layers* layers. Args: config: Zamba2Config c <V^8dQhRS[/#r7r#)rjrXs"r5rkZamba2Model.__annotate__s|r7cp\PW4WnVPVnVP Vn\ P!VP VPVP4Vn VPVn VP4Vn VPVn \VPVPR7VnVP"'d8VP$'d\&P)R4\+V4VnRVnVP14R#)rze`use_long_context` set to `True`: using rescaled `rope_theta` and extended `max_position_embeddings`.FN)rr*rd pad_token_id padding_idx vocab_sizer Embeddingr2 embed_tokenslayers_block_type get_layerslayersrrZrfinal_layernormruse_long_contextlogger warning_oncer_ rotary_embgradient_checkpointing post_init)r1rds&&r5r*Zamba2Model.__init__s&&t4 !.. ++LL):):F>/0DKK4N4NN!"6==/45I/J,%)*>%?N++22N3ST)//? . 0J0JJ"a'"8xX !yy)@)@$++BYBY`ef  / +VW k*;%F<}}V$$r7Nc<V^8dQhRS[PR,RS[PR,RS[PR,RS[R,RS[PR,RS[R,RS[S[,R S[S[ ,/#) rc input_idsNrrr inputs_embedsrrr) r,rrrrrrrrr )rjrXs"r5rkrGsE E ##d*E  t+E &&- E  E ((4/ E $;E +,E  ( (E r7c pVRJVRJ, 'd \R4hVfVPV4pTp\P!V4p V'dVf\ VP R7pVfbVeVP 4M^p \P!VP^,VPR7V ,pVP^4p\T;p \4'g/RVP RVRVRVRV/p R \R/V BR \R/V B/p VP P'dVP!WR 7p MRp \#VP$4F(wrV!VV VV R ,V R ,3RVR VR V RV/VBpK* VP'V4p\)TV'dVR7#RR7#)NzaYou cannot specify both input_ids and inputs_embeds at the same time, and must specify either one)rddevicerdrkrrrfull_attentionlinear_attention)rrr)last_hidden_staterr[) ValueErrorrMr,cloner rdget_seq_lengthr-r@rn unsqueezer!dictr r rrUrrPrQr )r1rjrrrrkrrrErpast_seen_tokenscausal_mask_mapping mask_kwargsrrelayers&&&&&&&, r5rMZamba2Model.forwards  -t"; < <s    --i8M% !&]!; 0*$++>O  CRC^==?de  << (;(;A(>}G[G[\_ooL'11!4L?-FF$++ .!? K!"4"C{"C"$C$Rk$R#  ;; # # #"&//-/"[ "&  )$++ 6 I!&#$67#$45  !0 $ %8 *  M!7,,]; &+/8O  >B  r7) rr[rdrMrQr\rVrPrNrJrUrK)NNNNNN) rPrQrRrSrr*rOrrrrMrTrU)rXs@r5rErEsF.%%N E E  E r7rEc]tRtRtRtR#)Zamba2ForCausalLMiAr[Nr\r[r7r5r}r}Ar]r7r}chaa]tRtRtoV3RlV3Rllt]]RV3RlRll44tRtVt V;t #)Zamba2ForSequenceClassificationiEc <V^8dQhRS[/#r7r#)rjrXs"r5rk,Zamba2ForSequenceClassification.__annotate__Fs|r7cf<\SV`V4\V4VnVP 4R#r()r)r*rErrW)r1rdr4s&&r5r*(Zamba2ForSequenceClassification.__init__Fs&   (  r7c`<V^8dQhRS[PR,RS[PR,RS[PR,RS[R,RS[PR,RS[PR,RS[R,R S[S[P,R S[S[,R S[ S[ ,/ #) rcrjNrrrrklabelsrlogits_to_keeprr) r,rrrrrrirrrr )rjrXs"r5rkrMs@ @ ##d*@  t+@ &&- @  @ ((4/ @   4'@ $;@ ell*@ +,@  1 1@ r7c VP!V3RVRVRVRVRV/V Bp V ^,p VPV 4p VeVP^,p MVP^,p VPPfV ^8wd \ R4hVPPfRpMVeWPP8gP V P\P4p\P!VPR,V P\PR7pVV,PR4pM.Rp\PVPP R 24V \P!WPR 7V3,pRpVe%VP !RR VR VR VRVP/V Bp\#VVV P$V P&V P(R7#)ae labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). rrrrkrNz=Cannot handle batch sizes > 1 if no padding token is defined.)rnr;z will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`rmlogitsr pooled_logitsrd)lossrrrErr:r[)rscorer@rdrIrrr<rnr,int32r-argmaxrSrTr4rP loss_functionr rrEr)r1rjrrrrkrrrrtransformer_outputsrEr batch_sizelast_non_pad_token non_pad_mask token_indicesrrs&&&&&&&&&, r5rM'Zamba2ForSequenceClassification.forwardKs(8!F!Fr!J !#    >>**+,ZZ  u||J}}MOaab   %%$-3CPY]YdYdhnD0 /??-;;*55   r7)r)NNNNNNNr) rPrQrRrSr*rrrMrTrUrVrWs@@r5rrEs9 @ @ @ @ r7r)r}rrEr)Fr$collections.abcr itertoolsrr,rrr* activationsr cache_utilsrr masking_utilsr r modeling_outputsr r modeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrutils.output_capturingrbamba.modeling_bambarllama.modeling_llamarrzamba.modeling_zambarrrrrrr r!r"configuration_zamba2r$ get_loggerrPrSModuler&rZr_rarrrrr rrEr}r__all__r[r7r5rs: $ &!.PYF&RR75-M   /   H %;;* L  0 j)nj)Z*z*Z' 'T1"<1hZ4Z<(<~!O!!BN *3N b ( H &DH V r7