+ j<<Rt^RIHt^RIt^RIHt^RIHt^RIHt^RI H t ^RI H t ^RI Ht^R IHtHtHtHtHtHt^R IHt!R R ]4t!R R]4t!RR]P4P64tR"Rlt!RR]4t!RR]P>4t !RR]PB4t"!RR]4t#!RR]4t$!RR]4t%!RR]4t&!R R!]]$4t'.R#Ot(R#)$zPyTorch Phimoe model.)CallableN)nn) GenericForSequenceClassification)ROPE_INIT_FUNCTIONS)maybe_autocast)OutputRecorder)LlamaAttention)MixtralDecoderLayerMixtralExpertsMixtralForCausalLM MixtralModelMixtralPreTrainedModelMixtralRotaryEmbedding PhimoeConfigc@a]tRt^(toRV3RlRlltRRltRtVtR#)PhimoeRotaryEmbeddingNc <V^8dQhRS[/#configr)format __classdict__s"j/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/phimoe/modular_phimoe.py __annotate__"PhimoeRotaryEmbedding.__annotate__)s O O| Oc(\PP4VPVnVPVnWnVP PR,VnVPVn VPR8wd\VP,Vn VPVP V4wq0n \P!VRR7Vn\P!VP4RR7VnR#) rope_typedefaultF) persistentN)rModule__init__max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrrope_parametersrcompute_default_rope_parameters rope_init_fnrattention_scalingBufferinv_freqcloneoriginal_inv_freq)selfrdevicer+s&&& rr"PhimoeRotaryEmbedding.__init__)s "("@"@$*$B$B! 44[A&*&J&J >>Y & 3DNN CD +/+<+>+;!NrcRp\P!V4^,pVPPR,R8wddV'd\W@PPR,8dVPPR,MVPPR,pVP VP4wrVVfTMTpVR ,P VP ^,R ^4P\PVPR7pVR,P4p\VPP\4'd2VPPR8wdVPPMRp \V R R 7;_uu_4Wx,P^^4p \P!W3R R 7p V P!4V,p V P#4V,p RRR4X PVP$4X PVP$43# +'giLF;i)Nrr original_max_position_embeddings long_mscale short_mscale)dtyper/mpscpuF) device_typeenableddim)NNNNN)r<Nr<)torchmaxrr&r(expandshapetofloatr/ isinstancetypestrr transposecatcossinr5)r.x position_idsmscaleseq_lenr+r)inv_freq_expandedposition_ids_expandedr8freqsembrIrJs&&& rforwardPhimoeRotaryEmbedding.forward8s))L)A- ;; & &{ 3y @W[[889[\\ ++M:[[00@  '+&7&7 &D#&,n"& ] # * *<+=+=a+@"a H K KRWR]R]fgfnfn K o !-Z 8 > > @'1!((--'E'E!((--[`J`ahhmmfk  U C C&>II!QOE))UN3C'')f$C'')f$C D vvaggqww// D Cs 'A I I )r)rr+r$r-r%r(rN)__name__ __module__ __qualname____firstlineno__r"rS__static_attributes____classdictcell__rs@rrr(s O O00rrc]tRt^QtRtR#)PhimoeAttentionNrVrWrXrYrZr_rrr^r^Qrr^cXa]tRt^Uto]V3RlRl4t]V3RlRl4tRtVtR#)PhimoeMultiplierc <V^8dQhRS[PRS[PRS[PRS[PRS[P/#)rscores multiplierselected_experts masked_gates mask_for_oner>Tensor)rrs"rrPhimoeMultiplier.__annotate__WsQ)) )LL) ,, ) ll ) ll )rc 6VPW#V4W%,#)a Forward pass for the custom autograd function. Args: ctx: Context object to save information for backward computation. scores (torch.Tensor): Input scores tensor. multiplier (torch.Tensor): Multiplier tensor. selected_experts (torch.Tensor): Tensor of selected experts. masked_gates (torch.Tensor): Masked gates tensor. mask_for_one (torch.Tensor): Mask for one tensor. Returns: torch.Tensor: Result of the forward pass. )save_for_backward)ctxrerfrgrhris&&&&&&rrSPhimoeMultiplier.forwardVs. jLI((rc4<V^8dQhRS[P/#)rgrad_at_outputrj)rrs"rrrlqs    rc VPwr#pW,pWAPR4,pVPRVVR7VRRRR3#)a  Backward pass for the custom autograd function. Args: ctx: Context object with saved tensors from the forward pass. grad_at_output (torch.Tensor): Gradient at the output. Returns: tuple[torch.Tensor, None, None, None, None]: Gradients for the inputs. )r;indexsrcNr=) saved_tensorsmul scatter_add_)rorrrfrgrhgrad_at_scores_expandeds&& rbackwardPhimoeMultiplier.backwardpsg695F5F2 l'4".1C1CB1G"G,," -  $       rr_N) rVrWrXrY staticmethodrSrzrZr[r\s@rrcrcUs+))2  rrcc4\P!4;_uu_4VPR RR7wrEVP4P VR7pW@, V, ^V,8pRRR4VP X\ R44pV'dqV\P!V\PR7P4P4, PR R7^,PR 4pMXp\P!VR R7pVPR VR7p V'dVPR RR7wr\P!W8H\P!V 4R 84p \P !R V R R 7P#V4p \$P'VV VVV 4p MT p \P(!VR V\ R44p \P!4;_uu_4V PR RR7wrEVP4P VR7pW@, V, ^V,8pRRR4V P V\ R44pV'dqV\P!V\PR7P4P4, PR R7^,PR 4pMXp\P!VR R7pVPR VR7pV'dVPR RR7wr\P!W8H\P!V 4P+4R 84p\P !R VR R 7P#V4p\$P'VVVVV4pMTp\P,!V V3R R7p \P,!W3R R7pV V3# +'giEL;i +'giEL;i)u, Sparse mixer function to select top-k experts and compute multipliers. Based on the paper: https://huggingface.co/papers/2409.12136 We first replace the TopK(·) function as random sampling of discrete variables in model training. Then, following Liu et al. (2023a) and Liu et al. (2023b), we apply Heun's third order method to approximate the expert routing gradient and construct a modified back-propagation to give a mathematically sound gradient estimation for expert routing. Args: scores (torch.Tensor): Input scores tensor. jitter_eps (float): Jitter epsilon for numerical stability. training (bool): Flag indicating if the model is in training mode. top_k (int): Number of top experts to select. Returns: tuple[torch.Tensor, torch.Tensor]: Multiplier and selected experts tensors. T)r;keepdim)minNz-inf) memory_formatr:)r;rtg?gioT?gK=U?)alphar=)r>no_gradr?absclamp masked_fillrC empty_likelegacy_contiguous_format exponential_log unsqueezesoftmaxgather logical_or rand_likeaddtype_asrcapplyscatteruniform_concat)re jitter_epstrainingtop_kmask_logits_thresholdmax_indfactorrhrg multiplier_o max_scoresrirf masked_scoresmasked_gates_top2selected_experts_top2multiplier_top2_omask_for_one_top2multiplier_top2s&&&& r sparsemixerrs$ )/D)I&##(=#>"7"@F!JqS]~ ^ %%&;U6]KL"""7"@F!JqS]~ ^ &112GvW"""#4EDbDbc SRS[  Yr] !( &7R@)00R?T0U/33D3I !,, ! , OOJ ' 0 0 2T 9  "IIf.?vNVVWhi*00   !    ,z?;DJ||%5$MSUV  G f sAO2AP2 P  P c]tRtRtRtR#) PhimoeExpertsir_Nr`r_rrrrrarrcVaa]tRtRtoV3RlV3RlltV3RlV3RlltRtVtV;t#)PhimoeTopKRouteric <V^8dQhRS[/#rr)rrs"rrPhimoeTopKRouter.__annotate__s44|4rc<\SV`VPVPRR7VPVnVP VnVP VnVPVnR#)FbiasN) superr" hidden_sizenum_local_expertsrouter_jitter_noiseinput_jitter_noisenum_experts_per_tokr num_expertsr.r __class__s&&rr"PhimoeTopKRouter.__init__sZ ++V-E-EER#)#=#= "(";";// !33rcx<V^8dQhRS[PRS[S[PS[P3,/#r hidden_statesreturn)r>rktuple)rrs"rrrs5 @ @U\\ @eELL%,,rrrrSrrr)r.r router_logitsrouting_weightsrgrs&& rrSPhimoeTopKRouter.forwards ===T44q8 U--m<EEd---sT5L5L/L M 6 ,7 &>&>^b^h^h- )/???r)rrrr) rVrWrXrYr"rSrZr[ __classcell__rrs@@rrrs!44 @ @ @rrcHaa]tRtRtoRtV3RltV3RlRltRtVtV;t #)PhimoeSparseMoeBlocki&a This implementation is strictly equivalent to standard MoE with full capacity (no dropped tokens). It's faster since it formulates MoE operations in terms of block-sparse operations to accommodate imbalanced assignments of tokens to experts, whereas standard MoE either (1) drop tokens at the cost of reduced performance or (2) set capacity factor to number of experts and thus waste computation and memory on padding. c<\SV`4VPVnVPVnVP VnVPVn \V4Vn \V4Vn VPVnR#rU)rr"r hidden_dimintermediate_sizeffn_dimrrrrrrouterrexpertsrrs&&rr"PhimoeSparseMoeBlock.__init__2si  ,,// !33// &v. $V, "(";";rcN<V^8dQhRS[PRS[P/#rrj)rrs"rr!PhimoeSparseMoeBlock.__annotate__<s' T TU\\ Tell TrcVPwr#pVP'daVP^8dPV\P!V4P RVP, RVP,4,pVPwr#pVP RV4pVPV4wrVpVPWV4pVP W#V4#)rrr=) rArrr>rrreshaperr) r.r batch_sizesequence_lengthr_rrgfinal_hidden_statess && rrSPhimoeSparseMoeBlock.forward<s2?2E2E/ Z ===T44q8 U--m<EEd---sT5L5L/L M3@2E2E/ Z%--b*= /3{{=/I,,"ll=O\"**: SSr)rrrrrrr) rVrWrXrY__doc__r"rSrZr[rrs@@rrr&s! < T T Trrc>aa]tRtRtoV3RlV3RlltRtVtV;t#)PhimoeDecoderLayeriJc&<V^8dQhRS[RS[/#)rr layer_idx)rint)rrs"rrPhimoeDecoderLayer.__annotate__Ks  |  rc<\SV`W4\P!VPVP RR7Vn\P!VPVP RR7VnR#T)epselementwise_affineN)rr"r LayerNormr rms_norm_epsinput_layernormpost_attention_layernorm)r.rrrs&&&rr"PhimoeDecoderLayer.__init__KsX + "||F,>,>FDWDWlpq(*   F$7$7D) %r)rrrVrWrXrYr"rZr[rrs@@rrrJs   rrc6]tRtRtR]!]^R7R]R]/tRt R#)PhimoePreTrainedModeliUr)rtr attentionsr_N) rVrWrXrYrrrr^_can_record_outputsrZr_rrrrUs"(8B+orrc>aa]tRtRtoV3RlV3RlltRtVtV;t#) PhimoeModeli]c <V^8dQhRS[/#rr)rrs"rrPhimoeModel.__annotate__^sgg|grc<\SV`V4\P!VPVP RR7VnR#r)rr"rrrrnormrs&&rr"PhimoeModel.__init__^s1  LL!3!39L9Laef r)rrrs@@rrr]sgggrrcBaa]tRtRtoV3RltRV3RlltRtVtV;t#)PhimoeForCausalLMicc<\SV`V4\P!VPVP VP PR7VnR#)rN) rr"rLinearr vocab_sizer lm_head_biaslm_headrs&&rr"PhimoeForCausalLM.__init__ds:  yy!3!3V5F5FT[[MeMef rc J<V'd|\VPR4'd`VP^,VPP^,8d-VP 4p WPP8:dRp\ S V`!R RVRVRVRVRVRVRV/VBp V #) r2N input_idspast_key_valuesattention_mask inputs_embedsrL use_cachelogits_to_keepr_)hasattrrrAr2get_seq_lengthrprepare_inputs_for_generation) r.rrrrrLrrkwargs past_length model_inputsrs &&&&&&&&, rr/PhimoeForCausalLM.prepare_inputs_for_generationis"  %GHH"dkk&R&RUV&VV)88:KkkJJJ"&w<    +  *  (  &    *    r)r)NNNNTr) rVrWrXrYr"rrZr[rrs@@rrrcsg ##rrc]tRtRtRtR#)PhimoeForSequenceClassificationir_Nr`r_rrr r s`crr )r)rrrr ))rcollections.abcrr>rmodeling_layersrmodeling_rope_utilsr utils.genericrutils.output_capturingrllama.modeling_llamarmixtral.modeling_mixtralr r r r r rconfiguration_phimoerrr^autogradFunctionrcrrrrr!rrrrrr __all__r_rrrs$ 7+41/&02&0R n ; u~~..; |xv N @ryy@(!T299!TH , 2g,g )*)Xd&FH]c r