+ "ݳj} Rt^RIHt^RIt^RIHt^RIHt^RIHtH t ^RI H t H t ^RI HtHt^RIHt]!]4t^R IHtR tR tR tRRR R^RRR./RRR R^/RRR R^/RRR R^//t!RR]P64tRRltRRltRRltRRlt RR lt!R!R"lt"R#R$lt#R%R&lt$R'R(lt%R)R*lt&R+R,lt'R-R.lt(R/R0lt)R1R2lt*R3R4lt+R5R6lt,R;R7R8llt-]R3R9R:llt.R# ]dR tRtLi;i) F$4$4$=$=#>>j!kl lf~v}%%!!%))X,>B"GHH VXt ,F ~ 0 0 9 9::hijj ( f $$rc V^8dQhR\PR\PR\PR\PR\P/#)r#self hidden_states top_k_index top_k_weightsr$r0)r's"r r(r(lsU``ii`||`\\`|| `  \\ `rc\'d4\PR\ RVPP 24VP 4^8XdeVPwrEpVPRV4pVPRVPR44pVPRVPR44pMRRrTVPR4pVPpVPR4pVP^4p VPp \P!WR7P^4PRV4P!R4p VP!R4P#VP$4p VP!R4p W,p\P&!V P^4WgVP$R7p\)V 4FpV V8HpVP+4'gK VV,p\-V\/V44p\1VR4'd$VP3V4pVP5V4pMRpVP5V4p\7VVV4pVP9V4pVP#VP$4VV&K WPR4,pVPWV4P;^R7pVeVPWEV4pV#) a[Forward using individual nn.Linear layers per expert. This implementation loops over experts and accesses per-expert containers (self._modules["0"], self._modules["1"], ...) each with gate_proj, up_proj, and down_proj as nn.Linear layers (or quantized equivalents), enabling proper quantization support. Expected module structure: - Numbered children (0, 1, ..., num_experts-1), each an _ExpertContainer with: - gate_proj: nn.Linear (in_features=hidden_dim, out_features=intermediate_dim) - up_proj: nn.Linear (in_features=hidden_dim, out_features=intermediate_dim) - down_proj: nn.Linear (in_features=intermediate_dim, out_features=hidden_dim) - act_fn: activation function - num_experts: number of experts - _apply_gate: optional custom gating function Args: self: The experts module hidden_states: Input tensor of shape (num_tokens, hidden_dim) top_k_index: Selected expert indices of shape (num_tokens, top_k) top_k_weights: Expert weights of shape (num_tokens, top_k) Returns: final_hidden_states: Output tensor of shape (num_tokens, hidden_dim) Using  experts forward for N)devicerHdtyperr7r9)r loggerdebugLINEAR_LOOP_IMPLr<rr8shapeviewsizerH num_expertsr2arange unsqueezeexpandreshapetorJzerosrangeanyr:strr=rrr?rsum)rArBrCrD batch_sizeseq_len hidden_dimrH num_top_k num_tokensrQ token_idxsample_weights expert_idsselected_hidden_statesout_per_sample expert_idxmask expert_inputexpertr.r/ gated_out expert_outfinal_hidden_statess&&&& r linear_loop_experts_forwardrmls>x'7&88MdnnNeNeMfgha*7*=*=' Z%**2z: !&&r;+;+;B+?@ %**2}/A/A"/EF "DG"''+  ! !F  $I##A&J""K Z7AA!DKKBPYZbbcefI"**2.11-2E2EFN$$R(J+5[[!2JUbUhUhiNK( Z'xxzz -d3 s:/ 6; ' ''' 5H^^L1FH^^L1F&tXv> %%i0  *}}^-A-ABt-)2$&>&>r&BBN)--jZPTTYZT[166zJW rcxV^8dQhR\PR\PR\P/#)r#rArBr$r0)r's"r r(r(s5FF F%,,FSXS_S_Frc>\'d4\PR\ RVPP 24VP R4pVPVPRV4p.p\VP4Fnp\V\V44pW5,pVPV4pVPV4p \WV 4p VPVP!V 44Kp \"P$!V^R7P'RV4#)aRun defused experts for models that feed experts as expert-major input batches. Llama4 is the current example: upstream code repeats and pre-weights tokens, then calls ``experts(hidden_states)`` where the leading dimension is laid out as ``[expert0_tokens, expert1_tokens, ...]``. This forward keeps that public contract while still executing per-expert ``nn.Linear`` modules internally. rFrGr7r9)r rKrLBATCHED_INPUT_IMPLr<rrPrOrQrXr:rZrrr?appendrr2stackrU) rArBr^ expert_inputsexpert_outputsrfrirhr.r/rjs && r batched_input_experts_forwardrusx'9&::OPTP^P^PgPgOhij##B'J!&&t'7'7ZHMND,,- s:/$0 ##L1 -&tv> f..y9: . ;;~1 - 5 5b* EErc$V^8dQhR\/#r"r%)r's"r r(r(sdrc\'g\PR4R#\\P 9dE\ \P \&\'d\PR\ R24R#)zRegister the linear_loop experts implementation with transformers. Returns: True if registration was successful, False otherwise. ztransformers.integrations.moe.ALL_EXPERTS_FUNCTIONS not available. linear_loop experts implementation not registered. Requires transformers >= 5.0.0Fz Registered 'z' experts implementationT) r*rKwarnrMr _global_mappingrmr rLrrr register_linear_loop_expertsrzs` !   - 4DDDB]--.>? 8V\\L1A0BBZ"[\ rcZV^8dQhR\PR\\,/#)r#modelr$rr1listdict)r's"r r(r(s"  ryy T$Z rc\VRR4p\VRR4pVf.#\P!V/4P\P.4p\ V\ 4'dV.#\V4#)zEReturn declarative experts-defusion specs for the current model type.configN model_type)r:rgetrEXPERTS_DEFUSE isinstancerr~)r|rrspecss& r _model_experts_defuse_specsrsl UHd +Ft4J   Z , 0 01E1Er JE%w ;rcDV^8dQhR\PR\/#r#r-r$rr1rZ)r's"r r(r( s"HHryyHSHrc`VPP RVPP 2#)zAReturn a stable import-style class path for matching model specs..)r<rr)r-s&r _module_class_pathr s-)) *!F,<,<,E,E+F GGrctV^8dQhR\PR\\,R\R,/#)r#r-rr$Nr})r's"r r(r(s.  ")) DJ 4RV; rc\V4pVPPpVF pVPR4pWRV09gKVu# R#)zIFind the first declarative experts-defusion spec that matches ``module``. module_classN)rr<rr)r-r module_path module_namespectargets&& r _matching_experts_defuse_specrsJ$V,K""++K.) ;/ /K rcHV^8dQhR\PR\RR/#)r#r-implementationr$Nr)r's"r r(r(s* V Vbii V V VrcV\8Xd\\V4VnR#V\8Xd\\ V4VnR#\ RV 24h)zGAttach a generic forward implementation directly to one experts module.Nz,Unsupported experts forward implementation: )rMrrmforwardrpru ValueError)r-rs&&r _install_instance_forwardrsL))#$?H++#$A6J CNCST UUrcfV^8dQhR\PR\\\3,/#r)rr1rrZ)r's"r r(r((s&%%ryy%T#t)_%rc /pVPVP3UUUu/uF@pVP4F)wr4\V\P 4'gK'W4bK+ KB pppp\ P4F7wrgVPV4pVfKVP4^8XgK3WqV&K9 V'g|VP4FgwrV PR4'dKVfK$VP4^8XgK;\'d\PRV 24RRR^/W&Ki V#uupppi)axDetect which expert projections exist in the module. This function scans the module for any registered 3D Parameter / Tensor attributes. It first checks known projection names, then discovers any unknown 3D registered tensors. Returns: Dict mapping projection names to their config, only for projections that exist as 3D registered Parameter / Tensor in the module. _z"Discovered unknown 3D projection: rTr) _parameters_buffersitemsrr2r3KNOWN_PROJECTION_PATTERNSrr8 startswithr rKrL) r-detectedregistrynametensor local_tensors proj_namerparam attr_names & r _detect_expert_projectionsr(sH  ++V__==H$NN,LD fell +  , =7<<> !!),  !1"(Y ?  - 3 3 5 I##C(( UYY[A%58V\\,Nyk*Z['6>QST&U# !6 O/s 7D= D=crV^8dQhR\PR\R\PR,/#)r#r-rr$Nrr1rZr2r3)r's"r r(r(Ps.  299 C ELL4= 11rc ~V^8dQhR\PR\R\R\\ \ 3,/#)r#rr is_transposedr$)r2r3rr&tupleint)r's"r r(r(ts9%%U\\%4%%QVWZ\_W_Q`%rcVP^,VP^,rCVPR^4pV'dY4rvMY4rgV^8dWu,V,pWg3#)a8Infer input and output dimensions for a projection. Args: param: The 3D projection tensor (num_experts, dim1, dim2) config: Projection config with is_input_proj and output_multiplier is_transposed: Whether weights are stored transposed Returns: (in_features, out_features) for the Linear layer r)rNr)rrrdim1dim2 multiplier in_features out_featuress&&& r _infer_dimensionsrtsZQQ$/3J$(\%)kA~#1J>  $$rcV^8dQhR\PR\R\R\R\ P R\ PR\R,/#) r#r-rrQrrJ target_devicer$N) rr1rZrr&r2rJrHr~)r's"r r(r(sgee eee e {{ e || e D[erc >\W4pVeVP4^8wdR#\PVRRR^/4p\ WgV4wrV R2p \W 4p \ V \ P4p VPp V PR8Hp\ V\P4'd VPMRpV'EgWVP4P4pV'dVP^^4pVP!4'gVP#4pVP%^4pV 'dV P4P4pVP!4'gVP#4pVP%^4p\ V \P4'd V PMRp\'W\)V44RpV 'd\'W \)V 44Rp \*'d\,P/RV R 24.p\3V4Fp\P4!WWRR 7pV'gXV,pVPR 8wdVP7WTR 7p\P!VVR 7VnV 'dUXV,pVPR 8wdVP7VVP:R 7p\P!VXR 7VnVP?V4K V# \0dLi;i)aUnfuse a single projection from a 3D registered tensor to Linear layers. Optimized to keep peak device memory low while preserving the module's original device placement: - Moves the full 3D tensor to CPU in a single transfer - Performs batch transpose on CPU if needed - Releases the original fused parameter before allocating defused linears - Re-materializes each expert linear back onto ``target_device`` Args: module: The experts module proj_name: Name of the projection attribute num_experts: Number of experts is_transposed: Whether weights are stored transposed dtype: Data type for the Linear layers target_device: Device for the Linear layers Returns: List of Linear layers, or None if projection doesn't exist NrTr_biasmetaFzReleased memory for z using to_meta())biasrJrHcpurIr) rr8rrrrr2r3rHtyperrrdetachr transpose is_contiguous contiguousunbindsetattrr r rKrL ExceptionrXLinearrVweightrJrrq)r-rrQrrJrrrrr bias_name bias_paramhas_bias source_deviceis_metaweight_requires_grad weights_cpu weight_slicesbias_cpu bias_slicesbias_requires_gradlinearsilinearrrs&&&&&& r _unfuse_single_projectionrs8 #6 5E } q(' * *9Nacd6e fF!2% OK+U#I':J*ell3HLLM  F*G2?! x(._compact_repr s6  ----/JC%!122##SL1""C<0 0 !$' '__&  LL))$/ 0&JC5kG !,G LL1SEWI. /& !OA& #A&<?1-.   a0 q5 LL1YKq #aSK=I J LL1YKs;-8 9>>#c)  U!33d: :HCrT __getitem__c,\V\V44#N)r:rZ)rAidxs&&r _getitem._install_compact_expert_repr.._getitem=s4S* *rcVP#r )rQ)rAs&r _len*_install_compact_expert_repr.._len@s## #rc3t"\VP4Fp\V\V44xK R#5ir )rXrQr:rZ)rArs& r _iter+_install_compact_expert_repr.._iterCs+4++,dCF++-s68)r<__dict____repr__rr=r __len____iter__)r-clsr rrrrs& @r _install_compact_expert_reprrs}   C'3<<7\\N)V!L)-C& 3 & &-s||*K + $ ,#  +LrcV^8dQhR\PR\R\\,R,R\/#)r#r-check_decoratorprojection_namesNr$)rr1r&r~rZ)r's"r r(r(LsCMM MMs)d*M Mrc lV'd1\P4UUu/uFwr4W29gK W4bK pppM \V4pV'gR#V'dA\V4'g0\P RVP P R24R#\\V44p\W4pVP^,p\VRR4p V fBVR9dRV9d RV9dRp M)VP^,VP^,rW8p VPp VPPR8wd VPMR p /p.pVP4EFwppVP!R 4pV'gK"\VV4pVeVP#4^8wdKIV 'd^M^pVP%\'V4VR 7pV R 2p\VV4pRp\)V\*P,4'd2VP#4^8XdVP%\'V4^R 7p\/\1VV44FPwpwpp\3VVVV4Ve\3VV R 2VV,V4\P!VR RR^/4VV&KR \5VV4Ve \5VV4VP7V4\8'gEKz\P RV RV RV R24EK VFpWSK VP;V4/pVFKp\=VVWW4pVfK\5VV4V R 2p\?VV4'd \5VV4VVV&KM V'gR#\AV4FVp\C4pVP4Fwpp\EVVVV,4K VPG\IV4V4KX \?VR4'gWn%RVn&RVn'VPQ4F pRVn'K \SV4R#uuppi)amConvert fused 3D expert weights to per-expert containers of nn.Linear layers. This function modifies the module in-place, replacing fused 3D Parameters with numbered _ExpertContainer children, each holding nn.Linear projections. Args: module: The experts module to unfuse check_decorator: If True, only unfuse if the module supports @use_experts_implementation decorator. Default is True. projection_names: Optional list of projection names to unfuse. If None, auto-detects from KNOWN_PROJECTION_PATTERNS. Returns: True if unfusing was successful, False if module doesn't match pattern FzSkipping unfuse for z.: does not support @use_experts_implementationrNrrrrrr7rrTrzSplit z -> z: z expertsrQ>rr)*rrrrrKrLr<rnextiterrrNr:rJrHrrr8chunkrrr2r3 enumerateziprdelattrrqr updaterr=rXrr add_modulerZrQ_unfused_experts_is_hf_initializedmodulesr) r-rrrrdetected_projectionsfirst_proj_name first_paramrQrrrrJrextra_projectionsfused_to_removerrr split_dim split_paramsrr bias_splitsr split_name split_param proj_linearsrrf containerrs &&& r _unfuse_experts_weights_inplacer8Ls*-F-L-L-N -N\TRVRjLDL-N   :&A :6BB +F,<,<,E,E+FFtuv4 456O(AK##A&KFOT:M 7 7&::#77!M$**1-{/@/@/C$ KM   E*5*<*<*A*AV*KK&&QVMO1779 6ZZ -  &vy9 =EIIK1, &AA {{3z? {B !k' +FI>  j%,, / /JNN4D4I$**3z?*BK,5c*l6S,T (A( K ( K O&+Fzl%4H+VW.Zde,E,I,I_d4GK- j ) -U  "  ! FI &y) 8V\\F9+T*R }T\"]^?:D  &  12L) +FI{[`p   FI &$+U+Ivy)) *&-L #*  K( $& "."4"4"6 Iw Iy'**= >#7#j/95 ) 6= ) )(#F !%F!#' "!( o s P0P0cfV^8dQhR\PR\R\\,/#)r#r|rr$)rr1rZr~)r's"r r(r(s3DD 99DD #Y Drc\4'g \R4h.p.p\V4p\V4pVP 4Fwrx\ Wv4'gK\ W4p V en\VRV PR4R7'dK\WR,4VPV4\'d\PRV R24K\V4'gKVPV4VPV4\'gK\PRV R 24K V'd\'d#\PR \V4 R 24\!4V'dk\#VR 4'dY\%VP&R R4p V 'dT MTp WP&n\'d\PRV R 24V#)adPrepare a model for MOE quantization using transformers' experts interface. This function: 1. Registers the linear_loop experts implementation with transformers 2. Sets model.config._experts_implementation = implementation 3. Unfuses all fused MOE expert weights into per-expert containers of nn.Linear After calling this function, the model's forward pass will use individual nn.Linear layers per expert, which can be quantized normally. Args: model: The model to prepare implementation: The experts implementation to use (default: "linear_loop") Returns: List of module names that were unfused zxFailed to register linear_loop experts implementation. This requires transformers >= 5.0.0 with MOE integration support.NFr)rr forward_implz[MoE Prep] Unfused 'z' via declarative spec'z[MoE Prep] Unfused z MOE experts modulesrexperts_implementationz,Set model.config._experts_implementation = ')rz RuntimeErrorrr named_modulesrrr8rrrqr rKrLinforr r=r:r_experts_implementation) r|r filter_rulesunfused_modulesdecorated_unfused_modulesexperts_defuse_specsmodule_name_filterrr-r saved_impl impl_to_sets &&& r "prepare_model_for_moe_quantizationrIs, ( ) ) P  O "6u=3LA++- )$CC ,VJ   ? !!XX&89! ! &f>.B C  " "4 (x(LL 0x(TU`Taab&cd r)TN)/rtypesrr2logbarrrdefuser.model_registryrrdefuser.utils.commonrrdefuser.utils.devicer r defuserr rrKtransformers.integrations.moer r* ImportErrorrMrprr1rr+r?rmrurzrrrrrrrrrrrr8rIrrr rRs>& 6W6  !C  !$_d,?LS^`iRjk/4)/5*=qA   ryy  ! %*`FF2* H  V%P - 2%8ePQhMd+DD]!! !sC33 DD