+ j:Rt^RIt^RIHt^RIt^RIHt^RIHtHtH t H t H t R.t Rt RtRtRt^tR R ltR R lt!R R] 4tRR] R] R] R] R] R] R2 ,]nRRlt]!]R7RR/RRll4tR#)z%Implementation of the Muon optimizer.N)MutableMapping)Tensor)_disable_dynamo_if_unsupported _params_doc _to_scalar OptimizerParamsTMuongHz>guV @gn@@c V^8dQhR\R\\\\3,R\R\R\/#)gradns_coefficientsns_stepsepsreturn)rtuplefloatint)formats"R/home/luca/screen-checking/.venv/lib/python3.14/site-packages/torch/optim/_muon.py __annotate__rs@'' '#(u)<#='IL'SX' 'c V^d8d \R4h\VP4^8wd \R4h\V4^8wd \R4hVwrEpVP4pVP ^4VP ^48d VP pVP VP4PVR74\V4FFpWwP ,p \P!WWVR7p \P!WzWtR7pKH VP ^4VP ^48d VP pV#)a Newton-Schulz iteration to compute the zeroth power / orthogonalization of G. We opt to use a quintic iteration whose coefficients are selected to maximize the slope at zero. For the purpose of minimizing steps, it turns out to be empirically effective to keep increasing the slope at zero even beyond the point where the iteration no longer converges all the way to one everywhere on the interval. This iteration therefore does not produce UV^T but rather something like US'V^T where S' is diagonal with S_{ii}' ~ Uniform(0.5, 1.5), which turns out not to hurt model performance at all relative to UV^T, where USV^T = G is the SVD. Implementation reference: https://github.com/KellerJordan/Muon/blob/master/muon.py with suggestions by @jxbz, @leloykun, and @YouJiacheng. zBNumber of steps must be less than 100 for computational efficiencyz)Input tensor gradient must be a 2D matrixz0Coefficients must be a tuple of exactly 3 values)min)betaalpha)r) ValueErrorlenshapebfloat16sizeTdiv_normclamprangetorchaddmm) r r rrabc ortho_grad_ gram_matrix gram_updates &&&& r_zeropower_via_newtonschulzr/s3 P   4::!DEE ?q KLLGA!J yy|diil"\\ OOJOO%+++45 8_ <</ kk k [[*M  yy|diil"\\ rcjV^8dQhR\R\R,R\PR\/#)r lr adjust_lr_fnN param_shaper)rstrr&Size)rs"rrrIs1  5 d  PU rc^VR,wr4VeVR8Xd/\P!\^W4, 44pW,#VR8Xd/R\P!\W444,pW,#VR8Xd%\P!W4, 4pW,#RpW,#)z.Default learning rate adjustment used by Muon.:Nr Noriginalmatch_rms_adamwg?spectral_unclampedg?)mathsqrtmax)r1r2r3ABadjusted_ratios&&& r _adjust_lrr@Is r?DA|z93q!%=1   * *tyyQ33   - -15)   rcaa]tRt^YtoRRRR]]]3]]R3V3RlV3Rlllt V3RlR lt ] P!4R R l4t R tVtV;t#) r gMbP?g?gffffff?TNc|<V^8dQhRS[RS[RS[RS[RS[RS[S[S[S[3,RS[RS[R S[R ,R R / #) r paramsr1 weight_decaymomentumnesterovr rrr2Nr)rrboolrrr4)r __classdict__s"rrMuon.__annotate__Zs... . .  .  .ueU23...Dj. .rc <\V\4'd!VP4^8wd \R4hRV8:g\RV 24hRV8:g\RV 24hRV8:g\RV 24hV eV R9d\RV R24hR VR VR VR VR VRVRVRV /p \S V`W4VP F@p V R,F0p V P^8wgK\RV P4 24h KB R#)zTensor lr must be 1-elementgz%Learning rate should be >= 0 but is: z momentum should be >= 0 but is: z$weight decay should be >= 0 but is: NzAdjust learning rate function z is not supportedr1rDrErFr rrr2rCzIMuon only supports 2D parameters whereas we found a parameter with size: )r7r8r9) isinstancernumelrsuper__init__ param_groupsndimr )selfrCr1rDrErFr rrr2defaultsgroupp __class__s&&&&&&&&&& rrO Muon.__init__Zs4 b& ! !bhhjAo:; ;byDRDIJ Jh?zJK Kl"CL>RS S  # = ) 0>OP  " L    3  L   *&&E8__66Q;$cdedjdjdlcmn%'rc h<V^8dQhRS[RS[S[,RS[S[,RS[S[,RS[/#)r rTparams_with_gradgradsmuon_momentum_bufsr)rlistrrG)rrHs"rrrIsIv,F|  !L   rcVR,FpVPfK\P!V4'd \R4hVPP'd \R4hVP V4VP VP4VP V,pRV9d4\P!VP\PR7VR&VP VR,4K R#)rCz(Muon does not support complex parametersz&Muon does not support sparse gradientsmomentum_buffer) memory_formatF) r r& is_complex RuntimeError is_sparseappendstate zeros_likepreserve_format)rRrTrYrZr[rUrds&&&&& r _init_groupMuon._init_groupsxAvv~"""#MNNvv"#KLL  # #A & LL JJqME -+0+;+;FF%*?*?,'(  % %e,=&> ?%!(rc RpVe.\P!4;_uu_4V!4pRRR4VPFrpVR,pVR,pVR,p.p.p.p VPVVVV 4p \ VVV VVVVR,VR,VR,VR,VR ,V R 7 Kt V# +'giL;i) z$Performs a single optimization step.Nr1rDrErFr rrr2) r1rDrErFr rrr2 has_complex)r& enable_gradrPrgmuon) rRclosurelossrTr1rDrErYrZr[rjs && rstep Muon.steps  ""$$y%&&EtB 0LZ(H-/ "$E/1 ** " K  ")!z* %&7 8%Lz*">2' !'< C%$s B88 C )N)__name__ __module__ __qualname____firstlineno__ DEFAULT_A DEFAULT_B DEFAULT_CEPSDEFAULT_NS_STEPSrOrgr&no_gradro__static_attributes____classdictcell__ __classcell__)rVrHs@@rr r YsZ!7@)Y6W(#'..`: ]]_%%%ru Implements Muon algorithm. .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \gamma \text{ (lr)},\ \lambda \text{ (weight decay)},\ \mu \text{ (momentum)},\ \textit{nesterov}\in\{True,False\},\\ &\hspace{13mm}(a,b,c)\ \text{ (NS coefficients)},\ \varepsilon \text{ (epsilon)},\ k \text{ (NS steps)},\ \theta_0 \text{ (params)},\ f(\theta) \text{ (objective)} \\ &\textbf{initialize} : B_0 \leftarrow 0 \text{ (momentum buffer)} \\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for}\ t=1\ \textbf{to}\ \ldots\ \textbf{do} \\[0.25ex] &\hspace{5mm} g_t \leftarrow \nabla_{\theta} f_t(\theta_{t-1}) \\[0.25ex] &\hspace{5mm} B_t \leftarrow \mu B_{t-1} + g_t \\[0.25ex] &\hspace{5mm} \widetilde{B}_t \leftarrow \begin{cases} g_t + \mu B_t, & \text{if nesterov}=True \\ B_t, & \text{if nesterov}=False \end{cases} \\[1.0ex] &\hspace{5mm} O_t \leftarrow \mathrm{NS}^{(a,b,c)}_{k}\!\big(\widetilde{B}_t;\ \varepsilon\big) \\[0.5ex] &\hspace{5mm} \theta_t \leftarrow \theta_{t-1} - \gamma\,\lambda\,\theta_{t-1} \quad\text{(decoupled weight decay)} \\[0.25ex] &\hspace{5mm} \gamma \leftarrow \mathrm{AdjustLR}\!\big(\gamma;\ \mathrm{shape}\!\big(\theta_t \big) \big) \\[0.25ex] &\hspace{5mm} \theta_t \leftarrow \theta_t - \gamma\, O_t \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\mathbf{return}\ \theta_t \\[-1.ex] &\rule{110mm}{0.4pt}s \end{aligned} Here, :math:`\mathrm{NS}^{(a,b,c)}_{k}(\cdot;\varepsilon)` denotes :math:`k` iterations of the Newton–Schulz orthogonalization operator parameterized by coefficients :math:`(a,b,c)` with numerical stabilization :math:`\varepsilon`. The purpose for :math:`\mathrm{AdjustLR}\!\big(\gamma;\ \mathrm{shape}\!\big(\theta_t \big) \big)` is to make the orthogonalized update scale consistently across rectangular matrices. Keller's original implementation scales the update by :math:`\sqrt{\max\!\left(1, \frac{A}{B}\right)}`, where :math:`A` and :math:`B` are dimensions of the matrix being optimized, which represent fan-out and fan-in for a Linear weight matrix. Moonshot's implementation focuses on matching :math:`RMS` of AdamW. The adjustment is computed as: :math:`\gamma \leftarrow {0.2}\gamma\,\sqrt{\max\!\left({A}, {B}\right)}` The method is adopted from `Muon is Scalable for LLM Training`_. Research results show that with this adjustment Muon can directly reuse the learning rate and weight decay tuned for AdamW. Jeremy Bernstein in `Deriving Muon`_ proposes a scaling condition on the spectral norm, which scales the update by :math:`\sqrt{\frac{A}{B}}`. This is similar to the Keller's "original" implementation but removes clamping down to 1. We provide these options for the learning rate adjustment: "original", which follows Keller's implementation, "match_rms_adamw", which refers to Moonshot's implementation, and "spectral_unclamped", which matches Bernstein's implementation. If `adjust_lr_fn` is not specified, the default is "original". For further details regarding the algorithm we refer to `Muon: An optimizer for hidden layers in neural networks`_, `Muon is Scalable for LLM Training`_, and `Deriving Muon`_. z Args: u. Note that Muon is an optimizer for 2D parameters of neural network hidden layers. Other parameters, such as bias, and embedding, should be optimized by a standard method such as AdamW. lr (float, Tensor, optional): learning rate (default: 1e-3). weight_decay (float, optional): weight decay (L2 penalty). (default: 0.1) momentum (float, optional): momentum factor (default: 0.95) nesterov (bool, optional): enables Nesterov momentum. Only applicable when momentum is non-zero ns_coefficients (tuple of three floats, optional): coefficients \(a,b,c\) for the Newton–Schulz orthogonalization polynomial (default: (z, zc)) eps (float, optional): term added to the denominator for numerical stability. (default: uY) ns_steps (int, optional): number of Newton–Schulz iteration steps. (default: a1) adjust_lr_fn (str, optional): function to adjust learning rate. One of "original", "match_rms_adamw", and "spectral_unclamped". If not specified, we will default to use "original". (default: None) Example: >>> # xdoctest: +SKIP >>> # Muon only supports 2D params; use a standard optimizer >>> # such as AdamW for biases, embeddings, and other non-2D >>> # parameters. >>> muon_params = [ ... p for p in model.parameters() if p.ndim == 2 ... ] >>> other_params = [ ... p for p in model.parameters() if p.ndim != 2 ... ] >>> optim_muon = torch.optim.Muon( ... muon_params, lr=0.02, momentum=0.95 ... ) >>> optim_adamw = torch.optim.AdamW( ... other_params, lr=3e-4, weight_decay=0.01 ... ) >>> optim_muon.zero_grad() >>> optim_adamw.zero_grad() >>> loss_fn(model(input), target).backward() >>> optim_muon.step() >>> optim_adamw.step() .. _Muon\: An optimizer for hidden layers in neural networks: https://kellerjordan.github.io/posts/muon/ .. _Muon is Scalable for LLM Training: https://arxiv.org/pdf/2502.16982 .. _Deriving Muon: https://jeremybernste.in/writing/deriving-muon c(V^8dQhR\\,R\\,R\\,R\R\R\R\R\\\\3,R \ R \R \ R ,R \RR / #)r rCrZr[r1rDrErFr rrr2Nrjr)r\rrrGrrr4)rs"rrr>s!/!/ L!/ <!/V !/  !/  !/!/!/5%./!/!/ !/*!/!/ !/rc \V4pV 'd \R4h\V4FwrW,pVP^8wd \R4hW,,pVP V^V, 4V'dVP W4MTp\ VWxV 4p\W:V P4pV P^W4,, 4V PVV)R7K R#)z$Complex parameters are not supportedz"Param gradient must be a 2D matrix)rN) rr enumeraterQlerp_lerpr/r@rmul_add_)rCrZr[r1rDrErFr rrr2rjiparamr bufupdate adjusted_lrs&&&$$$$$$$$$ r_single_tensor_muonr>s BB?@@f%x 99>AB B # $H %-53)3,V_PST 5;;?  1r(() 6+ .&r)single_tensor_fnforeachcBV^8dQhR\\,R\\,R\\,R\R,R\R\R\R \R \\\\3,R \ R \R \ R,R\RR/#)r rCrZr[rNr1rDrErFr rrr2rjr)r\rrGrrrr4)rs"rrrcs&& L& <&V & D[ &  &&&&5%./&& &*&& &rc hVeV'd \R4h\p V !VVVVVVVVV V V V R7 R#)zfFunctional API that performs Muon algorithm computation. See :class:`~torch.optim.Muon` for details. Nz%Foreach is not supported for Muon yet) r1rDrErFr rrr2rj)rar)rCrZr[rr1rDrErFr rrr2rjfuncs&&&$$$$$$$$$$ rrlrlbsI*wBCC D  !' ! rg)__doc__r:collections.abcrr&r optimizerrrrrr__all__ryrvrwrxrzr/r@r rrlrqrrrs, *  (     'T  t9tp:v  EFOKrR[Q\\^_h^ijaad`efXXhWi"j.wi \!/H 1DE&  &F&r