+ jo7aR t0t^RIt^RIt^RIHtHtHt^RIHt^RI H t H t ^RI t ^RI Ht^RI Ht^RIHt^RIHt.t]^k]P,!]4t!RR ]P24tR R ltR#) N)Callable CollectionMapping)deepcopy)Anyoverload)optim) ShardedTensor)FullyShardedDataParallelc&a]tRt^toRtRV3RlRlltV3RlRltV3RlRlt]RV3R lR ll4t ]V3R lR l4t RV3R lRllt ] V3RlRl4t V3RlRlt V3RlRlt V3RlRltV3RlRltV3RlRltRtVtR#)_NamedOptimizera` ``_NamedOptimizer`` takes a dict of parameters and exposes ``state_dict`` by parameter key. We replace the original key (number) in an optim to the fully qualified name (FQN) string. User can initialize the optim as they initialize a PyTorch optim, the only difference is that they also need to pass in the FQN of each parameters. Args: named_parameters (Mapping[str, Union[torch.Tensor, ShardedTensor]]): Mapping from FQN to parameter. optimizer_class (optim.Optimizer): The class of optimizer to instantiate. param_groups (Collection[Mapping[str, Any]]): `param_groups` to pass to optimizer if specified. The key of the inner map needs to be FQNs. Default: None module (nn.Module): the module whose parameters to be updated by the optimizer. args: arguments to pass to the optimizer constructor. kwargs: arguments to pass to the optimizer constructor. Example:: >>> # xdoctest: +SKIP("distributed") >>> from torch import optim >>> from torch.distributed.optim import _NamedOptimizer >>> >>> # Define the named optimizer. >>> m = Model(...) >>> named_optim = _NamedOptimizer(m.named_parameters(), optim.SGD) >>> # Forward pass + backward pass. >>> named_optim.step() >>> ... >>> # Call state_dict for the named optimizer returns an FQN state_dict. >>> named_optim.state_dict() Warning: This API is still in development and subject to change. TODO: Add tutorial for _NamedOptimizer. TODO: Add documentation in the docstring for the public attributes like self.param_groups and self.named_parameters. Nc<V^8dQhRS[S[S[PS[,3,RS[P RS[S[S[S[3,,R,RS[ PR,RS[ S[R3,RS[ S[S[3,R R/#) named_parametersoptimizer_class param_groupsNmoduleargs.kwargsreturn) rstrtorchTensorr r OptimizerrrnnModuletupledict)format __classdict__s"h/home/luca/screen-checking/.venv/lib/python3.14/site-packages/torch/distributed/optim/named_optimizer.py __annotate___NamedOptimizer.__annotate__?s)9)9!#u||m'C"CD)9)9!c!23d: )9  D )9 S#X )9sCx.)9 )9c\PPR4W0nVP 4\ V4VnVfVP P4MTpV!V.VO5/VBVnW@n Vf*\VP P44Vn M\P!R^R7VP P4UU u/uFwrWbK p pp .p VF?p V R,F/p W9d\!RV R24hV P#W,4K1 KA Wn VPPVnR#uup pi)z'torch.distributed.optim._NamedOptimizerNzvSince we pass in param_groups, we will use param_groups to initialize the optimizer, not all parameters of the module.) stacklevelparamszExpect param name z% found in param group but is missing.)r_C_log_api_usage_oncer_param_groups_checkrrvalues _optimizerrlistkeysordered_param_keyswarningswarnitems ValueErrorappend) selfrrrrrrparams_for_optimizerkeyparam param_to_keyr/groups &&&&&*, r!__init___NamedOptimizer.__init__?sT $$%NO;G   " $%5 6.:.BD ! ! ( ( *  *       &*4+@+@+E+E+G&HD # MMN  :>9N9N9T9T9VW9V:3EJ9VLW!# %"8__E0(07\]'--l.AB -&'9 # OO88Xs Ec<V^8dQhRR/#rrN)rr s"r!r"r#js//T/r$cVPeVPFp\V\4'g \R4hRV9d \R4hVR,p\V\P 4'dV.p\ V4pVFIp\V\P 4'dK%\R\P!V4,4h W!R&K R#R#)Nparam group must be a dictr'z#param group must contain key paramsz>optimizer can only optimize Tensors, but one of the params is ) r isinstancerAssertionErrorrrr- TypeErrortypename)r5 param_groupr'r8s& r!r*#_NamedOptimizer._param_groups_checkjs    (#00 !+t44()EFF;.()NOO$X.fell33$XFf#E%eU\\::'8:?..:OP$ )/H% 1 )r$c6<V^8dQhRS[S[S[3,/#rrrrr)rr s"r!r"r#}sWWDcNWr$c VPP4pVR,pVR,P4UUu/uFwr4VPV,VbK ppp.pVFxpVR,Uu.uFqPV,NK p pR\ V 4/p VP4FwrV R8wgK\ V 4W&K VP V 4Kz VPRVRV/4#uuppiuupi)z Return the ``state_dict`` of the optimizer. Instead of using number to index parameters, we will use module fully qualified name (FQN) as the key. rstater')r, state_dictr2r/sortedrr4_post_state_dict) r5rMrst_key state_val ret_state ret_groupsr:r8 param_keys ret_groupkvs & r!rM_NamedOptimizer.state_dict}s__//1 !.1 &0%8%>%>%@ %@!  # #F +Y 6%@   !EFKHoVoU11%88oJV!6*#56I =#+A;IL&   i ( "$$gy.*%UVV Ws C52C;c"<V^8dQhRRRR/#rclosureNrr?)rr s"r!r"r#s55D5D5r$cR#Nr?r5r[s&&r!step_NamedOptimizer.steps25r$c:<V^8dQhRS[.S[3,RS[/#)rr[rrfloat)rr s"r!r"r#s>>HRY/>E>r$cR#r]r?r^s&&r!r_r`s;>r$cV<V^8dQhRS[.S[3,R,RS[R,/#rZrb)rr s"r!r"r#s-55HRY/$65%$,5r$c :VPPVR7#)zr Perform a single optimization step. This will call :meth:`torch.optim.Optimizer.step` on the wrapped optimizer. r[)r,r_r^s&&r!r_r`s##G#44r$cJ<V^8dQhRS[S[PS[3,/#rI)rrrr)rr s"r!r"r#s"%%wu||S01%r$c.VPP#r])r,rL)r5s&r!rL_NamedOptimizer.states$$$r$c:<V^8dQhRS[S[S[3,RR/#)rrMrNrJ)rr s"r!r"r#s'h8h8$sCx.h8Th8r$c  VPP4pVPV4pVR,pVR,p\V4^8Xd \ R4h\ VP 4EFwrVWc9dK\W6,4\WE,48wd3\ R\WE,4 RV R\W6,4 24hWE,P4EFwrxWsV,9d\ RV RV R24hW6,V,p \V\4'd\V \4'g\h\VP44p \V P44p W8wd\ RV R V R V R V 24h\VP4V P44F8wrV PP4PV P4K: EK\V\ P"4'dI\V \ P"4'g\hVP4PV 4EK\%V 4WE,V&EK EK VR ,pVR ,p/pVF#p\'VR ,4pVV\)V4&K% /pVFEp.pVR ,F%pVP+VP V,4K' VV\)V4&KG VP4FwppVV9dKVV,p\V4\V48wd(\ R\V4 RV R \V4 R24hVF:pVV9d\ RV RV R24hVR 8wgK%\%VV,4VV&K< K VPP-V4R#)a@ Define the default behavior to load a state_dict for ``_NamedOptimizer``. Sample Code ``` my_model = MyModule() optimizer = _NamedOptimizer(my_model.named_parameters(), Adagrad) ... optim_state_dict = optimizer.state_dict() ... ... optimizer.load_state_dict(optim_state_dict) ... ``` Args: state_dict (dict[str, Any]) : A ``state_dict`` to load into the optimizer. Note that this state dict update is performed in place. .. note:: PyTorch is using lazy init to initialize the optim states. So it is possible that there is no optim state when user call ``load_state_dict`` and for ``_NamedOptimizer`` we make it stricter that users can only call ``load_state_dict`` after the state is initialized. By doing this, we can validate the optim ``state_dict`` to be loaded. rLzJExpects the optim to be initialized before load but found not initialized.zExpects equal length as z for parameter z but found: zExpects state z but not found.z"Expects equal number of shards as z but found z for /rr'z"Expects equal param_group size as z for group .zExpects group key z to be in group z in `state_dict` but is missing.N)r,rM_pre_load_state_dictlenr3 enumerater/r2rBr rC local_shardsziptensordetachcopy_rrrr-_gen_param_group_keyr4load_state_dict)r5rMnew_state_dictrL new_stateidx param_key state_keyrQ src_state_val num_shardsnum_new_shardsshard src_shardsrc_param_groupsnew_param_groups src_group_mapr:rT new_group_map new_group group_key src_grouprVs&& r!rx_NamedOptimizer.load_state_dicts6335..z: 7#"7+ y>Q \ ((?(?@NC%5#$IN(;; .s9>/B.C?S\R]]ijmnsn~jjAB)2(<(<(>$ )$44$( ?9+_]!& 0 ; i77%m]CC,,!$Y%;%;%=!>J%()C)C)E%FN!3(@@PP[\f[gglmvlwwxzCyDE-0!..0-2L2L2N-( ++-33I4D4DE- 5<<88%mU\\BB,,$$&,,];080GIN9-3)?AJ&n5).9 %EeHo.J>CM.z: ;& )IJ&x00 !!$"9"9)"DE1>GM.z: ; * %2$7$7$9 Iy -%i0I9~Y/ 8Y8H T]S^^ijmnwjxiyyz{I%$,QC/? {Jjk=#+IaL#9IaL %:$ ''7r$c:<V^8dQhRS[S[S[3,RR/#)rrFrN)rrr)rr s"r!r"r#s$99738+<99r$c \V\4'g \R4hVR,p\V\P4'dV.VR&M\ V4VR&VP P4UUu/uFwr4WCbK pppVR,F5pWE9d \R4hVPPWT,4K7 VPPV4VPPVn R#uuppi)z Add a param group to the :class:`_NamedOptimizer` s `param_groups`. Warning: This API is still in development and subject to change. rAr'z%some parameters are not in the moduleN)rBrrCrrr-rr2r3r/r4r,add_param_groupr)r5rFr'r7r8r9s&& r!r_NamedOptimizer.add_param_groups +t,, !=> >X& fell + +%+HK !$(LK !595J5J5P5P5RS5Rzs 5R S **E( !HII  # # * *<+> ?+ '' 4 OO88Ts= Dc<V^8dQhRR/#r>r?)rr s"r!r"r#-s   D  r$c VPP4FQpVP'gK\P!V4p\P P V4VnKS VPRR7R#)z Run a dummy optimizer step, which allows us to initialize optimizer state because we do lazy init for most optimizers. This allows doing in-place loading of optimizer state from a checkpoint. Nrg) rr+ requires_gradr zeros_likeautogradVariablegradr_)r5r8ts& r! init_state_NamedOptimizer.init_state-s_ **113E"""$$U+"^^44Q7 4 $ r$cR<V^8dQhRS[S[S[3,RS[S[S[3,/#rrMrrJ)rr s"r!r"r#:s+tCH~$sCx.r$c\VP\4'd/\P!VPVPVRR7#V#)T)is_named_optimizer)rBrFSDPoptim_state_dict_to_loadr,r5rMs&&r!ro$_NamedOptimizer._pre_load_state_dict:sA dkk4 ( (00 T__jT r$cR<V^8dQhRS[S[S[3,RS[S[S[3,/#rrJ)rr s"r!r"r#Cs+4S>d38nr$c\VP\4'd-\P!VPVPV4V#r])rBrroptim_state_dictr,rs&&r!rO _NamedOptimizer._post_state_dictCs5 dkk4 ( (  ! !$++t Kr$)r,rrr/r)NNr])__name__ __module__ __qualname____firstlineno____doc__r;r*rMrr_propertyrLrxrrrorO__static_attributes____classdictcell__)r s@r!r r s)V)9)9V//&WW4555 >>55%%h8h8T992   r$r cFV^8dQhR\\,R\/#)rrTr)r-r)rs"r!r"r"Ks((T#Y(3(r$c6RP\V44#)zFConcatenate all param keys as a unique identifier for one param group.rm)joinrN)rTs&r!rwrwKs 88F:& ''r$cVV^8dQh/^\9d\\,;R&#)r__all__)__conditional_annotations__r-r)rs"r!r"r"scr$)rloggingr0collections.abcrrrcopyrtypingrrrtorch.nnrr 'torch.distributed._shard.sharded_tensorr torch.distributed.fsdpr rr getLoggerrloggerrr rwr")rs@r!rs[99 AC   8 $ueooup (r$