+ jU! ^RIt^RIHt^RIHt^RIt^RIHuHt ^RI HuH t ^RI H t ^RIHt^RI Ht^RIHtR.t]P&!]4t!RR 4t!R R 4tR tR tRtRRltRt!RR4tR#)N) defaultdict)Lock)Tensor)RRef)functional_optim_mapDistributedOptimizerc8a]tRt^toRtV3RlRltRtVtR#)_FunctionalLocalOptimizercVUu.uFqUP4NK upVnV!VP.VO5/VBVnR#uupiN local_value _local_paramsoptimself optim_clslocal_params_rrefargskwargsrrefs&&&*, b/home/luca/screen-checking/.venv/lib/python3.14/site-packages/torch/distributed/optim/optimizer.py__init__"_FunctionalLocalOptimizer.__init__D=NO=NT..0=NOt11CDCFC PAc <V^8dQhRS[/#)autograd_ctx_id)int)format __classdict__s"r __annotate__&_FunctionalLocalOptimizer.__annotate__sCc\P!V4pVPUu.uFpW29d W#,MRNK ppVPP V4R#uupir ) dist_autograd get_gradientsrrstep)rrall_local_gradspgradss&& rr)_FunctionalLocalOptimizer.steps^'55oF''& '#$"6O D @' &  & sArrN)__name__ __module__ __qualname____firstlineno__rr)__static_attributes____classdictcell__r"s@rr r sDr%r c:a]tRt^'to]!4tRtRtRtVt R#)_LocalOptimizercVUu.uFqUP4NK upVnV!VP.VO5/VBVnR#uupir r rs&&&*, rr_LocalOptimizer.__init__1rrc\P!V4p\P;_uu_4VP 4F wr4WCnK VP P4RRR4R# +'giR#;ir )r'r(r7 global_lockitemsgradrr))rrr*paramr=s&& rr)_LocalOptimizer.step5sU'55oF  ( ( (.446 !  7 JJOO ) ( ( (s :A33 B r.N) r/r0r1r2rr;rr)r3r4r5s@rr7r7's&KDr%r7cH\P!\W.VO5/VB4#r )rpcrr7)rrrrs&&*,r_new_local_optimizerrB>s 88OIR4R6R SSr%cHVP4pVPV4R#r rr)local_optim_rrefr local_optims&& r_local_optimizer_steprHBs"..0K_%r%cL\W.VO5/VBp\P!V4#r )r rAr)rrrrrs&&*, r_new_functional_local_optimizerrJGs$ %i TT TV TE 88E?r%c4V^8dQhR\R\RR/#)rrFrreturnN)rr )r!s"rr#r#Ls$&&&-0& &r%cHVP4pVPV4R#r rDrEs&& r _functional_local_optimizer_steprNLs #..0K_%r%cRp.pVF#pVPVP44K% VeVhV# \dpTPT4TpRp?KSRp?ii;ir )appendwait Exception)rpc_futs exceptionresultsfutes& r _wait_for_allrXSsbIG  NN388: &  N   NN1 I s6 AAAc0a]tRt^btoRtRtRtRtVtR#)ra DistributedOptimizer takes remote references to parameters scattered across workers and applies the given optimizer locally for each parameter. This class uses :meth:`~torch.distributed.autograd.get_gradients` in order to retrieve the gradients for specific parameters. Concurrent calls to :meth:`~torch.distributed.optim.DistributedOptimizer.step`, either from the same or different clients, will be serialized on each worker -- as each worker's optimizer can only work on one set of gradients at a time. However, there is no guarantee that the full forward-backward-optimizer sequence will execute for one client at a time. This means that the gradients being applied may not correspond to the latest forward pass executed on a given worker. Also, there is no guaranteed ordering across workers. `DistributedOptimizer` uses a functional optimizer internally when one is available for the given ``optimizer_class``, so that optimizer updates are not blocked by the Python Global Interpreter Lock (GIL) in the case of multithreaded training (e.g. Distributed Model Parallel). This feature is currently enabled for most optimizers. Args: optimizer_class (optim.Optimizer): the class of optimizer to instantiate on each worker. params_rref (list[RRef]): list of RRefs to local or remote parameters to optimize. args: arguments to pass to the optimizer constructor on each worker. kwargs: arguments to pass to the optimizer constructor on each worker. Example:: >>> # xdoctest: +SKIP("distributed") >>> import torch.distributed.autograd as dist_autograd >>> import torch.distributed.rpc as rpc >>> from torch import optim >>> from torch.distributed.optim import DistributedOptimizer >>> >>> with dist_autograd.context() as context_id: >>> # Forward pass. >>> rref1 = rpc.remote("worker1", torch.add, args=(torch.ones(2), 3)) >>> rref2 = rpc.remote("worker1", torch.add, args=(torch.ones(2), 1)) >>> loss = rref1.to_here() + rref2.to_here() >>> >>> # Backward pass. >>> dist_autograd.backward(context_id, [loss.sum()]) >>> >>> # Optimizer. >>> dist_optim = DistributedOptimizer( >>> optim.SGD, >>> [rref1, rref2], >>> lr=0.05, >>> ) >>> dist_optim.step(context_id) c~\PPR4\\4pVF(pWVP 4,P V4K* V\9d8\PP'd\P!V4pMTpWq8gVn VP'd\pM\PRV4\ p.p VP#4F8wr\$P&!V VW{3V,VR7p V P V 4K: \)V 4VnR#)z,torch.distributed.optim.DistributedOptimizera3Creating the optimizer %s without a functional optimizer, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need a functional optimizer for this optimizer. )rrN)torch_C_log_api_usage_oncerlistownerrPrjit_state_enabledgetis_functional_optimrJloggerwarningrBr<rA rpc_asyncrXremote_optimizers) roptimizer_class params_rrefrrper_worker_params_rrefr> optim_ctoroptimizer_new_funcremote_optim_futsworker param_rrefsremote_optim_rref_futs &&&*, rrDistributedOptimizer.__init__s  $$%ST!,T!2 E ";;= 1 8 8 ?! 2 2szz7J7J7J-11/BJ(J#-#@  # # #!@  NN<   "6 #9#?#?#A F$'MM" .5 % !  $ $%: ;$B"//@!Ar%c  \P!V4VP'd\M\pVP Uu.uF+p\ P!VP4VW13R7NK- pp\V4R#uupi)a Performs a single optimization step. This will call :meth:`torch.optim.Optimizer.step` on each worker containing parameters to be optimized, and will block until all workers return. The provided ``context_id`` will be used to retrieve the corresponding :class:`~torch.distributed.autograd.context` that contains the gradients that should be applied to the parameters. Args: context_id: the autograd context id for which we should run the optimizer step. )rN) r'_is_valid_contextrdrNrHrhrArgr_rX)r context_idoptimizer_step_func optimizerrSs&& rr)DistributedOptimizer.steps '' 3''' -& "33  4 MM!#,  4   h s1B)rdrhN) r/r0r1r2__doc__rr)r3r4r5s@rrrbs6p#BJ  r%)logging collectionsr threadingrr[torch.distributed.autograd distributedautogradr'torch.distributed.rpcrA torch.jitr`rrutilsr__all__ getLoggerr/rer r7rBrHrJrNrXrr%rrs}# 22##&' " "   8 $  $.T&  & | | r%