+ –°³jÐ4ãóª€^RIt^RIHtHt^RIHt^RIHtHt^RI t ^RI H t .R Ot !RR4t!R R]4t!R R ]4t!R R4tR#)éN)ÚABCÚabstractmethod)Ú TracebackType)ÚAnyÚ NamedTupleÚJoinHookÚJoinableÚJoincóHa€]tRt^toRtV3RlRltV3RlRltRtVtR#)ra¯ This defines a join hook, which provides two entry points in the join context manager. Entry points : a main hook, which is called repeatedly while there exists a non-joined process, and a post-hook, which is called once all processes have joined. To implement a join hook for the generic join context manager, define a class that inherits from :class:`JoinHook` and override ``main_hook()`` and ``post_hook()`` as appropriate. có<€V^8„dQhRR/#©éÚreturnN©)ÚformatÚ __classdict__s"€Úb/home/luca/screen-checking/.venv/lib/python3.14/site-packages/torch/distributed/algorithms/join.pyÚ __annotate__ÚJoinHook.__annotate__sø€÷ ñ ˜4ñ óc ó€R#)zÆCall this hook while there exists a non-joined process to shadow collective communications in a training iteration. Training iteration i.e., in one forward pass, backward pass, and optimizer step. Nr©Úselfs&rÚ main_hookÚJoinHook.main_hookó‚rcó$<€V^8„dQhRS[RR/#)rÚis_last_joinerrN©Úbool)rrs"€rrr sø€÷  ñ  ©ð  °ñ  rc ó€R#)a Call hook after all processes have joined. It is passed an additional ``bool`` argument ``is_last_joiner``, which indicates if the rank is one of the last to join. Arguments: is_last_joiner (bool): ``True`` if the rank is one of the last to join; ``False`` otherwise. Nr)rrs&&rÚ post_hookÚJoinHook.post_hook rrrN) Ú__name__Ú __module__Ú __qualname__Ú__firstlineno__Ú__doc__rr"Ú__static_attributes__Ú__classdictcell__©rs@rrrsø‡€ñ ÷ ð ÷  ö  rcó´aa€]tRt^,toRt]V3RlV3Rll4t]V3RlRl4t]]V3RlRl44t ]]V3RlR l44t R t Vt V;t #) r aC This defines an abstract base class for joinable classes. A joinable class (inheriting from :class:`Joinable`) should implement :meth:`join_hook`, which returns a :class:`JoinHook` instance, in addition to :meth:`join_device` and :meth:`join_process_group` that return device and process group information, respectively. có<€V^8„dQhRR/#r r)rrs"€rrÚJoinable.__annotate__8sø€÷IñI˜$ñIrcóV<€\SV`4\P4VnR#©N)ÚsuperÚ__init__Ú _JoinConfigÚconstruct_disabled_join_configÚ _join_config)rÚ __class__s&€rr2ÚJoinable.__init__7sø€ä ‰ÑÔÜ'×FÑFÓHˆÖrcó <€V^8„dQhRS[/#©rr)r)rrs"€rrr.=sø€÷  ñ  ¡Xñ  rc ó€R#)aV Return a :class:`JoinHook` instance for the given :class:`Joinable`. Arguments: kwargs (dict): a :class:`dict` containing any keyword arguments to modify the behavior of the join hook at run time; all :class:`Joinable` instances sharing the same join context manager are forwarded the same value for ``kwargs``. Nr)rÚkwargss&,rÚ join_hookÚJoinable.join_hook<s€ñ rcó4<€V^8„dQhRS[P/#r9)ÚtorchÚdevice)rrs"€rrr.Ksø€÷ ñ ™UŸ\™\ñ rc ó€R#)zeReturn the device from which to perform collective communications needed by the join context manager.Nrrs&rÚ join_deviceÚJoinable.join_deviceIó€ñ rcó <€V^8„dQhRS[/#r9)r)rrs"€rrr.Qsø€÷ ñ ¡Cñ rc ó€R#)zfReturns the process group for the collective communications needed by the join context manager itself.Nrrs&rÚjoin_process_groupÚJoinable.join_process_groupOrDr)r5)r$r%r&r'r(rr2r<ÚpropertyrBrGr)r*Ú __classcell__)r6rs@@rr r ,swù‡€ñð÷IòIóðIð÷  óð  ðØ÷ óóð ðØ÷ óó÷ ð rcó@a€]tRt^VtoRt]R4tV3RltRtVt R#)r3zdThis includes all fields needed from a :class:`Joinable` instance for the join context manager side.c ó€\RRRR7#)z”Return a :class:`_JoinConfig` instance indicating that join-related logic should be disabled. e.g. if the caller is not in a join context manager. F©ÚenableÚthrow_on_early_terminationÚis_first_joinable)r3rrrr4Ú*_JoinConfig.construct_disabled_join_config]s€ô Ø°UÈeô ð rcó><€V^8„dQh/S[;R&S[;R&S[;R&#)rrNrOrPr)rrs"€rrÚ_JoinConfig.__annotate__Vs+ø‡‚ñ Lññ!%Ñ$ñ ñ Ñò rrN) r$r%r&r'r(Ú staticmethodr4Ú__annotate_func__r)r*r+s@rr3r3Vs#ø‡€Ùoð ñ óð ÷ƒrr3cóža€]tRt^htoRtRV3RlRlltV3RlRltV3RlRltRtV3R lR lt R t R t ] V3R lRl4t RtVtR#)r a This class defines the generic join context manager, which allows custom hooks to be called after a process joins. These hooks should shadow the collective communications of non-joined processes to prevent hanging and erroring and to ensure algorithmic correctness. Refer to :class:`JoinHook` for details about the hook definition. .. warning:: The context manager requires each participating :class:`Joinable` to call the method :meth:`notify_join_context()` before its own per- iteration collective communications to ensure correctness. .. warning:: The context manager requires that all ``process_group`` attributes in the :class:`JoinHook` objects are the same. If there are multiple :class:`JoinHook` objects, then the ``device`` of the first is used. The process group and device information is used for checking for non- joined processes and for notifying processes to throw an exception if ``throw_on_early_termination`` is enabled, both of which using an all- reduce. Arguments: joinables (List[Joinable]): a list of the participating :class:`Joinable` s; their hooks are iterated over in the given order. enable (bool): a flag enabling uneven input detection; setting to ``False`` disables the context manager's functionality and should only be set when the user knows the inputs will not be uneven (default: ``True``). throw_on_early_termination (bool): a flag controlling whether to throw an exception upon detecting uneven inputs (default: ``False``). Example:: >>> import os >>> import torch >>> import torch.distributed as dist >>> import torch.multiprocessing as mp >>> # xdoctest: +SKIP >>> import torch.nn.parallel.DistributedDataParallel as DDP >>> import torch.distributed.optim.ZeroRedundancyOptimizer as ZeRO >>> from torch.distributed.algorithms.join import Join >>> >>> # On each spawned worker >>> def worker(rank): >>> dist.init_process_group("nccl", rank=rank, world_size=2) >>> model = DDP(torch.nn.Linear(1, 1).to(rank), device_ids=[rank]) >>> optim = ZeRO(model.parameters(), torch.optim.Adam, lr=0.01) >>> # Rank 1 gets one more input than rank 0 >>> inputs = [torch.tensor([1.]).to(rank) for _ in range(10 + rank)] >>> with Join([model, optim]): >>> for input in inputs: >>> loss = model(input).sum() >>> loss.backward() >>> optim.step() >>> # All ranks reach here without hanging/erroring có<<€V^8„dQhRS[S[,RS[RS[/#)rÚ joinablesrNrO)Úlistr r )rrs"€rrÚJoin.__annotate__¦s-ø€÷"ñ"ᙕ>ð"ñð"ñ%)ñ "rc ó €\V4^8Xd \R4hWnVPUu.uFqUP!R/VBNK upVnW nW0nVP4VP4R#uupi)rz7The join context manager requires at least one joinableNr) ÚlenÚ ValueErrorÚ _joinablesr<Ú _join_hooksÚ_enableÚ_throw_on_early_terminationÚ_set_joinable_configsÚ_extract_dist_info)rrXrNrOr;Újoinables&&&&, rr2Ú Join.__init__¦sw€ô ˆy‹>˜QÔ ÜÐVÓWÐ WØ#Œà9=¿ºó Ù9H¨X× Ò Ñ ( Ô (¹ñ ˆÔðŒ Ø+EÔ(Ø ×"Ñ"Ô$Ø ×ÑÖ!ùò  s°Bcó<€V^8„dQhRR/#r r)rrs"€rrrZ¸sø€÷ &ñ & tñ &rc ó€\VP4^8:d\hRpVPF,p\VPVP VR7VnRpK. R#)zESet the :class:`_JoinConfig` of each participating :class:`Joinable`.TrMFN)r\r^ÚAssertionErrorr3r`rar5)rrPrds& rrbÚJoin._set_joinable_configs¸sU€ä ˆt‰Ó  1Ô $Ü Ð Ø ÐØŸœˆHÜ$/Ø—|‘|Ø+/×+KÑ+KØ"3ô%ˆHÔ !ð !&Ò ó (rcó<€V^8„dQhRR/#r r)rrs"€rrrZÅsø€÷ñ Dñrc ó€RpRpVPFApVfVPpMWP8wd \R4hVeK5VPpKC Wn\ P !VP4VnW nR#)as Extract the process group and device information from the joinables. If there are multiple joinables, then the context manager uses the first specified device. Preconditions: ``self._joinables`` is not ``None`` and is non-empty. Raises: ValueError If there are multiple conflicting ``process_group`` attributes among the ``Joinable`` objects. Nz7Using join context manager with multiple process groups) r^rGr]rBÚ_process_groupÚdistÚget_rankÚ_rankÚ_device)rÚ process_groupr@rds& rrcÚJoin._extract_dist_infoÅs~€ðˆ ØˆàŸœˆHØÒ$Ø (× ;Ñ ;‘ Ø×"=Ñ"=Ô=Ü ØMóððŒ~Ø!×-Ñ-’ñ(ð,ÔÜ—]’] 4×#6Ñ#6Ó7ˆŒ ØŽ rcó€R#r0rrs&rÚ __enter__ÚJoin.__enter__äs€™rcóf<€V^8„dQhRS[S[,R,RS[R,RS[R,/#)rÚtypeNÚvalueÚ traceback)rwÚ BaseExceptionr)rrs"€rrrZæs=ø€÷30ñ30á‘=Õ! DÕ(ð30ñ˜tÕ#ð30ñ! 4Õ'ñ 30rc  ó€VP'd V'dR#RpRp^pRp\P!R4V'g Wg8”d-\P!RV RVP RV R 2^R 7VP 4pV^8XdRpKUVP 'dVP4VPFp V P4K RpV^, pK§VPFp V PV4K R#) zŸ Repeatedly runs the main hooks until all processes join; then, runs the post-hooks. Raises: RuntimeError If ``throw_on_early_termination=True``. NFTièÚoncez+Detected uneven input skew of greater than z. This means that rank z has at least zz fewer inputs than other currently-active ranks. This level of skew could lead to performance degradation during training.)Ú stacklevel) r`ÚwarningsÚ simplefilterÚwarnroÚ_get_num_nonjoined_procsraÚ_notify_procs_to_terminater_rr") rrwrxryÚall_procs_joinedrÚiÚWARN_THRESHOLDÚnum_nonjoined_procsr<s &&&& rÚ__exit__Ú Join.__exit__æsû€ð||ˆ|ŸtÙ à ÐØˆà ˆØˆÜ×Ò˜fÔ%ç"ØÔ!Ü— ’ ØAØ%Ð&Ð&=Ø—z‘zl .°Ð0@ðA3ð3ð !õð#'×"?Ñ"?Ó"AÐ Ø" aÔ'Ø#'Ò à×3×3Ð3Ø×3Ñ3Ô5ð"&×!1Ô!1IØ×'Ñ'Ö)ñ"2ð"'ØQ•’ð×)Ô)ˆIØ × Ñ  Ö /ó*rc ó¨€\P!^VPR7p\P!WP R7VP 4#)zaReturn the number of non-joined processes by shadowing an all-reduce in the non-joined processes.©r@©Úgroup)r?ÚzerosrprmÚ all_reducerlÚitem)rr†s& rrÚJoin._get_num_nonjoined_procss9€ä#Ÿkšk¨!°D·L±LÔAÐÜ ŠÐ+×3FÑ3FÕGØ"×'Ñ'Ó)Ð)rc óº€\P!^VPR7p\P!WP R7\ RVP R24h)z¡Schedule an all-reduce to notify non-joined processes to terminate. Also raise a ``RuntimeError`` indicating that the current process has exhausted its inputs. rŠr‹zRank z exhausted all inputs.)r?ÚonesrprmrŽrlÚ RuntimeErrorro)rr’s& rr‚ÚJoin._notify_procs_to_terminate!sC€ô zŠz˜! D§L¡LÔ1ˆÜ Š˜×$7Ñ$7Õ8ܘU 4§:¡: ,Ð.DÐEÓFÐFrcó <€V^8„dQhRS[/#)rrd)r )rrs"€rrrZ+sø€÷5ñ5¡hñ5rc ó€\VR4'g\R\V4 R24hVPpVP'dVP 'gR#VP pVPp\P!^VR7p\P!WCRR7pVP'dS\P!^VR7p\P!WcR7VP4pV'd \R 4hV#) a¸ Notifies the join context manager that the calling process has not yet joined. Then, if ``throw_on_early_termination=True``, checks if uneven inputs have been detected (i.e. if one process has already joined) and throws an exception if so. This method should be called from a :class:`Joinable` object before its per-iteration collective communications. For example, this should be called at the beginning of the forward pass in :class:`DistributedDataParallel`. Only the first :class:`Joinable` object passed into the context manager performs the collective communications in this method, and for the others, this method is vacuous. Arguments: joinable (Joinable): the :class:`Joinable` object calling this method. Returns: An async work handle for the all-reduce meant to notify the context manager that the process has not yet joined if ``joinable`` is the first one passed into the context manager; ``None`` otherwise. r5zCheck that the z/ constructor calls the ``Joinable`` constructorNrŠT)rŒÚasync_opr‹zLDetected at least one rank that exhausted inputs. Throwing across all ranks.)Úhasattrrhrwr5rPrNrBrGr?r’rmrŽrOrrr“)rdÚ join_configr@rqr’ÚworkrÚ should_throws& rÚnotify_join_contextÚJoin.notify_join_context*sâ€ô4x ×0Ò0Ü Ø!¤$ x£.Ð!1ð2+ð+óð ð ×+Ñ+ˆ à×,×,Ð,°K×4F×4FÐ4FÙà×%Ñ%ˆØ ×3Ñ3ˆ ôzŠz˜! FÔ+ˆÜŠ˜tÀ4ÔHˆà × 1× 1Ð 1ä—K’K ¨&Ô1ˆEÜ OŠO˜EÕ 7Ø Ÿ:™:›<ˆLßÜ"ð1óððˆ r)rpr`r_r^rlroraN)TF)r$r%r&r'r(r2rbrcrtr‡rr‚rTrœr)r*r+s@rr r hsXø‡€ñ;÷z"ò"÷$ &ð &÷ðò>÷30ð30òj*ò Gð÷5óö5r)rr r )r~ÚabcrrÚtypesrÚtypingrrr?Útorch.distributedÚ distributedrmÚ__all__rr r3r rrrÚr¤sMðãß#Ýß"ã Ý ò +€÷ ñ ô<' ˆsô' ôT *ô ÷$xóxr