+ jA~^RIt^RIt^RIt^RIHt^RIHtHt^RIH t ^RI t ^RI H uH uHuHt^RIHtHt^RIHtHt^RIHt^RIHtHt^R IHtHtHt^R I H!t!H"t"H#t#^R I$H%t%^R I&H't'^R I(H)t)^RI*H+t+^RI,H-t-.ROt.]+!]/4t0]!RR44t1!RR4t2RRlt3RRlt4RRRllt5R#)N)Callable) dataclassfield)Any)get_default_numa_optionsjustknobs_check)eventsmetrics) WorkerSpec)create_healthcheck_serverHealthCheckServer)_AliveCallbackProxyLocalElasticAgentTORCHELASTIC_HEALTH_CHECK_PORT)DefaultLogsSpecs LogsSpecsSignalException)ChildFailedError)RendezvousParameters)parse_rendezvous_endpoint) get_logger) NumaOptions LaunchConfigelastic_launchca]tRt^/toRtRtRtRtRtRt ] !] R7t Rt ^tRtRtRt] !] R7tRtR tRtR tRtRtR tRtR tV3R ltRtVtR#)ra Creates a rendezvous config. Args: min_nodes: Minimum amount of nodes that the user function will be launched on. Elastic agent ensures that the user function start only when the min_nodes amount enters the rendezvous. max_nodes: Maximum amount of nodes that the user function will be launched on. nproc_per_node: On each node the elastic agent will launch this amount of workers that will execute user defined function. rdzv_backend: rdzv_backend to use in the rendezvous (zeus-adapter, etcd). rdzv_endpoint: The endpoint of the rdzv sync. storage. rdzv_configs: Key, value pair that specifies rendezvous specific configuration. rdzv_timeout: Legacy argument that specifies timeout for the rendezvous. It is going to be removed in future versions, see the note below. The default timeout is 900 seconds. run_id: The unique run id of the job (if not passed a unique one will be deduced from run environment - flow workflow id in flow - or auto generated). role: User defined role of the worker (defaults to "trainer"). max_restarts: The maximum amount of restarts that elastic agent will conduct on workers before failure. monitor_interval: The interval in seconds that is used by the elastic_agent as a period of monitoring workers. start_method: The method is used by the elastic agent to start the workers (spawn, fork, forkserver). metrics_cfg: configuration to initialize metrics. local_addr: address of the local node if any. If not set, a lookup on the local machine's FQDN will be performed. local_ranks_filter: ranks for which to show logs in console. If not set, show from all. event_log_handler: name of the event logging handler as registered in `elastic/events/handlers.py `_. duplicate_stdout_filters: If non-empty, duplicates stdout to a file containing only lines that match _any_ of the filter strings. duplicate_stderr_filters: If non-empty, duplicates stderr to a file containing only lines that match _any_ of the filter strings. virtual_local_rank: Enable virtual local rank mode for workers (defaults to False). When enabled, LOCAL_RANK is set to 0 for all workers and CUDA_VISIBLE_DEVICES is adjusted so each worker accesses its assigned GPU at device index 0. shutdown_timeout: Time in seconds to wait for graceful shutdown of workers before sending SIGKILL. Can also be set via TORCH_ELASTIC_SHUTDOWN_TIMEOUT environment variable. Defaults to 30 seconds. .. note:: `rdzv_timeout` is a legacy argument that will be removed in future. Set the timeout via `rdzv_configs['timeout']` N default_roleetcd)default_factoryg?spawnnullzSIGTERM,SIGINT,SIGHUP,SIGQUITFcRpVPR8wdVPVPR&MRVP9dWPR&VPf\4VnVPf\ P P4;'d,\ P P4VP8HpV'd0\4Vn\PRVP4VPf1\\ P"P%RR44VnR#VP^8d\'RVP 24hR# \dRpLi;i) itimeoutNFzUsing default numa options = %rTORCH_ELASTIC_SHUTDOWN_TIMEOUT30z+shutdown_timeout must be non-negative, got ) rdzv_timeout rdzv_configs logs_specsr numa_optionstorch accelerator is_available device_countnproc_per_node RuntimeErrorrloggerinfoshutdown_timeoutintosenvironget ValueError)selfdefault_timeoutshould_use_default_numas& _/home/luca/screen-checking/.venv/lib/python3.14/site-packages/torch/distributed/launcher/api.py __post_init__LaunchConfig.__post_init__}sK    "+/+<+! =t?P?PQ  ($' ?F%D ! " "Q &=d>S>S=TU '  0*/' 0s9$E*,E** E:9E:c<V^8dQh/S[;R&S[;R&S[;R&S[R,;R&S[;R&S[;R&S[;R&S[;R &S[S[S[3,;R &S[;R &S[;R &S[;R &S[;R&S[R,;R&S[S[S[3,;R&S[R,;R&S[;R&S[R,;R&S[;R&S[S[,R,;R&S[S[,R,;R&S[;R&S[R,;R&#) min_nodes max_nodesr/Nr)run_idrole rdzv_endpoint rdzv_backendr(r' max_restartsmonitor_interval start_methodlog_line_prefix_template metrics_cfg local_addrevent_log_handlerr*signals_to_handleduplicate_stdout_filtersduplicate_stderr_filtersvirtual_local_rankr3) r4rstrdictrfloatrlistbool)format __classdict__s"r< __annotate__LaunchConfig.__annotate__/slNmnNopqrD 'st uv wxyz{|sCx.>}~@AB!CDEF"Dj/GHc3h=IJd !KL#MN$+OPLLL!L+/"'"=K!J#'+L<1515$#']r[c@a]tRt^toRtRV3RlRlltRtRtVtR#)ra Launches a torchelastic agent on the container that invoked the entrypoint. 1. Pass the ``entrypoint`` arguments as non ``kwargs`` (e.g. no named parameters)/ ``entrypoint`` can be a function or a command. 2. The return value is a map of each worker's output mapped by their respective global rank. Usage :: def worker_fn(foo): # ... def main(): # entrypoint is a function. outputs = elastic_launch(LaunchConfig, worker_fn)(foo) # return rank 0's output return outputs[0] # entrypoint is a command and ``script.py`` is the python module. outputs = elastic_launch(LaunchConfig, "script.py")(args) outputs = elastic_launch(LaunchConfig, "python")("script.py") NcX<V^8dQhRS[RS[S[,R,RS[R,/#)r@config entrypointNhealth_check_server)rrrRr )rWrXs"r<rYelastic_launch.__annotate__s7888sNT)8/5 8r[c*WnW nW0nR#N_config _entrypoint_health_check_server)r9rgrhris&&&&r<__init__elastic_launch.__init__s  %$7!r[cn\VPVP\V4VPR7#))ri) launch_agentrnrorUrp)r9argss&*r<__call__elastic_launch.__call__s/ LL    J $ 9 9   r[rmrl) r\r]r^r_r`rqrvrbrcrds@r<rrs488  r[cvV^8dQhR\\,R,R\\,R\/#)r@rhNrureturn)rrRrUr)rWs"r<rYrYs/X^d%:$s)PSr[c\V\4'd VP#\V\4'd+V\P 8Xd\ RV4R4#V#R#)aRetrieve entrypoint name with the rule: 1. If entrypoint is a function, use ``entrypoint.__qualname__``. 2. If entrypoint is a string, check its value: 2.1 if entrypoint equals to ``sys.executable`` (like "python"), use the first element from ``args`` which does not start with hyphen letter (for example, "-u" will be skipped). 2.2 otherwise, use ``entrypoint`` value. 3. Otherwise, return empty string. c3D"TFq^,R8wgKVxK R#5i)r-N).0args& r< '_get_entrypoint_name..s>A# s  r) isinstancerr\rRsys executablenext)rhrus&&r<_get_entrypoint_namersR*h''""" J $ $  '>>C C r[cnV^8dQhR\R\\R,\R,3,/#)r@rdzv_parametersryN)rtuplerRr4)rWs"r<rYrYs.&&)& 3:sTz !"&r[cVPR8wdR#VPpVP4pV'g \R4h\ VRR7wr#VR8Xd\RV R24hW#3#)staticzKEndpoint is missing in endpoint. Try to add --master-addr and --master-port) default_portzport is missing in endpoint: z. Try to specify --master-port)NNr&)backendendpointstripr8r)rr master_addr master_ports& r<_get_addr_and_portrs(*''H~~H  Y   9PRSKb+H:5S T    %%r[c V^8dQhR\R\\,R,R\\,R\ R,R\ \\3,/#)r@rgrhNruriry)rrrRrUrr rSr4)rWs"r<rYrYs[b)b) b)3%b) s)b)+T1 b)  #s(^ b)r[c. VP'gE\\P!4P4p\ P RV4W@n\W4p\ PR/RVbRVPbRVPbRVPbRVPbRVPbRVPbR VPbR VPbR VP bR VP"P$bR VP&bRVP(bRVP*bRVP,bRVP.bRVP0b4\3R#RVPRVPRVPRVPRVPRVP4/VPBp\7V4wrxVP,\8P:R&Vfx\8P<!\>4p V eZ\ARRR7'dG\C\E4\ V 4^VP^p \aV \D4'dV PcV Pd4Rp \fPh!\fPj!VP&44V Pm4p\nPp!V Ps4VP(4VPu4'd\wVVPxR"7hVPzV 'dV P|P4## \Hd\ P RRR7RpELi;i \vdh\d4Rp \nPp!T P4TP(4h\Hd2\nPp!T P4TP(4hi;i T 'dT P|P4ii;i)$z3config has no run_id, generated a random run_id: %saRStarting elastic_operator with launch configs: entrypoint : %(entrypoint)s min_nodes : %(min_nodes)s max_nodes : %(max_nodes)s nproc_per_node : %(nproc_per_node)s run_id : %(run_id)s rdzv_backend : %(rdzv_backend)s rdzv_endpoint : %(rdzv_endpoint)s rdzv_configs : %(rdzv_configs)s max_restarts : %(max_restarts)s monitor_interval : %(monitor_interval)s log_dir : %(log_dir)s metrics_cfg : %(metrics_cfg)s event_log_handler : %(event_log_handler)s numa_options : %(numa_options)s signals_to_handle : %(signals_to_handle)s duplicate_stdout_filters : %(duplicate_stdout_filters)s duplicate_stderr_filters : %(duplicate_stderr_filters)s rhrArBr/rCrFrEr(rGrHlog_dirrKrMr*rNrOrPrrrLTORCHELASTIC_SIGNALS_TO_HANDLENzNai_infra/pytorch_distributed:torchelastic_enable_healthcheck_before_rendezvousF)default)alive_callbackportr#z>Started early health check server on port %s before rendezvousz)Failed to start early health check serverT)exc_info)rDlocal_world_sizerhru rdzv_handlerrGrHrrrLrMr*rOrPrQ)specr)rIrJr3ri)namefailuresr})BrCrRuuiduuid4r4r1warningrr2rArBr/rFrEr(rGrHr) root_log_dirrKrMr*rNrOrPrrLrr5r6getenvrrr rstart Exceptionr rDr rdzv_registryget_rendezvous_handlerrQrrIrJr3rr set_delegate_get_alive_timer initialize_metrics MetricsConfigrunr recordget_event_succeeded is_failedrr return_valuesrshutdownrget_event_failed)rgrhrurirCentrypoint_namerrrhealthcheck_portragentcb shutdown_rdzvresults&&&& r<rtrts ===TZZ\%%&LfU *:/B+Q)>$Q&%Q&)S,>;S,:2S,,S//%T)rrrtrl)6r5rrcollections.abcr dataclassesrrtypingrr+-torch.distributed.elastic.rendezvous.registry distributedelastic rendezvousregistryrtorch._utils_internalrrtorch.distributed.elasticr r *torch.distributed.elastic.agent.server.apir :torch.distributed.elastic.agent.server.health_check_serverr r :torch.distributed.elastic.agent.server.local_elastic_agentrrr)torch.distributed.elastic.multiprocessingrrr0torch.distributed.elastic.multiprocessing.errorsr$torch.distributed.elastic.rendezvousr*torch.distributed.elastic.rendezvous.utilsr'torch.distributed.elastic.utils.loggingrtorch.numa.bindingr__all__r\r1rrrrrtr}r[r<rs $( EEK5A  NEP>* = H  ll l^+ + \(&&b)b)r[