+ j^RIt^RIt^RIt^RIt^RIt^RIt^RIt^RIt^RIH t H t ^RI H t ^RI Ht^RIHtHtHt^RIHt^RIHt^RIHt^RIHt^R IHtHt^R IHtHtH t H!t!H"t"H#t#H$t$H%t%H&t&^R I'H(t(H)t).R;Ot*]PV!],4t-R!R0R14t?!R2R34t@!R4R54tA!R6R]!4tBR7R8ltCR9R:ltDR#)=N)ABCabstractmethod)Callable) dataclass)datetime timedeltatimezone)Enum)AnyStore)construct_and_record_rdzv_event NodeState) RendezvousClosedErrorRendezvousErrorRendezvousGracefulExitErrorRendezvousHandlerRendezvousInfoRendezvousParametersRendezvousStateErrorRendezvousStoreInfoRendezvousTimeoutError)_delay_PeriodicTimerRendezvousBackendRendezvousTimeoutRendezvousSettingsDynamicRendezvousHandlerc\\P!44V8d'\P!4V,P#R#)no_method_name)leninspectstackfunction)depths&x/home/luca/screen-checking/.venv/lib/python3.14/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.pyget_method_namer'4s/ 7==?e#}}u%... ca]tRt^>toRt]]V3RlRl44t]V3RlRl4t]R V3RlRll4t R t Vt R#) rz4Represent a backend that holds the rendezvous state.c <V^8dQhRS[/#returnstr)format __classdict__s"r& __annotate__RendezvousBackend.__annotate__Cs++c+r(c R#)zGet the name of the backend.Nselfs&r&nameRendezvousBackend.nameAr(cD<V^8dQhRS[S[S[3,R,/#r,r-N)tuplebytesToken)r0r1s"r&r2r3Gs"   5.5  r(c R#)a8Get the rendezvous state. Returns: A tuple of the encoded rendezvous state and its fencing token or ``None`` if no state is found in the backend. Raises: RendezvousConnectionError: The connection to the backend has failed. RendezvousStateError: The rendezvous state is corrupt. Nr5r6s&r& get_stateRendezvousBackend.get_stateFr:r(Nc b<V^8dQhRS[RS[R,RS[S[S[S[3,R,/#)r,statetokenNr-)r>r?r=bool)r0r1s"r&r2r3Vs:" " " #(4<" ueT! "T )" r(c R#)aSet the rendezvous state. The new rendezvous state is set conditionally: - If the specified ``token`` matches the fencing token stored in the backend, the state will be updated. The new state will be returned to the caller along with its fencing token. - If the specified ``token`` does not match the fencing token stored in the backend, the state won't be updated; instead the existing state along with its fencing token will be returned to the caller. - If the specified ``token`` is ``None``, the new state will be set only if there is no existing state in the backend. Either the new state or the existing state along with its fencing token will be returned to the caller. Args: state: The encoded rendezvous state. token: An optional fencing token that was retrieved by a previous call to :py:meth:`get_state` or ``set_state()``. Returns: A tuple of the serialized rendezvous state, its fencing token, and a boolean value indicating whether our set attempt succeeded. Raises: RendezvousConnectionError: The connection to the backend has failed. RendezvousStateError: The rendezvous state is corrupt. Nr5)r7rDrEs&&&r& set_stateRendezvousBackend.set_stateUr:r(r5N) __name__ __module__ __qualname____firstlineno____doc__propertyrr8rArH__static_attributes____classdictcell__r1s@r&rr>sO> ++    " " " r(c $a]tRt^{toRt]!^4tR]!RR7R]!^R7R]!^R7R]!^R7/tRV3R lR llt] V3R lR l4t ] V3R lRl4t ] V3RlRl4t ] V3RlRl4t V3RlRltV3RltRtVtR#)raJHold the timeout configuration of a rendezvous. Args: join: The time within which the rendezvous is expected to complete. last_call: An additional wait amount before completing the rendezvous once the rendezvous has the minimum number of required participants. close: The time within which the rendezvous is expected to close after a call to :py:meth:`RendezvousHandler.set_closed` or :py:meth:`RendezvousHandler.shutdown`. heartbeat: The time within which a keep-alive heartbeat is expected to complete. joiniXseconds last_callclose heartbeatNc n<V^8dQhRS[R,RS[R,RS[R,RS[R,RR/#)r,rUNrXrYrZr-r)r0r1s"r&r2RendezvousTimeout.__annotate__sQ   $  t#  4  t#    r(c,VPWW4R7R#))rUrXrYrZN) _set_timeouts)r7rUrXrYrZs&&&&&r&__init__RendezvousTimeout.__init__s %  r(c <V^8dQhRS[/#r+r\)r0r1s"r&r2r]sir(c VP#)zGet the join timeout.)_joinr6s&r&rURendezvousTimeout.joinszzr(c <V^8dQhRS[/#r+r\)r0r1s"r&r2r]9r(c VP#)zGet the last call timeout.) _last_callr6s&r&rXRendezvousTimeout.last_callr(c <V^8dQhRS[/#r+r\)r0r1s"r&r2r]syr(c VP#)zGet the close timeout.)_closer6s&r&rYRendezvousTimeout.close{{r(c <V^8dQhRS[/#r+r\)r0r1s"r&r2r]rgr(c VP#)z%Get the keep-alive heartbeat timeout.) _heartbeatr6s&r&rZRendezvousTimeout.heartbeatrkr(c.<V^8dQhRS[R,/#)r,timeoutsNr\)r0r1s"r&r2r]s// D(8/r(c VP4FRwr#VfVPV,pW0P8:d\RV RV R24h\ VRV,V4KT R#)NzThe z timeout (z) must be positive._)items_DEFAULT_TIMEOUTS_ZERO ValueErrorsetattr)r7rvr8timeouts&, r&r_RendezvousTimeout._set_timeoutss_%^^-MD006**$ 4vZy@S!TUU D#*g . .r(cJ<V^8dQh/S[;R&S[;R&S[;R&S[;R&#)r,rdrirnrsr\)r0r1s"r&r2r]{s96 789: ;<=r(r5)NNNN)rKrLrMrNrOrr{rzr`rPrUrXrYrZr___annotate_func__rQrRrSs@r&rr{s" aLE  #&Yr*2&Yq)    //r(FT)repreqfrozenc0a]tRt^toRtV3RltRtVtR#)raOHold the settings of the rendezvous. Attributes: run_id: The run id of the rendezvous. min_nodes: The minimum number of nodes to admit to the rendezvous. max_nodes: The maximum number of nodes to admit to the rendezvous. timeout: The timeout configuration of the rendezvous. keep_alive_interval: The amount of time a node waits before sending a heartbeat to keep it alive in the rendezvous. keep_alive_max_attempt: The maximum number of failed heartbeat attempts after which a node is considered dead. cb<V^8dQh/S[;R&S[;R&S[;R&S[;R&S[;R&S[;R&#)r,run_id min_nodes max_nodesr~keep_alive_intervalkeep_alive_max_attempt)r/intrr)r0r1s"r&r2RendezvousSettings.__annotate__sP* K+,N-.N/012#"34 5r(r5N)rKrLrMrNrOrrQrRrSs@r&rrsr()rorderrcBa]tRt^toRtV3RlRltV3RltRtVtR#) _NodeDesczDescribe a node in the rendezvous. Attributes: addr: The FQDN of the node or user specified local node address. pid: The id of the process in which the rendezvous handler runs. local_id: A process-wide unique id. c <V^8dQhRS[/#r+r.)r0r1s"r&r2_NodeDesc.__annotate__s99#9r(cRVP RVP RVP 2#)rx)addrpidlocal_idr6s&r&__repr___NodeDesc.__repr__s%))AdhhZq88r(c><V^8dQh/S[;R&S[;R&S[;R&#)r,rrr)r/r)r0r1s"r&r2rs) I HMr(r5N) rKrLrMrNrOrrrQrRrSs@r&rrs 99#r(rcXa]tRt^toRtV3RlRltR V3RlRlltV3RltRtVt R#) _NodeDescGeneratorzGenerate node descriptors. A node descriptor is a combination of an FQDN, a process id, and an auto- incremented integer that uniquely identifies a node in the rendezvous. c<V^8dQhRR/#r<r5)r0r1s"r&r2_NodeDescGenerator.__annotate__s$r(cH\P!4Vn^VnR#)rN) threadingLock_lock _local_idr6s&r&r`_NodeDescGenerator.__init__s^^% r(Nc4<V^8dQhRS[R,RS[/#)r, local_addrNr-)r/r)r0r1s"r&r2rs#PP3:PPr(c&VP;_uu_4VPpV;P^, unRRR4\T;'g\P!4\ P !4X4# +'giLM;i)N)rrrsocketgetfqdnosgetpid)r7rrs&& r&generate_NodeDescGenerator.generatesWZZZ~~H NNa N 77v~~'7hOO Zs &B B cF<V^8dQh/S[P;R&S[;R&#)r,rr)rrr)r0r1s"r&r2rs" >>Nr()rrrJ) rKrLrMrNrOr`rrrQrRrSs@r&rrs' PP!r(rcBa]tRtRtoRtV3RlRltV3RltRtVtR#)_RendezvousStateiaHold the state of a rendezvous. Attributes: round: The current round of the rendezvous. complete: A boolean value indicating whether the current round of the rendezvous is complete. deadline: The time at which the current round of the rendezvous will be considered complete if it is still waiting for nodes to join. closed: A boolean value indicating whether the rendezvous is closed. participants: A dictionary of the participants and their corresponding ranks. wait_list: A set of nodes that are waiting to participate in the next round of the rendezvous. redundancy_list: A set of nodes that are redundant in the current round and can join the next rendezvous without triggering re-rendezvous. last_heartbeats: A dictionary containing each node's last heartbeat time. c<V^8dQhRR/#r<r5)r0r1s"r&r2_RendezvousState.__annotate__3s""$"r(c^VnRVnRVnRVn/Vn\ 4Vn\ 4Vn/VnR#)rFN) roundcompletedeadlineclosed participantsset wait_listredundancy_listlast_heartbeatsr6s&r&r`_RendezvousState.__init__3sA    "u!r(c<V^8dQh/S[;R&S[;R&S[R,;R&S[;R&S[S[S[3,;R&S[S[,;R&S[S[,;R&S[S[S[3,;R &#) r,rrNrrrrrr)rrFrdictrr)r0r1s"r&r2rs4 J56N78o9: L;<y#~&&=>9~?@^#AB)X-..Cr()rrrrrrrrN rKrLrMrNrOr`rrQrRrSs@r&rrsD""Gr(rc4V^8dQhR\R\RR/#)r,rDsettingsr-N)rr)r0s"r&r2r2>s$"" "'9" "r(cVP'dNVP'g:Rp\PV4RVnV;P^, unR#R#\ VP4VP 8dER\ VP4 RVP R2p\PV4RVnR#R#)zHNo participants left in the rendezvous, marking rendezvous as incompleteFzNumber of participants z less than min_nodes z, clearing deadline in stateN)rrloggerdebugrr!rr)rDrmsgs&& r&_remove_participant_epiloguer>s ~~~!!!\C LL "EN KK1 K " u!! "X%7%7 7)#e.@.@*A)BC%//00LN  LL !EN 8r(ca]tRtRtoRt]]V3RlRl44t]V3RlRl4t]V3RlRl4t R t Vt R #) _RendezvousStateHolderiSz9Hold the shared rendezvous state synced with other nodes.c <V^8dQhRS[/#r+r)r0r1s"r&r2#_RendezvousStateHolder.__annotate__Xs##'#r(c R#)zGet the local state.Nr5r6s&r&rD_RendezvousStateHolder.stateVr:r(c.<V^8dQhRS[R,/#r<rF)r0r1s"r&r2r\s  dTk r(c R#)zRead or writes the latest state. Returns: A boolean value indicating whether the local state, in case marked as dirty, was successfully synced with other nodes. Nr5r6s&r&sync_RendezvousStateHolder.sync[r:r(c<V^8dQhRR/#r<r5)r0r1s"r&r2res--D-r(c R#)zMark the local state as dirty.Nr5r6s&r& mark_dirty!_RendezvousStateHolder.mark_dirtydr:r(r5N) rKrLrMrNrOrPrrDrrrQrRrSs@r&rrSsJC ##  --r(rca]tRtRtoRtRV3RlRllt]P3V3RlRllt] V3RlRl4t V3R lR lt V3R lR lt V3R lRlt V3RltRtVtR#)_BackendRendezvousStateHolderiia?Hold the rendezvous state synced with other nodes via a backend. Args: backend: The rendezvous backend to use. settings: The rendezvous settings. cache_duration: The amount of time, in seconds, to cache the last rendezvous state before requesting it from the backend again. c0<V^8dQhRS[RS[RS[RR/#)r,backendrcache_durationr-N)rrr)r0r1s"r&r2*_BackendRendezvousStateHolder.__annotate__s3  " %    r(cWn\4VnW nW0nRVnRVnRVn.VnR#)NF) _backendr_state _settings_cache_duration_token_dirty_last_sync_time _dead_nodes)r7rrrs&&&&r&r`&_BackendRendezvousStateHolder.__init__s<  &( !-  !r(c&<V^8dQhRS[RS[/#)r,message node_stater/r)r0r1s"r&r2rs  s  r(c\VPP R\4 2VPP VVR7R#).)r8rrrN)r __class__rKr'rrr7rrs&&&r&_record%_BackendRendezvousStateHolder._records:'NN++,Ao.?-@A>>((!  r(c <V^8dQhRS[/#r+r)r0r1s"r&r2rs'r(c VP#See base class.)rr6s&r&rD#_BackendRendezvousStateHolder.staterpr(c.<V^8dQhRS[R,/#r<r)r0r1s"r&r2rs;;dTk;r(c 2RpRpVP'dRRp\P!VP4pVPP WP 4pVeVwrpMvRpVP^8dBVP\\P!4VP, ^48dR#VPP4pVeVwrVe\P!V4VnM\!4VnV'dVP"'d\$P'\(P*4'dgRP-RVP"44pRV RVP.P0 R2pVP3VR 7\$P5V4W nRVn\P!4VnVP74V# \Pdp\R4ThRp?ii;i) rNFzAThe rendezvous state is corrupt. See inner exception for details.z, c3."TF pRV R2xK R#5i)'Nr5).0 dead_nodes& r& 5_BackendRendezvousStateHolder.sync..s!UDTyAi["2DTsz*As part of the sync operation the node(s) z( have been removed from the rendezvous 'z' since they had no heartbeat.r)rpickledumpsrrrHrrrmaxtime monotonicrAloads PickleErrorrrrr isEnabledForloggingDEBUGrUrrrr _sanitize) r7 state_bitsrEhas_set set_response get_responseexc node_listrs & r&r"_BackendRendezvousStateHolder.syncs#'  ;;;Gdkk2J==22:{{KL'-9* 7G##a'''3NN$t';';;Q, ==224L'$0!  ! $ll:6 +,DK t'''F,?,? ,N,N !UDDTDT!UUI=YKH#~~4455SU  LLL % LL   #~~/ 3%% *W s#G00H HHc<V^8dQhRR/#r<r5)r0r1s"r&r2rs)@)@4)@r(cVPp\P!\P4VP P VP P,, pVPP4UUu.uFwr4WB8gK VNK uppVn RpVPFspRV R2p\PV4VPVVPVRpVPP!V4VP"P!V4Ku V'd\%WP 4R#R#uuppi \dLpi;i \dLei;i \dKi;i)FzDetected dead node 'z"', removing it from the rendezvousTN)rrnowr utcrrrrryrrrrKeyErrorrremoverr)r7rD expire_timenodelast_heartbeatparticipant_removedrrs& r&r '_BackendRendezvousStateHolder._sanitizes[ ll8<<0 NN . .1V1V V ).(=(=(C(C(E (E$+ D(E  $))I( 3UVC LL %%i0 &&y1&*# &&y1 %%,,Y7%*,  ) ? = "        sB EEE,EE0 EE E-,E-0 E?>E?c<V^8dQhRR/#r<r5)r0r1s"r&r2rsDr(c RVnR#)zSee base class. If the local rendezvous state is dirty, the next sync call will try to write the changes back to the backend. However this attempt might fail if another node, which had the same state, also made changes and wrote them before us. TN)rr6s&r&r(_BackendRendezvousStateHolder.mark_dirtys  r(c<V^8dQh/S[;R&S[;R&S[;R&S[;R&S[;R&S[;R&S[;R&S[S[,;R&#) r,rrrrrrrr) rrrrr?rFfloatlistr)r0r1s"r&r2riso  "! !" M#$ L%&'(i )r()rrrrrrrrN)r)rKrLrMrNrOr`rRUNNINGrrPrDrr rrrQrRrSs@r&rrish *  =F<V^8dQh/S[;R&S[;R&S[;R&#)r,rrDrr7)r0r1s"r&r2r8 s+ O ! r()rrrDNrrSs@r&r5r5 s  !!#r(r5cDa]tRtRtoRt]RV3RlRll4tRtVtR#)_RendezvousOpExecutori9zExecute rendezvous operations.Ncr<V^8dQhRS[S[S[.S[3,RS[RS[S[.S[3,R,RR/#r, state_handlerrupdate_deadlineNr-rr5r"r&r)r0r1s"r&r2"_RendezvousOpExecutor.__annotate__=sT  !3U ;W DE  "9+u"45<    r(c R#)a+Execute a rendezvous operation. An operation is run inside a state machine and is expected to transition the rendezvous from one state to another. Args: state_handler: A callable that is expected to return the next state transition action based on the current state of the rendezvous. deadline: The time, in seconds, at which the operation will be considered timed-out. update_deadline: Function to generate a new operation deadline if the current node may participate in the next rendezvous. Nr5)r7r@rrAs&&&&r&run_RendezvousOpExecutor.run<r:r(r5rJ) rKrLrMrNrOrrErQrRrSs@r&r=r=9s(   r(r=c&a]tRtRtoRtV3RlRlt]P3V3RlRlltRV3RlR llt V3R lR lt V3R lR lt V3RlRlt V3RlRlt V3RlRltV3RlRltV3RlRltV3RlRltV3RlRltV3RltRtVtR#) _DistributedRendezvousOpExecutoriUaGExecute rendezvous operations using a shared state. Args: node: The node descriptor associated with the current rendezvous handler instance. state_holder: The ``RendezvousStateHolder`` to use to sync the rendezvous state with other nodes. settings: The rendezvous settings. c0<V^8dQhRS[RS[RS[RR/#)r,r state_holderrr-N)rrr)r0r1s"r&r2-_DistributedRendezvousOpExecutor.__annotate__hs3"""-"% "  "r(c*WnW nW0nR#rJ)_node _state_holderr)r7rrJrs&&&&r&r`)_DistributedRendezvousOpExecutor.__init__hs  )!r(c*<V^8dQhRS[RS[RR/#)r,rrr-Nr)r0r1s"r&r2rKrs#   s   RV  r(c  \VPP R\4 2VPP VVVP PVP PVP PR7R#)r)r8rrrhostnamerrN) rrrKr'rrrMrrrrs&&&r&r(_DistributedRendezvousOpExecutor._recordrs]'NN++,Ao.?-@A>>((!ZZ__ ZZ(( r(Ncr<V^8dQhRS[S[S[.S[3,RS[RS[S[.S[3,R,RR/#r?rB)r0r1s"r&r2rK}sZI0I0!3U ;W DEI0I0"9+u"45< I0  I0r(c 4RpV\P8wEdVPP4pVeV'd)RVP RVP P R2pM'RVP RVP P R2pVPVR7\PV4VPPVn \VPVPVP 4pV!Wr4pV\P8XdEKV\P8Xd\hV\P8Xd\ hV\P"8Xd\%^R7EKrV\P&8XdVP)4EM_V\P*8XdVP-4EM8V\P.8XdVP14EMV\P28XdVP54MV\P68XdVP94MV\P:8XdVP=4MV\P>8Xd@VPA4V'd'V!VP PBPD4pMKV\PF8XdVPI4M%V\PJ8XdVPM4VPPO4EKR#)rN The node 'zP' has successfully synced its local changes with other nodes in the rendezvous ''.z]' has a stale state and failed to sync its local changes with other nodes in the rendezvous 'rrV)(r&r3rNrrMrrrrrrDrr5r1rr2rr0rr' _keep_aliver(_add_to_participantsr)_add_to_wait_listr*_add_to_redundancy_listr+_remove_from_participantsr,_remove_from_wait_listr-_remove_from_redundancy_listr~rUr._mark_rendezvous_completer/_mark_rendezvous_closedr)r7r@rrAactionr rctxs&&&& r&rE$_DistributedRendezvousOpExecutor.run}sZ&((--/G"$TZZL1::>..:O:O9PPRT %TZZL1GGK~~G\G\F]]_a  S ) S!,,22DK$TZZdnnMC#31F'---++...,,%q!!W///$$&w:::--/w777**,w===002w???224w<<<//1wBBB557>>3I3I3N3N#Ow???224w===002""--/C'r(c<V^8dQhRR/#r<r5)r0r1s"r&r2rKsMMTMr(c,RVP RVPP R2pVPVR7\P V4\ P!\P4VPPVP&R#)rVz<' updated its keep-alive heartbeat time for the rendezvous ''. Pending sync.rN) rMrrrrrrrr rrrr7rs& r&rX,_DistributedRendezvousOpExecutor._keep_alivesp %%%&&6 8  S ! S2:,,x||2L ##DJJ/r(c<V^8dQhRR/#r<r5)r0r1s"r&r2rKs--d-r(c RVP RVPP RVPP R2pVP VR7\ PV4VPpVPPVP4^VPVP&VP4\VP4VPP8XdO\P !\"P$4VPP&P(,Vn\VP4VPP,8XdVP/4R#R# \dLi;i)rVz,' added itself to the participants of round  of the rendezvous 'rfrN)rMrrrrrrrrrrrrXr!rrrr rr~rXrrr_r7rrDs& r&rY5_DistributedRendezvousOpExecutor._add_to_participantss/ $P{{  !!5dnn6K6K5LL\ ^  S ! S   OO " "4:: . *+4::&  u!! "dnn&>&> > X\\*T^^-C-C-M-MM N u!! "dnn&>&> >  * * , ?   s3%E?? F  F c<V^8dQhRR/#r<r5)r0r1s"r&r2rKs  4 r(cRVP RVPP^, RVPP R2pVP VR7\ PV4VPVPP9d0VPPPVP4VPPPVP4VP4R#)rVz)' added itself to the wait list of round rkrfrN) rMrrrrrrrrrraddrXrgs& r&rZ2_DistributedRendezvousOpExecutor._add_to_wait_lists $M{{  1$%%9$..:O:O9PP` b  S ! S ::44 4 KK ' ' . .tzz : !!$**- r(c<V^8dQhRR/#r<r5)r0r1s"r&r2rKs   r(c\RVP RVPP^, RVPP R2pVP VR7\ PV4VPPPVP4VP4R#)rVz/' added itself to the redundancy list of round rkrfrN) rMrrrrrrrrrprXrgs& r&r[8_DistributedRendezvousOpExecutor._add_to_redundancy_lists $S{{  1$%%9$..:O:O9PP` b  S ! S ##'' 3 r(c<V^8dQhRR/#r<r5)r0r1s"r&r2rK s<<4?JD'+  t $@r(c<V^8dQhRR/#r<r5)r0r1s"r&r2rKDs"""r(cRVP RVPP R2pVPV\P R7\ PV4RVPn R#)rVz' marked the rendezvous 'z' as closed. Pending sync.rTN) rMrrrrrrrrrrgs& r&r`8_DistributedRendezvousOpExecutor._mark_rendezvous_closedDs_ $=dnn>S>S=TU   SY-@-@ A S! r(cJ<V^8dQh/S[;R&S[;R&S[;R&S[;R&#)r,rMrrNr)rrrr)r0r1s"r&r2rKUs9   *)!""!#r()rMrrrNrJ)rKrLrMrNrOr`rr$rrErXrYrZr[r\r]r^r_r`rrQrRrSs@r&rHrHUs $""=F!\@PB48d6RVP R2p\PV4\PD#RVP R2p\PV4M6RVP R2p\PV4M\PF#\V4'd\P#\P #) zThe rendezvous 'z,' is closed, terminating pending rendezvous.z The node z is in redundancy listz is removed from redundancy listrVzE' marking the rendezvous complete, quorum established within deadlinez-' can't complete rendezvous: deadline reachedz4' can't complete rendezvous: not enough participants)$rDrrrrrrr&r1rrr!rrrr'r0r-rr3rrr+rr,r2r)r*rrrrr rr.r()r7rbrrDris_participantrrollback_periods&&& r&r_RendezvousJoinOp.__call__is  <<<xx5000()<)<(==ij1#66'' ' 88u,, ,chhZ'=>C LL 5%%&#,,*@*@@%c**"---"<<'"#((+KL S!:::U%7%77 >>>n>> !nn >O 00"";;;88u."888(( ( >>>5%%&)?)??885??2"3333U''(CLL,B,BBHHE$9$997"999 ^ E&&'3<<+A+AA**+s||/E/EENN.>>HLL$>>$SXXJ/=>LL%";;;&sxxj0]^CLL%"388*,`a S!.. . c " "%% % ||r(r5NrrSs@r&rrfs0bbr(rc6a]tRtRtoRtV3RlRltRtVtR#)_RendezvousCloseOpiz'Represent a rendezvous close operation.c,<V^8dQhRS[RS[RS[/#rr)r0r1s"r&r2_RendezvousCloseOp.__annotate__s#....%.G.r(cVPP'd\P#\P !4V8d\P #\P#rJ)rDrr&r3rrr2r/rs&&&r&r_RendezvousCloseOp.__call__sC 99   >> ! >> h &(( (---r(r5NrrSs@r&rrs1..r(rc6a]tRtRtoRtV3RlRltRtVtR#)_RendezvousKeepAliveOpiz3Represent a rendezvous keep-alive update operation.c,<V^8dQhRS[RS[RS[/#rr)r0r1s"r&r2#_RendezvousKeepAliveOp.__annotate__rr(c\V4'd;\P!4V8d\P#\P #\P #rJ)rrrr&r2r'r3rs&&&r&r_RendezvousKeepAliveOp.__call__s> c " "~~(*,,,%% %~~r(r5NrrSs@r&rrs=r(rca]tRtRtoRt]!4t]R2V3RlRll4tV3RlRlt ] PR3V3RlR llt V3R lR lt ]V3R lR l4tV3RlRlt]V3RlRl4tV3RlRltV3RlRltV3RlRltV3RlRltV3RlRltV3RlRltV3RlRlt]V3R lR!l4tV3R"lR#ltV3R$lR%ltV3R&lR'ltV3R(lR)ltV3R*lR+ltV3R,lR-lt V3R.lR/lt!V3R0lt"R1t#Vt$R#)3rizCRepresent a handler that sets up a rendezvous among a set of nodes.Ncl<V^8dQhRS[RS[RS[RS[RS[RS[R,RS[R,R S[R S[/ #) r,rstorerrrrNr~rr)r/r rrr)r0r1s"r&r2%DynamicRendezvousHandler.__annotate__sy2F2F2F2F# 2F  2F  2F$J2F#T)2F!2F!$2Fr(c  VPPV4p \TTTT;'g \4\ VR7V R7p \ W;4p V!WVP W,4#)aCreate a new :py:class:`DynamicRendezvousHandler`. Args: run_id: The run id of the rendezvous. store: The C10d store to return as part of the rendezvous. backend: The backend to use to hold the rendezvous state. min_nodes: The minimum number of nodes to admit to the rendezvous. max_nodes: The maximum number of nodes to admit to the rendezvous. local_addr: The local node address. timeout: The timeout configuration of the rendezvous. keep_alive_interval: The amount of time a node waits before sending a heartbeat to keep it alive in the rendezvous. keep_alive_max_attempt: The maximum number of failed heartbeat attempts after which a node is considered dead. rVrr)_node_desc_generatorrrrrrr8) clsrrrrrrr~rrrrrJs &&&&&&&&&& r& from_backend%DynamicRendezvousHandler.from_backendshL''00<%     * *(* )2E F#9  5WG 47<<EEr(c <<V^8dQhRS[RS[RS[RS[RS[RR/#)r,rr backend_namerrJr-N)rrr/r r)r0r1s"r&r2r(sO+F+F+F%+F +F  +F - +F +Fr(cVP'g \R4hVP^8d\RVP R24hVPVP8d'\RVP RVP R24hWnW nW0nW@nWPn\VPVPVP 4Vn \P!4Vn RVnRVnRVnR#)z&The run id must be a non-empty string.zThe minimum number of nodes (z) must be greater than zero.zThe maximum number of nodes (z@) must be greater than or equal to the minimum number of nodes (z).N)rr|rr _this_noder _backend_name_storerNrH _op_executorrr_heartbeat_lock_keep_alive_timer_shared_tcp_store_server_bootstrap_store_info)r7rrrrrJs&&&&&&r&r`!DynamicRendezvousHandler.__init__(sEF F    !/0B0B/CC_`     2 2 2/0B0B/CD33;3E3E2FbJ  !) )< OOT//  )~~/!%<@%AE"r(c><V^8dQhRS[RS[RS[R,RR/#)r,rrrNr-)r/rr)r0r1s"r&r2rUs7    Dj    r(c \VPP R\4 2VPP VVVP PVP PVP PVR7R#)r)r8rrrrRrrrN) rrrKr'rrrrrr)r7rrrs&&&&r&r DynamicRendezvousHandler._recordUsf (NN++,Ao.?-@A>>((!__))##__-- r(c4<V^8dQhRS[P/#r+distTCPStore)r0r1s"r&r2rfs  DMM r(c6\P!VVRRR7#)T) host_nameport is_master multi_tenantr)r7 master_addr master_ports&&&r&_create_tcp_store_server1DynamicRendezvousHandler._create_tcp_store_serverfs}}!   r(c <V^8dQhRS[/#r+)r)r0r1s"r&r2ros,r(c VP#)z#Get the settings of the rendezvous.)rr6s&r&r!DynamicRendezvousHandler.settingsns~~r(c <V^8dQhRS[/#r+r.)r0r1s"r&r2rss""S"r(c VP#r)rr6s&r& get_backend$DynamicRendezvousHandler.get_backendss!!!r(c <V^8dQhRS[/#r+r)r0r1s"r&r2rxsKKKr(c 6\P!RR4R8g#)r"TORCH_DISABLE_SHARE_RDZV_TCP_STORE01)rgetenvr6s&r&use_agent_store(DynamicRendezvousHandler.use_agent_storewsyy=sCsJJr(c <V^8dQhRS[/#r+)r)r0r1s"r&r2r|sX X X r(c  RVP RVPP R2pVPVR7\P V4VP 4VPPP^8Xd \RR7\4p\4pVPVPPP4pVP P#W$4VP P#W4VP4VP%4VP'4wrVVP)4pRTP R TPPP R TPP R T R T R 2 pTPYR7\P T4\6P8!RR4R8Xd;\:P<!YWTPP>R7p \ATTTT 4#TPBeT^8XdTPDf~^p T^8XdBTPGTPP>T 4Tn"TPDPHp \:P<!TTTPP>T R7Tn!TPBf\JhT^8XdTPDf\Jh\ATTTTPB4# \*dIpTP\-T4P. R\1T4 2\2P4R7hRp?ii;i)rrVz5' attempts to join the next round of the rendezvous 'rWrrV: rNz' has joined round rkz ' as rank z in a world of size r)rrrrr)r)r server_port)rg333333?)&rrrrrinfo_stop_heartbeatsrNrDrrrr _get_deadliner~rUrrE_start_heartbeats _get_world _get_store ExceptiontyperKr/rFAILEDrrrbuildrrrrrrAssertionError) r7rexit_opjoin_oprr world_sizerebootstrap_store_infors & r&next_rendezvous(DynamicRendezvousHandler.next_rendezvous|s)*%%&b *  S ! C   ! ! # !!''--2x(')G')G))$..*@*@*E*EFH    ! !' 4    ! !'T5G5G H  " " $#0 DOO%E))&%%%   LL7++,Bs1vh7$++     s6A8AA$ A8$ A5 /A85A88 C ACC c <V^8dQhRS[/#r+r.)r0r1s"r&r2rs%%C%r(c .VPP#r)rrr6s&r& get_run_id#DynamicRendezvousHandler.get_run_ids~~$$$r(c <V^8dQhRS[/#r+r)r0r1s"r&r2rs$r(c  VP4VP4R# \dpRTP RTPP R\ T4P R2pTPT\PR7\PT4Rp?R#Rp?i\dIpTP\ T4P R \T4 2\PR7hRp?ii;i) rTrVz)' has failed to shutdown the rendezvous '' due to an error of type rrNFr)rrnrrrrrrKrrrrwarningrr/)r7exrrs& r&shutdown!DynamicRendezvousHandler.shutdowns   KKM T__-.NN))**DT"XEVEVDWWXZ  LL1A1AL B NN3   LL7++,Bs1vh7$++     s($ C>A3B(( C>5C>6AC99C>c<V^8dQhRR/#r<r5)r0r1s"r&r2rs   r(ch\4pVPVPPP4pVP P W4RVP RVPP R2pVPV\PR7\PV4R#)rVz' has closed the rendezvous 'rWrN)rrrr~rYrrErrrrrrr)r7oprrs& r&rnDynamicRendezvousHandler._closes  !%%dnn&<&<&B&BC b+4??++HI^I^H__ab SY-@-@ A Cr(c<V^8dQhRR/#r<r5)r0r1s"r&r2r(str(c@V!4pVeVP4R#R#rJ)rX) weak_selfr7s& r&_keep_alive_weak)DynamicRendezvousHandler._keep_alive_weak's {       r(c<V^8dQhRR/#r<r5)r0r1s"r&r2r-s$$T$r(c VP;_uu_4\4pVPVPPP 4pVP PW4RVP RVPP R2pVPVR7\PV4RRR4R# \d~pRTP RTPP R\T4P R2pTPT\ P"R7\P%T4Rp?LRp?ii;i +'giR#;i) rVz5' has sent a keep-alive heartbeat to the rendezvous 'rWrz?' has failed to send a keep-alive heartbeat to the rendezvous 'r rrN)rrrrr~rZrrErrrrrrrrKrrr )r7rrrrs& r&rX$DynamicRendezvousHandler._keep_alive-s+  ! ! !')B))$..*@*@*J*JKH $!!%%b3! 12--.b2 S ) S!" !# $  12##'>>#8#8"99STXY[T\TeTeSffgi SY5E5E Fs##  $" ! !s1:EA)C E A3EEE  E E c<V^8dQhRR/#r<r5)r0r1s"r&r2rDs ' '4 'r(c&\VPPVP\P !V44VnVP PRVPP 24VP P4R#)RendezvousKeepAliveTimer_N) rrrrweakrefrefrset_namerrstartr6s&r&r*DynamicRendezvousHandler._start_heartbeatsDso!/ NN . .0E0Ew{{SWGX"  '''(@(@'A B  $$&r(c<V^8dQhRR/#r<r5)r0r1s"r&r2rOs(($(r(cZVPfR#VPP4R#rJ)rcancelr6s&r&r)DynamicRendezvousHandler._stop_heartbeatsOs#  ! ! )  %%'r(c6<V^8dQhRS[S[S[3,/#r+)r=r)r0r1s"r&r2rUsLLE#s(OLr(cVPPpVPVP,\ VP43#rJ)rNrDrrr!)r7rDs& r&r#DynamicRendezvousHandler._get_worldUs8""((!!$//2C8J8J4KKKr(c&<V^8dQhRS[RS[/#)r,rr-r )r0r1s"r&r2rZs33353r(cRVPP RVPPP 2p\ P !W!4#)ztorch.rendezvous.r)rrrNrDrr PrefixStore)r7r key_prefixs&& r& _wrap_store$DynamicRendezvousHandler._wrap_storeZsH 5 56a8J8J8P8P8V8V7W X  22r(c <V^8dQhRS[/#r+r )r0r1s"r&r2ras--E-r(c8VPVP4#rJ)r0rr6s&r&r#DynamicRendezvousHandler._get_storeas ,,r(c&<V^8dQhRS[RS[/#)r,r~r-)rr")r0r1s"r&r2rds::Y:5:r(cV\P!4VP4,#rJ)rr total_seconds)r7r~s&&r&r&DynamicRendezvousHandler._get_deadlineds~~'"7"7"999r(c<V^8dQh/S[;R&S[;R&S[;R&S[;R&S[;R&S[;R&S[P;R&S[R,;R &#) r,rrrrrNrrNr) rrr/r rr=rrr)r0r1s"r&r2rsu  "! M*)('^^#&,,r() rrrrrrrrNrr)NN)%rKrLrMrNrOrr classmethodrr`rr$rrrPrrrrrrrrrrn staticmethodrrXrrrr0rrrrQrRrSs@r&rrs@M./2F2F2Fh+F+F`!* 1 1   "  ""KKX X t      %%0   $$. ' '(( LL 33--::A r(cJV^8dQhR\R\R\R,/#)r,paramskeyr-N)rr/r)r0s"r&r2r2hs'&&-&C&IrUs  #$!22 #W   *    8 $ G: : zE/E/P %-  . 8 d$t,99-9(PP6+"+"\"*-S-,b$:bJd$!!2 C 8w"'<w"t eeP..A:0A:H &Cr(