+ jhKaRt0t^RIt^RIt^RIt^RIt^RIt^RIt^RIt^RIt^RI t ^RI t ^RI t ^RI t ^RI t ^RIt^RIt^RIt^RIt^RIHt^RIHt^RIHt^RIHt^RIHt^RIHtHtHt^RIH t ^R I!H"t"H#t#^R I$H%t%^RI&t&^RI't&^RI(t&^RI)H*t+^RI,H-t-^R I.H/t/^R I0H1t1^R I2H3t3^RI4H5t5^RI6H7t7^RI8H9t9H:t:H;t;Ht>H?t?H@t@HAtAHBtBHCtCHDtDHEtE^RIFHGtGHHtHHItIRtJRtKRtLRtMRtNRtO]5'd5^RIPtPRF*wtQtR]PP!]Q4'gK R]T!4]R&K, RRltU]P!]W4tX]XP]P4.ROt[RR.t\]@;'g ]A;'g]Dt]!RR]#4t^/R]^!^HR4bR]^!^IR4bR]^!^WR 4bR!]^!^JR"4bR#]^!^KR$4bR%]^!^MR&4bR']^!^PR(4bR)]^!^QR*4bR+]^!^RR,4bR-]^!^SR.4bR/]^!^TR04bR1]^!^UR24bR]^!^LR34bR4]^!^NR54bR6]^!^OR74bR8]^!^VR94bR:]^!^XR;4bR<]^!^YR=4/Ct_]!R>R?44t`R@taRAtbRBtcRCtdRDteREtfRFtgRGRHlthRIR/RJltiRKRLltjRMRNltkROtlRPtmRQtnRRtoRStpRTtqRUtrRVtsRWttRXtuRYtvRRZltwR[txR\ty]>?z" $)*$56$5qd$56  I7s< Fcudaxpuc,a]tRt^toV3RltRtVtR#)TestSkipc2<V^8dQh/S[;R&S[;R&#)r& exit_codemessager/r.)r3 __classdict__s"r4r5TestSkip.__annotate__sN Lr6N)__name__ __module__ __qualname____firstlineno____annotate_func____static_attributes____classdictcell__rbs@r4r]r]sr6r]backend_unavailablez5Skipped because distributed backend is not available.small_worldsizez Skipped due to small world size. odd_worldsizezSkipped due to odd world size.no_cudazCUDA is not available.z multi-gpu-1zNeed at least 1 CUDA devicez multi-gpu-2zNeed at least 2 CUDA devicesz multi-gpu-3zNeed at least 3 CUDA devicesz multi-gpu-4zNeed at least 4 CUDA devicesz multi-gpu-5zNeed at least 5 CUDA devicesz multi-gpu-6zNeed at least 6 CUDA devicesz multi-gpu-7zNeed at least 7 CUDA devicesz multi-gpu-8zNeed at least 8 CUDA devicesz#c10d not compiled with NCCL support skipIfRocmzTest skipped for ROCmno_peer_accessz'Test skipped because no GPU peer accessgenericzHTest skipped at subprocess level, look at subprocess log for skip reason importerrorz"Test skipped due to missing importno_acceleratorzaccelerator is not available.c]tRt^t/t0Rm]R&]!4]R&0Rm]R&0Rm]R&/t0Rm]R&0Rm]R&0Rm]R&0Rm]R &]!4]R &]'dR 0]R &]'d R0]R &Rt R#Rt R#) DistTestCasesxcclallgather_coalescedr zsendrecv anysourcez cpu barriergpurZddpsubgrouppluginhcclhpur[rdN>mpiuccr$rx>rr$rx>rgloor$rx>rrr$) rerfrgrhskip_collectivesetbackend_featurerrrjrdr6r4rwrwsO-KO)* #OH,CO()%.wrappers XX HHZ *44 5L12 9002Z? HHZ*ZL 9:DD E 8 ..0:= HHZ*ZL 9:DD E 8 ..0:= HHZ*ZL 9:DD ET$V$$r6r rrsf r4skip_if_no_gpurs" 4[ % % Nr6c0a\S4V3Rl4pV#)c<\PR,R8wdQ\\PR,4^8d,\P!\ R,P 4S!V/VB#)BACKENDrrrnrGrHr/rrrr_rrrs*,r4r(skip_if_small_worldsize..wrappersR JJy !U *BJJ|4L0MPQ0Q HHZ 12<< =T$V$$r6rrsf r4skip_if_small_worldsizer  4[%% Nr6c0a\S4V3Rl4pV#)c<\PR,R8wdX\\PR,4^,^8Xd,\P!\ R,P 4S!V/VB#)rrrrorrs*,r4r&skip_if_odd_worldsize..wrappersW JJy !U *BJJ|4L0MPQ0QUV0V HHZ0:: ;T$V$$r6rrsf r4skip_if_odd_worldsizerrr6caaVV3RlpV#)c6<a\S4VVV3Rl4pV#)c<SR8XdT\PP4S8d1\P!\ RS 2,P 4R#S!V/VB#r$rN)r0rZrrrrr_)rrr'rns*,r4rCrequire_n_gpus_for_nccl_backend..decorator..wrappersM& UZZ%<%<%>%Bj$45??@T,V,,r6r)rrr'rsf r4 decorator2require_n_gpus_for_nccl_backend..decorators t -  - r6rd)rr'rsff r4require_n_gpus_for_nccl_backendrs r6c RpV#)c0a\S4V3Rl4pV#)c<^RIHpHpS!V/VB# \d0\P !\ R,P4R#i;i)r)AutoModelForMaskedLM BertConfigrtN) transformersrr ImportErrorrrrr_)rrrrrs*, r4r?import_transformers_or_skip..decorator..wrappersA >IT,V,, >M2<<= >s6A  A rrsf r4r.import_transformers_or_skip..decorators t >  >r6rd)rs r4import_transformers_or_skiprs  r6c,\'d&\PP4V8dR#\'d&\P P4V8dR#\ 'd&\PP4V8dR#R#)TF)rr0rZrrrrr[)xs&r4at_least_x_gpur$sYyUZZ,,.!3xEII**,1xEII**,1 r6c$V^8dQhR\/#r&r-bool)r3s"r4r5r5.sr6cj\V^,RR4p\V4^8XgVfR#V!V4R#)r_handle_test_skipNFT)getattrlen)rmsgrs&& r4_maybe_handle_skip_if_lt_x_gpur.s5Q).decorator..wrapper?szz&&((UZZ-D-D-F!-KT,V,,xEII2249T,V,,xEII2249T,V,,%**"9"9";";xx88T,V,,"Zs#34I1$8I8IJJ,,-Kr6r)rrrrsf r4r#skip_if_lt_x_gpu..decorator>s t .  .r6rd)rrrsfd r4skip_if_lt_x_gpur6s" r6c$V^8dQhR\/#)r&rr/)r3s"r4r5r5Rs3r6caV3RlpV#)a2 Decorator to request a specific world size for a test. The test harness can read this attribute to set the number of ranks to spawn. If there are fewer than `n` CUDA devices available, the test should be skipped by the harness. Usage: @require_world_size(3) def test_something(self): ... c<SVn\PP4p\P !VS8RS RV 24!V4#)z requires z GPUs, found )_required_world_sizer0rZrunittest skipUnless)r availablers& r4r&requires_world_size..decorator^sN$%!JJ++- "" Nis- {C  r6rd)rrsf r4requires_world_sizerRs r6c<V^8dQhR\R\R\/#)r&objdefaultr-)r r/)r3s"r4r5r5hs!ssr6c\VR4'd,\VP4'dVP4M VPp\ W4pVP p\ V4# \dTu#i;i)z Returns the requested world size for the currently running unittest method on `obj` if annotated via `@require_world_size(n)`, else returns `default`. _current_test_name)hasattrcallabler_testMethodNamerrr/ Exception)rr test_namefnvalues&& r4get_required_world_sizerhsx s011hs?U?U6V6V  " " $$$  S $''5z sA*A-- A=<A=caaVV3RlpV#)c6<a\S4VVV3Rl4pV#)c^<SR8wd S!V/VB#\PP4'd,\PP4S8d S!V/VB#\RS 2,p\ WP 4'g#\P!VP4R#R#r) r0rZrrrrr`rrr_)rrrr'rrs*, r4r9nccl_skip_if_lt_x_gpu..decorator..wrapper~s& T,V,,zz&&((UZZ-D-D-F!-KT,V,,"Zs#34I1$8I8IJJ,,-Kr6r)rrr'rsf r4r(nccl_skip_if_lt_x_gpu..decorator}s t .  .r6rd)r'rrsff r4nccl_skip_if_lt_x_gpur|s  r6c2VP4pRV9d \R4hRV9d \R4hRV9d \R4hVR,pVPR4R 8XdTMVPR4^,pWC9d\RV RV 24hR #) iterationz(Expected 'iteration' in ddp_logging_data has_errorz(Expected 'has_error' in ddp_logging_dataerrorz$Expected 'error' in ddp_logging_dataz Exception raised from zDid not find expected z in ddp logging data error: N)_get_ddp_logging_dataAssertionErrorfindsplit) model_DDP err_substrddp_logging_data logging_erractuals&& r4verify_ddp_error_loggedrs 668**GHH**GHH&&CDD"7+K ??5 6" <    8 9! <  $VH,H V  !r6c0a\S4V3Rl4pV#)a6 Convenience decorator to set/unset TORCH_NCCL_BLOCKING_WAIT flag. Note that use of this decorator will override the setting of TORCH_NCCL_ASYNC_ERROR_HANDLING for the particular test. After the test, both TORCH_NCCL_BLOCKING_WAIT and TORCH_NCCL_ASYNC_ERROR_HANDLING will be restored to their original values. c<\PR,p\PR\PR,pR\PR&S!V/VBpTVeV\PR&VeV\PR&## \dRpLti;i \dRpLoi;i R\PR&i;i TeT\PR&TeT\PR&ii;i)TORCH_NCCL_ASYNC_ERROR_HANDLINGNTORCH_NCCL_BLOCKING_WAIT1)rGrHKeyError)rr cached_nccl_async_error_handlingcached_nccl_blocking_waitretrs*, r4r(with_nccl_blocking_wait..wrappers 4;=::1< , <=  946JJ*5 % 69BJJ1 2 S''C0;4 <=)49R 5651 4/3 , 4 -(, % -69BJJ1 20;4 <=)49R 565s@(BB# C B B # B30B62B33B66C 1C?rrsf r4with_nccl_blocking_waitrs% 4[ S SD Nr6caV3RlpV#)zC Runs a test for each distributed debug level specified in levels. c4<a\S4VV3Rl4pV#)c<\PPRR4pSFapV\PR&\P!4S!V/VBp\P !4VfKNV\PR&Kc X#)TORCH_DISTRIBUTED_DEBUGN)rGrHgetr1set_debug_level_from_envbarrier)rr old_levellevelrrlevelss*, r4r:with_dist_debug_levels..decorator..wrappersq '@$GI8= 45--/D+F+ (.decorators t   r6rd)rrsf r4with_dist_debug_levelsrs $ r6cJ\\P!4'*R4#)z+c10d was not compiled with the Gloo backend)rr1is_gloo_availablerdr6r4 requires_gloor! )  " " $$5 r6c 4\'gR#\P!4'g \R4#\ \ P PP4V8RV R\ P PP4 RV 24#)cV#rrd)fs&r4'requires_nccl_version..sr6+c10d was not compiled with the NCCL backendz0Requires NCCL version greater than or equal to: z , found: z , reason: ) rr1is_nccl_availablerrr0rZr$version)rrs&&r4requires_nccl_versionr s 9  ! ! # #* 9  . JJOO # # % />wiyQVQ[Q[Q`Q`QhQhQjPkkuvyuz {  r6c\RR4#)zC Require NCCL shrink support (NCCL available and version >= 2.27). z Need NCCL 2.27+ for shrink_group)r&)r rdr6r4requires_nccl_shrinkr#s !*L MMr6cJ\\P!4'*R4#)r)rr1rrdr6r4 requires_ncclr%rr6cJ\\P!4'*R4#)z+c10d was not compiled with the XCCL backend)rr1is_xccl_availablerdr6r4 requires_xcclr(rr6cJ\\P!4'*R4#)z*c10d was not compiled with the UCC backend)rr1is_ucc_availablerdr6r4 requires_uccr+! )  ! ! ##4 r6cJ\\P!4'*R4#)z*c10d was not compiled with the MPI backend)rr1is_mpi_availablerdr6r4 requires_mpir/r,r6cVf\p\;QJdRV4F 'gK RM RM !RV44p\V'*RV 24#)a Decorator to skip tests if no accelerator communication backend (NCCL, XCCL, HCCL) is available. Args: backends (Optional[List[str]]): Specific accelerator backends to check (e.g., ["nccl", "xccl", "hccl"]). If None, checks all supported accelerator backends (NCCL, XCCL, HCCL). Returns: callable: A decorator that skips the test if no specified accelerator backend is available. c3"TF@pR\PR\PRR/PVR4!4xKB R#5i)r$rxr~c\#r)rrdr6r4r=requires_accelerator_dist_backend...5sHr6cR#Frdrdr6r4rr36sur6N)r1rr'r ).0r's& r4 4requires_accelerator_dist_backend..1sV  G D** D** $  #g} %  & ( (  sAA TFz5No accelerator communication backend available among )ACCELERATOR_DIST_BACKENDSanyr)backendsbackend_availables& r4!requires_accelerator_dist_backendr=#s_,     *  ?zJ r6c\PP4;'d&\P!\ P ^4p\V'*R4#)rz"multicast support is not available)r0rZrrhas_multicast_supportrCUDAr)r?s r4requires_multicast_supportrA@sO ! G G  2 2:??A F * !!, r6c\'dO\'dARR.pVF4pV\PP ^4P 9gK3R# R#R#R#)gfx942gfx950TF)rrr0rZget_device_properties gcnArchName) arch_listarchs r4#evaluate_platform_supports_symm_memrIKsKy >!8,I!5::;;A>JJJ"r6c\4#r)rIrdr6r4rrZs / 1r6cn\P!\\R,P4!V4#)z&Skips a test for ROCm multiprocess UTsrq)rskipIfrrr`)rs&r4skip_if_rocm_multiprocessrM^s# ??>:l+C+K+K LT RRr6c>V^8dQhR\\R3,/#)r&rH.)tupler.)r3s"r4r5r5cs  sCx r6caV3RlpV#)z4Skips a test for given ROCm archs - multiprocess UTsc<Rp\'dL\PP^4PP R4^,pVS9dRS 2p\ P!VRJV4!V4#)N:z0skip_if_rocm_arch_multiprocess: test skipped on )rr0rZrErFrrrL)rreasonproprHs& r4r1skip_if_rocm_arch_multiprocess..decoratorfse >::33A6BBHHMaPDt|KD6RvT16:4@@r6rd)rHrsf r4skip_if_rocm_arch_multiprocessrVcsA r6caV3RlpV#)z:Skips a test for ROCm based on ROCm ver - multiprocess UTsc<Rp\'d\\PP4pVP R^R7^,p\ ;QJd#.RVP R44FNK 5M!RVP R444pVeSeV\ S48d RV RS R2p\P!VRJV4!V4#)N-maxsplitc38"TFp\V4xK R#5irrr6rs& r4r7Lskip_if_rocm_ver_lessthan_multiprocess..decorator..z&O7N!s1vv7N.z-skip_if_rocm_ver_lessthan_multiprocess: ROCm z is available but z required rr.r0rhiprrOrrLrrS rocm_versionrocm_version_tuplers& r4r9skip_if_rocm_ver_lessthan_multiprocess..decoratorus >u}}001L'--cA->qAL!&&O|7I7I#7N&O&O|7I7I#7N&O!O "*?%g6HI[H\\novnwxABvT16:4@@r6rdrrsf r4&skip_if_rocm_ver_lessthan_multiprocessrirs A r6caV3RlpV#)zSSkips a test for ROCm if the version is at least the given tuple - multiprocess UTsc<Rp\'d\\PP4pVP R^R7^,p\ ;QJd#.RVP R44FNK 5M!RVP R444pSeV\ S48d RV RS R2p\P!VRJV4!V4#)NrYrZc38"TFp\V4xK R#5irrr]s& r4r7Kskip_if_rocm_ver_atleast_multiprocess..decorator..r_r`raz=skip_if_rocm_ver_atleast_multiprocess: known failure on ROCm z (>= )rbrds& r4r8skip_if_rocm_ver_atleast_multiprocess..decorators >u}}001L'--cA->qAL!&&O|7I7I#7N&O&O|7I7I#7N&O!O "'9U7^'KXYkXllqryqzz{|vT16:4@@r6rdrhsf r4%skip_if_rocm_ver_atleast_multiprocessrps A r6c<\\PR8HR4#)win32z8This unit test case is not supported on Windows platform)rrplatformrdr6r4 skip_if_win32rts ) B r6c\V^8dQhR\PR\R\R\/#)r&r*majorminorr-)r0r*r/r)r3s"r4r5r5s2FFFcF#F$Fr6cVPR8wdR#\PPeR#\PP V4W38#)z Returns True if the device's compute capability is (major, minor) or higher. Error out if the device is not a CUDA device. Returns False if device is a RoCM device. Returns True if device is a non-CUDA device. rZTF)typer0rrcrZget_device_capability)r*rvrws&&&r4sm_is_or_higher_thanr{sB{{f }}$ :: + +F 3~ EEr6 localhost)minutesc \4pV'dH\V\^R7, 4p\PP P WWV4#\P !VVVVVVR7#)zD Creates a TCP store. Retries if the chosen port is already in use. ) milliseconds)wait_for_workers use_libuv)rr/rr0classes dist_c10dTCPStorer1) addrr) is_mastertimeoutr jit_classrporttimeout_milliseconds &&&&&&& r4create_tcp_storersn  D!'I1,E"EF}}&&//  /B  }}    -   r6i!DISTRIBUTED_TESTS_DEFAULT_TIMEOUT300test_ddp_uneven_inputsitest_DistributedDataParalleltest_join_kwargsc$V^8dQhR\/#)r& lazy_initr)r3s"r4r5r5s  T r6c\PR8XgVf"\PP RVR7#\PP WR7#)rrz 127.0.0.1)hostnamer interfacer)rrsr1ProcessGroupGloo create_devicers&&r4rrsY ||w)"3$$22 I3  $$223  r6c$V^8dQhR\/#rr)r3s"r4r5r5sIICIr6cb\PVPR4R,\4#rar)TIMEOUT_OVERRIDEr rTIMEOUT_DEFAULT)test_ids&r4 get_timeoutrs#    c 22 6 HHr6c#F"\4\4r\P\Pr2Wu\n\n\P\P3xW#u\n\nR# Y#u\n\ni;i5ir)r rstdoutstderr)new_outnew_errold_outold_errs r4captured_outputrsaz8:Wzz3::W2!( CJjj#**$$!( CJ CJs3B!8B.B!BB!c<V^8dQhR\R\R\/#)r&r(r) num_inputsr)r3s"r4r5r5s!((S(c(s(r6c RRRllpRRlp\V^R7\V^R7\V^R7\V^R7\V^R7\V^R73UUu.uF_p\V4Uu.uFpV!W ,V,W!,4NK! up\V4Uu.uFqd!WRV,4NK up3NKa upp#uupiuupiuuppi)z Generate a number of basic test cases for sparse reduction. These cover tensors with a varying number of sparse dimensions and a varying number of dense dimensions. The only reduction operation we support is sum. cHV^8dQhR\R\R\R\/#)r&r(r) sparse_dims dense_dimsr)r3s"r4r50simple_sparse_reduce_tests..__annotate__s( ? ?s ? ?# ?s ?r6c 8\P!\P!V^,4^V^,34pV.\V4Uu.uFp^NK up,p\V^, 4FHp\P!V\P !^V^,434pVP V4KJ \P!V^,.\V4Uu.uFp^NK up,4p\P!WGV4#uupiuupi) r0reshapearangerTcatzerosappendonessparse_coo_tensor)r(r)rrindices_shapevaluess&&&& r4generate,simple_sparse_reduce_tests..generates-- TAX 6D1H F 5+<=+> >*Hs D# D c$V^8dQhR\/#r&r)r)r3s"r4r5r s  C r6c |\\P\V4Uu.uF q !W!4NK up4#uupir)r operatoraddrT)rr)r(s&& r4 compute_sum/simple_sparse_reduce_tests..compute_sum s6 LLE* >{{ G##%%*:*@*@*B*B#l73C3C2D$EFF  G  iikJ.G #&>wiq$IJJsA *B+8B+B+*B+caa]tRtRtoR&t^ tV3RlRlt]V3RlRl4t]V3RlRl4t Rt R'V3R lV3R lllt V3R lV3R llt V3R lV3Rllt V3RlRltV3RlRltV3RlRlt!RR]4t]V3RlRl4t]V3RlRl4tV3RlRltV3RlRltV3RlR ltV3R!lR"lt]V3R#lR$l4tR%tVtV;t#)(MultiProcessTestCaseic <V^8dQhRS[/#rr)r3rbs"r4r5!MultiProcessTestCase.__annotate__sr6cR#r5rdselfs&r4_should_stop_test_suite,MultiProcessTestCase._should_stop_test_suitesr6c <V^8dQhRS[/#rr)r3rbs"r4r5rsdr6cR#)Trdrs&r4destroy_pg_upon_exit)MultiProcessTestCase.destroy_pg_upon_exitsr6c <V^8dQhRS[/#rr)r3rbs"r4r5r""C"r6c\#rDEFAULT_WORLD_SIZErs&r4r)MultiProcessTestCase.world_size!!r6cXa\S4V3Rl4p\P!W 4#)cr<VPVP8XdVPS4R#S!4R#r)r(MAIN_PROCESS_RANK_join_processesrrs&r4r1MultiProcessTestCase.join_or_run..wrappers(yyD222$$R(r6r types MethodTyperrrs&f r4 join_or_run MultiProcessTestCase.join_or_run, r    ..r6c*<V^8dQhRS[RS[RR/#r& method_name methodNamer-Nr)r3rbs"r4r5r%8; r6c<VR8wdTp\SV`V4\W4p\WVP V44R# \ d/pTR8wd\ RTP RT 24ThRp?R#Rp?ii;irunTestzno such test method in : Nsuper__init__rsetattrrAttributeError ValueError __class__rrr rer,s&&& r4r(MultiProcessTestCase.__init__  "$K % +B Dt'7'7'; < Y&!-dnn-=R |L' &A A<#A77A<c<V^8dQhRR/#rrd)r3rbs"r4r5rstr6c<\SV`4/Vn.Vn.VnVP Vn\P!RR7;_uu_4pVPVn RRR4/Vn R# +'giL;i)FdeleteN) r'setUpspecial_return_code_checksskip_return_code_checks processesrr(rNamedTemporaryFiler? file_name pid_to_pipe)rrr,s& r4r6MultiProcessTestCase.setUpsj  13'.0$**  ( ( 6 6!VVDN77 6s A88 B c<V^8dQhRR/#rrd)r3rbs"r4r5rs$r6cx<\SV`4VPFpVP4K .VnR#r)r'tearDownr9 terminate)rpr,s& r4r@MultiProcessTestCase.tearDowns. A KKM r6c <V^8dQhRS[/#rr)r3rbs"r4r5r((C(r6cNVP4PR4R,#ridrrs&r4r'MultiProcessTestCase._current_test_nameswwys#B''r6c<V^8dQhRR/#rrd)r3rbs"r4r5rs+++r6c .Vn\\VP44Fp\P P 4wr4V!VPPR\V4,VVP4VPV3R\VRR4/R7pVP4\PRW%P 4W0P"VP &VPP%V4K R#)process fake_pgF)targetr?rrStarted process %s with pid %sN)r9rTr/r)r0rPiper,_runr.rr;rstartloggerinfopidr<r)rprocr( parent_conn child_connrs&& r4_start_processes%MultiProcessTestCase._start_processess#doo./D&+&;&;&@&@&B #K~~**#d)+++-NN wtY> G MMO KK8$ L,7  W[[ ) NN ! !' *%0r6c<V^8dQhRR/#rrd)r3rbs"r4r5rs$$$$r6c\PPR4\PP R4P pVP V4R# \dLIi;i)spawnN)r0rset_start_methodr get_contextrrY)rrVs& r4_spawn_processes%MultiProcessTestCase._spawn_processess[   ! ! 2 27 ;$$009AA d#    sA A+*A+c]tRtRt^tRtR#)MultiProcessTestCase.EventirdN)rerfrgrh GET_TRACEBACKrjrdr6r4Eventrcs r6rec <V^8dQhRS[/#)r&r(r)r3rbs"r4r5rsr6c\PRV4\PP W.4pW9EdVP 'd\PRV4R#VP 4p\PRWB4V\PP8Xd\P!RR7;_uu_4p\P!V4VP4VP!^4VP#VP%44\PRV4RRR4W9gEK4R# +'giL;i)z*Starting event listener thread for rank %sz:Pipe closed for process %s, stopping event listener threadNzReceived event %s on process %szr+)modezProcess %s sent traceback)rSdebugr connectionwaitclosedrecvrTrrerdrr: faulthandlerdump_tracebackflushseeksendread) parent_pipe signal_piper( ready_pipeseventtmp_files&&& r4_event_listener$MultiProcessTestCase._event_listeners A4H)4499;:TUK)%%%LLT#((* =uK066DDD!44$??8$33H= ( a(#((9 $?F@)@?s A-E  E c0<V^8dQhRS[RS[RS[RR/#r&r(rr;r-Nra)r3rbs"r4r5rs-...#&.36. .r6c PV!V4pWnW6nVPW$4R#r)r(r;run_testclsr(rr;rtrrs&&&&&, r4rQMultiProcessTestCase._runs#9~ " i-r6c$<V^8dQhRS[RR/#)r&rr-Nr)r3rbs"r4r5r&s77#7t7r6c\PPRR7wr4\P!\ P W#VP3RR7pVP4\PR8wd5\PR8wd \PPR4R\PR&\P !4\#W4!4VeVP=R 4Vf \?R 4hVPA4VPC4VPD'd\FPH!4R #R # \$P&dXp\(P+RTPTT4\P,!\.R ,P04R p?LR p?i\2d\(P5R \6P8!4TP\ P:4TP=\6P8!44\P,!\ P:4ELsi;i TeTP=R 4Tf \?R 4hTPA4TPC4i;i \>\J3dR #i;i) F)duplexT)rNrdaemonrrdarwinrTORCH_SHOW_CPP_STACKTRACESz4Process %s skipping test %s for following reason: %srsNz;Caught exception: %s exiting process %s with exit code: %sz-Expected event_listener_thread to not be None)&r0rrP threadingThreadrryr(rRrrs_C'_set_print_stack_traces_on_fatal_signalrGrHr set_rng_seedrrSkipTestrSrTrrr_rr traceback format_excTEST_ERROR_EXIT_CODErrrrcloserr1destroy_process_groupr+)rrrtsignal_recv_pipesignal_send_pipeevent_listener_threadses&&& r4r~MultiProcessTestCase.run_test&s-2-B-B-G-Gu-G-U* ) 0 0'77;!  ##% <<7 "s||x'? HH < > ? @ + %%d+$,$%TUU ! & & (    #J/  sQE?J(I-A F?:I? I B IIIIAJ%(J=<J=c<V^8dQhRR/#rrd)r3rbs"r4r5r_s&&&r6c.p\VP4Fnwr#VPeKVPVP,pVP \ PP4VPW$34Kp VFwrTVP^4'dSVP'd\PRV4KGVP!4p\P#RWV4Ko\P#RV4K R# \d\PRT4EK$i;i \d\PRT4Ki;i)Nz>Encountered error while trying to get traceback for process %sz5Pipe closed for process %s, cannot retrieve tracebackz)Process %s timed out with traceback: %sz6Could not retrieve traceback for timed out process: %s) enumerater9rr<rUrrrrerdrConnectionErrorrS exceptionpollrlrTrmr)rpipesrrpiper(rs& r4_get_timedout_process_traceback4MultiProcessTestCase._get_timedout_process_traceback_s&#DNN3JA''' 4II288FFGLL!+ 4 JD 99Q<<{{{ S ! $ ILLEtLLPRV! '$$X4#   T s6 ;D(E;E&E;E!D=<D=!E%$E%c<V^8dQhRR/#rrd)r3rbs"r4r5rs**T*r6cf\VP44p\P!4pRp\VP4F{wrVVP \ P8XgK&\RV RVP R24\PP4pVFpVP4K RpM V'dM\;QJd&RVP4F 'dK RM RM!RVP44'dM~\P!4V, p W8dDVP4\RV R24VPFpVP4K M\P!R4EKk\P!4V, p VP!W4VP"P%4Fp V P'4K R # TP"P%4Fp T P'4K i;i) FTProcess z terminated with exit code z", terminating remaining processes.c3<"TFqPRJxK R#5ir)r)r6rBs& r4r77MultiProcessTestCase._join_processes..sF~!zz-~szTiming out after z" seconds and killing subprocesses.g?N)rrHrrr9rrrprintr0ractive_childrenrAallrsleep_check_return_codesr<rr) rrrrsubprocess_errorrrBracr elapsed_timers && r4r$MultiProcessTestCase._join_processessdggi(YY[  & %dnn5DAzz%9%N%NN&qc)DQZZLPrs+0*?*?*O*O*Q"1BLLN#2+/(6$3Ft~~F333Ft~~FFF))+ 2$88:+G94VW"^^ , 399;3L  $ $R 6((//1 2((//1 2s+7G=.A G= G=G=9$G=B,G==3H0c<V^8dQhRR/#rrd)r3rbs"r4r5rsJ J tJ r6c  ~VP'g\PR4R#VP^,p\VP4UUu.uF)wrEVP\ P 8XgK&WE3NK+ pppV'deRpVFQwrHVPVP,P4p VRV R\ P RV R2, pKS \V4h\VP4F%wrEVPeK\RV RV R 24h WP9dR#\P4F|p VPV P8XgK \'d3\P!R VP#4V P$4R#\&P(!V P$4h ^p WP*9dVP*V,p VP-VPV R V R VP R VP 2R7R#uuppi)z Checks that the return codes of all spawned processes match, and skips tests if they returned a return code indicating a skipping condition. zAD D c$V^8dQhR\/#rr)r3s"r4r5r5dsr6cv\P!.RORR7P^8H# \dR#i;i)z If shell command `fi_info -p efa -t FI_EP_RDM` returns exit code 0 then we assume that the machine has Libfabric EFA interfaces and EFA software components installed, see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-start.html. F)check)fi_infoz-pefaz-t FI_EP_RDM) subprocessrun returncodeFileNotFoundErrorrdr6r4has_efarcsA NN;5 j     s &) 88c.\4'dRR.#R#)aw If the machine has Libfabric EFA interfaces and EFA software components installed it may cause 'RuntimeError: In operator() at tensorpipe/common/ibv.h:172 "": Operation not supported' if tensorpipe uses InfiniBand transport, so we exclude it from tensorpipe transports, see https://github.com/pytorch/pytorch/issues/73885 and https://github.com/pytorch/pytorch/issues/65022 shmuvN)rrdr6r4 tp_transportsrws$IIE4=/4/r6cjaaaSf\\VSR7#Ro\S4VVV3Rl4pV#)z# Wrapper to use with a test method )rr)caaaa\4o\P!4pV3RloVVV3Rlp.p\S4F>p\P !W5SV3R7pVP 4VPV4K@ V#)c><S\PP8H#rr1r_worldworldsr4world_is_validaspawn_threads_and_init_comms.._run_test_method_with_multi_threads..world_is_validD11888 8r6c<\P!RVSVR7S!4S!4'd\P!4R#R# \dUp\PP T\ P!434\P!T4Rp?LRp?ii;i S!4'd\P!4ii;i)threadedr'r(r)r@N) r1r BaseExceptionMultiThreadedTestCaseexception_queueputrexc_infor#exception_handler)r(world_pgr@excallbackrr)s&&& r4workerYspawn_threads_and_init_comms.._run_test_method_with_multi_threads..workers  # #"*E  1 "##..0$! %55994:PQ!22 "##..0$s*A B)A B$B,$B))B,,%CrNr)r!r1 HashStorerTrrrRr) r)r( global_storer)threadsr(trrs ff @@r4#_run_test_method_with_multi_threadsIspawn_threads_and_init_comms.._run_test_method_with_multi_threadssj$&~~'  9 1 *%D  E<5PQA GGI NN1 & r6c\<aaa\PPPR4S!SVVVV3Rl4p\P VS4\PPPR4R# \PPPR4i;i)Tc<S!S.SO5/SB#rrd)rrrrsr4r?spawn_threads_and_init_comms..wrapper..sD$?$?$?r6FN)r0r_distributed_c10d_set_thread_isolation_moder! _join_threads)rrrr.r0rr)sfjl r4r-spawn_threads_and_init_comms..wrappersv ""==dC I9?G " / / > HH & & A A% HEHH & & A A% Hs &B+B+)rspawn_threads_and_init_commsr )rrr)rr0sf&f @r4r9r9sD  | ('j  > 4[ I I Nr6c:aa]tRtRtoRt]P !4tRtRt RV3RlV3Rlllt Rt Rt V3RlV3R llt V3R ltR t]R 4tR t]R4t]R4t]V3RlRl4t]V3RlRl4tRR^/RlltRR^/RlltRtVtV;t#)r!ia  Test runner that runs all tests with the in-proc process group using multiple threads with the threaded process group. Each test spawns world_size threads and run the test method in each thread. Difference from regular MultiProcess test runner: Must explicitly defines SetUp and call self._spawn_threads() to run the tests. Cannot use setUp / tearDown (must use perThreadSetup / perThreadShutdown) to set up / tear down each thread when running each test. No global state possible How bad of a limitation is this? cXa\S4V3Rl4p\P!W 4#)c<VPVP8XdVPVPS4R#S!4R#r)r(MAIN_THREAD_RANKr7r.rs&r4r2MultiThreadedTestCase.join_or_run..wrappers.yyD111""4<<4r6rrs&f r4r!MultiThreadedTestCase.join_or_runrr6c*<V^8dQhRS[RS[RR/#rr)r3rbs"r4r5"MultiThreadedTestCase.__annotate__r!r6c<VR8wdTp\SV`V4\W4p\WVP V44R# \ d/pTR8wd\ RTP RT 24ThRp?R#Rp?ii;ir#r&r-s&&& r4r(MultiThreadedTestCase.__init__r0r1cR#rrdrs&r4perThreadSetUp$MultiThreadedTestCase.perThreadSetUps r6cR#rrdrs&r4perThreadTearDown'MultiThreadedTestCase.perThreadTearDowns r6c<V^8dQhRR/#rrd)r3rbs"r4r5rAs 7 7t 7r6c z<\SV`4VPVn.VnR\ P R&R#)zy setUp only set up things in the main thread, if you want to configure things in the spawned threads, use perThreadSetUp rrN)r'r6r=r(r.rGrHrs&r4r6MultiThreadedTestCase.setUps1  ))  36 /0r6c 2<\SV`4.VnR#)z tearDown only set up things in the main thread, if you want to configure things in the spawned threads, use perThreadTearDown N)r'r@r.rs&r4r@MultiThreadedTestCase.tearDowns  r6c a\PPPR4VPp\ 4o\ P!4VPn V3RlpV!4'g \R4h\VP4Ffp\P!VPPWVP3R7pVP!4VP"P%V4Kh R#)z[ class method to spawn threads and run test, use this method in the SetUp of your TestCase Tc><S\PP8H#rrrsr4r.world_is_validrr6z Invalid worldr+N)r0rr5r6rr!r1r,r,r-rrTr)rrrQrRr.r)rrrr(r/rs& @r4_spawn_threads$MultiThreadedTestCase._spawn_threadss ""==dC++ $&&*nn&6# 9/ /$//*D  ~~**)4??1SA GGI LL   " +r6c V!V4pW%n\VR4'dY\P!4Vn\ P VPn\ PVPn VPWV4R#)_tlsN) r(rrlocalrUr _precision precision_rel_tolrel_tolrun_test_with_threaded_pg)rrr(r)rrs&&&&, r4rQMultiThreadedTestCase._runsa9~ 4 !)DI"*"5"5DII  ( 1 1DII  &&y Cr6c \P!RVVVPPR7VP 4\ W4!4\P!4VP4R# \ dQpTPPT\P!434\P!T4Rp?L}Rp?ii;i \P!4TP4i;i)zT Run the current test associated with `test_name` using the threaded process group. rrN)r1rr,r-rErr r"r#rr$r#r%rrH)rrr(r)r's&&&& r4r[/MultiThreadedTestCase.run_test_with_threaded_pg,s !..--   % D $ &  & & (  " " $   $ $dCLLN%; <  . .     & & (  " " $s+A77 CAC C CC'C<c \p\V4FmwrEVP\^V44VP 4'gK8\ P PV\\RV R24R334Ko \P!4.pVP P4'g.VP P4pVPV4KM\4\P P"P%R4VP'WcV4R# \4\P P"P%R4i;i)rzRank failed to join in under rNF)rrrrrr!r"r# TimeoutErrorr#rrr rr"r0rr5r6r)rr.rridxthread failed_ranksfailures&&& r4r7#MultiThreadedTestCase._join_threadsCs! I(1  C7O,??$$)99== , ,&CG9H$U!"!%    2  # # %L))//11--113##G,2 # $ HH & & A A% H  r: # $ HH & & A A% Hs>D0 A/D0;-D005E%c RpR pVEFYwrgV^,p\V\P4'd;\P RVVV4V^8d\ R,P pKgKi\V\4'd*RV RV R2p \PV 4\V 4h\V\4'dMRP\P!V!4p \PRW4VRV RV R2, pEK \V\4'gEK#\VP 4\"JgEKCV^8gEKMVP pEK\ \%V4^8d \V4hV^8d~\ P'4Fcp WZP 8XgK\('d%\P R VV P*4R #\P!V P*4h R #R #) rz3Thread %s skipping test %s for following reason: %srszThread rz seconds z'Caught exception: %s exiting thread %sz exited with exception: rrNr) isinstancerrrSrTrr_r`rrrrrformat_exception SystemExitrycoder/rrrr`) rrcrr error_msg skip_coder(r$excrrs &&&& r4r)MultiThreadedTestCase._check_return_codesas  *ND1+C#x0011 I  q= *9 5 ? ?I!C..v%DWIZX S!"3''C++ggi88(CD GSwtf,EcU"MM C,,>S(Y] #I++0 y>A y) ) q="))+.$} T LL &// ==, r6c <V^8dQhRS[/#rr)r3rbs"r4r5rAr r6c\#rr rs&r4r) MultiThreadedTestCase.world_sizerr6c <V^8dQhRS[/#rr)r3rbs"r4r5rArEr6cNVP4PR4R,#rrGrs&r4r(MultiThreadedTestCase._current_test_nameswwys#B''r6r(c PVPV8XdVPWV4R#R#)z The reason why we have this util function instead of self.assertEqual is all threads are sharing one CPU RNG so the assertion result is only reliable on rank 0 N)r(rrryrr(s&&&&$r4assertEqualOnRank'MultiThreadedTestCase.assertEqualOnRanks% 99    Q3 ' r6cNVPV8XdVPW4R#R#r)r(assertNotEqualrvs&&&&$r4assertNotEqualOnRank*MultiThreadedTestCase.assertNotEqualOnRanks! 99     % r6)rUr(r.rrr)rerfrgrh__doc__rrr"r=rr(rErHr6r@rRrrQr[r7rrr)rrxr|rjrkrrs@@r4r!r!s kkmO/&   7 7#. D D%.;;:/>/>b""((((&1&&r6r!cPaa]tRtRtoV3RlV3RlltV3RlRltRtVtV;t#)SaveForwardInputsModuleich<V^8dQhRS[S[PS[P3,RS[RR/#r&forward_inputscast_forward_inputsr-NrnnModuler0Tensorr)r3rbs"r4r5$SaveForwardInputsModule.__annotate__s;77RYY 457"7  7r6ct<\SV`4\P!^d^d4VnWnW nR#)dN)r'r(rLinearlrrrrrr,s&&&r4r( SaveForwardInputsModule.__init__s. 3$,#6 r6cN<V^8dQhRS[PRS[P/#r&rr-r0r)r3rbs"r4r5rs'TTT%,,Tr6cWPV&TPVP'd4VPVPPP 44#T4#r)rrrtoweightdtyperrs&&r4forwardSaveForwardInputsModule.forwardsJ$%D!vv43K3K3Kadd466==../SSQRSSr6)rrr rerfrgrhr(rrjrkrrs@@r4rrs!77TTTr6rcPaa]tRtRtoV3RlV3RlltV3RlRltRtVtV;t#)SaveForwardInputsModelich<V^8dQhRS[S[PS[P3,RS[RR/#rr)r3rbs"r4r5#SaveForwardInputsModel.__annotate__s;--RYY 45-"-  -r6cp<\SV`4\W4Vn\W4VnWnR#r)r'r(rc1c2rrs&&&r4r(SaveForwardInputsModel.__init__s. ).N).N,r6cN<V^8dQhRS[PRS[P/#rr)r3rbs"r4r5rs####%,,#r6c^WPV&VPVPV44#r)rrrrs&&r4rSaveForwardInputsModel.forwards'$%D!wwtwwqz""r6)rrrrrs@@r4rrs--###r6rc#"V'g \PPV4\PP4;p'd VPMRpVf\ P !V4pR\PR&R\PR&V'dxV'dX\PPPPP4p\ P!RVVVR7M\ P!W VR7\PP!4\PP"P$P'4Rx\PP!4\PP"P$P'4V'd\ P(!4R#R# \PP!4\PP"P$P'4T'd\ P(!4ii;i5i) cpuNr| MASTER_ADDR6789 MASTER_PORTfaker)r'r(r))r0rrrryr1get_default_backend_for_devicerGrHtesting _internalrrM FakeStorerrrutilscountersclearr)r(r)r'init_pgrMaccrr@s&&&&& r4_dynamo_dist_per_rank_initrs  **40"--AACCSC%55kB +BJJ} &BJJ} MM++77??IIKE  # #%    # #G: V MM MM  &&()   $$**,   & & (    $$**,   & & ( s4A IAI&I.B?I.G#2A1I#A0IIcVaa]tRtRtoRt]V3Rl4t]V3Rl4tRtVt V;t #)#DynamoDistributedSingleProcTestCaseiz Test harness for single-process dynamo distributed tests, initializes dist process group. Prefer this for simple tests, as it's easier to debug. c <\SV`4VPP\P !\ PRRRR/44^Vn\PP4PpV RVP 2Vn WP9dRM VP.Vn\P !\P"!V4VP^R7R#)rr|r12355rRNr(r))r' setUpClass _exit_stack enter_contextrrrGrHr(r0rrryr* device_idsr1rr)rr*r,s& r4r.DynamoDistributedSingleProcTestCase.setUpClasss  %% JJ !;!7  ""668==xq + !'::!5CHH:   / / 7chhST r6cN<\P!4\SV` 4R#r)r1rr' tearDownClassrr,s&r4r1DynamoDistributedSingleProcTestCase.tearDownClasss ""$ r6rd) rerfrgrhr~rrrrjrkrrs@@r4rrs7  (   r6rc\a]tRtRtoRt]V3RlRl4t]V3RlRl4tRt Vt R#) "DynamoDistributedMultiProcTestCaseia Use this for tests that actually run on multiple GPUs. Decorate tests with @skip_if_lt_x_gpu(ngpu) Note: MultiProcTestCase spawns processes per test and is slow. Prefer MultiThreadedTestCase for most tests. Perhaps use this one sparingly for integration tests. c <V^8dQhRS[/#rr)r3rbs"r4r5/DynamoDistributedMultiProcTestCase.__annotate__ s00C0r6c>\PP4#r)r0rrrs&r4r)-DynamoDistributedMultiProcTestCase.world_sizes  --//r6c0<V^8dQhRS[RS[RS[RR/#r|ra)r3rbs"r4r5r$s- . . .#& .36 .  .r6c \P!\P!44V!V4pWnW6nVP W$4R#r)r addHandlerlogging NullHandlerr(r;r~rs&&&&&, r4rQ'DynamoDistributedMultiProcTestCase._run#s< W00239~ " i-r6)r;r(N) rerfrgrhr~rr)rrQrjrkrls@r4rrs200 . .r6rcaa]tRtRtoRtRtRtRt]!^xR7t Rt Rt ] V3RlRl4t ] V3RlRl4t] RR l4t] R 4t] V3R lR l4t] R 4t] V3RlRl4t] V3RlRl4t] V3Rl4t] R4t] V3Rl4tV3RlV3RlltRtRV3RlV3RllltV3RltRtVtV;t#) MultiProcContinuousTesti0N)secondsFc.<V^8dQhRS[R,/#rr)r3rbs"r4r5$MultiProcContinuousTest.__annotate__AsC$Jr6c R#)z ProcessGroup backend str. To be customized by sub test classes, e.g. "nccl". Otherwise we return None -- lazily decided by tensor. Nrd)rs&r4 backend_str#MultiProcContinuousTest.backend_str@r6c <V^8dQhRS[/#rr)r3rbs"r4r5rKs  C r6cb\PP4pVfR#VP#)Nr)r0rrry)r curr_devices& r4r#MultiProcContinuousTest.device_typeJs+'';;=  r6c R#)zs ProcessGroup init options. To be customized by sub test classes, e.g. ProcessGroupNCCLOpTest Here we return None. Nrd)rhigh_priority_streams&&r4optsMultiProcContinuousTest.optsQrr6c RVf \R4h\V4\PR&\P !W24p\P !VP4VVVVP4VPR7\PP4Vn R#)Nz!Expected rdvz_file to not be None LOCAL_RANK)r'r)r(r@ pg_optionsr) rr.rGrHr1rFrrrrrrrW)rr(r) rdvz_filer@s&&&& r4_init_pg MultiProcContinuousTest._init_pgZs   !DE E$'t9 < y5 OO%!xxzKK  &&99;r6c$<V^8dQhRS[RR/#)r&rr-Nr)r3rbs"r4r5ros   4 r6c VPR^R7R,pV!V4pVPVnVPVn\WC4p\P !4V!R/VBR#)rarZNrrd)rsplitr(r)rrr)rrrrrrs&&, r4_run_test_given_id*MultiProcContinuousTest._run_test_given_idns_NN3N3B7 9~HH ..$* !!# &r6c\aRp^Tu;8:dV8gM\RV RV 24hWnW nRpVPWV4\PR4TP4p \PRT 24T fMPTe(TP\P!T44KYTP!T 4TPT 4K~\PR 4T'g\4P6!4R#R# \dVp\ TRR4o\ T3Rl\P44R4p T 'dT PpRp?EL#hRp?ii;i \"Ed-p\%T\P4'dTPT4Rp?EK[\%T\4'dt\ TRR4o\ T3R l\P44R4p T 'd7TP\P!T P44Rp?EKRp\&P(!4p R P+\,P.!T !4p \1R T 24p YnTPT 4Rp?EKJRp?ii;i) Fz*Expected 0 <= rank < world_size, got rank=z , world_size=Nrjc3L<"TFqPS8XgKVxK R#5irr_r6vr_s& r4r77MultiProcContinuousTest._worker_loop..sL/q;;)3K/$ $zSetup completeTz Got test c3L<"TFqPS8XgKVxK R#5irrrs& r4r7rsT$7q;;);S$7rrzException in worker process: zTerminating ...)rr(r)rrirnextrrr`rSrir r#rrrr rgrr$rrrhr __cause__r1r)rr(r)r task_queuerraised_exceptioninit_skip_reasonr' skip_entryrr$tb_str enhanced_exr_s&&&&&& @r4 _worker_loop$MultiProcContinuousTest._worker_loop~s> T&J&   &'   & & ( E FD1ILJ--/LJ#-#5#5  >! 2b("3"344$((,b*-- 'FD 9I"&TJ$5$5$7T"J "(,,X->->z?Q?Q-RS #' <<>!;!;X!FG*-KF8+TU (*% $$[117 2sDD6"E3 E0A E+*E++E03 J+?1J&7BJ&AJ&&J+c<V^8dQhRR/#rrd)r3rbs"r4r5rsNNTNr6c v.Vn.Vn.Vn\P!RR7;_uu_4pVP VnRRR4\PPR4\\V44EF p\PP4p\PP4p\PPVPR\!V4,RW1VP WE3R7pVP#4VPP%V4VPP%V4VPP%V4\&P)RW6P*4EK R# +'giELW;i \dELHi;i)Fr4Nr]rLT)rNr?rrrO)r9 task_queuescompletion_queuesrr:r?rr0rr^rrTr/rrrr.rRrrSrirU)rr)rr(rrrs&& r4r`(MultiProcContinuousTest._spawn_processessI  "  ( ( 6 6!FFCM7    ! ! 2 27 ; #j/*D..446J$44::< ++33''#d)+ zT 4G MMO MM  ) OO " ": .  ! ! ( ()9 : LL94 M+7 6 6   sFF) F& ) F87F8c&<V^8dQhRS[RS[/#)r&rr-)r.r/)r3rbs"r4r5rs##r6c `\P!VRR4p\V\4'd(\P V4pVP V4pM VPpVR8XdF\P!V4P4pV^8Xd\P!RV R24hV#)z Get world_size, handling both class variable and property definitions. Properties are instance-level and need special handling in class methods. r)NzNo z devices available) inspectgetattr_staticrgrobject__new__fgetr)r0rrrr)rrworld_size_attr temp_instancer)s&& r4_get_world_size'MultiProcContinuousTest._get_world_sizes"00lDI ox 0 0#NN3/M(--ma } M  S^^,!%r6c <VPPRR4'g\SV` 4R#\P RVP R24VPFpVPR4K VPFpVP4K \P!VP4\PRVP R24\SV` 4R# \dL@i;i)z{ Class-scope test fixture. Run once for entire test class, after all tests finish. Tear down the process group if spawned. rFNzJoining z workerszClass z finished)rr r'rrSrir)rr#r9rrGrrrrTre)rrrr,s& r4r%MultiProcContinuousTest.tearDownClassHs|| 4e<< G ! #  x/x89//J NN4 *}}G LLN%  IIcmm $  fS\\N)45     s C22 D?Dc<V^8dQhRR/#rrd)r3rbs"r4r5rgs&&t&r6c <\SV`4VPP4VPVnVPP 'd(\P!RVP4 24h\VP4FMwr\PRV RVP4 24VPVP44KO R#)z% Test fixture. Run before each test. zPrevious test failed, skipping z Sending Rank r%N)r'r6r,rrr( poison_pillrrrHrrrSrir#)rrrr,s& r4r6MultiProcContinuousTest.setUpgs   002**  >> % % %##&Edggi[$QR R't'7'78MA LL=2dggi[9 : NN4779 %9r6cXa\S4V3Rl4p\P!W 4#)c F<VPVP8XEd}\PRVP 4 24Rp\ \ VPVP44EFwpwr4\W4\VP 44R7pVeK3\V\P4'dTpKW\V\4'dV\PRV RVP 4 RVP P" 24RVP nTpKWPP 48wd \'RV RVP 4 24h\PR V R VP 4 24EK! VeVhR#S!4R#) zWaiting for workers to finish NrzDetected failure from Rank z in: z(, skipping rest of tests in Test class: TzExpected rv == self.id(), got z != zMain proc detected rank z finished )r(rrSrirHrrr9rrrrgrrr rr,rerr)rdeferred_exceptionrrBrrvrs& r4r>MultiProcContinuousTest._worker_run_main_wait..wrapper}ssyyD222 =dggi[IJ&*"09(>(>?1,A,?[5KB*5 !"h&7&788-/* !"m449!E$'')MEEI^^E\E\D]_6:2-/* WWY,.1&1&f  <&&*,/d} r6rcaa]tRtRto]t]V3RlRl4t]V3RlRl4t V3RlRlt ]V3Rl4t V3R lV3R llt R t V3R ltR tVtV;t#)C10dTorchCommsTestBaseic <V^8dQhRS[/#rr)r3rbs"r4r5#C10dTorchCommsTestBase.__annotate__s3r6c<RV9dR#RV9dR#RV9dR#R#rrdrs&r4r'C10dTorchCommsTestBase.backends$ V  f_ f_r6c <V^8dQhRS[/#rr)r3rbs"r4r5r/s((C(r6clVPp\V4'dV!4pVPV4#r)rrr')rrs& r4r"C10dTorchCommsTestBase.backend_strs,oo K %-K{{;''r6c$<V^8dQhRS[RR/#)r&r*r-Nr)r3rbs"r4r5r/s Q Q3 Q4 Qr6c R\R\R\R\R\R\ /pVP V4pW29d(W#,'gVPRV R24R#R#R#) rrxr$rcclncclxrcclxz torchcomms z backend is not availableN)TORCHCOMM_HAS_GLOOTORCHCOMM_HAS_XCCLTORCHCOMM_HAS_NCCLTORCHCOMM_HAS_RCCLTORCHCOMM_HAS_NCCLXTORCHCOMM_HAS_RCCLXr'skipTest)rr* backend_flagsrBs&& r4_skip_if_backend_unavailable3C10dTorchCommsTestBase._skip_if_backend_unavailablesj & & & & ( (  ||F+  (1L1L MMK ~5NO P2M (r6c<R\PPnR\P R&\ \44\P R&\ V4\P R&\ V4\P R&V\P R&\SV`%WV4VP4pRV9gRV9d\PP4pV'd]\P!VP R V 24p\P!V4\PP!V4R #\#R V R 24hR #) Tr|rrr:r;TORCHCOMM_STORE_PATHr$rxrRrrN)r0rconfiguse_torchcommsrGrHr.rr'rrrrr*ryrrr)rr(r)rr'rr*r,s&&&& r4rC10dTorchCommsTestBase._init_pgs26  /$/ =!$'(8$9 =!'*4y #$'*: #$-6 )* 95//# W ' 1++??AK)9)9(:!D6&BC((0!!226:"[\c[ddz{!2r6c<V^8dQhRR/#rrd)r3rbs"r4r5r/str6c<VPPp\PRVP 4V4\ V4'dV!4pVP \V44\SV`%4R#)z&Setting up test: %s on device type: %sN) r,rrSrirHrrBr.r'r6)rrr,s& r4r6C10dTorchCommsTestBase.setUpsWnn00  =twwy+V K %-K ))#k*:;  r6c\P!V4P4p\VP4Uu/uF q3W2,.bK up#uupirrrs&& r4r%C10dTorchCommsTestBase.rank_to_device rrc&<V^8dQh/S[;R&#rr)r3rbs"r4r5r/s(r6rd)rerfrgrhr r)rr'rrrBrr6rrirjrkrrs@@r4r-r-sh(J(( Q Q*NAr6r-cV^8dQh/^\9d \;R&^\9d\PR,;R&#)r&PLATFORM_SUPPORTS_SYMM_MEMNr)__conditional_annotations__rrr)r3s"r4r5r5s?lDmt32 $ $t +2ur6))rr:)rxr;)r$r<)r9r?)r8r>)r$rxr~r)NFr)NTF)rQrn functoolsrrrrrrGrr rrrrrrrcollections.abcr contextlibr dataclassesrdatetimerenumrrr r ior typingr r unittest.mockrr0torch._dynamo.test_casetorch.cuda.nccltorch.distributedrr1torch.nnrtorch._C._autogradrtorch._C._distributed_c10drtorch._logging._internalrrErtorch.testing._internalr$torch.testing._internal.common_utilsrrrrrrrrrrrrr 5torch.testing._internal.distributed.multi_threaded_pgr!r"r#r:r;r<r=r>r?rI_backend_flagis_backend_builtglobalsrY getLoggerrerSsetLevelINFOr9rHAS_ACCELERATORr]rrwrrrrrrrrrrrrrrrrr r#r%r(r+r/r=rArIrPrMrVrirprtr{rrr/getenvrrrrrrrrrrrrrZrr rrrcacherrr9r!rrrrr test_caserrrr-r5)rQs@r4rpsV    $%!,,"  )7.C0%  & &x 0 0#GIe ?D   8 $ 4E?33x338z 8 Cx$FG  Xb"BC  x45 8B =>8B >?8B >?8B >?8B >?8B >?8B >?8B >? HR> ? (267!"hr#LM#$x V%*8B DE+,hr#BC- 4 ** **&6   U8,(" ,+\4  N: $+1$ S *"F"  a   :O"))$GOPOc"C+.'( I22(X2/32 " K6QAuzz'>'>'@ AB8N 5N.5NpD &0 3E7tl&Hl&^ Tbii T #RYY # #)#)L  %--*A*A*J*J  F.)<.8OhOd BN4BNr6