+ jwaRWtX0t^RIt^RIt^RIt^RIt^RIHt^RIHt^RI H t ^RI t ^RI t ^RI HuHt^RIHt^RIHt^RIHt^RIHtHtHt^R IHt]P8!]4t!R R ]4t!R R ]4t !RR]4t!]!RR7!RR44t"RXt#]!PH]"!RRR4]!PJ]"!RRR4]!PL]"!RRR4]!PN]"!RRR4]!PP]"!RRR^d]#,R7]!PR]"!RRR^K]#,R7]!PT]"!RRR 4/t+]^k] PX]!PN] PZ]!PJ] P\]!PH] P^]!PL/t0]^k] PX]!PR] PZ]!PR] P\]!PP] P^]!PP/t1]^k]PdR!R"l4t3]PdR#R$l4t4R%R&lt5R'R(lt6R)R*lt7RYR+R,llt8RYR-R.llt9R/R0lt:R1R2lt;!R3R4]4t<!R5R6]4t=!R7R8]4t>^t?^t@.RZO.R[O.tA.R\O.R]O..R^O.R_O..R`O.RaO..tB.RbO.RcO.RdO.ReO.tC.RfO.RfO.RgO.RhO.tD.RfO.RfO.RiO.RjO.tE.RkO.RlO.RmO.RnO.tF.RoO.RpO.RqO.tG] PX^] PZ^] P\^] P^^ /tH] PXR:] PZR;] P\R<] P^R=/tI]^kR9tJ] PXR] PZR>] P\R] P^R/tK]^kR?R@ltLRARBltMRCRDltNRERFltORGRHltPRIRJltQRKRLltRRrRMRNlltSRORPltTRQRRltURSRTltVRsRURVlltWR#)tN) dataclass)IntEnumAny)optimization_hint)normalize_function)ir)get_dtype_sizesnode_args_kwargs sympy_product)Vc.]tRt^t^t^t^t^t^t^t Rt R#) NCCL_COLLN) __name__ __module__ __qualname____firstlineno__ ALL_REDUCE ALL_GATHERREDUCE_SCATTER ALL_TO_ALL UNSUPPORTEDP2P__static_attributes__r^/home/luca/screen-checking/.venv/lib/python3.14/site-packages/torch/_inductor/comm_analysis.pyrrs JJNJK Crrc&]tRt^!t^t^t^t^tRtR#)NVIDIA_GPU_TYPErN) rrrrVOLTAAMPEREHOPPER BLACKWELLrrrrrr!s E F FIrrc2]tRt^(t^t^t^t^t^t^t ^t Rt R#)InterconnectTyperN) rrrr NVLINK_V100 NVLINK_A100 NVLINK_H100 NVLINK_B200IB_HDRIB_NDRPCIErrrrr%r%(s%KKKK F F Drr%T)frozenc4a]tRt^2toRt^tV3RltRtVtR#)InterconnectProfilezIEmpirically calibrated interconnect parameters for saturation estimation.cJ<V^8dQh/S[;R&S[;R&S[;R&S[;R&#) bus_bw_GBpslatency_per_hop_usbase_latency_usmin_saturation_bytes)floatint)format __classdict__s"r __annotate__ InterconnectProfile.__annotate__2s9     !rrN) rrrr__doc__r5__annotate_func__r__classdictcell__)r9s@rr/r/2sS !"rr/B@D@$@g6@F@gN@g>@,@gI@g(@)r5g&@g.@c$V^8dQhR\/#r1return)bool)r8s"rr:r:bsTrc^RIp\PP4'd$\PP 4^8dR#VP .RORR^R7pVP ^8Xd(VPPR4^,pRV9#\PP^^4# TP\3dL:i;i \\3dR#i;i)zIDetect NVLink via nvidia-smi topology, falling back to peer access check.NT)capture_outputtexttimeoutLegendNV)z nvidia-smitopoz-m) subprocesstorchcuda is_available device_countrun returncodestdoutsplitTimeoutExpiredOSErrorcan_device_access_peerAssertionError RuntimeError)rOresultrNs r _has_nvlinkr^as :: " " $ $ (?(?(AA(E   (       !==&&x03D4<  "zz00A66  % %w /   L )s%AC CCCC21C2c$V^8dQhR\/#rE)r)r8s"rr:r:}s&&o&rca\PP4'd\PP4^8d\PP ^4P pV^ 8d\ P#V^ 8d\ P#V^8d\ P#V^8d\ P#\PPP\PPP4;'gRoRS9d\ P#RS9d\ P#RS9d\ P#\;QJdV3RlR4F 'gK RM RM!V3RlR44'd\ P#\ P#) rV100A100H100c3,<"TF qS9xK R#5iNr).0gpugpu_infos& r get_gpu_type..s A(@H_(@TF)B100B200B300)rPrQrRrSget_device_propertiesmajorrr#r"r!r utils collect_env get_gpu_inforTany)rqris @r get_gpu_typerv|s< zz  UZZ%<%<%>%B 00399 B;",, , A:")) ) A:")) ) A:"(( ({{&&33EKK4K4K4O4OPVVTVH $$$ 8 %%% 8 %%%  A(@ A A(@ A A A(((%%%rc0V^8dQhR\R\/#)r1 group_sizerF)r7r%)r8s"rr:r:s??C?,<?rc\PP4'd\PP4M^p\ 4p\ P !W, 4^8XdE\4'g\P#\PV\P4#\PV\P4#)zEAuto-detect interconnect type from GPU generation and group topology.)rPrQrRrSrvmathceilr^r%r, _GPU_TO_INTRAgetr' _GPU_TO_INTERr*)rx gpus_per_nodegpu_gens& rdetect_interconnectrs161H1H1J1JEJJ++-PQMnG yy+,1}}#(( (  *:*F*FGG   W&6&=&= >>rc0V^8dQhR\R\/#)r1 kernel_namerF)strr)r8s"rr:r:s%%c%i%rc aSf \R4hRS9d\P#RS9d\P#RS9d\P#\ ;QJdV3RlR 4F 'gK RM RM!V3RlR 44'd\P #\ ;QJdV3RlR 4F 'gK RM RM!V3RlR 44'd\P#\P#) Nzkernel_name must not be None all_reduce all_gatherreduce_scatterc3,<"TF qS9xK R#5irfrrgcommrs& rrj7get_collective_type_from_kernel_name..s H-GT[ -GrlTFc3,<"TF qS9xK R#5irfrrs& rrjrs M-LT[ -Lrl) all_to_allalltoall)isendirecv batch_p2p) r[rrrrrurrr)rsfr$get_collective_type_from_kernel_namers;<<{"###  $### [ ('''  H-G H H-G H H H###  M-L M M-L M M M}}$$$rcDV^8dQhR\PR\/#r1noderF)r IRNoder)r8s"rr:r:s66bii6I6rc\V\P4'g\RV 24hVPpVf \ R4h\ V4#)z!node is not a collective kernel: z(node.python_kernel_name must not be None) isinstancer _CollectiveKernel ValueErrorpython_kernel_namer[r)rnames& rget_collective_typersO dB00 1 1>7v>??rc"]tRt^t^t^t^tRtR#)NCCL_HWrN)rrrrNVLINKPCINETrrrrrrs F C Crrc]tRt^t^t^tRtR#) NCCL_ALGOrN)rrrrTREERINGrrrrrrs D Drrc"]tRt^t^t^t^tRtR#) NCCL_PROTOrN)rrrrLLLL128SIMPLErrrrrrs B E Frr8@g^@n@g w@g@9@c$V^8dQhR\/#rEr6)r8s"rr:r:s5555rc\PPPpVe \ V4#\ 4'g\ #\P\4R4#)zEReturn intra-node bandwidth in GB/s. Config overrides auto-detection.r) rP _inductorconfig intra_node_bwr6r^_PCIE_INTRA_NODE_BW_GPU_NVLINK_BWr}rvoverrides rget_intra_node_bwrsI%%33HX ==""   lne 44rc$V^8dQhR\/#rEr)r8s"rr:r:s8858rc\PPPpVe \ V4#\ P \4R4#)zOReturn inter-node (IB/RoCE) bandwidth in GB/s. Config overrides auto-detection.r)rPrr inter_node_bwr6_GPU_INTER_NODE_BWr}rvrs rget_inter_node_bwrs<%%33HX  ! !,.$ 77rc0V^8dQhR\R\/#)r1nrF)r7)r8s"rr:r:s  c c rcRV^8:d^#^pV^8dV^,pV^, pKV#)z4Integer log2, matching NCCL's log2i (floor of log2).r)rrs& r_log2irs/Av A a% a Q Hrc2V^8dQhR\R,/#r1rFNr)r8s"rr:r:s!!edl!rcVPpVf \R4h\VRR4pVPR ,p^RIHpV!V4p\ PPV4p\ P!RV 24p\V4p\V4wrRV9dV R,V ^,,p \ PPWWR7;_uu_4p V!V /V Bp \ PPPP!V 4RRR4X P"p V f \R 4hV ^8dR#V R , pV# +'giL?;i) Nzsnode.node must not be Nonerra)_resolve_process_groupzcuda:all_gather_into_tensor_out:rNNgroupdevice,NCCL time estimator did not produce a result@@r)rr[getattrrrrrP distributedget_rankrevalr _time_estimatorops_c10d_functional wait_tensordefaultestimated_time)snodekernelpy_kernel_namepg_namerpgrankrfnargskwargstime_estimatorw est_time_us est_time_mss& r/estimate_nccl_collective_runtime_nccl_estimatorrs: ZZF ~:;;V%92>N""2&GI  (B!!**2.D\\E$. )F n B$U+LD$~5Bx$q'!    * * * C C~    ""..66q9 D!//KKLLQ#K  D Cs 39d V^, pM@V\6P@8Xd^V^, ,pMV\6PB8Xd^pMR #V\(P08Xd V\6P88Xd VR,pM6V\(P*8XdVRV,V, ,pM VR,pV^8:dR #V \PD8Xd\FPHM\FPJp\LV,V,p\NV,V,V,p\N\FPP,V,V,pV^8dRMRpV\P48XdV^8d VR ,p\%VV4pV\(P*8Xd^V^8Xd^M-V\6P88Xd^V^, ,MV^, pVVV, V,VV,,, pMpV\(P08Xd\V\6P88XdGV^V\%V^4,^, V,\SV4V,,,, pV\(P08Xd^V\6P88XdIV^@8d\SV^, 4M^p^Tu;8:d^8dMMV\TV,V,,pVVR V,, ,pV#)zCompute NCCL estimated time in us for a given (algo, proto) pair. Mirrors ncclTopoTuneModel bandwidth/latency computation and ncclTopoGetAlgoTime from NCCL tuning.cc. Returns -1 if the (algo, proto) combination is disabled for this configuration. ?gp= ף?gq= ףp?ig(\??gq= ףp?gg@rgg?gy 5?g98?)+rPrQrRrSrzr{rvrr_GPU_NCHANNELSr}rr%r+r*rrminmaxllMaxBwsrrrperChMaxRingLL128BwsrperChMaxTreeLL128BwsrrrperChMaxTreeBwsrrrrr,rrrbaseLathwLatrrtreeCorrectionFactor) rrxrrrrnNodesnRanks compCapIndexindex2bwIntrabwInter nChannels interconnectis_ibbwnic_efficiency multi_railbusBwindex1llMaxBwfactornsteps bandwidthintraHwlatintraLatinterLat netOverhead nInterStepslogSizetime_uss &&&&& r_nccl_algo_timer=s271H1H1J1JEJJ++-PQM YYz1 2F F>L!Q;VaZAF  !G!G""<3I'z2L -446F6M6MN NE    A:K A:%K {   ! W !s3!q0@'A AA n A:fj1J>4GJ * B E#q[\aFvv&G y~~%:=="8GUS[)  EZ-=-=$= ] # ,\:6B B   EZ]]$:EY'1  EZ-=-=$=$k} FN ,\:6B B   EZ->->$> 9'' ' i/,2OPV2W&WXE z  y###fqj! )**I,@,@A A! %% %fqj!    y~~$)*>*>">CK  S6\F23 CK A~ *-=-B-BBgkkG $- CW~d#E*HW[[!$'.H!#K !!!fqjs 8[)H y~~{ &*i.B.B&B!vz"QR   $0;3III   9'' ' 13vq>)A-9F6NX*>">)/2&1%1  "  -e4W= =IFfy011G Nrc tV^8dQhR\R\R\R\\\\3,/#)r1rrxrrF)r7rtupler6)r8s"rr:r:s;,, ,, , 5#s? ,rc \R4p\Pp\Pp\P \P3FpV\P 8XdV\ P8wdK/\P\P\P3F+p\WW&V4p^Tu;8:d V8gK!MK%TpTpTpK- K W4V3#)zFind the best (algo, proto) for a given collective and return (time_us, algo, proto). Iterates all supported (algo, proto) combinations and picks the one with minimum estimated time, mirroring NCCL's algorithm selection. inf) r6rrrrrrrrrr=) rrxr best_time best_algo best_protorrts &&& r_nccl_best_algo_timerFse IIJ0 9>> !di.B.B&B  mmZ%5%5z7H7HIEDFAA! !!  " J 1  ++rcHV^8dQhR\R\R\R\/#)r1tensor_storage_size_bytesrxrrFr)r8s"rr:r:s,"03;D rcV^8:d^#V\P8Xd^#\WV4wppV^8gV\R48Xd^#VR, #)aDReturns estimated NCCL collective runtime in milliseconds (ms). Uses a multi-algorithm, multi-protocol model aligned with NCCL's ncclTopoTuneModel and ncclTopoGetAlgoTime (tuning.cc). Evaluates Ring and Tree algorithms across LL, LL128, and SIMPLE protocols, selecting the (algo, proto) pair that minimizes estimated time. rAr)rrrFr6)rHrxrr<_s&&& r%estimate_nccl_collective_runtime_implrKsSQ y$$$()BPTUMGQ{gu- V rcHV^8dQhR\R\R\R\/#)r1rxrtarget_efficiencyrFr)r8s"rr:r:s0!8!8!8 !8!8 !8rcr\P\P\P3pW9d\ RV RV 24hV^8:d^#\ \ V4,pV\P\P39d V^, pM(V\P8Xd^V^, ,pM^#VPWTP,,pVR,pVPR,pVRV, , p \W,V,4p \WP4#)zCompute min message size for target BW efficiency using empirical profiles. Uses per-interconnect profiles calibrated to real NCCL behavior (algorithm, protocol, channel selection) rather than the analytical model. zUnsupported collective z, expected one of gư>geAr) rrrrr[INTERCONNECT_PROFILESrr4r3r2r7rr5) rxrrM _SUPPORTEDprofiler3total_latency_us latency_sbw_bytes_per_s eff_ratio min_bytess &&& rcompute_min_saturation_bytesrWs && (@(@)BVBVWJ %dV+=j\ J  Q#$7 $CDG $$i&>&>??a %% %j1n%..:T:T1TT 4'I((3.N!S+<%<=II)N:;I y66 77rcDV^8dQhR\PR\/#r)r rr6)r8s"rr:r:s  299  rc\\V4p\V4p\V4p\WV4#)zReturns estimated NCCL collective runtime in milliseconds (ms). Uses the multi-algorithm, multi-protocol analytical model aligned with NCCL's tuning.cc to select the best (algo, proto) pair for the given collective size and topology. )rrrrK)rrHrxrs& r estimate_nccl_collective_runtimerZs6!@ E*40J t $D 0!t rcXV^8dQhR\PPR\/#r1fx_noderFrPfxNoder7)r8s"rr:r:s",&,&,&3,&rcaaRoVPVPr!\V4pVPRR4RRloRVV3Rllp\P !\ PPVW34VPPRR4pSeVf^#\V\\34'd\V3RlV44pSV,#\V\ P4'dS!V4pSV,#^#) zSEstimate the size of a collective operation in bytes, including inputs and outputs.NoutcDV^8dQhR\PR\/#)r1rErF)rPTensorr7)r8s"rr:1estimate_fx_collective_size..__annotate__s"JJ JJrch\VP44\VP4,#rf)rrr r)rEs&r tensor_bytes1estimate_fx_collective_size..tensor_bytess!%affh/.2IIIrcLV^8dQhR\PP/#)r1r)rPr_r`)r8s"rr:res--588==-rc<VPPRR4p\V\P4'gR#Sf^oSS!V4, oR#)valN)metar}rrPrd)rinp_val input_bytesrgs& r add_inp_bytes2estimate_fx_collective_size..add_inp_bytessG((,,ud+'5<<00   K|G,, rrkc3x<"TF/p\V\P4'gK%S!V4xK1 R#5irf)rrPrd)rgrErgs& rrj.estimate_fx_collective_size..s) %/:a3NOLOOZs"::)r r dictpoppytree tree_map_onlyrPr_r`rlr}rlistr?sumrd)r]r r ro output_val output_bytesrnrgs& @@restimate_fx_collective_sizer{sK<<& &\F JJudJ--     !!%.Jj0*tUm,, %/    %% J - -#J/   %%rcXV^8dQhR\PPR\/#r\r^)r8s"rr:r:s" = =UXX]] =s =rcV^RIHp\V4pV!V4'gV#V^,#)zEstimate the memory footprint of a collective operation in bytes. This returns the total bytes that need to be live concurrently in memory. For all_reduce, we divide by 2 since it can be done in-place. )is_all_reduce_tensor)#torch._inductor.fx_passes.bucketingr~r{)r] is_all_reducers& r'estimate_fx_collective_memory_footprintrs,  'w /D$W--4<419.__annotate__Vs;;EDL;rc<aa^RIHpHpV!S4p\PP P V4VP8XdR#\P!R4pVPV4pTP'gR#\P!SS34wrVRT3RlloRTT3RlloTUu.uF pS!T4NK pp\P!YV4wrSPp \!T \P"P$4'g\'R\)T 4 24h\PP+Y#R 7;_uu_4p T !T/T Bp \!T \,\.34'd>T F6p \P0P2P4P7T 4K8 M3\P0P2P4P7T 4RRR4X P8pTf \'R 4hT^8dR#TR , pT# \dR#i;iuupi +'giLV;i) r)rBackendNrQc8V^8dQhR\P/#rE)rPrd)r8s"rr:[estimate_nccl_collective_runtime_from_fx_node.._nccl_estimate..__annotate__is  ELL rcD<\P!SfTMS.VVR7#)N)rr)rPempty)rrrrs&&&r_tensorVestimate_nccl_collective_runtime_from_fx_node.._nccl_estimate.._tensoris&;;%-M? rc0V^8dQhR\R\/#)r1erFr)r8s"rr:rps  c c rc:<\V\PP4'dS!VPR,4#\V\P 4'd7S!\ VP44.VPVP4#V#)rk) rrPr_r`rlrdrrrr)rrto_real_tensors&rr]estimate_nccl_collective_runtime_from_fx_node.._nccl_estimate..to_real_tensorpsg!UXX]]++%affUm44!U\\** 6qvvx @A177AHHUUHrrrrr)rrrrPrdistributed_c10d get_backendFAKEr _get_backendr\supports_time_estimateru tree_flattentree_unflattentargetr_ops OpOverloadr[typerrwr?rrrrr)rrrrbackend flat_argsflat_args_pytree_speca real_args real_kwargsrr r rEr rrrr r]rr rs @@r_nccl_estimateEestimate_nccl_collective_runtime_from_fx_node.._nccl_estimateVsV #J /    - - 9 9" = Mf% oof-G---+1+>+>f~+N(     1:: 1^A& :!'!6!6y!X ^^"ejj3344 CDH:N    . ./  I--A!dE]++AII..::BB1E **66>>qA %33   !OP P ?!C' _  *;  s%%H';H9BH>' H65H6> I )rrrrPrrall_to_all_singlerr{rrr[rr r rrrrrrrrK)r]rrrrHopt_args_kwargsrrxrrrr rr sff& @@@r-estimate_nccl_collective_runtime_from_fx_noder*sYK~~33EEMMM#$?$H!$1!'..#&&EgnnEUVWW( \\~~%) O ?@@"LD&G$VL%9:J(4J gnnejj&;&; < <?W^^@T?U V   00C0C0E FD;;z$&  "  0!t rcV^8dQh/^\9d\\\3,;R&^\9d\\\3,;R&^\9d\\\3,;R&^\9d\\\ 3,;R&^\9d\\\ 3,;R&#)r1rOr|r~rr)__conditional_annotations__rsr%r/rr6)r8s"rr:r:sF  t,.AAB GdtO%556eptO%556qn _e+,o H D%/0I ri)i)g333333@rC @)gffffff@rCr)333333?g?@)rffffff?g333333 @)rrr)rg@g@)g@g!@rC)g@rrC)C@rgffffff4@)gU@g6@g3@)ga@rBgA@)gq@gV@Q@)4@rr)皙YB@rr)r@r@r@)rrg=@)gK@gQ?@r)g:@g2@rA)rg7@g1@)gYC@g33333D@r?)rgffffffE@r)rrrr?皙?ffffff?rrrrr皙?rrrrrrrrrrr)rrrrrrrrrrrrrrrrrrrrrrrr)rrrrrrrrrrrrrrrrrrrrrrrr)r)NT)Yrrloggingrzr dataclassesrenumrtypingrrrPtorch.utils._pytreerr_pytreeru%torch.fx.experimental.symbolic_shapesrtorch.fx.operator_schemasrrar r r r virtualizedr getLoggerrlogrrr%r/_MBr(r'r)r&r+r*r,rOr r!r"r#r|r~ lru_cacher^rvrrrrrrrrrr _NUM_ALGOS _NUM_PROTOSr r!rrrrr"rrrrrrrrr=rFrKrWrZr{rrr:)rs@rrs, ! $$C8CC!  g w  $"""  "5dD$"G  "5dD$"G  "5dD$"G  "5dD$"G0 dDsSy0 dDrCx.tT4@ F  +77,88,88/;; : +22,33,33/66 :  4 &&4?%"6H @ g           ,     66qRp1ABr 5EEu 0 4DDt 458 !HRj,8,!8R ,&^ =ppr