+ je\aR@tX0t^RIt^RIt^RIt^RIt^RIt^RIt^RIHt^RI H t ^RI H t ^RI HtHtHt^RIt^RIHuHt^RIHuHuHt^RIHt^RIHt^RIHt^R IH t H!t!H"t"H#t#H$t$^R I%H&t&^R I'H(t(^R I)H*t*H+t+H,t,H-t-H.t.H/t/^R I0H1t1H2t2^RI3H4t4^RI5H6t6]Pn!]84t9Rs:]^kRs;]^kRRlt<]PzRARRll4t>]PzRARRll4t?]P!RRR7!RR44tA]P!RRR7!RR]A44tBRRltCRR ltDR!R"ltE]F!4tG]^kR#R$ltHR%R&ltI/tJ]^kR'R(ltKR)R*ltL!R+R,4tMRBR-R.lltN] RBR/R0ll4tOR1RR2RR3R/R4R5lltPR1R/R6R7lltQ!R8R9]4tR!R:R;]4tS!R<R=]PP4tV!R>R?]PP4tWR#)CN) defaultdict)Sequence)cache)Anycast TypedDict)_are_we_tracing) _MeshLayoutone_step_redistribute_cost)_StridedShardNotDecodableError DTensorSpec ShardOrderShardOrderEntry TensorMeta)assert_no_mixed_partial_types) DeviceMesh)_is_shard_like _StridedShardPartial Placement ReplicateShard) FloatLikeType IntLikeType)get_active_debug_mode)not_noneFc0V^8dQhR\R\/#)costreturn)rfloat)formats"g/home/luca/screen-checking/.venv/lib/python3.14/site-packages/torch/distributed/tensor/_redistribute.py __annotate__r%9sCCmCCc\V\4'dV#\V\P4'dVPP pM \V4#VP pV'g \V4#\VPP4pVPpVPV4'd\VPV44#\VPVRR74#)N)fallback) isinstancer"torchSymFloatnodeexpr free_symbolsr shape_envvar_to_hint_overrideissubsetxreplaceoptimization_hint)r r-r.r/hint_overridess& r$_redistribute_cost_sort_keyr59s$ $''yy~~T{$$L T{,,-I33N^,,T]]>233 ,,TD,A BBr&Tc$V^8dQhR\/#)renabledbool)r#s"r$r%r%Ns9898d98r&c#:"\pVsRxVsR# Tsi;i5i)u Context manager to control the redistribution planning strategy for DTensor operations. This context manager allows you to choose between two algorithms for computing the sequence of collective operations needed to redistribute a DTensor from one placement to another: - **Graph-based**: Uses Dijkstra's algorithm to find the minimum-cost path through all possible placement transformations. This approach considers the global cost of all collective operations and finds the optimal sequence. Best for complex redistribution patterns where reducing communication cost and memory overhead is critical. - **Greedy**: Uses a heuristic approach that makes locally optimal choices at each step. This is faster to compute but may not produce the globally optimal transformation sequence. Best for simple redistribution patterns or when planning speed is more important than optimal communication. **Default Behavior (without this context manager):** When this context manager is NOT used, the algorithm selection follows this priority: 1. **Non-default shard orders** → Always use graph-based algorithm (required for correctness) 2. **Explicit `use_graph_based_transform` parameter** to `_gen_transform_infos_non_cached` → Use the specified algorithm (True = graph-based, False = greedy) 3. **No explicit parameter** (default case) → Use greedy algorithm for faster planning **Behavior with this context manager:** This context manager overrides the default selection by setting the global flag `_FORCE_MIN_COST_REDISTRIBUTION_PLAN`, which takes precedence over the explicit `use_graph_based_transform` parameter (but not over non-default shard order requirements). **Cache Considerations:** The redistribution planner caches transform info for performance via the `@cache` decorator on `_gen_transform_infos`. If you need to change the algorithm selection for the same input specs, clear the cache using `_gen_transform_infos.cache_clear()` to ensure the new setting takes effect and doesn't reuse cached results from a previous run. Args: enabled (bool): If True, forces the use of the graph-based algorithm. If False, forces the use of the greedy algorithm. Default: True N)#_FORCE_MIN_COST_REDISTRIBUTION_PLAN)r7 old_values& r$ use_min_cost_redistribution_planr=Ms'j4I*1'8 .7+i+ c$V^8dQhR\/#)rdisabledr8)r#s"r$r%r%sAA$Ar&c#:"\pVsRxVsR# Tsi;i5i)a Context manager to disable the transform optimization pass that merges consecutive same-type collectives into single flattened operations. When the optimization is disabled, ``_optimize_transform_infos`` becomes a no-op and returns the original list of ``_TransformInfo`` objects unchanged. This is useful for debugging or isolating issues related to the flattened collective merging logic. The flag can also be set directly:: torch.distributed.tensor._redistribute._DISABLE_REDISTRIBUTE_TRANSFORM_OPTIMIZATION = True Args: disabled (bool): If True (default), disables the optimization. If False, explicitly enables it (the normal default). N),_DISABLE_REDISTRIBUTE_TRANSFORM_OPTIMIZATION)r@r<s& r$+disable_redistribute_transform_optimizationrCs'*=I3;0A 7@4y4r>frozenslotscDa]tRt^toRtV3RlRltV3RltRtVtR#)_TransformInfocVP^8d\hVP^,VP^,8Xd \R4hR#)rzQTransformInfo should only be created if it is an op with some effect, not a no-opN)mesh_dimAssertionErrorsrc_dst_placementsselfs&r$ __post_init___TransformInfo.__post_init__sH ==1   " "1 %)@)@)C C c  Dr&c.<V^8dQhRS[R,/#rr!Nstr)r# __classdict__s"r$r%_TransformInfo.__annotate__sd r&c jVPwrVP4'dVP4'dR#VP4'd\V4'dR#\V4'dVP4'dR#\V4'd\V4'dR#R#)z Return a key for grouping transforms by communication type. Returns None for local ops (no communication needed), or a string that identifies the collective type for potential grouping/merging. all_reducereduce_scatter all_gather all_to_allN)rL is_partial is_replicater)rNsrcdsts& r$_comm_type_key_TransformInfo._comm_type_keys** >>   0 0 2 2 ^^  ."5"5# C S%5%5%7%7 C ^C%8%8r&cd<V^8dQh/S[;R&S[S[S[3,;R&S[S[,;R&#)rrJrL logical_shape)inttuplerrr)r#rUs"r$r%rVs8Mi233 K(( r&N) __name__ __module__ __qualname____firstlineno__rOr`__annotate_func____static_attributes____classdictcell__rUs@r$rHrHsr&rHcFa]tRt^toRtRtV3RlRltV3RltRtVt R#)_FlattenedTransformInfoa  Represents a flattened transform that combines multiple mesh dimensions into a single collective operation using a flattened DeviceMesh. Note: inherits the fields from _TransformInfo. Gets an __init__ with parent fields, followed by child fields, and runs parent validation (post_init) Nc<V^8dQhRR/#rRrf)r#rUs"r$r%$_FlattenedTransformInfo.__annotate__str&c\PV4VPe,VP^8g\RVP 24hR#R#)Nz"avg_scale must be > 1 if set, got )rHrO avg_scalerKrMs&r$rO%_FlattenedTransformInfo.__post_init__sL$$T* >> %>>A%$88HI& &r&c`<V^8dQh/S[;R&S[S[R3,;R&S[R,;R&#)rmesh.original_mesh_dimsNrtrrerd)r#rUs"r$r%rrs8 c3h' Tz !r&rf) rgrhrirj__doc__rtrOrkrlrmrns@r$rprps"!I%r&rpc V^8dQhR\R\\,R\\\\,3,RR/#)rtransform_infocurrent_placementsshard_order_dictr!N)rHlistrdictrd)r#s"r$r%r%sA.5.5".5Y.53S >*.5 .5r&c VPwr4\V\4'dVPpM VP3p\V\ \ ,4'dVPp\4pVFUp\W&,4^8Xd\RV RV RV RV 24hVPW&,P44KW \V4V8Xg$\R\V4 RV RV RV RV 2 4h\V\ \ ,4'd7VPp W9d.W)&VFp W),PV 4K VFp WAV &K R#)zk Update current_placements and shard_order_dict in-place to reflect the effect of a single transform step. z?Invalid shard_order update. No entries left to pop for src_dim z. transform_info=z, current_placements=z, shard_order=z^Mismatch between expected and removed mesh dims during shard_order update. Expected to remove z, but removed N)rLr)rprxrJrrdimsetlen ValueErroraddpopappend) r|r}r~ src_placement dst_placement mesh_dimssrc_dim removed_dim_dst_dimrJs &&& r$"_update_shard_order_and_placementsrs$2#D#D M."9::"55 #,,. -!677##e A#,-2 Ui00@A**<)=>##3"46 OO,599; <9~,.y>". >""0!12&&8%9:/0 2 -!677##  *(*  %!H  % , ,X 6"'48$r&cdV^8dQhR\R\\R3,R\R,/#rrwr.r!Nry)r#s"r$r%r%s1 !&sCx$r&craVP4pVPoSfR#\;QJd.V3RlV4FNK 5M!V3RlV44pVPV4p\ VP 4.4pVP P4FpVPV8XgKVu# R#)z Query for an explicitly created flattened mesh using layout comparison. Searches root_mesh._flatten_mapping for a mesh whose layout matches the expected flattened layout for the given dims. Pure Python layout math. Nc36<"TFpSV,xK R#5iNrf).0imesh_dim_namess& r$ 5_get_flattened_mesh_by_layout_impl..'s;AnQ''s) _get_root_meshrre_get_slice_mesh_layoutr collapse_flatten_mappingvalues_layout)rwr root_mesh dim_names sliced_layoutexpected_layoutflattened_meshrs&& @r$"_get_flattened_mesh_by_layout_implrs##%I((N;;;;;I// :M!=#9#9#;"<=O$44;;=  ! !_ 4! !> r&cdV^8dQhR\R\\R3,R\R,/#rry)r#s"r$r%r%6s1?? ?!&sCx?$?r&c^RIp\4'dkVPPP'dE\ W4fR#^RIHpVPPPV\V44#\ W4#)a* Query for an explicitly created flattened mesh using layout comparison. When tracing with compile_on_one_rank, delegates to a custom op so the flattened mesh appears as a call_function node derived from mesh (a graph input) rather than as a get_attr constant holding an unpicklable ProcessGroup. N) device_mesh) torch.compiler.configr compilerconfigcompile_on_one_rankrtorch.distributed._opsrops_get_flattened_submeshr)rwrr*rs&& r$_get_flattened_mesh_by_layoutr6sd!U^^22FFF .d > F;yy$$;;D$y/RR -d >>r&cV^8dQhR\R\\R3,R\\R3,R\\R3,R\R\R\R R /#) rrr.src_placementsdst_placementsnum_ops comm_typereasonr!N)rrerdrrT)r#s"r$r%r%Ssr999S#X9)S.)9)S.) 9  9  9 9 9r&c\V4W3pV\9dR#\PV4VPpVe1VU u.uF qV ,NK p p RP RV 44p MRRP RV44 RV 2p Rp VR8XdR V R 2p M&VR 8XdRp MVR 8XdR V R2p M\ RV 24h\ PWW4W]4R#uup i)a Warn once per (mesh, dims, reason) about inability to flatten operations. Args: device_mesh: The device mesh being used mesh_dims: Tuple of mesh dimensions that could not be flattened src_placements: Source placements for the redistribution dst_placements: Target placements for the redistribution num_ops: Number of sequential operations that will be performed comm_type: Type of collective operation (e.g., "reduce_scatter") reason: Either "no_flattened_mesh" or "uneven_tensor_shape" Nz, c3."TF pRV R2xK R#5i)"Nrf)rnames& r$r:_warn_flatten_optimization_not_possible..ps?YTqa[Yszdims c38"TFp\V4xK R#5irrS)rds& r$rrrs$?YSVVYz of a%While redistributing from %s to %s, %d sequential %s operations will be performed. This is suboptimal: multiple collective operations have higher latency (separate kernel launches and synchronization points) and may give inconsistent results between ranks due to different reduction orders. %sno_flattened_meshz&To optimize, flatten mesh dimensions [z0] so DTensor can use a single operation instead.uneven_tensor_shapenon_ascending_mesh_dimsz0it is not possible to merge non-ascending order z operations.zUnexpected reason: )z Unfortunately, because the tensor dimension is not evenly divisible by the product of the mesh dim sizes that would need to be flattened for the optimization to work, it can not be optimized.)hash_warned_flatten_issuesrrjoinrKloggerwarning)rrrrrrr cache_keyrrrdims_strcommon_warning reason_msgs&&&&&&& r$'_warn_flatten_optimization_not_possiblerSs *k"I6I**y) //N!09: 1A&& :99?Y??499$?Y$??@[MR `$$=hZGwx ( (  , ,>yk V 26(;<< NN7;sCc V^8dQhR\\,R\R\\R3,R\\R3,R\\\ ,,/#)rtransform_infosrr.rr!)rrHrrerrp)r#s"r$r%r%scpp.)pp)S.)p)S.) p  .2 23 pr&c laaa\V4^8dV#\'dV#\0R m4oRV3RllpRRloRVV3Rllp.p^pV\V48Ed@W,pVP4p V!V 4'gVP V4V^, pKSVP p V.p V^,p V \V48ddV!W ,P44'dBS!W ,P V 4'd#V P W ,4V ^, p KsV!V 4wrV eVP V 4MMVP V 4VR 9d6\\RV 444p\SVVV\V 4V V4T pEKP\PRW4V#) at Optimize transform infos by merging consecutive same-type collectives into a single flattened operation when a matching flattened DeviceMesh exists. Merging requirements: - Operations must be consecutive in the transform list (no reordering). Notably, redistributing from P, P, P -> R, S, R is not optimized here and cannot be optimized due to optimization needing to fuse non-contiguous reductions, leaving this pattern vulnerable to numerics issues and suboptimal perf - Operations must have the same comm type (e.g., all allgather or all reduce_scatter) - Operations must have identical src_dst_placements (e.g., can't merge Partial->Shard(0) with Partial->Shard(1)) - A flattened mesh covering the relevant dimensions must exist - For reduce_scatter, tensor dim must be evenly divisible by flattened mesh size For nested sharding, the merged operation uses the logical_shape from the outermost mesh dimension (smallest mesh_dim index) which represents the global tensor shape needed for correct padding/unpadding. TODO: - all_to_all operations are excluded from merging, but it may be possible to merge them in some cases. c>V^8dQhR\R,R\/#)rkeyNr!)rTr9)r#s"r$r%/_optimize_transform_infos..__annotate__s++#*++r&c<VS9#)z:Check if a comm type key represents a mergeable operation.rf)rMERGEABLE_COMM_TYPESs&r$ is_mergeable/_optimize_transform_infos..is_mergeables***r&cV^8dQhR\\\3,R\\\3,R\/#)rp1p2r!)rerr9)r#s"r$r%rs8  )Y& ' -29i3G-H  r&c"W8XdR#Vwr#VwrEW58wdR#VP4'dVP4'gR#\\V4p\\V4pRR0pVPV9;'dVPV9#)z Check if two src_dst_placements can be merged. Allows merging of Partial("sum") and Partial("avg") since they can be combined: perform sum reduction, then scale by avg mesh dims afterward. TFsumavg)r\rr reduce_op) rrsrc1dst1src2dst2partial1partial2mergeable_reduce_opss && r$are_placements_mergeable;_optimize_transform_infos..are_placements_mergeables 8   <!!doo&7&7&& %u~   "6 6 ; ;""&:: r&cV^8dQhR\\,R\\R,\R,3,/#)rinfosr!N)rrHrerprT)r#s"r$r%rs9i i N#i &-sTz9 :i r&c <aa\V4^8dR#V^,PoV^,P4p\;QJd VV3RlV4F 'dK RM RM!VV3RlV44'g \ R4h\ ;QJd.RV4FNK 5M !RV44p\ \ V44pVR8Xd W#8wdR#MVR8XdW#RRR1,8wdR#\SV4pVfR#SwrVVR8Xd-\\V4Po\VV3R lR 7pMVR8Xdo\\V4Po\VV3R lR 7pVPS,p\P!V3R lV44p W,^8wdR#MVR 8Xd V^,pM\RV 24hRp Tp VP!4'd5\P!V3RlV44p V ^8dT p \#R4p W3p \%^V VPVVV R7R3#)a Try to create a flattened transform from 2+ same-type transforms. Returns (result, failure_reason) where: - result is the FlattenedTransformInfo if successful, None otherwise - failure_reason is None if successful, or one of: - "too_few_transforms": Less than 2 transforms provided - "no_flattened_mesh": No flattened mesh exists for the required dimensions - "uneven_tensor_shape": For reduce_scatter, tensor dim not evenly divisible Nc3J<"TFpS!VPS4xK R#5ir)rL)rinforfirst_placementss& r$rJ_optimize_transform_infos..try_create_flattened..s)  %T%<%<>N O Os #FTz5All transforms must have mergeable src_dst_placementsc38"TFqPxK R#5irrJ)rrs& r$rrs:ED--ErrYrZc*<VPS,#rrcx affected_dims&r$I_optimize_transform_infos..try_create_flattened..aool6Sr&)rc*<VPS,#rrrs&r$rrrr&c3Z<"TF pSPVP4xK" R#5ir)sizerJrrrs& r$rrs&2.try_create_flattenedsK u:>- -!866!H++- s  sss    !G E:E:EE:E::   !23 ( (,66- , &TrT22666{DTU  !, ,$  $s+//L ,STN * *s+//L ,STN,::<HO(, 2>  II!E qy! $U^ '- $#4,::##3#     r&c38"TFqPxK R#5irr)rgs& r$r,_optimize_transform_infos..os(CUUrz5_optimize_transform_infos original: %s, optimized: %s>rZrXrY)rrr) rrB frozensetr`rrLextendrerrrdebug)rrrrrr resultrr current_keyr}groupj flattenedfailure_reasonrrrs&f&& @@r$_optimize_transform_infosrs: ?a33%%ST++ @i i X>@F A c/" "!))+ K(( MM$  FA  "44'+f E O$ $_/>>@AA("557I LL+ , FA%9$?!  MM) $ MM% " "&(CU(C"CD 7""J"  LL? Mr&c4V^8dQhR\R\RR/#)rr dtensor_metar!DTensorRedistributePlannerrr)r#s"r$r%r%s&%%%%"%r&c\4'd \W4#\P!V4V3pV\9d\W4pV\V&\V,#)a Factory function to get or create a DTensorRedistributePlanner instance. This function provides transparent caching of planner instances based on device mesh and dtensor meta. Multiple calls with the same parameters will return the same cached instance for better performance. Args: device_mesh: The device mesh for the planner dtensor_meta: TensorMeta of the DTensor to redistribute Returns: A DTensorRedistributePlanner instance (potentially cached) )r rweakrefref_planner_cache)rrrplanners&& r$get_redistribute_plannerr$sR)+DD[)<8I&,[G$+y! ) $$r&cV^8dQhRR/#rRrf)r#s"r$r%r%s$r&c.\P4R#)z8Clear the cache of DTensorRedistributePlanner instances.N)r"clearrfr&r$ clear_redistribute_planner_cacher(sr&cTa]tRtRtoRt]P !RRR7!RR44tRt] V3RlR l4t ] V3R lR l4t ] R V3R lRll4t V3RlRlt V3RlRltV3RlRltV3RlRltV3RlRltV3RlRltV3RlRltV3RlRltRtVtR #)!riac This class is used to plan the collective calls to transform the local shard of the DTensor from its current spec to the target spec. Suppose there are N tensor dimensions and M mesh dimensions, the total possible state size will be (N+2)*M*M!. Note: Use get_redistribute_planner() factory function instead of direct instantiation for automatic caching. TrDca]tRtRto]P !RRRRR7tRtRtRt V3RlR lt V3R lR lt V3R lR lt V3Rlt RtVtR#)$DTensorRedistributePlanner.DistStateiNF)defaultinitreprcomparecX\P!VPVP4#r)rformat_shard_order_str placementstensor_dim_to_mesh_dimrMs&r$__str__,DTensorRedistributePlanner.DistState.__str__s%55++ r&c"VP4#r)r4rMs&r$__repr__-DTensorRedistributePlanner.DistState.__repr__s<<> !r&cP\PVRVP44R#)_hashN)object __setattr__ _compute_hashrMs&r$rO2DTensorRedistributePlanner.DistState.__post_init__s"   ""$ r&c <V^8dQhRS[/#rr!rd)r#rUs"r$r%1DTensorRedistributePlanner.DistState.__annotate__s R Rc Rr&cVVPe VP#VP4#r)r:r=rMs&r$__hash__-DTensorRedistributePlanner.DistState.__hash__s#!%!74:: QT=O=O=Q Qr&c <V^8dQhRS[/#r@rA)r#rUs"r$r%rBs  3 r&cD\VPVP34#r)rr2r3rMs&r$r=2DTensorRedistributePlanner.DistState._compute_hashs$OO// r&c&<V^8dQhRS[RS[/#)rotherr!)r;r9)r#rUs"r$r%rBs   4 r&c\V\P4'gR#VPVP8wdR#VPVP 3VPVP 38H#)F)r)r DistStater:r2r3)rNrJs&&r$__eq__+DTensorRedistributePlanner.DistState.__eq__sde%?%I%IJJzzU[[(++  ,, r&c`<V^8dQh/S[S[R3,;R&S[;R&S[R,;R&#)r.r2r3Nr:)rerrrd)r#rUs"r$r%rBs9)S.))!+*Tz r&rf)rgrhrirj dataclassesfieldr:r4r7rOrDr=rMrkrlrmrns@r$rLr+sW(--u5%    "  R R    Ir&rLc a\V\\,4'd3\;QJd.V3RlV4FNK 5#!V3RlV44#V#)z.s.s- /  DOseEl C C/s 11)reritems)rs&r$_dict_to_ShardOrder.DTensorRedistributePlanner._dict_to_ShardOrdersLu $QWWY/ u u $QWWY/   r&cL<V^8dQhRS[RS[S[S[S[,3,/#rY)rrrdr)r#rUs"r$r%rZs($$z$d3S >.B$r&c x\\4pVF$p\VP4WP&K& V#)z1Convert ShardOrder to dict with tensor dim as key)rrrr])rtensor_mesh_dim_dictentrys& r$_ShardOrder_to_dict.DTensorRedistributePlanner._ShardOrder_to_dicts7 +40E59%//5J !1!1 2##r&Nc p<V^8dQhRS[RS[S[,RS[S[R3,RS[R,RS[RS[/#) rrwrr.src_shard_orderN use_strided_shard_as_shard_orderr!)rrrHrerrr9rT)r#rUs"r$r%rZs`G$G$G$!.1G$Y^,G$$d* G$ +/ G$ G$r&c h\V4VP8wd%\R\V4 RVP 24hV'd\P!W RR7wr#Vf\P !V4p\ V4p\PV4p\P\V4V4pV.p.p VFsp \V \4p \WV4\P\V4\PV44p VPV 4V PV 4Ku \!V^,4.p \#V 4FIwrV 'dRMRpV PV4V P\!W^,,44KK RP%V 4#)aL Generate a string representation of the sequence of state transitions (placements and shard orders) as described by the given transform_info. Args: mesh: The DeviceMesh used for the redistribution. transform_infos: A sequence of _TransformInfo objects describing each transformation step. src_placement: The initial tuple of Placement objects. src_shard_order: (Optional) The initial ShardOrder representing the mapping of tensor dimensions to mesh dimensions. If None, the default shard order is computed from src_placement and mesh. use_strided_shard_as_shard_order: If True, normalize _StridedShard placements into regular Shard placements with an explicit shard_order before stringifying. Returns: A string showing the sequence of DistState transitions, separated by '->'. zdimensions mismatch z vs Trkz-->z->)rndimrKr&_normalize_placements_into_shard_ordercompute_default_shard_orderrrrgrLrer)rprrarrT enumerater)rwrrrjrk cur_placementr~ cur_state state_listis_flattened_listr| is_flattened new_state trace_partsr separators&&&&& r$stringify_transform_infos4DTensorRedistributePlanner.stringify_transform_infoss6 }  * &s='9&:$tyykJ  ,BB!$ +M  ")EEmTO]+ 5II  /88 - /   )+-N%n6MNL ./? 3<<m$*>>?OPI   i (  $ $\ 2.:a=)* ():;OA!-4I   y )   s:!e#45 6 <ww{##r&c*<V^8dQhRS[RS[RR/#)rrrr!Nr)r#rUs"r$r%rZDs)$$$!$  $r&c WnVP4'g\hVf\hW n\ VP 4Vn\4Vn\4Vn VP4R#)z Initialize DTensorRedistributePlanner. Args: device_mesh: The device mesh for this planner dtensor_meta: TensorMeta of the DTensor to redistribute N) r_is_current_rank_part_of_meshrKrrshapetensor_dimensionr"strided_shard_placements_in_targetpartial_reduce_ops_in_targetsetup_cost_callbacks)rNrrs&&&r$__init__#DTensorRedistributePlanner.__init__Dsd'88::   ( #L$6$6 7FIe/69e) !!#r&c<V^8dQhRR/#rRrf)r#rUs"r$r%rZ[s++ +r&c 6aaRV3RlloV3RlpVSnR#)z Set up the cost function for different collective operations. Uses communication time estimation based on actual tensor sizes and mesh topology for accurate cost modeling. cDV^8dQhR\PR\/#)rstater!)rrLr)r#s"r$r%EDTensorRedistributePlanner.setup_cost_callbacks..__annotate__ds"  -77   r&ct<\SPVPSPVPRR7#)F)rwr2 tensor_meta shard_orderrk)rrr2rr3)rrNs&r$ state_to_specFDTensorRedistributePlanner.setup_cost_callbacks..state_to_specds9%% ++ --!8816  r&c4<\S!V4S!V44#rr ) src_state dst_staters&&r$ cost_functionFDTensorRedistributePlanner.setup_cost_callbacks..cost_functionos-i(- *B r&N)r)rNrrsf @r$r/DTensorRedistributePlanner.setup_cost_callbacks[s    +r&cT<V^8dQhRS[S[R3,RS[RS[RS[3,/#)rr2.tensor_mesh_dim_tupler!r+)rerrrr")r#rUs"r$r%rZvsBdd)S.)d *d 4e; < dr&c  /p\PV4pVPVPV4V4pVEFpVPpWG,R,p\ W,\ 4'gK=\VP4Fp Wy8XdK WG,P4p WI,PV 4\V4p \ V 4W&VPVPV 4\PV44p VPVV 4W<&WG,PV 4WI,P4K EK VFpVPpWG,R,p\ W,\ 4'gK<WG,P4p \V4p \4W&VPVPV 4\PV44p WG,PV 4VPVV 4W<&K \V4Fiwr\ V \ 4'gK\V4p \4W&VPVPV 4V4p VPVV 4W<&Kk \V4Fwr\ V \4'gK\VP4Fp \V4p \ V 4W&WI,PV4VPVPV 4\PV44p VPVV 4W<&WI,P4K K \V4Fwr\ V \ 4'gK\VP4Fp \V4p \ V 4W&WI,PV4VPVPV 4\PV44p VPVV 4W<&WI,P4K K \V4Fwr\ V \4'gKVP"Fp\V4p \!V4W&V Uu0uF'p\ V\ 4'gKVP$kK) pp\'V4^8d VRR08wdKiVPVPV 4V4p VPVV 4W<&K K VFpVPpWG,R,p\ W,\(4'gK<WG,P4p \V4p \4W&VPVPV 4\PV44p WG,PV 4VPVV 4W<&K VP*'gV#\V4Fwr\ V \4'gKVP*FpVP,p \V4p VW&WI,PV4VPVPV 4\PV44p VPVV 4W<&WI,P4K K V#uupi)rrr)rrgrLrTr]r)rrangerrrrrarrrrrrrrrrr)rNr2rall_next_staterecur_dist_staterfsrc_tensor_dim src_mesh_dimdst_tensor_dim move_mesh_dimnew_placements dist_state placementrJrppartial_reduce_opsstrided_shard_objs&&& r$get_next_state)DTensorRedistributePlanner.get_next_statevs'dMO9MM !  NN: & ! +E"--N/?CLj6>>"'(=(=">!3!5 D H H J $4;;MJ!%j!105n0E-!^^NN>2.BB, .2-?-?".* %4;;MJ$488:+#? +B+E"--N/?CLj6>>0@DDFM!*-N,5KN )~.*>>?STJ ! 0 7 7 F)-););*N &+*(1'< #Li11!*-N+4;N (~.0EJ*.););*N &(=$-Z#8 Hi33"'(=(=">!%j!1+0+@($4;;HE!^^NN>2.BB, .2-?-?".*%488:#?$9,$-Z#8 Hi11"'(=(=">!%j!1+0+@($4;;HE!^^NN>2.BB, .2-?-?".*%488:#?$90$-Z#8 Hi33!>> !%j!1+29+=(*8&)7A:a;QKAKK#&)*Q.3E%QV3W!^^NN>24I .2-?-?".*?$9:+E"--N/?CLj6 FF0@DDFM!*-N,5KN )~.*>>?STJ ! 0 7 7 F)-););*N &+&666! !$-Z#8 Hi33%)%L%L!!2!6!6!%j!1+<($4;;HE!^^NN>2.BB, .2-?-?".*%488:!&M$9*Q&s [;[cB<V^8dQhRS[RS[RS[S[S[3,/#)r current_staterr!)rLrr")r#rUs"r$r%rZs/&3< i r&c H/pVPVP4pVPFpVPpWF,R,pVPV,pVPV,p \ V\ 4'd%\ V 4\JdV PR8XgKWF,P4\VP4p WV&VP\V 4VPV44p WF,PV4RW;&K V#)aGenerate explicit-only Shard -> Partial("sum") target steps. Zero weight means no communication; allocation and copy costs are omitted. This edge is generated only when Partial("sum") is the requested target, so path search cannot use it as an intermediate step to another layout. rr)rgr3r]r2r)rtyperrrrrLrerar) rNrr next_statesrrfr]rJcurrenttargetr2 next_states &&& r$#_get_shard_to_partial_target_states>DTensorRedistributePlanner._get_shard_to_partial_target_statessJL ..}/S/ST "99E))J".r2H#..x8G))(3F7E**LG+$$-  # ' ' )m667J#)x j!4#;#;K#HJ  # * *8 4&)K #':(r&c:<V^8dQhRS[RS[RS[R,/#)rrrr!r+)rLr)r#rUs"r$r%rZs0@ @ "@ /8@ 4 5@ r&c  4^RIp^pRVRW.3.p\4pV'dVPV4wrxrW8XdV #W9dK-VPV 4VP V P V P 4p V PVPW44V P4FVwrW9gK \\V4V ,pW.,pV^, pVPV\V4VVV V34KX K\RV RV 24h)z Find the min cost path from src_state to dst_state using Dijkstra's algorithm. Args: src_state: The source state dst_state: The destination state Returns: A list of states representing the min cost path from src_state to dst_state NrzNo path found from src_state z to dst_state )heapqrheappoprrr2r3updaterr`rrheappushr5rK)rNrrrcounterpqvisitedrr rpathrrtransition_costnew_costnew_paths&&& r$find_min_cost_path-DTensorRedistributePlanner.find_min_cost_paths0  7CK 8 9 %.3mmB.? +Aq ) ' KK &--((44K   88R 0;/@/@/B+ ,#C@H#l2HqLGNN7A#$&$   0C+I;nYK P  r&c T<V^8dQhRRRS[RS[S[R3,RS[S[,/#)rrr+rJfull_tensor_shape.r!)rdrerr)r#rUs"r$r%rZs@!!9!!!c? ! k  !r&c \V4pVPEF2pVPpVPp\ V4^8:d\ hVFpW8XdK VP V,p \V \4'dPV PWF,VPPVR7VPPV44wrMs\V \4'dPV PWF,VPPVR7VPPV44wrM\RV 24hWV&K EK5 V#)rrzUnsupported placement type: )rr3r]rrrKr2r)rlocal_shard_size_and_offsetrr_sym_get_coordinaterr) rNrrJrnew_logical_shaperfr]rmdimrnew_sizers &&&& r$get_logical_shape,DTensorRedistributePlanner.get_logical_shapes4 044E/F55E))JI9~"$$!#%006 i//"+"G"G)5((--t-<((<.__annotate__+s/ 9 9 9 5C(*4 5 9r&cVP'd3\P!VPVPRR7wrW3#VP f\ RV 24hVPVP 3#)TrmzMissing shard_order field in )rkrrpr2rwrr)rrrs& r$_try_normalize_spec\DTensorRedistributePlanner.generate_graph_based_transform_infos.._try_normalize_spec+su444FF 9=,&22##+$'DTF%KLL(8(888r&z@Multiple mesh_dims are different between cur_state and nxt_staterJrLrcr)r)rrr itertoolschainrrrrLrpairwiser2rrziprKrrrH)rNrrrrrrjrdst_shard_orderrrrr state_pathrt nxt_stateupdate_mesh_dimrJrs nxt_placementrcs&&&& r$$generate_graph_based_transform_infos?DTensorRedistributePlanner.generate_graph_based_transform_infos"s 9"+>h*G'*=h*G' (I)]3377;;IF(#HI)W--1155i6I6IJINN>C NN>C 02,,YB $-$6$6z$B I##y';';;"$@I ,,i.B.BCA(>%x1B) (..*)84A=3Q.;A %C0r&c<<V^8dQhRS[RS[RS[S[,/#)rrrr!)rrrH)r#rUs"r$r%rZks3~~~~ n  ~r&c  \VP4pV.p.pVPP^8XdmVP^,VP^,8wdBVP \ ^VP^,VP^,3VR74V#\VP4FwrgWF,p\V\4'dW`PP^, 8dVPPVR7p VPWP,V VPPV44wr\V4p WVP&VP V 4KKVP V4K \VP4p \VP4pVP^8Ed\\!\#V 444FpW,pW,p\V\4'dVPp..pp\\%W44FcwpwppWo8dMVVP'V4'dVP V4VP'V4'gKRVP V4Ke VV8wd \)4pVV8wgKVP \ VVV3WO,R74VW&K \\%W44F;wpwppVV8wgKVP \ VVV3WO,R74VW&K= V#)ap Generate the transform infos from the source placements to the target placements. To transform from source to target placement it might have multiple steps, i.e. it might decompose Si -> Sj into Si -> R -> Sj. This would detect if there're mis-aligned/nested shardings between src/dst placements. E.g. Suppose the redistribution to perform is (Shard(0), Shard(0)) -> (Replicate(), Shard(0)), in this case Shard(0) -> Shard(0) for mesh dimension 1 actually needs resharding, because in the former is a nested-sharding of a tensor already sharded dimension 0, whereas the latter is the first sharding on tensor dimension 0. rr)rrrror2rrHrrr)rr_local_shard_size_and_offsetrr num_shardsreversedrrris_shardr)rNrrinitial_logical_shapemesh_dims_to_logical_shaperrr^current_logical_shape mesh_dim_sizelocal_shard_sizerrr}target_placementsrJrr shard_dimcurrent_mesh_shardingtarget_mesh_shardingsrs&&& r$generate_greedy_transform_infos:DTensorRedistributePlanner.generate_greedy_transform_infosks$!%X^^ 4&;%<"02    A %""1%)<)$A !#u%%'',,q00$($4$4$9$91$9$EM*-*J*J-gg6%((<.s%$7E ++E227s#%FTc3B"TFp\V\4xK R#5ir)r)r)rrs& r$rrs!=A 1m$$=s) rrranyr2r;rrKr$rrr r) rrrrrjrhas_non_default_orderhas_strided_sharddrprs &&& r$_gen_transform_infos_non_cachedrs{ &&K**O**O!$$%7$$%7$!=8&&=)<)<==8&&=)<)<= 1$(! , 8$G! " *$)!# " C! V!FFO ==hQ  . V!AA(UO   VsEE10E1c lV^8dQhR\R\R\R,R\\,/#rr)r#s"r$r%r%(s9     $d{  . r&c\VVV4#r)r)rrrs&&&r$_gen_transform_infosr 's +! r&async_opr is_explicitcV^8dQhR\PR\R\R\R\R,R\R\P/#) r local_tensor current_spec target_specr rNr r!)r*Tensorrr9)r#s"r$r%r%4sc,,   $d{  \\r&c  VPVP8wd \R4hVPf \R4h\ VP 4\ VP 4TpVPpVP 4'gV#\4'd\VVV4pM \VVV4p\VVVP VP 4p \4p V e`V PVVP VP \PVV VP VPVP4VR7M\ P"!4p T ;_uu_4V EFp \%V \&4'dV Pp MTp V P(pV P*wpp\%V\,4'g\%V\,4'dWJd \/R4hV P1VR7pVV8XdTpKV^8XdTpKVP34'dVP54'd[\7\8V4pVP;W V4p\%V \&4'd!V P<eWlP<, pEMVP?4'd/\7\@V4pVPCW WPD4pEMK\%V\,4'dVPCWWPD4pEM\GRV RV R 24hVP?4'Ed\7\@V4pVP54'd\\7\8V4pVPIW VV4p\%V \&4'd!V P<eWlP<, pEMmVP34'd$VPKW WPMV44pEM4VP?4'dW\7\@V4pVPNVPN8wd*VPQVV VV PDVPN4pEM\%V\,4'dBVPCWWPD4pVPKVV VV PMV44pEMq\R V R V 24hVP54'dVP34'd%\7\8V4pVPSW V4pEM\UV4'dm\%V\@4'd%\WV4\8JgVPXR 8wd\GRV RV R 24hVP[VV VV PD4pEMVV8wd\/R V RV R24hTpEMv\%V\,4'Ed`VP54'dG\7\8V4pVP;W V4pVP]VW~VPMV44pEMVP34'd#VP]WWPMV44pMVP?4'dP\7\@V4pVPCW WPD4pVP]VW~VPMV44pMf\%V\,4'd@VPCWWPD4pVP]VW~VPMV44pM\R V R V 24hV'g1\%V\^P`4'dVPc4pTpEK RRR4V# +'giT#;i)z This redistribute the local tensor (torch.Tensor) from the current DTensorSpec to the target DTensorSpec, which involves the necessary collective calls to transform the local shard of the DTensor from its current spec to the target spec. z)Cross device mesh comm not supported yet!NzUuse_strided_shard_as_shard_order should be initialized in DTensorSpec.__post_init__())r z:_StridedShard redistribute assumes no flattened transformsrzredistribute from z to z not supported yetzUnexpected placement z& for redistribute to target placement rz&Redistribution from one partial type (z) to another (z) is unsupported.)2rwrrkrrr2rr rr rrrecord_redistribute_callsrr{r contextlib nullcontextr)rprJrLrrKrr]r\rr _reduce_valuertrr_to_replicate_tensorrc RuntimeError_reduce_shard_value_replicate_to_shardrr_to_new_shard_dim_partition_valuerrr_to_partial_tensor_replicate_to_strided_shardfuncolAsyncCollectiveTensorwait)r rrr rr new_local_tensorrroptimized_transform_infos debug_moderedistribute_contextr| mesh_to_userrr num_chunks partial_speccurrent_placementtarget_placement shard_spec replicateds&&&$$$ r$redistribute_local_tensorr,4s/ K,,,!"MNN44< c  ","9"9:!+"8"89###K  4 4 6 69   %  /   % !: !'(J  ! ,,   # #  " " & @ @)''((==  $ -  # # %$  7N.*ABB,11 ) ''A,??OGV '=11Z 5V5V1(T%))1)5J& #/ Q$0 ""$$%%''#'#9L'3'A'A$1($ #>3JKK*44@+;>V>V+V(%%''(,UG(<%'8'M'M$16R6R($ 77'.'C'C$16R6R($',WIT&AST""#'v#6 %%''#'#9L'3'G'G$16F($ #>3JKK*44@+;>V>V+V())++'7'K'K$16U6UVW6X($%%''!%eW!5J!~~)9)=)==+5+G+G('*88,00 ,( 77!(!=!=$16R6R"J(8'K'K"##77: ($%/y8^_e^fg""$$''))#'#8L'3'D'D$1($$G,,&w66">!!$4#8#8#: +L[8 ^ _  ^ s\)C]2*]2A/]20B ]2;B]28]2 C]2]25C*]2 A]2<7]24B<]217]22 ^ c tV^8dQhRRR\R\PR\PR\/#)r grad_outputdtensor.DTensor previous_spec out_dtypeop_dtyper )rr*dtyper9)r#s"r$r%r%6sI[["[[{{ [ kk [  [r&c W0PP8wdVPPVR7p\VPP VPP \VPVP4VR7VPPR7p\VP VP VPVPR7pMVPpVPp.p\VP VP 4Fnwr\V4'gVP4'd2V P4'dVP!\#44K]VP!V 4Kp \VP \%V4VPVPR7p\'VVVVR7p V PV8wdV PV4p \VP \%V4\VPVP4V PR7VPR7p W3#)ao Common function for redistributing a distributed tensor during backward and twice-backward backpropagation steps. Args: grad_output: The output gradient tensor. previous_spec: DTensorSpec prior to redistribution. out_dtype: dtype to cast the returned gradient to. Pinned by autograd to match the dtype of the input of the node above this one. op_dtype: dtype to run the backward collective at. async_op: whether to perform the DTensor redistribute operation asynchronously or not. Default: False Returns: A :class:`torch.Tensor` object. A :class:`DTensorSpec` object. r3rstrider3rwr2rrk)r2rrk)r rrk) _local_tensorr3tor_specrr2rrr7rkrrrr]r\rrrer,) r.r0r1r2r r rnormalized_placementsrroutputrs &&$$$ r$_redistribute_backwardr?6s2,,222"0033(3C """.."((33"!''"))+ .9->->-_-_  $**$//$00-:-[-[  #00 "(( .0|66 8P8PQ 7 # #w';';'='=6CTCTCVCV ! ( ( 5 ! ( ( 0 R  !!./!--)6)W)W M' F||y 9% !! #$##%%',, *7)W)W D <r&c,a]tRtRtoV3RltRtVtR#)_BackwardDtypeConfigicZ<V^8dQh/S[P;R&S[P;R&#)rr2r1)r*r3)r#rUs"r$r%!_BackwardDtypeConfig.__annotate__s#kk{{r&rfNrgrhrirjrkrlrmrns@r$rArAr&rAc,a]tRtRtoV3RltRtVtR#) _DtypeConfigicf<V^8dQh/S[P;R&S[P;R&S[;R&#)rr2r1backward_options)r*r3rA)r#rUs"r$r%_DtypeConfig.__annotate__s0kk{{+*r&rfNrDrns@r$rGrGrEr&rGcXa]tRtRto]V3RlRl4t]V3RlRl4tRtVtR#) Redistributeic J<V^8dQhRRRS[RS[S[R3,RS[RS[/#)rinputr/rr2.r  dtype_config)rrerr9rG)r#rUs"r$r%Redistribute.__annotate__sOH H !H  H )S.) H  H #H r&c nW@nVR,pVR,VnVR,VnVR,pVR,pWqPP8wdyVPP VR7p \ VVPP\VPVP4VR7VPPR7p MVPp VPp Wn V PV8wd)\ W#V PR7p \V V V VRR7p MT p T p V PV8wdYV P V4p \ VV P\VPVP4VR7V PR 7p \ P"!V V VP$R 7#) rIr2r1r5r6r8)rT)r r r9 requires_grad)r  bwd_op_dtype bwd_out_dtyper:r3r;rr<r2rrr7rkrrr,dtensorDTensorrS) ctxrNrr2r rObwdr2r1r rrr>s &&&&&& r$forwardRedistribute.forwards -.z? , + - **00 0 ..111AL&  ;;11&++ <<>" 271]1] L!..L ;;L'  " "j 0%\5M5MK/! F"F&K <<9 $YYy)F%&&&++ <<># 2=1]1] K  --   r&c<V^8dQhRR/#)rr.r/rf)r#rUs"r$r%rPs  #4 r&c\PVVPVPVPVP 4pVRRRR3#r)NestedRedistributeapplyrr rTrU)rXr.output_dtensors&& r$backwardRedistribute.backwardsR+11     LL                r&rfN) rgrhrirjrrZrarlrmrns@r$rLrLs.H H T  r&rLc\a]tRtRtoRt]V3RlRl4t]V3RlRl4tRtVt R#) r^ia This class is used to make the redistribution of a DTensor twice-differentiable. This is called during the `Redistribute.forward`. Therefore, `NestedRedistribute.forward` is called during the first backward pass, and `NestedRedistribute.backward` is called during the second backward pass. Note: `NestedRedistribute.backward` is not differentiable, and therefore triple backward is not yet supported. c ^<V^8dQhRRRS[RS[RS[PRS[P/#)rr.r/r0r r2r1)rr9r*r3)r#rUs"r$r%NestedRedistribute.__annotate__ sG   '  #    ++  ;;  r&cW0nVPPVnW@n\ VVVVVR7wrgWpn\P!VVVPR7#))r1r2r rR) r r:r3original_dtyper2r?rrVrWrS)rXr.r0r r2r1r>rs&&&&&& r$rZNestedRedistribute.forwardsg (66<< -       %33   r&c<V^8dQhRR/#)r grad2_outputr/rf)r#rUs"r$r%re,s  $5 r&cVPp\PVVVPVPVP 4pVRRRRR3#r)rr^r_r r2rg)rXrjr0r`s&& r$raNestedRedistribute.backward+sZ(( ,11   LL LL              r&rfN) rgrhrirjrzrrZrarlrmrns@r$r^r^s3    D  r&r^cV^8dQh/^\9d\R,;R&^\9d \;R&^\9d8\\\\\R3,\ 3,,;R&^\9d=\ \\P\,\3,R3,;R&#)rNr;rB.rrr") __conditional_annotations__r9rrerdrTrr  ReferenceTyperr)r#s"r$r%r%sZ87TD[7[f;:d:gZ FEE#uS#X";<=E[ @ '   +Z 78 "Ar&)Tr)YrnrrPrloggingrr  collectionsrcollections.abcr functoolsrtypingrrrr*)torch.distributed._functional_collectives distributed_functional_collectivesrtorch.distributed.tensor._apitensor_apirVr torch.distributed._mesh_layoutr *torch.distributed.tensor._collective_utilsr &torch.distributed.tensor._dtensor_specr rrrrtorch.distributed.tensor._utilsr$torch.distributed.tensor.device_meshr(torch.distributed.tensor.placement_typesrrrrrr torch.typesrrtorch.utils._debug_modertorch.utils._typing_utilsr getLoggerrgrr;rBr5contextmanagerr=rCrrHrprrrrrrrr"r$r(rrr r,r?rArGautogradFunctionrLr^r%)rns@r$rs #$'' :://E6QJ;39.   8 $ 48#7 6;,:C( 9898x AA8 d$/  0 F d$/n04.5b>?4ADE9xpn %4 A A H8v     .2 D[  [|9 +9+ [ 5>>**[ |C 00C r&