+ j=Rt^RIt^RIHtHt^RIHt^RIHt^RI H t H t ^RI t ^RI HtHtHtHtHtHtHtHtHtHtHtHt]!RR7!R R 44t]!RR7!R R ]],44tR#) zj Definition of CuTe inspired Layouts for DeviceMesh internal bookkeeping and functions to manipulate them N)IteratorSequence) dataclass)product)NoReturnoverload) as_tuplecoalesce complement compositionflattenIntTupleis_intis_tupleLayout make_layoutmatch_structuresuffix_productT)frozenca]tRt^toRtRV3RlRlltV3RlRltV3RlRltV3R lR ltV3R lR lt V3R lRlt V3RlRlt V3RlRlt V3RlRlt V3RlRltV3RlRlt]V3RlRl4tV3RltRtVtR#) _FlatLayouta A canonical CuTe layout for a single dimension of a DeviceMesh Utility class for representing an integer layout by borrowing ideas from CuTe Layout Algebra. See https://github.com/NVIDIA/cutlass/blob/main/media/docs/cpp/cute/02_layout_algebra.md for more details. Each layout is represented as a list of sizes and strides. We use it as a way for mechanical bookkeeping of the integers such as ranks in a SPMD mesh, and the transformation on top of it. Lots of methods of layout like coalesce, composition, complement, etc. are borrowed from pycute. https://github.com/NVIDIA/cutlass/blob/6dd13d42784ee5bfa232d2441e6b9a021c5c6290/python/pycute/layout.py#L137,L257 Note this is a CuTe-inspired layout, because CuTe uses co-lexicographic way in linearization while PyTorch is using lexicographic. So even though the CuTe documentation can still be referenced, the implementation will be different from that of PyCute's. This layout is _not_ itself subdivided into multiple dimensions. It might internally sometimes use multidimensional tuple to represent "irregular" layouts (e.g., flattening non-adjacent dims), but this should be considered an opaque implementation detail. This class guarantees that all equivalent layouts are encoded as the same normalized representation, and thus compare equal. This is achieved by flattening and coalescing compatible adjacent dimensions (which includes removing all dimensions of size 1). Nc8<V^8dQhRS[RS[R,RR/#)shapestrideNreturn)r )format __classdict__s"_/home/luca/screen-checking/.venv/lib/python3.14/site-packages/torch/distributed/_mesh_layout.py __annotate___FlatLayout.__annotate__=s&88h84848cN\V4'g)\V4'g\R\V4 24hVeTM \ V4p\V4'g)\V4'g\R\V4 24h\ W4'g\ RV RV R24h\\W44p\VP4p\VP4pVR8Xd VR 8XdR pR p\PVRV4\PVRV4R#) z"shape must be a tuple or int, got Nz#stride must be a tuple or int, got zsizes z and strides z don't matchrr))rr TypeErrortyperr ValueErrorr rr rrobject __setattr__)selfrrcoalesced_layout flat_shape flat_strides&&& r__init___FlatLayout.__init__=sve}}@e NO O!->%3HvA$v,PQ Qu--veWM&NO O#F5$9:-334 .556   +"5JK 4*54;7r!c <V^8dQhRS[/#rr)r)rrs"rrr Ts   r!c\R4h)zWYou should never need to know the length of the internal representation of a FlatLayout RuntimeErrorr,s&r__len___FlatLayout.__len__Ts e  r!c&<V^8dQhRS[RS[/#rir)intr)rrs"rrr Ys  S X r!c\R4h)zOYou should never need to index into the internal representation of a FlatLayoutr5r,r<s&&r __getitem___FlatLayout.__getitem__Ys ]  r!c <V^8dQhRS[/#r3r)rrs"rrr ^s//6/r!c~VP'g \^^4#\VPVP4#r#)rrrr7s&r to_pycute_FlatLayout.to_pycute^s,zzz!Q< djj$++..r!c <V^8dQhRS[/#r3r=)rrs"rrr cs%%s%r!cB\P!VP4#N)mathprodrr7s&rnumel_FlatLayout.numelcsyy$$r!c"<V^8dQhRRRR/#)rlayout _MeshLayoutrr&)rrs"rrr fs((-(M(r!c \VP4VP44p\\VP4\VP 44UUu.uFwr4\ W44NK ppp\V4#uuppi)u By-dimension composition allows one layout to "select from" or "filter through" another layout. Think of it as function composition: (self ∘ layout)(input) = self(layout(input)) between two layouts. This function is a wrapper of pycute's composition. Mental model about how to understand the composition logic: - The LEFT layout (self) defines the "output space" - what indices are possible - The RIGHT layout (layout parameter) acts as a "selector" - which specific indices to pick - The composition only generates indices that the left layout could originally produce, but the right layout determines which indices to be picked. - The stride of the composition layout will not be smaller than the stride of the right layout, because when picking the indices the composition will at least follow the right layout's stride to move forward. Example: self = (6,2):(2,1) # sizes=(6,2), strides=(2,1) layout = (3:2) # sizes=(3,), stride=(2,) self o layout = (3:2) Returns: Layout being composed. )r rEziprrrrrQ)r,rPresultrr result_axess&& rr _FlatLayout.compositionfst.T^^-v/?/?/AB"%Xfll%;Xfmm=T!U !U   &!U  ;''  sBc$<V^8dQhRS[RR/#)r world_sizerrrH)rrs"rrr s88S8]8r!c v\VP4V4p\VPVP4#)u Compute the "complement layout" relative to a given world_size. A complement layout fills in the "missing" factor so that: self repeat a layout of complement(self, world_size) will get a complete world_size. We use ⊗ to denote the repeat operation. Example: self = (4:1) # size=4, stride=1 world_size = 8 Then: complete needed factor = 8 / 4 = 2 complement(self, 8) = (2:1) Together they form: (4:1) ⊗ (2:1) = (4,2):(2,1) which has world_size = 4 * 2 = 8, as required. In distributed terms, complement() is often used to derive the "other" rank grouping when splitting processes into 2D meshes. For a visualized explanation, see https://x.com/ezyang/status/1962364978393981433/ )r rErrr)r,rXrTs&& rr _FlatLayout.complements,,DNN,j96<<77r!c0<V^8dQhRS[S[,/#r3)listr=)rrs"rrr s% % T#Y% r!c  \RVP4!Uu.uF(p\R\WP444NK* up#uupi)a This function computes the all ranks specified by the layout starting from zero. How it works: 1. we enumerate every possible coordinate (like a nested for-loop). If sizes = (2, 3), we get the following coordinates: (0,0), (0,1), (0,2), (1,0), (1,1), (1,2) 2. For each coordinate, we compute a linear rank index as: all_ranks_from_zero = sum(coord[i] * strides[i] for i in range(ndim)) Example A: sizes = (2, 3) # 2 rows, 3 cols strides = (3, 1) # row-major layout coords = (0,0) -> 0*3 + 0*1 = 0 (0,1) -> 0*3 + 1*1 = 1 (0,2) -> 0*3 + 2*1 = 2 (1,0) -> 1*3 + 0*1 = 3 (1,1) -> 1*3 + 1*1 = 4 (1,2) -> 1*3 + 2*1 = 5 result = [0, 1, 2, 3, 4, 5] Example B: sizes = (2, 3) strides = (1, 2) # non-standard / strided layout coords = (0,0) -> 0*1 + 0*2 = 0 (0,1) -> 0*1 + 1*2 = 2 (0,2) -> 0*1 + 2*2 = 4 (1,0) -> 1*1 + 0*2 = 1 (1,1) -> 1*1 + 1*2 = 3 (1,2) -> 1*1 + 2*2 = 5 result = [0, 2, 4, 1, 3, 5] c38"TFp\V4xK R#5irJ)range).0ss& r 2_FlatLayout.all_ranks_from_zero..s"@Z588Zc36"TFwrW,xK R#5irJr&)r`cras& rrbrcs:"9$!"9s)rrsumrSr)r,coords& rall_ranks_from_zero_FlatLayout.all_ranks_from_zerosMH!"@TZZ"@A A :#e[["9: :A   s.AcF<V^8dQhRS[RS[S[S[,,/#)rrXr)r=r\)rrs"rrr s#  s tDI r!c  VPV4P4UUu.uF)pVP4Uu.uF q2V,NK upNK+ upp#uupiuuppi)u Build global ranks specified by the layout via two-level ranks composition. The nested list forms the Cartesian product of all ranks for one layout and offset regarding filling up the world_size with the layout. The final global ranks are the addition of these two. The result is a list of lists: one sublist per layout. This rank list will be used to build the communicator underlying the layout and the given `world_size`. Example: world_size = 16 self.size = 4 self.stride = 1 ranks = [0, 1, 2, 3] offsets = [0, 4, 8, 12] result = [ [0+0, 0+1, 0+2, 0+3], # → [0, 1, 2, 3] [4+0, 4+1, 4+2, 4+3], # → [4, 5, 6, 7] [8+0, 8+1, 8+2, 8+3], # → [8, 9, 10,11] [12+0, 12+1, 12+2, 12+3], # → [12,13,14,15] ] )r ri)r,rXoffsetranks&& r global_ranks_FlatLayout.global_rankss\2//*5IIK K(,'?'?'A B'Atd]]'A BK  B sAA AAc <V^8dQhRS[/#r3bool)rrs"rrr sGGdGr!c\\\VPRR74VP8H#)Treverse)tuplesortedrr7s&r check_sorted_FlatLayout.check_sorteds!VDKK674;;FFr!c <V^8dQhRS[/#r3rr)rrs"rrr s# # $# r!c aa\VP4^8dR#\\\VPVP4RR7!woo\ ;QJd9VV3Rl\ \S4^, 44F 'dK R# R#!VV3Rl\ \S4^, 444#)u Check if the layout has any overlap between the ranks it generates. If there is overlap, we return False, otherwise True. The layout is supposed to be injective i.e, aside from indice 0, indices from each dim of the layout must be non-overlapping. Example 1 - Valid (no overlap): Layout: sizes=(2,3), strides=(6,1) - Dim 1: stride=1, span=3*1=3, covers indices [0,1,2] - Dim 0: stride=6, span=2*6=12, covers indices [0,6] → No overlap since 6 > 3 Example 2 - Invalid (overlap): Layout: sizes=(2,3), strides=(2,1) - Dim 1: stride=1, span=3*1=3, covers indices [0,1,2] - Dim 0: stride=2, span=2*2=4, covers indices [0,2] → Overlap! stride=2 < span=3, so indices [0,2] are duplicated Example 3 - Invalid (overlap): Layout: sizes=(4,2), strides=(1,1) - Dim 1: stride=1, span=4, covers indices [0,1,2,3] - Dim 0: stride=1, span=2, covers indices [0,1] → Overlap! stride is same for two dims, so indices [0,2] are duplicated Returns: bool: True if no overlap, False if overlap detected Truc3<"TF;pSV,SV^,,SV^,,,,^8HxK= R#5i)r$Nr&)r`r<rrs& rrb/_FlatLayout.check_orthogonal..s; + 1IAq1u5 6! ;+sAAF)lenrrSrxrallr_)r,rrs&@@rcheck_orthogonal_FlatLayout.check_orthogonals: tzz?Q VC TZZ$@$OP s 3v;?+ ss s s 3v;?+   r!cF<V^8dQhRS[S[S[S[3,,/#r3)rrwr=)rrs"rrr s!,,8E#s(O#<,r!c B\VPVP4#)z5Iterate over (size, stride) pairs for each dimension.)rSrrr7s&rsizes_and_strides_FlatLayout.sizes_and_stridess4::t{{++r!cZ<V^8dQh/S[S[R3,;R&S[S[R3,;R&#)r.rrrwr=)rrs"rrr s/< c?=> #s(O?r!r&rJ)__name__ __module__ __qualname____firstlineno____doc__r0r8r@rErMr r riroryrpropertyr__annotate_func____static_attributes____classdictcell__rs@rrrs>88.    // %%((<882% % N  8GG# # J,,[r!rcXa]tRtRtoRtV3RlRlt]R"V3RlRll4tV3RlR lt] V3R lR l4t ] V3R lR l4t V3RlRlt V3RlRlt V3RlRlt ] V3RlRl4tV3RlRltV3RlRltV3RlRltV3RlRltV3RlRltV3R ltR!tVtR#)#rQia A multi-dimensional structure consisting of a series of dimension-less layouts This class represents the layout of a full DeviceMesh, where the overall top-level ndim and "logical" shape are well defined, but each individual mesh axis is squashed and normalized into a canonical _FlatLayout. It only contains methods that need to make use of this multi-dimensional structure (i.e., which access the ndim or the top-level sizes). Everything else should go on _FlatLayout and accessed by first calling .collapse(). c4<V^8dQhRS[S[,RR/#)raxesrN)rr)rrs"rr_MeshLayout.__annotate__s 66Xk26t6r!cF\PVR\V44R#)rN)r*r+rw)r,rs&&rr0_MeshLayout.__init__s4t5r!Nc`<V^8dQhRS[S[R3,RS[S[R3,R,RR/#)rsizes.stridesNrrQr)rrs"rrr#s:  #s(O .3CHo.D   r!c6Vf\\V44p\V4\V48wd$\R\V4 R\V4 24h\;QJd.R\ W44FNK 5M!R\ W444pV!V4#)Nz1sizes and strides must have the same length, got z and c3B"TFwr\V3V34xK R#5irJr)r`rads& rrb1_MeshLayout.from_sizes_strides..,s"M9L[!t,,9Ls)r rrr)rwrS)clsrrrs&&& rfrom_sizes_strides_MeshLayout.from_sizes_strides"s ?nU34G u:W %CCJ.CsD)$^^--))rrrrr7s&rrE_MeshLayout.to_pycute@s3 tyy>Q !Q< D$))DEEr!c4<V^8dQhRS[S[R3,/#)rr.r)rrs"rrrFs99sCx9r!c\;QJd.RVP4FNK 5#!RVP44#)c3@"TFqP4xK R#5irJ)rMrs& rrb._MeshLayout.top_level_sizes..Gs8idZZ\\ir)rwrr7s&rtop_level_sizes_MeshLayout.top_level_sizesEs,u8dii8u8u8dii888r!c <V^8dQhRS[/#r3rH)rrs"rrrIs//s/r!cB\P!VP4#rJ)rKrLrr7s&rrM_MeshLayout.numelIsyy--..r!c <V^8dQhRS[/#r3rH)rrs"rrrLs)))r!c>VP4P4#rJ)rEcosizer7s&rr_MeshLayout.cosizeLs~~&&((r!c <V^8dQhRS[/#r3r)rrs"rrrOs , ,+ ,r!c \;QJd.RVP4FNK 5M!RVP44p\;QJd.RVP4FNK 5M!RVP44p\W4#)z Merge all axes into a single _FlatLayout. This is used to "forget" the multi-dimensional structure of this object and recover a "flat" (and coalesced) representation. c38"TFqPxK R#5irJ)rrs& rrb'_MeshLayout.collapse..Vs8idzzirdc38"TFqPxK R#5irJ)rrs& rrbrWs:  rd)rwrr)r,shapesrs& rcollapse_MeshLayout.collapseOs[8dii88dii88%: :%%: ::6++r!c.<V^8dQhRS[RS[RRRR/#)rstartendrPrQrrH)rrs"rrrZs)%%C%c%=%]%r!c p\VP4p\VP4WAV%\V4#)z Replace (out-of-place) the start:end slice with the given list of layouts Returns the concatenation of self[:start] + layout + self[end:]. )r\rrQ)r,rrrPnew_axess&&&& rsplice_MeshLayout.spliceZs/  ?"6;;/s8$$r!cN<V^8dQhRS[PRS[P/#)rrank_mapr)torchTensor)rrs"rrrds#/-/- /-/-r!c VP^8wd\hVP4'g\hVP4VP 48d\hVP 4pVP VP44pVPVPVP,VPVP,4P!R.VPO5!#)a Leverage layout as an index for mesh tensor that re-maps the indexes after layout transformation to actual device ranks. With this method, the cute layout serves as the backend of indices bookkeeping for the mesh tensor when it comes to flatten, unflatten and slicing operations. The actual mesh tensor still represents the actual device assignment and ranks. We need this function to specify device allocation and create backend for a mesh. Although any transform of mesh tensors can be treated as a view or subset of mesh tensor, we do need to use the actual view or sub-tensor for DeviceMesh and its backend creation. The shape of the `rank_map` must be 1D and contiguous. Examples: Case 1 - Consecutive ranks, full world: original_mesh_tensor = [[0,1],[2,3]] # 2x2 mesh, ranks 0-3 world_size = 4 layout = Layout(2:2) Return: [[0,2],[1,3]] Case 2 - Non-consecutive ranks: original_mesh_tensor = [[10,20],[30,40]] # custom rank assignment world_size = 4 layout = Layout(2:2) Return: [[[10,30],[20,40]]] Args: rank_map: The concrete mesh tensor with actual device ranks Returns: torch.Tensor: A tensor representing the actual device allocation from rank_map ) ndimAssertionError is_contiguousrMrrr as_stridedrrreshaper)r,r self_layoutcomplement_layouts&& rremap_to_tensor_MeshLayout.remap_to_tensordsD ==A  %%'' >> dkkm + mmo '228>>3CD""  # #k&7&7 7  $ ${'9'9 9  '-++- -r!c:<V^8dQh/S[S[R3,;R&#)r.r)rwr)rrs"rrrs  S !!r!r&rJ)rrrrrr0 classmethodrr8rr@rrErrrMrrrrrrrrs@rrQrQs 66   55 99 FF 99//)) , ,%%/-/-mr!rQ)rrKcollections.abcrr dataclassesr itertoolsrtypingrrrtorch.distributed._pycuterr r r r r rrrrrrrrQr&r!rrs| .!%       $n,n,n,b $D-(;'D-D-r!