+ jD ^RIt^RIt^RIt^RIt^RIHtHt^RIHt^RI H t ^RI t ^RI H uHuHt^RIH uHuHt^RIHt^RIHt^RIHt^RIHt^RIHtHtHt^R I H!t!H"t"^R I#H$t$H%t%^R I&H't'H(t(^R I)H*t*H+t+H,t,H-t-H.t.^R I/H0t0H1t1H2t2H3t3H4t4.R8Ot5] PlPnt7RRlt8!RR] PrPt4t;!RR] PrPt4t<!RR] Pz4t>R9R^/RRlllt?] !R4R:RRll4t@R;RRlltA]R4tB]R4tCR9R R!lltDR"RR#] PR$R%R&RR'R/R(R)lltFR"RR#] PR$R%R&RR'R/R*R+lltGR"RR#] PR$R%R&RR'R/R,R-lltHR"RR#] PR$R%R&RR'R/R.R/lltIRPtN]!4R#)=N)CallableSequence)Any) deprecated)1mark_subclass_constructor_exportable_experimental) LazyString)maybe_run_for_local_tensor)_mesh_resources"_register_distributed_opaque_types DeviceMesh)check_tensor_metamesh_broadcast) DTensorSpec TensorMeta) Redistributeredistribute_local_tensor)assert_no_mixed_partial_typescompute_global_tensor_info%compute_local_shape_and_global_offsetExplicitRedistributionContextnormalize_to_torch_size) _StridedShardPartial Placement ReplicateShardDTensorcdV^8dQhR\\R3,R\\R3,/#) placements.returntupler)formats"^/home/luca/screen-checking/.venv/lib/python3.14/site-packages/torch/distributed/tensor/_api.py __annotate__r&?s.in% 9c>c.pVFEpVP4'dVP\44K4VPV4KG \V4#)a Normalize gradient placements by converting Partial to Replicate. See the gradient placement guarantees documented in DTensor.from_local's docstring for why Partial forward placements map to Replicate gradient placements. We do this for both from_local and to_local backward. ) is_partialappendrr#)r normalizedps& r%_normalize_placements_for_gradr-?sI#%J  <<>>   ik *   a   r'cXa]tRt^fto]V3RlRl4t]V3RlRl4tRtVtR#)_ToTorchTensorcB<V^8dQhRRRS[S[,R,/#)rinputrgrad_placementsN)rr)r$ __classdict__s"r%r&_ToTorchTensor.__annotate__hs)222"),t32r'cVPVnW nVPR4VPpVP 4VnVPV4#)F)_spec dtensor_specr2set_materialize_grads _local_tensorstridelocal_tensor_strideview_as)ctxr1r2 local_tensors&&& r%forward_ToTorchTensor.forwardgsS !;;- !!%(** "."5"5"7 ##L11r'cB<V^8dQhRS[PR,/#)r grad_outputN)torchTensor)r$r3s"r%r&r4ys c c 5<<$#6c r'c a a a aaVfR #VPpVPpVPpVPpVf\ VP 4p^RIHo ^RIH o V 3RloV 3RloV 3Rlo V VV3RlpV V3RlpWBP 8XdV!VPVP4VP4'dV!VPVPVP4'dlV!VPVP4VP4V!VPVPVP4\P!V4pMQ\WVP 4wr\V 4p \!VV\#VPV VP$R7R 7p\&P)VVP*VP VPVPR 7R3#) N)check_contiguous_sizes_strides)guard_or_falsec<\V4\V48H;'dM\;QJd(V3Rl\W44F 'dK R# R#!V3Rl\W444#)c3<<"TFwrS!W8H4xK R#5iN.0abrGs& r% J_ToTorchTensor.backward..strides_provably_match..s!53E41qv&&3EFT)lenallzipr:otherrGs&&r%strides_provably_match7_ToTorchTensor.backward..strides_provably_matchsav;#e*,536v3E5  536v3E52 r'c<\V4\V48g;'gM\;QJd(V3Rl\W44F 'gK R# R#!V3Rl\W444#)c3<<"TFwrS!W8g4xK R#5irJrKrLs& r%rPK_ToTorchTensor.backward..strides_provably_differ..s!43E41qv&&3ErRTF)rSanyrUrVs&&r%strides_provably_differ8_ToTorchTensor.backward..strides_provably_differsav;#e*,436v3E4  436v3E41 r'c<\V4\V4u;8Xd\V48XgR#R#S!WRR7;'d S!WRR7#)FT) false_if_dde)rS)shaper:rWrFs&&&r% strides_are_contiguous_for_shapeA_ToTorchTensor.backward..strides_are_contiguous_for_shapesPJ#f+;U;<1DRR0DQ Rr'c`<S!W4;'gS!WV4;'gS!W4'*#rJrK)rbr:rWrcr^rXs&&&r%strides_may_match2_ToTorchTensor.backward..strides_may_matchs9&v5>>3E5I>>.v== r'c<aaS!SS4'gS!VSS4'dR#\SS4F#wr4\P!W48HVV3Rl4K% R#)Nc<RS RS 2#)z-Expected matching DTensor local strides, got z and rK)rWr:sr%G_ToTorchTensor.backward..check_stride_matches..sGxuUZT[\r')rUrC_check)rbr:rWrNrOrcrXs&ff r%check_stride_matches5_ToTorchTensor.backward..check_stride_matchessP%1%GGFE* F+r'rbr:dtype tensor_meta)rbr:NN)r7meshr2rrr-r torch._prims_commonrF%torch.fx.experimental.symbolic_shapesrGrbr:r;copyrr#rrrpr from_local device_mesh)r=rBr7rtr2 dtensor_metarfrm grad_spec_ tensor_striderFrGrcr^rXs&& @@@@@r%backward_ToTorchTensor.backwardxs   ''   --#//   "<\=T=TUOFH     R  " 66 6!!!;#5#5#79P9P"""C$;$;\=P=P !!!;#5#5#79P9P  !""C$;$;\=P=P   ,/I9<#:#:  A"-0M#&&,,(&,,I   %%$$oo ''     r'rKN __name__ __module__ __qualname____firstlineno__ staticmethodr?r~__static_attributes____classdictcell__r3s@r%r/r/fs-22 c c r'r/c\a]tRt^to]RV3RlRll4t]V3RlRl4tRtVtR#)_FromTorchTensorNc<V^8dQhRS[PRS[RS[S[R3,RS[RS[P R,RS[S[R3,R,R S[S[R3,R,R R /#) rr1ryr . run_checkrbNr:r2r!r)rCrDr r#rboolSizeint)r$r3s"r%r&_FromTorchTensor.__annotate__sFF||F F)S.) F  F zzD Fc3h$&Fy#~.5F Fr'c W0nW nWpnVPR4V'd V'dYVrMRV'g8V'g0\ WV4wr\ P !V 4\V 4rM\RV RV R2R4hVP4'gVP^VPR7pMsV'dlV'*;'dV'*p \WR7\V4F:wrVP4'gKVP4p\!WV R7K< \#VV\%VV VP&4R7p\)VP+V4VVPR7pV#) Fz Found shape:z , stride:.z3Please pass both shape and stride at the same time. requires_grad)check_shape_stride)mesh_dimrq)forward_input_placementsforward_input_device_meshr2r8rrCrr# RuntimeError_is_current_rank_part_of_mesh new_emptyrr enumerate is_replicate contiguousrrrrprr<)r=r1ryr rrbr:r2 tensor_shaper} global_shape global_strideridx placement dist_spec dist_tensors&&&&&&&& r%r?_FromTorchTensor.forwards\(2$(3%- !!%( V*/-v+EJ+ 'L+0**\*BE-DX-ugYvha8E  88::OOAU5H5HOIE &+!9!96z  e K#,J"7))++",,.E"5D #8  "   MM%  -- r'c<V^8dQhRR/#)rrBzDTensor | NonerK)r$r3s"r%r&r*s7J7J#37Jr'cVPpVPpVPpVfR#VeTpM \V4pVPV8wdsVP p\ VVVP PR7p\P!VV\R44VPp\VVV4p V RRRRRR3#VP4RRRRRR3#)NrqcR#)z?Implicit redistribution occurred in DTensor.from_local backwardrKrKr'r%rj+_FromTorchTensor.backward..Rs]r')NNNNNNN)rrr2r-r r6rrrrobserve_redistributionrr9rto_local) r=rBrrr2normalized_placements current_spec target_specr>outputs && r%r~_FromTorchTensor.backward)s#&#?#? $'$A$A!--  ; ;  &$3 !$B(% !  ! !%: :(3(9(9L%)%'--99K * @ @] '44L.F4tT4= =##%tT4tTIIr'rK)NNNrrs@r%rrs6FFFP7J7Jr'rc aa]tRtRtoRtRR.t]P!4tV3RlRlt ] PPt ] P]V3Rl44tRtR t]R 4tV3R lR ltR tR-Rlt]R.Rl4t]R/RRRRRRRR/V3RlRlll4tRR/V3RlRlltR/RRRRRR/V3RlRllltRR/V3RlR llt]V3R!lR"l4t]V3R#lR$l4tV3R%lR&lt V3R'lR(lt!R)t"R*t#V3R+lt$R,t%Vt&V;t'#)0riday ``DTensor`` (Distributed Tensor) is a subclass of ``torch.Tensor`` that provides single-device like abstraction to program with multi-device ``torch.Tensor``. It describes the distributed tensor sharding layout (DTensor Layout) through the :class:`DeviceMesh` and following types of :class:`Placement`: * :class:`Shard`: Tensor sharded on the tensor dimension ``dim`` on the devices of the ``DeviceMesh`` dimension * :class:`Replicate`: Tensor replicated on the devices of the ``DeviceMesh`` dimension * :class:`Partial`: Tensor is pending reduction on the devices of the ``DeviceMesh`` dimension When calling PyTorch operators, ``DTensor`` overrides the PyTorch operators to perform sharded computation and issue communications whenever necessary. Along with the operator computation, ``DTensor`` will transform or propagate the placements (DTensor Layout) properly (based on the operator semantic itself) and generate new ``DTensor`` outputs. To ensure numerical correctness of the ``DTensor`` sharded computation when calling PyTorch operators, ``DTensor`` requires every Tensor argument of the operator be DTensor. .. note:: Directly using the Tensor subclass constructor here is not the recommended way to create a ``DTensor`` (i.e. it does not handle autograd correctly hence is not the public API). Please refer to the `create_dtensor`_ section to see how to create a ``DTensor``. r9r6cD<V^8dQhRS[PRS[RS[RR/#)rr>specrr!r)rCrDrr)r$r3s"r%r&DTensor.__annotate__s7  ll      r'c`\PPWW#R7pW$nWnV#r)rCrD_dtensor__new__r6r9)clsr>rrrs&&&$ r%__new__DTensor.__new__s3 LL ( ( t ) &r'c $<\SV`4R#)a Construct a DTensor from a local tensor, device mesh, and placement and other tensor properties (i.e. shape, requires_grad, strides, etc). .. note:: This is not a public API and it's only supposed to be used by the operator implementations and internals. If you want to construct a DTensor from a local tensor, consider using ``DTensor.from_local``, if you want to construct a DTensor from a "global" tensor (where you already have tensor initialized and want to shard this tensor), consider using ``distribute_tensor``. N)super__init__)selfargskwargs __class__s&*,r%rDTensor.__init__s r'c~RVP RVPP RVPP R2#)zDTensor(local_tensor=z, device_mesh=z , placements=))r9r6rtr rs&r%__repr__DTensor.__repr__s=&t'9'9&:.HYYfgkgqgqg|g|f}}~r'c RR.VPPVPPVPPVP33#)zM protocol to inform how to flatten a DTensor to local tensor for PT2 tracing r9ry)r6r rr shard_orderrrs&r%__tensor_flatten__DTensor.__tensor_flatten__sK  / JJ ! ! JJ " " JJ " "    2   r'cVf \R4hVR,pVR,pVwrgr\VVVPR7p \VVV VR7p \ VV V R7#)NzEExpecting spec to be not None from `__tensor_flatten__` return value!r9ryro)rrrr)AssertionErrorrrprr) inner_tensors flatten_spec outer_size outer_strider>rtr old_tensor_metarrunflatten_tensor_metaunflatten_specs &&&& r%__tensor_unflatten__DTensor.__tensor_unflatten__s   W %_5 ]+BN? [ *!''!  %  -#    '   r'c <V^8dQhRS[/#rr!)str)r$r3s"r%r&rs P P# Pr'c \VPP4pVPP 4\VP 4,V,p\ P!VP4^R7P4#)zr Return a stable hash for AOT autograd caching. [See note: Tensor subclass stable hashing for AOT autograd cache] ) digest_size) rr9devicer6 _stable_hashrhashlibblake2bencode hexdigest)rr cache_datas& r%_stable_hash_for_caching DTensor._stable_hash_for_cachingsdT''../ZZ,,.T5G5G1HH6Q z002CMMOOr'c\\;QJd&RVP4F 'gK RM RM!RVP44'gV#VPUu.uF%p\V\4'd \ 4MTNK' ppVP VP VR7#uupi)c3B"TFp\V\4xK R#5irJ) isinstancerrMr,s& r%rP6DTensor.__coerce_tangent_metadata__..sC?a:a))?TFryr )r]r rrr redistributery)rr,r s& r%__coerce_tangent_metadata__#DTensor.__coerce_tangent_metadata__ssC4??CsssC4??CCCKBF// BQQ:a11IKq 8/    T-=-=* UU s+B)NcVVeR#VwppVPVPVR7#)Nr)rry)rr expected_typer r|s&&& r%#__coerce_same_metadata_as_tangent__+DTensor.__coerce_same_metadata_as_tangent__s=  $ ,Q1  ((!!  r'cRp\PPPWS4p\PPPYT;'g/4pV!V/VB#)c\V\4'd@\V4\Jd-\VPVPVP R7#V#r)rrtyper9r6r)ts&r%unwrap*DTensor.__torch_dispatch__..unwrapsC!W%%$q'*@OOGG"#// Hr')rCutils_pytreetree_map)rfunctypesrrrs&&&&& r%__torch_dispatch__DTensor.__torch_dispatch__sS {{""++F9$$--fllCT$V$$r'rFrbr:r2c<V^8dQhRS[PRS[R,RS[S[,R,RS[RS[P R,RS[S[R3,R,R S[S[,R,R R /#) rr>ryNr rrbr:.r2r!r) rCrDr rrrrr#r)r$r3s"r%r&rs| | ll| $&| Y'$.|  | zzD | c3h$&| "),t3|  | r'c  \V\4'd\R\V4 R24hT;'g\P !4pVP pWpPP8wd$VP'gVPV4pVf.\VP4Uu.uF p\4NK ppM\V4p\V4Fwr\V \\ ,4'gK(V P"^8gK;V P"VP,p \V 4\ Jd\!WP$R7W)&K\V 4\JgK\V 4W)&K \'V4\(P+TT\-V4TTTVe\-V44#R4#uupi)a Create a :class:`DTensor` from a local torch.Tensor on each rank according to the ``device_mesh`` and ``placements`` specified. Args: local_tensor (torch.Tensor): local torch.Tensor on each rank. device_mesh (:class:`DeviceMesh`, optional): DeviceMesh to place the tensor, if not specified, must be called under a DeviceMesh context manager, default: None placements (List[:class:`Placement`], optional): the placements that describes how to place the local torch.Tensor on DeviceMesh, must have the same number of elements as ``device_mesh.ndim``. Keyword args: run_check (bool, optional): at a cost of extra communications, perform sanity check across ranks to check each local tensor's meta information to ensure correctness. If have :class:`Replicate` in ``placements``, the data on first rank of the device mesh dimension will be broadcasted to other ranks. default: False shape (torch.Size, optional): A List of int which specifies the size of DTensor which build on top of `local_tensor`. Note this needs to be provided if the shape of ``local_tensor`` are different across the ranks. If not provided, ``shape`` will be computed assuming the given distributed tensor is evenly sharded across ranks. default: None stride (tuple, optional): A List of int which specifies the stride of DTensor. If not provided, ``stride`` will be computed assuming the given distributed tensor is evenly sharded across ranks. default: None grad_placements (List[:class:`Placement`], optional): specifies the expected input gradient placements. The input gradient (a plain tensor) will be redistributed to this placement before exiting DTensor. If not specified, follows the default placement guarantees below. default: None Returns: A :class:`DTensor` object Raises: ValueError: If ``placements`` contains mixed :class:`Partial` reduce types (e.g., both ``Partial("sum")`` and ``Partial("max")``). All Partial placements must use the same reduce operation. .. note:: When ``run_check=False``, it is the user's responsibility to ensure the local tensor passed in is correct across ranks (i.e. the tensor is sharded for the ``Shard(dim)`` placement or replicated for the ``Replicate()`` placement). If not, the behavior of the created DTensor is undefined. .. note:: ``from_local`` is differentiable, the `requires_grad` of the created `DTensor` object will depend on if `local_tensor` requires_grad or not. .. note:: During backward, ``from_local`` provides the following gradient placement guarantees. For each mesh dimension, the gradient placement maps as follows: +---------------------+--------------------+ | Forward Placement | Gradient Placement | +=====================+====================+ | ``Shard`` | ``Shard`` | +---------------------+--------------------+ | ``Replicate`` | ``Replicate`` | +---------------------+--------------------+ | ``Partial`` | ``Replicate`` | +---------------------+--------------------+ When the forward placement is :class:`Partial`, we always redistribute the gradient to :class:`Replicate` instead of keeping it :class:`Partial`. This may not be the most efficient option, but it avoids ambiguity and provides clearer gradient semantics to users. zryr rrbr:r2rr|rrnormalized_dims &&&$$$$ r%rxDTensor.from_localsZ lG , ,NtT`OaNbbij "GG_%E%E%G !--  --22 2<;O;O;O'??;7L  /4[5E5E/FG/F!)+/FJGJj)J"+J"7i)>?? }}q()29J9J)J ?m;.; .=S=S/JO")_5.3N.CJO#8 &j1  %%   *    &5&AE/ "  HL  +Hs2G cX<V^8dQhRS[S[,R,RS[P/#rr2Nr!rrrCrD)r$r3s"r%r&rs+??"*9"5"<? ?r'c d\P!4'gVPpM:Ve"\V\4'g \ V4p\ P W4p\VRR4'd;\VRR4'g(W PJdVPV4pRVn V#)aT Get the local tensor of this DTensor on its current rank. For sharding it returns a local shard of the logical tensor view, for replication it returns the replica on its current rank. Keyword args: grad_placements (List[:class:`Placement`], optional): the placements describes the future layout of any gradient layout of the Tensor returned from this function. `to_local` converts DTensor to local tensor and the returned local tensor might not be used as the original DTensor layout later in the code. This argument is the hint that user can give to autograd in case the gradient layout of the returned tensor does not match the original DTensor layout. If not specified, we will assume the gradient layout remains the same as the original DTensor and use that for gradient computation. Returns: A :class:`torch.Tensor` or ``AsyncCollectiveTensor`` object. it represents the local tensor on its current rank. When an ``AsyncCollectiveTensor`` object is returned, it means the local tensor is not ready yet (i.e. communication is not finished). In this case, user needs to call ``wait`` to wait the local tensor to be ready. .. note:: ``to_local`` is differentiable, the ``requires_grad`` of the local tensor returned will depend on if the `DTensor` requires_grad or not. _is_paramFT) rCis_grad_enabledr9rr#r/rgetattrr<r)rr2results&$ r%rDTensor.to_locals8$$&&''F*:ou3U3U"'"8#))F* 4e , ,W K6 6 +++ /#F  r'async_op forward_dtypebackward_dtypec <V^8dQhRS[R,RS[S[,R,RS[RS[P R,RS[P R,RR/#) rryNr rrrr!r)r rrrrCrp)r$r3s"r%r&rspv v $&v Y'$.v  v {{T) v  d*v  v r'c T;'g VPpVf \R4h\V4p\V4EFFwrgVP 4'dzVP V,V8wdbVP V,p\ V\4'd%\V4\JdVPR8Xg\RV RV R24hK\ V\4'd<VP^8d+\VPVP,4W&&K\ V\4'gKVP^8gEK\VPVP,VPR7W&&EKI \V4pVP P"p T;'gT p\$P&!TTTTRTRTR RT;'gT RV //4#) a$ ``redistribute`` performs necessary collective operations that redistribute the current DTensor from its current placements to a new placements, or from its current DeviceMesh to a new DeviceMesh. i.e. we can turn a Sharded DTensor to a Replicated DTensor by specifying a Replicate placement for each dimension of the DeviceMesh. When redistributing from current to the new placements on one device mesh dimension, we will perform the following operations including communication collective or local operation: 1. ``Shard(dim)`` -> ``Replicate()``: ``all_gather`` 2. ``Shard(src_dim)`` -> ``Shard(dst_dim)``: ``all_to_all`` 3. ``Replicate()`` -> ``Shard(dim)``: local chunking (i.e. ``torch.chunk``) 4. ``Partial()`` -> ``Replicate()``: ``all_reduce`` 5. ``Partial()`` -> ``Shard(dim)``: ``reduce_scatter`` 6. ``Shard(dim)`` -> ``Partial("sum")``: local zero-filling. This explicit-only conversion materializes a global-shape local tensor on every rank and does not communicate. Of conversions to ``Partial``, this public API supports only ``Shard(dim)`` -> ``Partial("sum")``. Each rank places its local shard at the corresponding offset in a zero-filled logical-shape tensor, so summing across ranks reconstructs the original logical tensor. ``redistribute`` would correctly figure out the necessary redistribute steps for DTensors that are created either on 1-D or N-D DeviceMesh. Args: device_mesh (:class:`DeviceMesh`, optional): DeviceMesh to place the DTensor. If not specified, it would use the current DTensor's DeviceMesh. default: None placements (List[:class:`Placement`], optional): the new placements that describes how to place the DTensor into the DeviceMesh, must have the same number of elements as ``device_mesh.ndim``. default: replicate on all mesh dimensions Keyword args: async_op (bool, optional): whether to perform the DTensor redistribute operation asynchronously or not. Default: False forward_dtype (torch.dtype, optional): cast the local tensor to ``forward_dtype`` before running the forward collective. The result DTensor will be in ``forward_dtype``. If ``None``, no conversion is performed. Default: None backward_dtype (torch.dtype, optional): cast the gradient to ``backward_dtype`` before running the backward collective. After the collective the gradient is always cast back to the input DTensor's dtype. If ``None``, the backward collective runs at the input DTensor's dtype (not ``forward_dtype``). Default: None Returns: A :class:`DTensor` object .. note:: ``redistribute`` is twice-differentiable, which means user do not need to worry about the backward formula of the redistribute operation, or its compatibility with autograd for second-order gradients. Higher-order differentiation has not been tested (but may work). .. note:: ``redistribute`` currently only supports redistributing DTensor on the same DeviceMesh, Please file an issue if you need to redistribute DTensor to different DeviceMesh. z&placements is needed for redistribute!sumzCan not redistribute from z to z9, only Shard to Partial(sum) redistribution is supported!rop_dtype out_dtypebackward_options)ryrr rr)r rrrr reduce_opr r rrr#r9rprr) rryr rrrirsource input_dtypes &&&$$$ r%rDTensor.redistributesR"55T%5%5  GH H*% %j1LA##%%$//!*< *I+vu--Y72!++u4&4VHD LRR5 Iu--)--!2C %immdii&? @ I}55)--!:K -MMDII-Is/AA"*9"5"<A Ar'c VP\4.VPP,RR7p\P W!4#)aB Return the full tensor of this DTensor. It will perform necessary collectives to gather the local tensors from other ranks in its DeviceMesh and concatenate them together. It's a syntactic sugar of the following code: ``dtensor.redistribute(placements=[Replicate()] * mesh.ndim).to_local()`` Keyword args: grad_placements (List[:class:`Placement`], optional): the placements describes the future layout of any gradient layout of the full Tensor returned from this function. `full_tensor` converts DTensor to a full torch.Tensor and the returned torch.tensor might not be used as the original replicated DTensor layout later in the code. This argument is the hint that user can give to autograd in case the gradient layout of the returned tensor does not match the original replicated DTensor layout. If not specified, we will assume the gradient layout of the full tensor be replicated. Returns: A :class:`torch.Tensor` object that represents the full tensor of this DTensor. .. note:: ``full_tensor`` is differentiable. F)r r)rrryr r/r)rr2 redist_ress&$ r% full_tensorDTensor.full_tensor>sF4&&! }t'7'7'<'<) r6r?rCr@r9rDrrCrDr)rrCs& r%rDDTensor.__create_chunk_list__sq 224  4%%'> ? ?%%;;= = **ELL 9 9.t45 59: :r'cVP4\VPR4'dVPPV4#\ VP\ P 4'dVP4#\R4h)__get_tensor_shard__r>) r6r@r9rGrrCrDrr)rindexs&&r%rGDTensor.__get_tensor_shard__sh 224 4%%'= > >%%::5A A **ELL 9 9==? "9: :r'cf<V^8dQh/S[P;R&S[;R&S[P;R&#)rr9r6_op_dispatcher)rCrDr op_dispatch OpDispatcher)r$r3s"r%r&rds6,<<-. /6 ,,I7r'rKrJ)rKNrs)(rrrr__doc__ __slots__rLrMrKrrCrDr_disable_dynamorrrrrrrrr classmethodrrxrrr+propertyryr r6r=rDrG__annotate_func__rr __classcell__)rr3s@@r%rrds.!'*I0;/G/G/IN  ll**G 6 7 @    8 P PV %%,|  | $( | *.| 7;| | | |??C??Bv  v -1 v .2v v pA?CAA>%%   ; ;;.;{r' src_data_rankc V^8dQhR\PR\R,R\\,R,R\ R,R\ /#)rtensorryNr rUr!)rCrDr rrrr)r$s"r%r&r&sXkk LLkd"k#d*k : k  kr'c \PPR4T;'g\P!4pVP pVR8Xd^RIHpV!WV4#VP'g \R4hW@PP8wd$VP'gVPV4pVf-\VP 4Uu.uF p\#4NK pp\%V4VP 8wd&\'R\%V4 RVP R 24h\)V4\+V\,4'dfVP.V8wd\'R VP. R V R 24hVP0\3V48wd\'R VP0 R V R24hV#VP54p \7V4p\9V4EF_wr\+V \:\<,4'dV P>^8dV P>VP ,M V P>p \+V \:4'd(\:P@!WWV4p \;V 4W*&K\<P@!V V VV VV PBR7p \=WPBR7W*&K\+V \"4'd\"PD!WW4p EK\+V \F4'd-\"PD!WW4p V PIWV 4p EKP\RV RV R24h \3V4pV f \KR4h\MVV\OVPQ4VPS4VPTR7R7p \-V PWVPX4V VPXR7# \dpRp\T4ThRp?ii;iuupi)a Distribute a leaf ``torch.Tensor`` (i.e. nn.Parameter/buffers) to the ``device_mesh`` according to the ``placements`` specified. The rank of ``device_mesh`` and ``placements`` must be the same. The ``tensor`` to distribute is the logical or "global" tensor, and the API would use the ``tensor`` from first rank of the DeviceMesh dimension as the source of truth to preserve the single-device semantic. If you want to construct a DTensor in the middle of the Autograd computation, please use :meth:`DTensor.from_local` instead. Args: tensor (torch.Tensor): torch.Tensor to be distributed. Note that if you want to shard a tensor on a dimension that is not evenly divisible by the number of devices in that mesh dimension, we use ``torch.chunk`` semantic to shard the tensor and scatter the shards. The uneven sharding behavior is experimental and subject to change. device_mesh (:class:`DeviceMesh`, optional): DeviceMesh to distribute the tensor, if not specified, must be called under a DeviceMesh context manager, default: None placements (List[:class:`Placement`], optional): the placements that describes how to place the tensor on DeviceMesh, must have the same number of elements as ``device_mesh.ndim``. If not specified, we will by default replicate the tensor across the ``device_mesh`` from the first rank of each dimension of the `device_mesh`. Keyword args: src_data_rank (int, optional): the rank of the source data for the logical/global tensor, it is used by :meth:`distribute_tensor` to scatter/broadcast the shards/replicas to other ranks. By default, we use ``group_rank=0`` on each DeviceMesh dimension as the source data to preserve the single-device semantic. If passing ``None`` explicitly, :meth:`distribute_tensor` simply uses its local data instead of trying to preserve the single-device semantic via scatter/broadcast. Default: 0 Returns: A :class:`DTensor` or ``XLAShardedTensor`` object. Raises: ValueError: If ``placements`` contains mixed :class:`Partial` reduce types (e.g., both ``Partial("sum")`` and ``Partial("max")``). All Partial placements must use the same reduce operation. .. note:: When initialize the DeviceMesh with the ``xla`` device_type, ``distribute_tensor`` return `XLAShardedTensor` instead. see `this issue `__ for more details. The XLA integration is experimental and subject to change. ztorch.dtensor.distribute_tensorxla)xla_distribute_tensorDTo use DTensor API with xla, you must install the torch_xla package!NzY`distribute_tensor` should be used to distribute leaf tensors! but found non-leaf tensor!zW`placements` must have the same length as `device_mesh.ndim`! Found placements length: z, and device_mesh.ndim: rz-Cannot distribute a DTensor with device mesh z to a different device mesh z,Cannot distribute a DTensor with placements z to a different placements z-. do you want to call `redistribute` instead?rz8Trying to distribute tensor with unsupported placements z on device mesh dimension !z(distributing a tensor should not be Nonero)rtr rrr)-rC_C_log_api_usage_oncer rrtorch_xla.distributed.spmdrZ ImportErroris_leafrrrrr r r rrSr5rrrryr r#detachr rrrr _make_shard_tensorr_make_replicate_tensorr_partition_valuerrrsizer:rprequires_grad_r)rWryr rUrrZemsgr|r>rr placement_dimrs&&&$ r%distribute_tensorrksh HH  !BCCC!A!A!CK))Ke * )jI I >>> g  mm(((;'+01A1A+BC+Baik+B C :+***((+J'88PQ\QaQaPbbc e  "*-&'""    ,?@R@R?ST..9]!=    j 1 1>v?P?P>QR--7L9*+   ==?Lj!J#J/ i!6 7 7/8}}q/@  +imm )U++$77!=  #( "6 ,??! !!*!7!7   #0!0F0F#  9 - -$;;3L 7 + +$;;3L%55lQTULJ9+Uopsottuv A0Fz"JGHH  ++-==?,,  D ##F$8$89 **  G *XCc" ) *DsOO' O$OO$zAPlease use `distribute_tensor` with `src_data_rank=None` instead.cxV^8dQhR\PR\\,R\R,RR/#)rr+r ryNr!r)rCrDrrr )r$s"r%r&r&ZsC"W"W"W"Wd""W "Wr'c\WVRR7#)a4 Locally shards a full tensor based on indicated sharding arrangement, and returns a DTensor containing the local shard. .. warning:: This is a private API that is subject to change. It skips the communication otherwise required by `distribute_tensor`. It is only applicable to cases where all ranks have the same `full_tensor`. For example, in distributed inference all ranks load from the same checkpoint. This API will not check for data equality between ranks, it is thus user's responsibility to ensure the `full_tensor` is the same across ranks. Args: full_tensor (torch.Tensor): the full tensor to be sharded. placements (Sequence[:class:`Shard`]): the placements that describes how to place the local tensor on DeviceMesh. device_mesh (:class:`DeviceMesh`, optional): DeviceMesh to place the DTensor. Must have same dimension as the number of placements. If not specified, would be retrieve from current context. Returns: A :class:`DTensor` object with the shard as its local tensor. Examples: >>> # xdoctest: +SKIP("need world_size and rank") >>> device_mesh = dist.init_device_mesh("cuda", (world_size,)) >>> full_tensor = torch.arange(world_size, device=f"cuda:{rank}") >>> dtensor = _shard_tensor(full_tensor, [Shard(1)], device_mesh) N)rU)rk)r+r rys&&&r% _shard_tensorrnYsF [zQU VVr'c V^8dQhR\PR\R,R\\\P\.R3,R,R\\P\ \.R3,R,R\\P\ \.R3,R,R\P/#)rmoduleryN partition_fninput_fn output_fnr!)nnModuler rrr)r$s"r%r&r&smm IImd"mCJ7=>Em 3 3T9:TA m C4d:;dB m YY mr'cdaaa\PPR4\VRR4pV'd \ R4hS;'g\ P !4oSPpVR8Xd^RIH pV!VSVSS4#R R lp Vf"VP4F p V !V S4K M+VP4FwrV!WS4V !V S4K Se\\P!S4P 4p V ^8Xd5\"P$!R \&^R 7VP)VV3R l4M-V ^8XdVP)VV3Rl4M\+RV R24hSe\\P!S4P 4p V ^8Xd5\"P$!R\&^R 7VP-VV3Rl4M-V ^8XdVP-VV3Rl4M\+RV R24hRVnV# \dpRp \T 4ThRp?ii;i)a9 Convert a module's parameters/buffers to :class:`DTensor` s, and optionally convert its inputs/outputs at runtime, using the three user-provided callbacks. The behavior of the three callbacks: 1. ``partition_fn`` shards the module's parameters/buffers *before* runtime execution (i.e. it converts plain ``torch.Tensor`` parameters into :class:`DTensor` parameters). Any parameter/buffer not converted by ``partition_fn`` is replicated across ``device_mesh`` afterwards. If ``partition_fn`` is ``None``, *all* parameters/buffers are replicated. 2. ``input_fn`` / ``output_fn`` convert the module's inputs/outputs *during* runtime execution (e.g. convert plain tensor inputs into :class:`DTensor`, or convert :class:`DTensor` outputs back into plain ``torch.Tensor``). Args: module (:class:`nn.Module`): user module to be partitioned. device_mesh (:class:`DeviceMesh`): the device mesh to place the module. If ``None``, the current mesh from the ambient mesh context is used. partition_fn (Callable): a function with signature ``partition_fn(name: str, module: nn.Module, device_mesh: DeviceMesh) -> None``. It is called once for *every* submodule (including ``module`` itself), where ``name`` is the submodule's fully-qualified name (``""`` for the top-level module) as returned by :meth:`nn.Module.named_modules`. The function is expected to mutate ``module`` in place, typically by calling :func:`distribute_tensor` and :meth:`register_parameter` to replace selected parameters with :class:`DTensor` parameters. Its return value is ignored. If not specified, all module parameters are replicated across ``device_mesh``. input_fn (Callable): a function with signature ``input_fn(module: nn.Module, inputs: Inputs, device_mesh: DeviceMesh) -> Inputs``, where ``Inputs`` is the tuple of positional arguments to ``forward`` (one entry per positional argument, e.g. ``(x,)`` for ``forward(self, x)``; keyword arguments are not forwarded to ``input_fn``). Installed on the top-level ``module`` as a ``forward_pre_hook``, it returns the new argument tuple -- the same structure as ``inputs``, typically with each ``torch.Tensor`` entry converted to a :class:`DTensor` -- which replaces ``inputs`` for the actual ``forward`` call. output_fn (Callable): a function with signature ``output_fn(module: nn.Module, outputs: Outputs, device_mesh: DeviceMesh) -> Outputs``, where ``Outputs`` is the return type of ``module.forward`` (a single tensor, or any nested structure of tensors such as a tuple/dict), with the tensors typically being :class:`DTensor` s. Installed on the top-level ``module`` as a ``forward_hook``, it returns the replacement output of the same structure -- typically with each :class:`DTensor` converted back to a plain ``torch.Tensor`` via :meth:`DTensor.to_local` or :meth:`DTensor.full_tensor`, or redistributed to different placements. Returns: A module whose parameters/buffers are all :class:`DTensor` s (the same ``module`` object, mutated in place). .. warning:: ``input_fn`` and ``output_fn`` previously took two arguments ``(inputs, device_mesh)`` / ``(outputs, device_mesh)``. That form is deprecated; use the three-argument form documented above. Example:: import torch.nn as nn from torch.distributed.tensor import ( DTensor, Replicate, Shard, distribute_module, distribute_tensor, init_device_mesh, ) class MyModule(nn.Module): def __init__(self) -> None: super().__init__() self.fc = nn.Linear(8, 8) def forward(self, x): return self.fc(x) mesh = init_device_mesh("cuda", (4,)) def shard_params(name, mod, mesh): if isinstance(mod, nn.Linear): for pname, param in mod.named_parameters(recurse=False): mod.register_parameter( pname, nn.Parameter(distribute_tensor(param, mesh, [Shard(0)])) ) # Replicate the input across the mesh before forward. def to_replicated(mod, inputs, mesh): return tuple( distribute_tensor(x, mesh, [Replicate()]) if isinstance(x, torch.Tensor) else x for x in inputs ) # Convert the DTensor output back to a plain torch.Tensor after forward. def to_local(mod, outputs, mesh): return outputs.full_tensor() if isinstance(outputs, DTensor) else outputs sharded = distribute_module( MyModule(), mesh, partition_fn=shard_params, input_fn=to_replicated, output_fn=to_local, ) .. note:: When initializing the ``DeviceMesh`` with the ``xla`` device_type, ``distribute_module`` returns an ``nn.Module`` with PyTorch/XLA SPMD annotated parameters. See `this issue `__ for more details. The XLA integration is experimental and subject to change. ztorch.dtensor.distribute_module_distribute_module_appliedFzhdistribute_module should only be called once on a module, but it has already been called on this module!rY)xla_distribute_moduler[NcHV^8dQhR\PR\RR/#)rmrtr!N)rtrur )r$s"r%r&'distribute_module..__annotate__s*RR299RJR4Rr'c \4.VP,pVPP4Fiwr4VfK \ V\ 4'dK#VP V\P!\VPW4VPR74Kk VPP4F<wr5VfK \ V\ 4'dK#\WQV4VPV&K> R#)Nr) rr  _parametersitemsrrregister_parameterrt Parameterrkdatar_buffers)rzrtfull_replicatekeyparambuffers&& r%replicate_module_params_buffers:distribute_module..replicate_module_params_bufferss$+2----/JC E7)C)C$$LL)%**dK&+&9&90::++-KC!*VW*E*E"3F."Q 3.r'zDeprecating input_fn that takes two arguments (inputs, device_mesh), please use input_fn that takes in (module, inputs, device_mesh) instead!) stacklevelc<S!VS4#rJrK)r|inputsryrrs&&r%rj#distribute_module..Is (6;"?r'c<S!WS4#rJrK)modrryrrs&&r%rjrNs HS+$Fr'z-input_fn should take in 3 arguments, but got z arguments!zDeprecating output_fn that takes two arguments (inputs, device_mesh), please use output_fn that takes in (module, inputs, device_mesh) instead!c<S!VS4#rJrKrroutputsryrss&&&r%rjr`s Yw -Lr'c<S!WS4#rJrKrs&&&r%rjrds Ys[-Qr'z.output_fn should take in 3 arguments, but got T)rCr]r^rrr rrr_rxr`modules named_modulesrSinspect signature parameterswarningswarn FutureWarningregister_forward_pre_hookr5register_forward_hookrw)rpryrqrrrsalready_distributedrrxrhrirsubmodnamenum_argss&f&ff r%distribute_modulers+B HH  !BC!&*FN =  CC!A!A!CK))Ke * ) \8Y R,nn&F +FK @'#002LD { 3 +FK @3 w((2==> q= MM[    , ,? ]  , ,F ?zU w((3>>? q= MM\    ( (L ]  ( (Q @ +V )-F% Mk *XCc" ) *s6H H/H**H/c TpTpV^8dDV^8d=W!, V^, , p WV ,,pWyV^, ,,pM V^8dTpV!Wx3RV/VB#)rstepsrK) init_opstartend total_steps local_stepsoffsetr local_start local_endsteps &&&&&&, r%_linspace_local_tensorrusiKIQ;? a0tm+ +/":: q ; G G GGr'c TpTp V^8dDV^8d=W!, V^, , p WV ,,pWV^, ,,p M V^8dTp V!W3RVRV/VB#)rrbaserK) rrrrrrrrrrrs &&&&&&&, r%_logspace_local_tensorrsnKIQ;? a0tm+ +/":: q ; R R4 R6 RRr'cV^8dQhR\PR\R,R\\,R,R\ /#)rrfryNr r!)rCrr rrr)r$s"r%r&r&sK^^ **^d"^#d* ^  ^r'c ,T;'g\P!4pVPVR&T;'gS\;QJd'.R\ VP 44FNK 5M !R\ VP 444pVP \ V48wd \R4hVR,\P8wd \R4h\PPV4p\VVVV\P\P39R7wrgV\PJdVP!R4pV!Wh3/VBp EMV\PJdKVP!R4p VP!R4p \#VV V V^,V^,V^,3/VBp EMV\PJd]VP!R4p VP!R4p VP!R 4p \%VV V V V^,V^,V^,3/VBp EMV\P&JgV\P(JdVP+R \P,!44p \/WV 4p\1V\V4VR 7p\2P4!V4'd6\2P6'g \2P8!V4\2n\2P6f\h\2P6P;V4;_uu_4V!V3/VBp R R R 4M V!V3/VBp \1V\V4\/VVX P<4R 7p\?V VVR ,R7# +'giLQ;i)rc36"TFp\4xK R#5irJ)r)rMr|s& r%rP'_dtensor_init_helper..s$R:QQY[[:Qsz6mesh dimension does not match the length of placementslayoutzlayout value not supported!) skip_offset fill_valuerrrrprqNrr) r rrr#r r rSrrCstrided _prims_commonmake_contiguous_strides_forrlinspacelogspacefullpoprrrandrandngetget_default_dtyperrrandomis_rng_supported_mesh _rng_trackerOffsetBasedRNGTracker_distribute_regionrpr)rrfryr r torch_stride local_shape global_offsetrr>rrrrprrrs&&&&, r%_dtensor_init_helperrsCC!A!A!CK"..F8RRuu$R% @P@P:Q$Ruu$R% @P@P:Q$RRJ3z?*UVV h5==(:;;&&BB4HL"G ENNENN#CC "K%**ZZ - {A&A ENN " 7#jj-    G N !     ENN " 7#jjzz&!-     G N !      EJJ 'U[["8 7E$;$;$=> U; ;j(9{S  ' ' 4 4V=P=P=P"(">">{"KF     &  3 3D 9 9";9&9L: 9{5f5  j       D  _-  !: 9s & N N rprrFryr c V^8dQhR\PR,R\PR\R\R,R\ \ ,R,R\/#rrpNrrryr r!rCrprrr rrr)r$s"r%r&r&b(( ;; ( LL( ( d" ( #d* ( (r'c X\V4p\\PVVVVVVR7#)a  Returns a :class:`DTensor` filled with the scalar value 1, with the shape defined by the variable argument ``size``. Args: size (int...): a sequence of integers defining the shape of the output :class:`DTensor`. Can be a variable number of arguments or a collection like a list or tuple. E.g.: ones(1,2,3..) or ones([1,2,3..]) or ones((1,2,3..)) Keyword args: dtype (:class:`torch.dtype`, optional): the desired data type of returned :class:`DTensor`. Default: if ``None``, uses a global default (see :func:`torch.set_default_dtype`). layout (:class:`torch.layout`, optional): the desired layout of returned DTensor. Default: ``torch.strided``. requires_grad (bool, optional): If autograd should record operations on the returned :class:`DTensor`. Default: ``False``. device_mesh: :class:`DeviceMesh` type, contains the mesh info of ranks placements: a sequence of :class:`Placement` type: ``Shard``, ``Replicate`` Returns: A :class:`DTensor` object on each rank rprrryr )rrrConesrprrryr rf torch_sizes$$$$$* r%rrs4<).J  # r'c V^8dQhR\PR,R\PR\R\R,R\ \ ,R,R\/#rr)r$s"r%r&r&rr'c X\V4p\\PVVVVVVR7#)a0 Returns a :class:`DTensor` filled with uninitialized data. The shape of the :class:`DTensor` is defined by the variable argument ``size``. Args: size (int...): a sequence of integers defining the shape of the output :class:`DTensor`. Can be a variable number of arguments or a collection like a list or tuple. E.g.: empty(1,2,3..) or empty([1,2,3..]) or empty((1,2,3..)) Keyword args: dtype (:class:`torch.dtype`, optional): the desired data type of returned :class:`DTensor`. Default: if ``None``, uses a global default (see :func:`torch.set_default_dtype`). layout (:class:`torch.layout`, optional): the desired layout of returned :class:`DTensor`. Default: ``torch.strided``. requires_grad (bool, optional): If autograd should record operations on the returned :class:`DTensor`. Default: ``False``. device_mesh: :class:`DeviceMesh` type, contains the mesh info of ranks placements: a sequence of :class:`Placement` type: ``Shard``, ``Replicate`` Returns: A :class:`DTensor` object on each rank r)rrrCemptyrs$$$$$* r%rrs4<).J  # r'c V^8dQhR\PR,R\PR\R\R,R\ \ ,R,R\/#rr)r$s"r%r&r&Jsd,, ;;  , LL ,  ,d",#d*, ,r'c Z\V4p\\PVVVVVVVR7#)aq Returns a :class:`DTensor` filled with ``fill_value`` according to ``device_mesh`` and ``placements``, with the shape defined by the argument ``size``. Args: size (int...): a sequence of integers defining the shape of the output :class:`DTensor`. Can be a variable number of arguments or a collection like a list or tuple. E.g.: ones(1,2,3..) or ones([1,2,3..]) or ones((1,2,3..)) fill_value(Scalar): the value to fill the output tensor with. Keyword args: dtype (:class:`torch.dtype`, optional): the desired data type of returned :class:`DTensor`. Default: if ``None``, uses a global default (see :func:`torch.set_default_dtype`). layout (:class:`torch.layout`, optional): the desired layout of returned DTensor. Default: ``torch.strided``. requires_grad (bool, optional): If autograd should record operations on the returned :class:`DTensor`. Default: ``False``. device_mesh: :class:`DeviceMesh` type, contains the mesh info of ranks. placements: a sequence of :class:`Placement` type: ``Shard``, ``Replicate`` Returns: A :class:`DTensor` object on each rank )rrprrryr )rrrCr)rfrrprrryr rs&&$$$$$ r%rrJs8B).J  #  r'c V^8dQhR\PR,R\PR\R\R,R\ \ ,R,R\/#rr)r$s"r%r&r&yslBB ;;  B LL BBd"B#d*B Br'c TVeF\;QJdRV4F 'gK RM RM !RV44'd \R4h\V\P4'd:\P !VP 4^8HR4VP4p\V\P4'd:\P !VP 4^8HR4VP4p\V34p\\PVVVVVVVVR7 #)a Returns a :class:`DTensor` of size ``steps`` whose values are evenly spaced from `start` to `end`. Args: start (float or :class:`DTensor`): the starting value for the set of points. If :class:`DTensor`, it must be 0-dimensional end (float or :class:`DTensor`): the ending value for the set of points. If :class:`DTensor`, it must be 0-dimensional steps (int): size of the constructed :class:`DTensor` Keyword args: dtype (:class:`torch.dtype`, optional): the data type to perform the computation in. Default: if ``None``, uses the global default dtype (see :func:`torch.set_default_dtype`) when both `start` and `end` are real, and corresponding complex dtype when either is complex. layout (:class:`torch.layout`, optional): the desired layout of returned :class:`DTensor`. Default: ``torch.strided``. requires_grad (bool, optional): If autograd should record operations on the returned :class:`DTensor`. Default: ``False``. device_mesh: :class:`DeviceMesh` type, contains the mesh info of ranks placements: a sequence of :class:`Placement` type: ``Shard``, ``Replicate`` Returns: A :class:`DTensor` object on each rank c3B"TFp\V\4xK R#5irJrrrs& r%rPlinspace..%WJqjM&B&BJrTFz2linspace does not support _StridedShard placementscR#z:linspace only supports 0-dimensional start and end tensorsrKrKr'r%rjlinspace..Pr'cR#rrKrKr'r%rjrrr')rrrprrryr ) r]rrrCrDrlr itemrrr) rrrrprrryr rs &&&$$$$$ r%rrysJ##%WJ%W###%WJ%W"W"WQRR%&& IIK1  P  #u||$$ GGIN P hhj(%2J   #  r'c V^8dQhR\PR,R\PR\R\R,R\ \ ,R,R\/#rr)r$s"r%r&r&slFF ;;  F LLFFd"F#d*F Fr'c VVeF\;QJdRV4F 'gK RM RM !RV44'd \R4h\V\P4'd:\P !VP 4^8HR4VP4p\V\P4'd:\P !VP 4^8HR4VP4p\V34p \\PV VVVVVVVVR7 #)a` Returns a :class:`DTensor` of size ``steps`` whose values are evenly spaced from ``base`` :sup:`start` to ``base`` :sup:`end`, inclusive, on a logarithmic scale with base ``base``. Args: start (float or :class:`DTensor`): the starting value for the set of points. If :class:`DTensor`, it must be 0-dimensional end (float or :class:`DTensor`): the ending value for the set of points. If :class:`DTensor`, it must be 0-dimensional steps (int): size of the constructed :class:`DTensor` Keyword args: base (float, optional): base of the logarithm function. Default: ``10.0``. dtype (:class:`torch.dtype`, optional): the data type to perform the computation in. Default: if ``None``, uses the global default dtype (see :func:`torch.set_default_dtype`) when both `start` and `end` are real, and corresponding complex dtype when either is complex. layout (:class:`torch.layout`, optional): the desired layout of returned :class:`DTensor`. Default: ``torch.strided``. requires_grad (bool, optional): If autograd should record operations on the returned :class:`DTensor`. Default: ``False``. device_mesh: :class:`DeviceMesh` type, contains the mesh info of ranks placements: a sequence of :class:`Placement` type: ``Shard``, ``Replicate`` Returns: A :class:`DTensor` object on each rank c3B"TFp\V\4xK R#5irJrrs& r%rPlogspace..rrTFz2logspace does not support _StridedShard placementscR#z:logspace only supports 0-dimensional start and end tensorsrKrKr'r%rjlogspace..rr'cR#rrKrKr'r%rjrrr')rrrrprrryr ) r]rrrCrDrlr rrrr) rrrrrprrryr rs &&&&$$$$$ r%rrsP##%WJ%W###%WJ%W"W"WQRR%&& IIK1  P  #u||$$ GGIN P hhj(%2J    #  r'c V^8dQhR\R\PR,R\PR\R,R\ \ ,R,R\/#rrrpNrryr r!rrCrprr rrr)r$s"r%r&r&b))) ;; ) LL ) d" ) #d* ) )r'c X\V4p\\PVVVVVVR7#)aK Returns a :class:`DTensor` filled with random numbers from a uniform distribution on the interval ``[0, 1)``. The shape of the tensor is defined by the variable argument ``size``. Args: size (int...): a sequence of integers defining the shape of the output :class:`DTensor`. Can be a variable number of arguments or a collection like a list or tuple. E.g.: ones(1,2,3..) or ones([1,2,3..]) or ones((1,2,3..)) Keyword args: dtype (:class:`torch.dtype`, optional): the desired data type of returned :class:`DTensor`. Default: if ``None``, uses a global default (see :func:`torch.set_default_dtype`). layout (:class:`torch.layout`, optional): the desired layout of returned DTensor. Default: ``torch.strided``. requires_grad (bool, optional): If autograd should record operations on the returned :class:`DTensor`. Default: ``False``. device_mesh: :class:`DeviceMesh` type, contains the mesh info of ranks. placements: a sequence of :class:`Placement` type: ``Shard``, ``Replicate`` Returns: A :class:`DTensor` object on each rank r)rrrCrrrprryr rfrs$$$$$* r%rrs4>).J  # r'c V^8dQhR\R\PR,R\PR\R,R\ \ ,R,R\/#rr)r$s"r%r&r&3rr'c X\V4p\\PVVVVVVR7#)aJ Returns a :class:`DTensor` filled with random numbers from a normal distribution with mean 0 and variance 1. The shape of the tensor is defined by the variable argument ``size``. Args: size (int...): a sequence of integers defining the shape of the output :class:`DTensor`. Can be a variable number of arguments or a collection like a list or tuple. E.g.: ones(1,2,3..) or ones([1,2,3..]) or ones((1,2,3..)) Keyword args: dtype (:class:`torch.dtype`, optional): the desired data type of returned :class:`DTensor`. Default: if ``None``, uses a global default (see :func:`torch.set_default_dtype`). layout (:class:`torch.layout`, optional): the desired layout of returned DTensor. Default: ``torch.strided``. requires_grad (bool, optional): If autograd should record operations on the returned :class:`DTensor`. Default: ``False``. device_mesh: :class:`DeviceMesh` type, contains the mesh info of ranks. placements: a sequence of :class:`Placement` type: ``Shard``, ``Replicate`` Returns: A :class:`DTensor` object on each rank r)rrrCrrs$$$$$* r%rr3s4>).J  # r'c V^8dQhR\R\PR,R\PR\R,R\ \ ,R,R\/#rr)r$s"r%r&r&_sb&&& ;; & LL & d" & #d* & &r'c X\V4p\\PVVVVVVR7#)a Returns a :class:`DTensor` filled with the scalar value 0. Args: size (int...): a sequence of integers defining the shape of the output :class:`DTensor`. Can be a variable number of arguments or a collection like a list or tuple. E.g.: zeros(1,2,3..) or zeros([1,2,3..]) or zeros((1,2,3..)) Keyword args: requires_grad (bool, optional): If autograd should record operations on the returned :class:`DTensor`. Default: ``False``. dtype (:class:`torch.dtype`, optional): the desired data type of returned :class:`DTensor`. Default: if ``None``, uses a global default (see :func:`torch.set_default_dtype`). layout (:class:`torch.layout`, optional): the desired layout of returned :class:`DTensor`. Default: ``torch.strided``. device_mesh: :class:`DeviceMesh` type, contains the mesh info of ranks placements: a sequence of :class:`Placement` type: ``Shard``, ``Replicate`` Returns: A :class:`DTensor` object on each rank r)rrrCzerosrs$$$$$* r%rr_s48).J  # r') rrkrrrrrrrrrrsrJ)NNNN)g$@)Orwrrrcollections.abcrrtypingrtyping_extensionsrrC"torch.distributed.tensor._dispatch distributedrW _dispatchrL torch.distributed.tensor._random_randomrtorch.nnrttorch._export.wrappersrtorch._loggingrtorch.distributed._local_tensorr torch.distributed.device_meshr r r *torch.distributed.tensor._collective_utilsr r&torch.distributed.tensor._dtensor_specrr&torch.distributed.tensor._redistributerrtorch.distributed.tensor._utilsrrrrr(torch.distributed.tensor.placement_typesrrrrr__all__opsatenr-autogradFunctionr/rrDrrkrnrrrrrrrrrrrrrrxrKr'r%rs1 .( 8811T%F YJ  yy~~Nv U^^,,v rBJu~~..BJJD ;ellD ;Nk !" kk\ OP"WQ"WJml H H S S^B( $(!==( ( &* ( .2 (V( $(!==( ( &* ( .2 (V,!% , !== ,  ,&*,.2,^B !% B !== B B&*B.2BJF !% F!==F F&*F.2FFR))!%)!== ) &* ) .2 )X))!%)!== ) &* ) .2 )X&&!%&!== & &* & .2 &Z   "$r'