+ jKh^RIt^RIHt^RIt^RIHt^RIHtHt^RIH t R.t R R^/RRlllt R#) Nfnmatch)_mesh_resources DeviceMesh) ParallelStyleparallelize_module src_data_rankc V^8dQhR\PR\R,R\\\ \3,,R,R\ R,R\P/#)module device_meshNparallelize_planr return)nnModulerrdictstrint)formats"f/home/luca/screen-checking/.venv/lib/python3.14/site-packages/torch/distributed/tensor/parallel/api.py __annotate__rsf@ @ II@ d"@ $d3 +=&>>E@ : @ YY @ c Xa \PPR4T;'g\P!4pVf\ P !R^R7V#\V\4'dW2n VPW4#\V\4'dVP4FwrEVR8Xd\WV4KVPR4pV^,o \\!V 3RlVP#444pV'g%\ P !RV RS R V R 2^R7KVP%^4VF?wrV'd$RP'V4p \V VW/VR 7K0\V VVVR 7KA K V#\)R \+V4 R 24h)a+ Apply Tensor Parallelism in PyTorch by parallelizing modules or sub-modules based on a user-specified plan. We parallelize module or sub_modules based on a parallelize_plan. The parallelize_plan contains :class:`ParallelStyle`, which indicates how user wants the module or sub_module to be parallelized. User can also specify different parallel style per module fully qualified name (FQN). Note that ``parallelize_module`` only accepts a 1-D :class:`DeviceMesh`, if you have a 2-D or N-D :class:`DeviceMesh`, slice the DeviceMesh to a 1-D sub DeviceMesh first then pass to this API(i.e. ``device_mesh["tp"]``) Args: module (:class:`nn.Module`): Module to be parallelized. device_mesh (:class:`DeviceMesh`, optional): Object which describes the mesh topology of devices for the DTensor. If not specified, the call must be under a DeviceMesh context. parallelize_plan (Union[:class:`ParallelStyle`, Dict[str, :class:`ParallelStyle`]], optional): The plan used to parallelize the module. It can be either a :class:`ParallelStyle` object which contains how we prepare input/output for Tensor Parallelism or it can be a dict of module FQN and its corresponding :class:`ParallelStyle` object. If not specified, the call will do nothing at the moment. Keyword args: src_data_rank (int, optional): the rank of the source data for the logical/global tensor, it is used by :meth:`distribute_tensor` to scatter/broadcast the shards/replicas to other ranks. By default, we use ``group_rank=0`` on each DeviceMesh dimension as the source data to preserve the single-device semantic. If passing ``None`` explicitly, :meth:`parallelize_module` simply uses its local data instead of trying to preserve the single-device semantic via scatter/broadcast. Default: 0 Return: A :class:`nn.Module` object parallelized. Example:: >>> # xdoctest: +SKIP("distributed") >>> from torch.distributed.tensor.parallel import parallelize_module, ColwiseParallel >>> from torch.distributed.device_mesh import init_device_mesh >>> >>> # Define the module. >>> m = Model(...) >>> tp_mesh = init_device_mesh("cuda", (8,)) >>> m = parallelize_module(m, tp_mesh, {"w1": ColwiseParallel(), "w2": RowwiseParallel()}) >>> .. note:: For complex module architecture like Attention, MLP layers, we recommend composing different ParallelStyles together (i.e. ``ColwiseParallel`` and ``RowwiseParallel``) and pass as a parallelize_plan, to achieve the desired sharding computation. z4torch.distributed.tensor.parallel.parallelize_modulezNo parallelize_plan is provided and auto-parallel is not supported at the moment, so this parallelize_module call will do nothing.) stacklevel.c*<\V^,S4#)rr)ttokens&r$parallelize_module..fsgadE2rzParallelize plan key 'z6' could not be resolved: no submodule matching token 'z ' in module z, skipping this plan entry.)r zLExpect Union[ParallelStyle, Dict[str, ParallelStyle]] for parallelize_plan, z found!)torch_C_log_api_usage_oncerget_current_meshwarningswarn isinstancerr _applyritemsrsplitlistfilternamed_childrenpopjoin TypeErrortype) r r rr module_pathparallelize_style path_splitsmatched_children_ submodule leaf_pathrs &&&$ @rrrsn HH  !WXCC!A!A!CK  N  "M22)6&&&v;; $d + +.>.D.D.F *Kb "68IJ%++C0K NE#2))+  $ ,[M:4497,vhO01 !   OOA  0  # 5I&!#"6&3 '!#)&3 !1?/Gb  ""&'7"8!9 B  r)NN) r&rr"torch.nnrtorch.distributed.device_meshrr'torch.distributed.tensor.parallel.styler__all__rrrr?s8 EA @ !" @ @ r