+ ja0t$^RIHt^RIt^RIt^RIHtHt^RIH t ^RI H t H t H t HtHtHt^RIt^RIHt^RIHt^RIHt^R IHtHt]'d ^R IHt^R IHt]P<!]4t !R R ]4t!]"]"]#R3,]R,3,t$R]%R&!RR]&4t']!RRR7!RR44t(]!RRR7!RR](44t)](]),t*R]%R&]!RR7!RR44t+]!RRR7!RR44t,]!RRR7!RR 44t-R!R"lt.R#R$lt/!R%R&4t0!R'R(] 4t1R)R*/R+R,llt2R-t3RMR.R/llt4RMR0R1llt5]!RR7!R2R344t6R4R5lt7] R6R/R7R8ll4t8] R9R:l4t8R6R*/R;R<llt8RNR=R>llt9] ROR?R@ll4t:] RORARBll4t:RNRCRDllt:RER*RFR*/RGRHllt;RERRFR*/RIRJlltsP fg & &)\ ,,==?,, ..   r"c V^8dQhRRRR/#)rdevicetorch.device | strrr>r)rs"rr r?Vs   2 | r"c ~\W4pVP4'dVPVP4V#)zReconstruct a tensor on ``device`` from this metadata. Args: device: Target device for the tensor. Returns: An empty strided tensor on ``device``. )_make_tensor_from_metais_floating_pointrequires_grad_r;)r%rGts&& r to_tensor_TensorMeta.to_tensorVs6 #4 0    T// 0r"c V^8dQhRRRR/#rotherr4r list[str]r)rs"rr r?dskir"c W8Xd.#.pVPVP8wd,VPRVP RVP 24VPVP8wd,VPRVP RVP 24VPVP8wd,VPRVP RVP 24V#)zReturn field-by-field differences with ``other``. Args: other: Metadata to compare against. Returns: List of human-readable difference strings (empty if equal). zshape mismatch:  vs zstride mismatch: zdtype mismatch: )r6appendr8r9r%rRdiffss&& rget_diff_TensorMeta.get_diffds =I :: $ LL+DJJ_DTensorMeta.s UZZ^r")default_factoryr5 global_shape)defaultr7 global_strideztuple[Placement, ...] placementsrrNrrc V^8dQhRRRR/#)rdtensorrrr^r)rs"rr _DTensorMeta.__annotate__s  g , r"c  |VPp\VPPVPP 4VP VP VPVP 4VPPVP'd\VP4MRVPR7 #)zCreate metadata from a DTensor. Args: dtensor: The DTensor to extract metadata from. Returns: Metadata capturing both local and global attributes. ) r6r8r9r;rergrhrrr) device_meshr^ _local_tensorr6r8r9r;_specrhrtuple_layout)rjrms& r from_dtensor_DTensorMeta.from_dtensors)) ''--((//1--!// !..*}}//5@5O5O5Ok001UW#++  r"cV^8dQhRR/#)rrr.r)rs"rr rks77 7r"c 2VPVP3#)zd5CTCTBUV   LL< = r"r)r(r)r*r+r,rrer[rgrhrrr\rrpropertyrwrrYr-rr"rr^r^|s  %5KLL*L%*2%6M?6).)J% ',B&7NO7&+'K#  877!2**r"r^ TensorMeta)r2cx]tRtRt$RtRtR]R&RtR]R&RtR]R&Rt R]R &R R lt R R lt RRlt Rt R#) _StageMetaizPConsolidated tensor metadata for a pipeline stage's forward and backward passes.Ntuple[TensorMeta, ...] | Noneinputsoutputs$tuple[TensorMeta | None, ...] | None input_grads output_gradscV^8dQhRR/#rrr:r)rs"rr _StageMeta.__annotate__s   r"c \;QJdHRVPVPVPVP34F 'gK R# R#!RVPVPVPVP344#)z)Check if any metadata field is populated.c3*"TF pVRJxK R#5ir$r).0vs& r %_StageMeta.has_any..s U TMUsTF)anyrrrrrvs&rhas_any_StageMeta.has_anysws kk4<<1A1A4CTCTU ss s s kk4<<1A1A4CTCTU   r"cV^8dQhRR/#rr)rs"rr rsdr"c VPVP3FJpV'gK \;QJdRV4F 'gK RM RM !RV44'gKIR# R#)z3Check if any input/output metadata is DTensor type.c3T"TFq'gK \V\4xK R#5ir$)rBr^)rms& rr*_StageMeta.has_dtensors..sM%Q18Z<88%s ((TF)rrr)r%metass& r has_dtensors_StageMeta.has_dtensorssFkk4<<0EuM%MM%MMM1r"cV^8dQhRR/#rr)rs"rr rsDDDr"c JVPRJ;'dVPRJ#)z-Check if forward metadata is fully populated.N)rrrvs&ris_complete_for_forward"_StageMeta.is_complete_for_forwards"{{$&CC4<r)rs"rr r 6s&  r"cr\P!VPVPVPVR7#)zCreate a tensor from metadata. Args: meta: Metadata with shape, stride, and dtype. device: Target device for the tensor. Returns: Empty tensor preserving the exact memory layout. )sizer8r9rG)r` empty_stridedr6r8r9)rrGs&&rrJrJ6s/    ZZ{{jj  r"c V^8dQhRRRR/#)r tensor_metasrrrr)rs"rr r Ks66(6"6r"ctaRRlo\;QJd.V3RlV4FNK 5#!V3RlV44#)zDerive gradient metadata from tensor metadata. Returns metadata with the same shape/stride/dtype but ``requires_grad=False``. Entries where the source has ``requires_grad=False`` become ``None``. c V^8dQhRRRR/#)rrrrzTensorMeta | Noner)rs"rr (_derive_grad_metas..__annotate__Ts   j  %6  r"cVP'gR#\V\4'dR#\VPVP VP RR7#)NFrA)r;rBr^r4r6r8r9)rs&r derive_one&_derive_grad_metas..derive_oneTsC a & &''88''   r"c34<"TF pS!V4xK R#5ir$r)rrrs& rr%_derive_grad_metas..`s5 1A s)rp)rrs&@r_derive_grad_metasrKs,   55 55555 5 55r"cf]tRtRtRtRRRlltRRltRR ltR R ltR R lt RRlt Rt R#) _MeshCacheiczCache for :class:`DeviceMesh` objects keyed by ``(mesh_dim_names, mesh_layout)``. Assumes all pipeline stages share the same rank tensor (true for TorchTitan-style frameworks where meshes derive from a common world). Nc V^8dQhRRRR/#)r get_mesh_cbzGetMeshCallback | NonerNoner)rs"rr _MeshCache.__annotate__js(($:(d(r"c /VnWnR#r$_cache _get_mesh_cb)r%rs&&r__init___MeshCache.__init__js68 'r"c V^8dQhRRRR/#)rkeyr.rrr)rs"rr rns  L Z r"c WP9dVPV,#Vwr#VPf\RV RV R24hVPW#4pVf\RV RV R24hW@PV&V#)zReturn a cached mesh, or create one via the callback. Args: key: Cache key ``(mesh_dim_names, mesh_layout)``. Returns: The ``DeviceMesh``. Raises: PipeliningMetadataError: If not cached and no callback provided. z+Mesh not found in cache for mesh_dim_names=z, mesh_layout=z`, and no get_mesh callback provided. Provide a get_mesh callback or use DTensors in static mode.z>Mesh lookup failed: callback returned None for mesh_dim_names=z6. Ensure all stages use meshes from the same universe.)rrr0)r%rrrrzs&& rget_mesh_MeshCache.get_meshns ++ ;;s# #&)#    $)=n=MN*m,NO    = <)""0!1 }MGH   C r"c$V^8dQhRRRRRR/#)rrr.rzrrrr)rs"rr rs!  | : $ r"c "W PV&R#)zAdd a mesh to the cache.Nr)r%rrzs&&&rput_MeshCache.puts Cr"c V^8dQhRRRR/#)rtensorstuple[torch.Tensor | None, ...]rrr)rs"rr rs , ,+J ,t ,r"c VFp\V\4'gKVPpVP'd\ VP4MRpVP pWE3pW`P 9gKqW0P V&K R#)zJExtract and cache meshes from any :class:`DTensor` instances in *tensors*.Nr)rBrrmrrprqr)r%rr=rz dim_namesrrs&& rupdate_from_tensors_MeshCache.update_from_tensorssiF&'**)):>:M:M:ME$"5"56SU "ll  .kk)'+KK$r"c V^8dQhRRRR/#)rrr.rr:r)rs"rr rs"" ""r"c WP9#r$r)r%rs&&r __contains___MeshCache.__contains__skk!!r"cV^8dQhRR/#)rrintr)rs"rr rs   r"c ,\VP4#r$)lenrrvs&r__len___MeshCache.__len__s4;;r"rr$) r(r)r*r+r,rrrrrrr-rr"rrrcs+ ( D  ,"  r"rc8]tRtRtRtRtRt]RRl4tRt R#) InferenceModeiaPipeline-level metadata inference mode, determined collectively across all PP ranks. The mode is set by the schedule (not individual stages) because ``has_backward`` is only known at schedule creation time and all stages must agree to avoid P2P hangs. .. attribute:: STATIC All stages have sufficient metadata; runtime inference is skipped. .. attribute:: DYNAMIC At least one stage requires runtime metadata inference. staticdynamicc$V^8dQhRRRRRR/#)rrrstage_has_backwardr:rr)rs"rr InferenceMode.__annotate__s!$r"c VP4'gR#VP4'gR#V'gR#VPeVPfR#R#)zDetermine whether dynamic metadata inference is needed for a stage. Args: meta: Stage metadata from user-provided args. stage_has_backward: Whether a backward pass will be performed. Returns: ``True`` if dynamic inference is needed. TF)rrrr)clsrrs&&&r needs_dynamicInferenceMode.needs_dynamicsS++--  """    #t'8'8'@r"rN) r(r)r*r+r,STATICDYNAMIC classmethodrr-rr"rrrs% FGr"rdetachFcV^8dQhRR/#)rrr:r)rs"rr r s$r"c \V4wr#V'diVUu.uFNp\V\P4'd*VP 4P VP 4MTNKP pp\WS4pWe3#V#uupi)aFlatten ``args`` into a list, optionally detaching tensors. Args: args: Nested arguments to flatten. detach: If ``True``, detach tensors while preserving ``requires_grad``. Returns: ``(new_args, flat_detached_args)`` when ``detach=True``; ``flat_args`` list otherwise. )rrBr`TensorrrLr;r)argsr flat_argstreespeca flat_detachednew_argss&$ r flatten_argsrs't,I   !U\\** HHJ % %aoo 6   "-:&&  sABc\VRR7#)zHFlatten and detach. Deprecated: use ``flatten_args(args, detach=True)``.T)r)r)rs&rflatten_args_detachrs T **r"c(V^8dQhRRRRRRRR/#)rpp_sizer num_stagesstylestrrzdict[int, int]r)rs"rr r s, !*-r"cl/pVR8Xd\V4F pW@,W4&K V#VR8XdyW,^8wd\RV RV R24h^p\V4FFpWSV&V^,V,^8XdKW@,^,^8Xd V^, pK=V^,pKH V#\RV R24h)z Compute the stage id to rank mapping for either a looped or V-style schedule. Most commonly num_stages == pp_size * 2, but this function can be used to compute the mapping for any number of stages per rank. looprz num_stages z% must be evenly divisible by pp_size z for V scheduleszStyle z is not supported.)range ValueError)rrrmapping stage_index rank_indexs&&& rgenerate_stage_to_rank_mappingrsG  ,K#.#8G -( N% #  1 $j\)NwiWgh   ,K#-K a7*a/&!+q0a a - N6%(:;< list of stage indices assigned to that rank. )ritemsrVvaluessort)rrr stage_to_rankrank_to_stagesstage_idrankstagess&&& rgenerate_rank_to_stage_mappingr%sq37NM,.N'--/  %#%N ##H-0 !'') * r"c:]tRtRt$RtR]R&R]R&R]R&R tR #) PipeInfoi?z6 Captures information for a pipeline (`Pipe` object). zfx.Graphgraphrrr:has_loss_and_backwardrNrrr"rrr?s OOr"rc V^8dQhRRRR/#)rr=r>rrr)rs"rr r Os// //r"c\V\4'd\PV4#\P V4#)aExtract metadata from a tensor. Handles both plain Tensor and DTensor correctly: DTensors are dispatched to ``_DTensorMeta.from_dtensor`` which captures local shard attributes plus global shape/placement info, while plain tensors use ``_TensorMeta.from_tensor``. Args: tensor: A plain tensor or DTensor. Returns: ``_TensorMeta`` for plain tensors, ``_DTensorMeta`` for DTensors. )rBrr^rrr4rDrCs&rextract_tensor_metar Os3&'""((00&&v..r" allow_nonec$V^8dQhRRRRRR/#)rrtuple[torch.Tensor, ...] | Noner!Literal[False]rrr)rs"rr r ds((( ,((# (r"cR#r$rrr!s&$rextract_tensor_metasr'cs %(r"c$V^8dQhRRRRRR/#)rr&tuple[torch.Tensor | None, ...] | Noner! Literal[True]rrr)rs"rr r ls(// 3//* /r"cR#r$rr&s&$rr'r'ks ,/r"c$V^8dQhRRRRRR/#)rrAtuple[torch.Tensor | None, ...] | tuple[torch.Tensor, ...] | Noner!r:rrr)rs"rr r ss(!"!" N!"!"* !"r"cVfR#.pRpVFRp\V\P4'dVP\ V44K?RpVPR4KT V'gV'd \ R4h\ V4#)ajExtract metadata from a tuple of tensors. Args: tensors: Tuple of tensors (may include ``None`` when ``allow_none=True``). allow_none: If ``True``, preserve ``None`` elements (for gradients). Returns: Tuple of ``TensorMeta``, or ``None`` if ``tensors`` is ``None``. Raises: PipeliningMetadataError: If ``None`` found and ``allow_none=False``. NFTz_None values are not allowed in tensor metadata tuples. Use allow_none=True for optional values.)rBr`rrVr r0rp)rr!metas_with_nonehas_nonerMs&$ rr'r'ss~"/1OH  a & &  " "#6q#9 :H  " "4 (  (% 7    !!r"c$V^8dQhRRRRRR/#)rr=r>rr:rr)rs"rr r s!!! !d!|!r"cV'dVP4MTp\V\4'dVP4#V#)uConvert a DTensor to its local shard, or return a plain tensor as-is. When ``detach=True``, the tensor is detached before conversion — this applies to both DTensors and plain tensors. Args: tensor: A tensor that may be a DTensor. detach: If ``True``, detach before ``to_local()`` to avoid redistribution during backward. Returns: The local tensor component. )rrBrto_local)r=rmaybe_detached_tensors&& rto_local_if_dtensorr5s706FMMO6'11$--//  r"c$V^8dQhRRRRRR/#)rrzCtorch.Tensor | tuple[torch.Tensor, ...] | list[torch.Tensor] | Noner!r$rr#r)rs"rr r s&** M**%*r"cR#r$rrr!s&&rvalidate_and_normalize_to_tupler9s'*r"c$V^8dQhRRRRRR/#)rrzQtorch.Tensor | tuple[torch.Tensor | None, ...] | list[torch.Tensor | None] | Noner!r*rr)r)rs"rr r s*11 1  1 , 1r"cR#r$rr8s&&rr9r9s.1r"c$V^8dQhRRRRRR/#)rrztorch.Tensor | tuple[torch.Tensor, ...] | tuple[torch.Tensor | None, ...] | list[torch.Tensor] | list[torch.Tensor | None] | Noner!r:rr-r)rs"rr r s+- -  - - G- r"c VfR#\V\P4'dV3#\V\\34'd\ V4Fgwr#VfV'g\ RV R24hK"\V\P4'dKD\ RV R\V4P R24h \V\4'd \V4#T#\ R\V4P R24h)aNormalize ``args`` to a tuple and validate that all elements are tensors. Args: args: A single tensor, tuple/list of tensors, or ``None``. allow_none: If ``True``, permit ``None`` elements (for gradients). Returns: Tuple of tensors, or ``None`` if ``args`` is ``None``. Raises: PipeliningMetadataError: On non-tensor values (or ``None`` when ``allow_none=False``). Nz Stage arg[zF] is None. Stage args must be tensors. Use kwargs for optional values.z ] has type zC. All stage args must be tensors. Use kwargs for non-tensor inputs.z$>%J4PT:K^K^J__` a  r"raise_on_mismatchwarn_on_mismatchc 0V^8dQhRRRRRRRRR RR R /#) rdescrexpectedractualztorch.Tensor | TensorMetarDr:rErrSr)rs"rr r sLBB BB &B  B  BBr"c\V\P4'd \V4pMTp\ V4\ V4JdR\ V4P R\ V4P 2.pV'd\ V RV^, 24hV'd+\P!V RV^, R2\^R7V#VPV4pV'dbV'd \ V RRPV4 24hV'd3\P!V RRPV4 R2\^R7V#) a, Compare expected metadata against actual tensor or metadata. This is the unified validation/comparison function that uses get_diff() from metadata classes. Works with both plain tensors and DTensors. For plain tensors: compares shape/stride/dtype/requires_grad. For DTensors: compares all properties including global shape and placements. Args: desc: Description for error/warning messages. expected: Expected tensor metadata (_TensorMeta or _DTensorMeta). actual: Actual tensor or metadata to compare against. raise_on_mismatch: If True, raise PipeliningMetadataError on mismatch. warn_on_mismatch: If True, issue a warning on mismatch. Returns: List of differences (empty if metadata matches). Raises: PipeliningMetadataError: If raise_on_mismatch=True and differences exist. ztype: expected , got : z: Metadata type mismatch. z.. Using dynamically inferred metadata instead. stacklevelz; z: Metadata mismatch. ) rBr`rr rAr(r0warningswarn UserWarningrYjoin)rGrHrIrDrE actual_meta type_diffrXs&&&$$ rvalidate_metadatarUs3>&%,,'')&1   H~T+..d8n556fT+=N=W=W-?@?@   Lr"c 0V^8dQhRRRRRRRRR RR R /#) rrGrrHrrIz,tuple[torch.Tensor | TensorMeta | None, ...]rDr:rErrSr)rs"rr r 6sD66 6+6 96  6  66r"c h\V4\V48wdXV R\V4 R\V4 2pV'd \V4hV'd\P!V\^R7V.#.p\ \ WRR74FwpwrVfV fKVeV feT RT RVfRMR R V fRMR 2pV'd \V4hV'd\P!V\^R7VPV4K~\V RV R 2VV VVR 7p VPV 4K V#) aValidate metadata for a tuple of tensors element-wise. Args: desc: Description prefix for error/warning messages. expected: Tuple of expected metadata (may include ``None`` for grads). actual: Tuple of actual tensors or metadata to compare against. raise_on_mismatch: If ``True``, raise on the first mismatch. warn_on_mismatch: If ``True``, issue warnings for mismatches. Returns: Aggregated list of difference strings. Raises: PipeliningMetadataError: If lengths differ or on mismatch. z : expected z tensors, got rMTstrict[z ]: expected rmetadatarK])rDrE) rr0rOrPrQr@ziprVrUextend) rGrHrIrDrEmsg all_diffsrBexpactrXs &&&$$ rvalidate_tensors_metadatarc6s6. 8}F #k#h-s6{mL )#. .  MM#{q 9u I"3x#EF :C ;3;  ;#+&!L3;J(OP!$v*=? !-c22 c;1=   S ! !fAaSN  /-   +G, r"c ,V^8dQhRRRRRRRRR R /#) rr rrztuple[torch.Tensor, ...]gradsris_inputr:rrr)rs"rr r osADDD "D +D D  Dr"c V'dRMRpV R2pV R2p\V4\V48wd.\RV RV R\V4 RV R\V4 R 2 4h\\WR R 74EF wpwrVP'g6V e2\RV RV R V RV R V R\ V 4P R2 4hVP'd2V f.\P!RV RV R V RV R V R2 \^R7\V\4'gKVP'gKV fK\V \4'dK\RV RV R V RV R V R\ V 4P R2 4h R #)u Validate the args↔grads contract for static mode. Enforces four rules for each (arg, grad) pair: 1. len(args) must equal len(grads). 2. If arg.requires_grad is False, grad must be None. 3. If arg.requires_grad is True and grad is None, emit a warning (this is legal at pipeline boundaries but may indicate a bug). 4. If arg is a DTensor with requires_grad=True and grad is not None, grad must also be a DTensor. Args: stage_index: The stage index for error messages. args: Tuple of forward tensors. grads: Tuple of gradient tensors (can include None). is_input: True for input_args/input_grads, False for output_args/output_grads. Raises: PipeliningMetadataError: If any hard rule (1, 2, or 4) is violated. inputoutput_args_gradszStage rLz length (z) does not match zo). Each forward tensor must have a corresponding gradient entry (use None for tensors that don't require grad).TrXNrZz] has requires_grad=False, but z] is not None (zE). Non-differentiable tensors must have None as their gradient entry.z] has requires_grad=True, but zT] is None. This is legal at pipeline boundaries but may indicate a missing gradient.rMz,] is a DTensor with requires_grad=True, but z] is za, expected DTensor or None. DTensor gradients may have different placements than forward tensors.) rr0r@r]r;rAr(rOrPrQrBr) r rrerfkind args_name grads_namerBrCgrads &&&& r'validate_static_arg_grad_correspondencerpos47HD&I6J 4yCJ%[MJ__conditional_annotations__ __future__rloggingrO dataclassesrrenumrtypingrrr r r r r`r torch.distributed._mesh_layoutrtorch.distributed.tensorrtorch.utils._pytreerrtorch.distributed.device_meshr(torch.distributed.tensor.placement_typesr getLoggerr(loggerrrprr.r[ RuntimeErrorr0r4r^rrrrrJrrrrrrrrr r'r5r9rUrcrp)rqs@rrsZ#"(NN 6,<8B   8 $h c3ht1C CD iDGlG $d#HH$HV $d#@;@$@H$l2 I2  DDD6 $d#**$*  $d#  $ *60@ @ P0D0p6+ D4    /( ("%( ( / /!"!"H!( * *  1 1- jB $ B # BJ6 # 6 # 6rDr"