+ je^RIt^RIt^RIt^RIt^RIt^RIt^RIHtHt^RI H t ^RIH t ^RI H t ^RIHtHt^RIHt^RIHtHtHt^R IHt^RIt^RIHt^RIHt^RIHuH t!^R I"H#t#^R I$H%t%^R I&H't'H(t(H)t*^R I+H,t,^RI-H.t.H/t/^RI0H1t1^RI2H3t3H4t4H5t5^RI6H7t7^RI8H9t9H:t:H;t;^RIt>H?t?^RI@HAtAHBtBHCtCHDtD^RIHEtEHFtF^RIGHHtI^RIJHKtKHLtLHMtMHNtNHOtO^RIPHQtQHRtRHStSHTtTHUtUHVtVHWtW^RIXHYtY]V'd*]Z!^][!^]PP444t^M^t^]T'd Rt_Rt`]PP4t^M:]U'dRt_Rt`M-]W'd Rt_Rt`]PP4t^MR t_R!t`^t^!R"R#]4tb!R$R%]4tc!R&R']P]4teR(R)ltfR*tgRtR+R,llthRuR-ltiR.tjR/tkRvR0R1lltlR2R3ltmR4R5ltn!R6R74to!R8R9]e4tp!R:R;]e4tq!R<R=]q4tr!R>R?]q4ts!R@RA]e4tt!RBRC]t4tu!RDRE]P4tv!RFRG]q4tw!RHRI]P4tx!RJRK]P4tz!RLRM]P4t{]PRNROl4t}]PRPRQl4t~]PRRRSl4t]PRTRUl4t]PRVRWl4t]]PRXRYl44t]]PRZR[l44t]]PR\R]l44t]]PR^R_l44tR`RaltRwRbRcllt!RdRe]M4t!RfRg4t!RhRi]]L4t!RjRk]]K4tRxRlRmllt!RnRo]P4t!RpRq]P4t!RrRs]P4tR#)yN)ABCabstractmethodCallable) nullcontext)deepcopy)autoEnumwraps)Anycast no_type_check)mock) checkpoint) DeviceMesh) CPUOffload fully_shardFullyShardedDataParallel) TrainingState)FSDPParamGroupRegisterPostBackwardFunction)#NO_RESHARD_AFTER_FORWARD_STRATEGIES)BackwardPrefetchMixedPrecisionShardingStrategy)ShardedGradScaler)always_wrap_policyModuleWrapPolicywrap)distribute_tensorDTensorShard)ColwiseParallelparallelize_moduleRowwiseParallelSequenceParallel)TransformerDecoderLayerTransformerEncoderLayer)DistributedDataParallel)MultiProcContinuousTestMultiProcessTestCaseMultiThreadedTestCase run_subtests TEST_SKIPS) FILE_SCHEMAget_cycles_per_ms set_rng_seed TEST_CUDATEST_HPUTEST_WITH_ROCMTEST_XPU) has_tritoncudancclzhpu:0hcclxpuxcclcpuglooc2]tRt^]t]!4t]!4tRtR#) FSDPInitModeN)__name__ __module__ __qualname____firstlineno__r NO_FSDP RECURSIVE__static_attributes__rAd/home/luca/screen-checking/.venv/lib/python3.14/site-packages/torch/testing/_internal/common_fsdp.pyr@r@]sfGIrIr@c@]tRt^ft]!4t]!4t]!4tRtR#)DEVICEInitModerAN) rBrCrDrEr DEVICE_BEFORE DEVICE_AFTER DEVICE_NEVERrHrArIrJrLrLfsFM6L6LrIrLca]tRt^otoRt]V3RlRl4t]V3RlRl4t]V3RlRl4t] ]V3RlR l44t R t Vt R #) FSDPTestModelzVThis defines the interface expected from all models used commonly for FSDP unit tests.cH<V^8dQhRS[S[PR3,/#)return.)tupletorchTensor)format __classdict__s"rJ __annotate__FSDPTestModel.__annotate__ts"  5s):#; rIc R#)z*Returns an input for the model as a tuple.NrAselfdevices&&rJ get_inputFSDPTestModel.get_inputs rIc4<V^8dQhRS[P/#rSrTrVrW)rXrYs"rJrZr[ys   rIc R#)z,Returns the loss given the input and output.NrA)r^inputoutputs&&&rJget_lossFSDPTestModel.get_lossxrbrIc<V^8dQhRR/#rSrTNrA)rXrYs"rJrZr[~s  D rIc R#)zwr#VP\P!R48wgK,VP4W&K@ V^8XdTMR.p\P !V4\ \\\P3,V^,4pVF pW,P\4W&K" V#)r=N) itemsr_rVr=rbroadcast_object_listrdictrrWrr)rankrrrrs&& rJ_broadcast_state_dictrs(--/ <<5<<. .%*YY[J "0 19Z$ /Eu%d3 ,-uQx8J !+!7!:!:;!G ! rIcDV^8dQhR\PR\/#)rSrrecurser)rXs"rJrZrZs 2 2299 2t 2rIc\P!WR7;_uu_4\\VP 444uuRRR4# +'giR#;i)a? Returns the full unsharded parameters of ``model``. Any FSDP-managed parameters offloaded to CPU are moved to GPU in the returned list. Args: recurse (bool): If ``False``, only unshards the parameters immediate to ``model``; if ``True``, recurses through the module hierarchy rooted at ``model``. )rN)rrrlistr)rrs&&rJget_full_paramsrs:   8 8U--/01 9 8 8 8s "A  A cDV^8dQhR\PR\/#)rSrmove_to_devicer)rXs"rJrZrZs>>299>d>rIc@V'dVP\4#T#r)rr)rrs&&rJ_move_to_devicers$2588K ==rIcDV^8dQhR\PR\/#)rSr wrap_fsdpr)rXs"rJrZrZs"DDBIID$DrIc4V'gV#\V.VO5/VB#rr)rrrsrts&&*,rJ_maybe_wrap_fsdprs!5CtE'CD'CF'CCrIc\a]tRt^toV3RlRltV3RlRltV3RlRltRtRtVt R #) DummyProcessGroupc&<V^8dQhRS[RS[/#)rSrsizeint)rXrYs"rJrZDummyProcessGroup.__annotate__sSrIcWnW nR#r_rank_size)r^rrs&&&rJ__init__DummyProcessGroup.__init__s   rIc <V^8dQhRS[/#rdr)rXrYs"rJrZrcrIcVP#r)rr^s&rJrDummyProcessGroup.rank zzrIc <V^8dQhRS[/#rdr)rXrYs"rJrZrrrIcVP#r)rrs&rJrDummyProcessGroup.sizerrIcB\P!4pRpWCnV#)cd\PP4pVP^4V#)r)rVfuturesFuture set_result)futures rJ get_future/DummyProcessGroup.allreduce..get_futures'+0==+?+?+AF   a MrI)rMockr)r^rsrt dist_waitrs&*, rJ allreduceDummyProcessGroup.allreduces IIK   *rIrN) rBrCrDrErrrrrHr|r}s@rJrrs-  rIrc|aa]tRtRtoV3RlV3RlltRtRtRtRt] R V3RlR ll4t R t R t Vt V;t#) TransformerWithSharedParamsi cF<V^8dQhRS[PRS[RS[RS[/#)rSrdevice_init_modeadd_bn deterministic)rrrLr)rXrYs"rJrZ(TransformerWithSharedParams.__annotate__ s9((  ()( (  (rIc<\SV`4VP4VnVP4VnV'd\ P !^4^p^p\P!WV4Vn \P!V^^^RR7Vn \P!We4Vn VPPVPnVPRVPPP!V344VPR\ P"!VP$\ P&R74^VnV'd*\ PP+VP(4M\ PP-4VnV\0P28XdVP5\64pV'dVP94R#R#)rg?)d_modelnum_encoder_layersnum_decoder_layersdim_feedforwarddropout vocab_bias long_buffer)dtypeN)superrrrrrV manual_seedru Embedding embed_tokens Transformer transformerLinear output_projweightregister_buffernew_ones zeros_likerlongbs BatchNorm1dIdentitybnrLrMrreval)r^rrrrd_vocabr __class__s&&&&& rJr$TransformerWithSharedParams.__init__ s] JJL **,    a LL:>>    99W6#'"3"3":":  $++22;;WJG      T__EJJ ? 39%((&&tww/uxx?P?P?R ~;; ;77;'D  IIK rIc@\P!^VP,4\P!^ VR7P ^VP 4p\P!VP ^,VR7P ^VP 4pW#3#rr_)rVrrarangeviewr)r^r_srctgts&& rJr`%TransformerWithSharedParams.get_input7sj !dii-(ll2f-221dgg>ll477Q;v6;;AtwwGzrIcVPV4pW0P,VPPV4,pVPV4pVP V4pVP W44pVP V4#r)rrrtype_asrr r )r^src_idstgt_idsrrxs&&& rJforward#TransformerWithSharedParams.forward=sr(OO#d&6&6&>&>s&CC(ggcl   S &""rIcVwr4\PPVPRVP R44VPR4RR7#)rsum) reduction)ru functional cross_entropyrr)r^rgrhrrs&&& rJri$TransformerWithSharedParams.get_lossEsG}}** KKFKKO ,chhrle+  rIc&VP4R#rbackwardrns&&rJrp(TransformerWithSharedParams.run_backwardK  rIc<V^8dQhRS[PRS[RS[RS[S[S[3,R,RS[RS[RS[PS[ ,/#) rSrfsdp_init_moder fsdp_kwargsNrrrT) rrr@rLrrr rrurvr)rXrYs"rJrZrOs}KJKJ  KJ$KJ)KJ#s(^d* KJ  KJ  KJ T KJrIc Vf/pV\P8Xd/\V\4'd V^,pMTp\ WbWT4#V\P 8XdRV9d\ \\04pMVPR4pRV9dFVR,\P\P09d\V\4'gRpMTp\V\4'd V^,p MTp \ WWT4p \V V3RV/VBp V\P8XdV P!\"4p V #\%RV 24h)a Initializes a :class:`TransformerWithSharedParams` instance. Args: fsdp_init_mode (FSDPInitMode): If ``NO_FSDP``, then does not wrap any modules with FSDP. If ``RECURSIVE``, then wraps with top-level FSDP. By default, the top-level FSDP uses the ``ModuleWrapPolicy`` for encoder and decoder layers, but a different auto wrap policy may be specified via ``fsdp_kwargs``. device_init_mode (DEVICEInitMode): Determines model movement to DEVICE. fsdp_kwargs (Optional[Dict[str, Any]]): Optional keyword arguments forwarded to the FSDP constructor. deterministic (bool): Whether to make the model deterministic across constructions. add_bn (bool): Whether to include batch norm in the model. Nauto_wrap_policysharding_strategyUnsupported FSDP init mode: )r@rF isinstancerUrrGrr)r(popr HYBRID_SHARD_HYBRID_SHARD_ZERO2rrLrNrr ValueError) rr5rr6rrpgr8fsdp_pg tformer_pgm fsdp_models &&&&&& rJrx TransformerWithSharedParams.initNsW6  K \11 1%''1X.f |55 5!4#3//$ $/??3E#F ${2 34$113C3W3WXY"5%00%''"1X " +fA"2 J  >#>#>>']];7  77GHIIrIcVP.#r)r rs&rJget_ignored_modules/TransformerWithSharedParams.get_ignored_moduless  !!rI)rrrr rr r)NFT)rBrCrDrErr`r&rirpr{rxrGrHr| __classcell__rrYs@@rJrr sN((T # KJKJKJZ""rIrcvaa]tRtRtoV3RlV3RlltRtRtRtRt] R V3RlR ll4t R t Vt V;t #) NestedWrappedModuleicF<V^8dQhRS[PRS[RS[RS[/#rSrrrrrrrrL)rXrYs"rJrZ NestedWrappedModule.__annotate__s9      )    rIc <aaa\SV`4SP4VnSP4VnV\ P 8HpVVV3RlpV'd\P!^4\P!\\P!^^4V4V!\P!V!\\P!^^4V44\\P!^^4V444V!\\P!^^4V44\\P!^^4V44Vn R#)c2<S'd\VS3/SB#V#rrlayerr6rrs&rJ _maybe_wrap1NestedWrappedModule.__init__.._maybe_wrapE58K88LrIN)rrrrrrLrMrVrru Sequentialrr module r^rrrrr6rrUrs &ff&&l rJrNestedWrappedModule.__init__s JJL **,)^-I-II     a mm BIIaO^ <   !R0@. QR#BIIb"$5~F   "a(8.I J BIIaO^ <   rIc\P!^VP,4\P!^^VR73#r)rVrrrandr]s&&rJr`NestedWrappedModule.get_inputs. !dii-( 1a/11rIc$VPV4#rrYr^r%s&&rJr&NestedWrappedModule.forward{{1~rIc&VP4pV#r)r)r^rgrhros&&& rJriNestedWrappedModule.get_lossszz| rIc&VP4R#rr0rns&&rJrp NestedWrappedModule.run_backwardr3rIc <V^8dQhRS[PRS[RS[RS[S[S[3,R,RS[RS[P/#)rSrr5rr6NrrT) rrr@rLrrr rrurv)rXrYs"rJrZrPsd+J+J  +J$+J)+J#s(^d* +J  +J  +JrIc  Vf/pV\P8Xd\VRVVR7#V\P8Xd?\V3RRRVRV/VBpV\P 8XdVP \4pV#\RV 24h)a Initializes a :class:`NestedWrappedModule` instance. Args: fsdp_init_mode (FSDPInitMode): If ``NO_FSDP``, then does not wrap any modules with FSDP. If ``RECURSIVE``, then wraps some nested modules with FSDP but not the top-level module. The model may later be wrapped with a top-level FSDP external to this method if desired. device_init_mode (DEVICEInitMode): Determines model movement to DEVICE. fsdp_kwargs (Optional[Dict[str, Any]]): Optional keyword arguments forwarded to the FSDP constructor. deterministic (bool): Whether to make the model deterministic across constructions. FrrrrTrrr:) r@rFrLrGrLrNrrr?)rr5rr6rrDs&&&&& rJrxNestedWrappedModule.inits.  K \11 1&!1+   |55 5,"2,   J >#>#>>']];7  77GHIIrIrYrrNF)rBrCrDrErr`r&rirpr{rxrHr|rIrJs@@rJrLrLsF  @2+J+J+J+JrIrLcLaa]tRtRto]RV3RlV3Rlll4tRtVtV;t#)AlwaysWrapNestedWrappedModuleic p<V^8dQhRS[PRS[RS[RS[S[S[3,R,RS[/#rSrr5rr6Nrrrr@rLrrr r)rXrYs"rJrZ*AlwaysWrapNestedWrappedModule.__annotate__sN  $)#s(^d*   rIc J<\\\` V\PVVVR7pV\P8XdV#V\P 8XdKT;'g/p\ V3R\/VBpV\P8XdVP\4pV#R#)z Initializes a :class:`NestedWrappedModule` instance, but unlike :meth:`NestedWrappedModule.init`, for the ``RECURSIVE`` init mode, this wraps with top-level FSDP and the ``always_wrap_policy()`` auto wrap policy. )rr5rr6rr8N) rrprxr@rFrGrrrLrNrr)rr5rr6rrrDrs&&&&& rJrx"AlwaysWrapNestedWrappedModule.inits )+H '//-#'    \11 1L |55 5%++KeX6HXKXJ>#>#>>']];7   6rIrArn) rBrCrDrEr{rxrHr|rIrJs@@rJrprps!rIrpczaa]tRtRtoV3RlV3Rllt]V3RlRl4t]R V3RlRll4tRtVt V;t #) NonUniformReqGradNWMi cF<V^8dQhRS[PRS[RS[RS[/#rNrO)rXrYs"rJrZ!NonUniformReqGradNWM.__annotate__!s9( (   ( ( ) (  ( rIc <aaa\\V` 4SP4VnSP 4VnV\ P8HpVVV3RlpV'd\P!^4\P!\\P!^^4V4V!\P!V!\\P!^^4V44\\P!^^4V444V!\P!\\P!^^4V4\\P!^^4V4444VnR#)c2<S'd\VS3/SB#V#rrrSs&rJrU2NonUniformReqGradNWM.__init__.._maybe_wrap4rWrIN)rrLrrrrrLrMrVrrurXrr rYrZs &ff&&l rJrNonUniformReqGradNWM.__init__!s !413JJL **,)^-I-II     a mm BIIaO^ <   !R0@. QR#BIIb"$5~F   #BIIb!$4nE#BIIaO^D   rIc<V^8dQhRR/#rlrA)rXrYs"rJrZrzLs(($(rIcVP4F4wr#\P!W4'dK#VPR4K6 R#)FN)rrematchrequires_grad_)r req_grad_masknps&& rJ_set_nonuniform_req_grad-NonUniformReqGradNWM._set_nonuniform_req_gradKs4**,DA88M--  '-rIc p<V^8dQhRS[PRS[RS[RS[S[S[3,R,RS[/#rrrs)rXrYs"rJrZrzRsU+J+J  +J$+J)+J#s(^d* +J  +JrIc  \P!R4pV\P8Xd'\ VRVVR7p\P We4V#V\P 8XdZVf/p\ V3RRRVRV/VBpV\P8XdVP\4p\P Wu4V#\RV 24h) a Initializes a :class:`NestedWrappedModule` instance, but unlike :meth:`NestedWrappedModule.init`, it wraps a second :class:`torch.nn.Sequential` container to enable the desired non-uniform ``requires_grad`` ``use_orig_params=True`` tests. For both ``RECURSIVE`` and ``NO_FSDP`` init modes, freezes all parameters except the last two to validate ``ShardedGradScaler`` support for ranks with no (non-zero sized) local shards in FSDP ``use_orig_params=True`` mode. zmodule\.2.*\.1.*FrkrTrrr:) rcompiler@rFrxrrGrLrNrrr?)rr5rr6rreq_grad_pattern ddp_modelrDs&&&&& rJrxNonUniformReqGradNWM.initQs(::&9: \11 1,!1+ I ! 9 9) V  |55 5" -"2,   J >#>#>>']];7 9 9* W 77GHIIrIrmrn) rBrCrDrErr{rrxrHr|rIrJs@@rJrxrx sF( ( T(( +J+J+J+JrIrxcvaa]tRtRtoRtV3RlV3RlltRtRtRtRt ] V3R lR l4t R t Vt V;t#) ModuleWithDelayizThis class wraps a :class:`FSDPTestModel` to optionally add a delay after computing the loss and/or before the gradient reduction.c@<V^8dQhRS[PRS[RS[/#)rSrYdelay_after_loss_msdelay_before_reduction_ms)rurvr)rXrYs"rJrZModuleWithDelay.__annotate__s-    ! $' rIcH<\SV`4W nW0nWnR#r)rrrrrY)r^rYrrrs&&&&rJrModuleWithDelay.__init__s! #6 )B& rIc8VPPV4#r)rYr`r]s&&rJr`ModuleWithDelay.get_inputs{{$$V,,rIc$VPV4#rr`ras&&rJr&ModuleWithDelay.forwardrcrIczVPPW4pVP^8d\'g \'d*\ P !VPR, 4V#\'dB\PP\VP\4,44V#r) rYrirr4r6timesleepr3rVr8_sleeprr1res&&& rJriModuleWithDelay.get_losss}{{##E2  # #a 'x88 433d:;  !!#d&>&>ARAT&T"UV rIcaa\PPoVV3Rlp\P!RV4;_uu_4SP P V4RRR4R# +'giR#;i)cP<SP^8d\'dC\PP \ SP\ 4,44M?\'g \'d(\P!SPR, 4S!V/VB#r) rr3rVr8rrr1r4r6rr)rsrtorig_reduce_scatterr^s*,rJ_delayed_reduce_scatter=ModuleWithDelay.run_backward.._delayed_reduce_scattersq--19JJ%%D::=N=PPQXJJt==DE&77 7rIz'torch.distributed.reduce_scatter_singleN)rV distributedreduce_scatter_singlerpatchrYrp)r^rorrsf& @rJrpModuleWithDelay.run_backwards\#//EE 8ZZ 57N   KK $ $T *    s A)) A: c H<V^8dQhRS[S[,RS[RS[RS[RS[/#)rS module_class model_argsrr model_kwargs)typerQr r)rXrYs"rJrZrs@  =)  ! $'    rIc <\VP!V/VBVV4#)a Args: module_class (Type[FSDPTestModel]): Wrapped module class to which to add delays. model_args: Positional arguments forwarded to the ``module_class`` ``init()``. delay_after_loss_ms (int): Delay after computing the loss/before the optimizer step (in ms). delay_before_reduction_ms (int): Delay before reduce-scattering gradients (in ms). model_kwargs: Keyword arguments forwarded to the ``module_class`` ``init()``. )rrx)rrrrrs&$$*,rJrxModuleWithDelay.inits**   z :\ :  %  rI)rrrY)rBrCrDrErzrr`r&rirpr{rxrHr|rIrJs@@rJrrsBF  -+$   rIrc^a]tRtRto]]P RR^^3V3RlRll4tRtVt R#)NestedWrappedModuleWithDelayiNFc|<V^8dQhRS[PRS[RS[RS[S[S[3,R,RS[RS[RS[/#) rSrr5rr6Nrrr rrr@rLrrr rr)rXrYs"rJrZ)NestedWrappedModuleWithDelay.__annotate__sb     $ ) #s(^d*    !  $' rIc D\P\VVVVVVVR7#)rr5rr6rrr)rrxrLrs&&&&&&&rJrx!NestedWrappedModuleWithDelay.inits4## )-#' 3&?$  rIrA) rBrCrDrEr{rLrNrxrHr|r}s@rJrrs3,:+F+F-1##$)*   rIrc8aa]tRtRtoV3RltRtRtVtV;t#)DummyDDPic0<\SV`4WnR#r)rrrY)r^rYrs&&rJrDummyDDP.__init__s  rIc&VP!V/VB#rr`r^rsrts&*,rJr&DummyDDP.forwards{{D+F++rIr` rBrCrDrErr&rHr|rIrJs@@rJrrs,,rIrcjaa]tRtRtoV3RlV3RlltRtRt]R V3RlRll4tRt Vt V;t #) MixtureOfExpertsic L<V^8dQhRS[PRS[RS[RS[RS[/#)rSrrrdelay_before_free_msr)rrrrLr)rXrYs"rJrZMixtureOfExperts.__annotate__sC2 2   2 2 ) 2 " 2  2 rIc <\SV`VVVVR7WnW@nW nV\ P 8HVnV'd(\P!^*VP,4^p^ p^p \\P!Wx4VP4p \RV P444VnV P4F p RV nK V'd\P!^4\\P!W4VP4p V'dG\P$P'VP4.4p \)W3/VBp \)W3/VBp \P*!\\P!W4VP4V V \\P!W4VP44VnR#))rrrrc3@"TFqP4xK R#5ir)numel).0rs& rJ ,MixtureOfExperts.__init__.. s$L8K1WWYY8KTN)rrrrrrLrMrrVrrrrur r)rnum_expert_paramsexpertr new_grouprrXrY)r^rrrrrr6d_expertd_sharedd_inputrrshared expert_grouprs&&&&&&, rJrMixtureOfExperts.__init__s{ -'   $8!"..2N2NN    b499n - 8!>@S@ST!$$L8I8I8K$L!L""$AAH%    a  8!>@S@ST  ,,66L&>+>F&7;7Fmm BIIg8$:M:M N   BIIh8$:M:M N   rIcaaSP^8dSP^,p\V\4'dq\P P PPoVV3Rlp\P!RV4;_uu_4SPV4uuRRR4#SPV4# +'giL!;i)rc.<\'dC\PP\ SP \ 4,44M?\'g \'d(\P!SP R, 4S!V/VB#)r) r3rVr8rrrr1r4r6rr)rsrt orig_reshardr^s*,rJ_delayed_reshard2MixtureOfExperts.forward.._delayed_reshard(sc y )) 9 9 {{1~ s B>> C cVP4VP'g\P!4;_uu_4VP 4Fp\ VR4'dKVP fK'VP PVP4\PPVP VPR7K RRR4R#R# +'giR#;i)rNr) r1rrVrrhasattrgraddiv_rr all_reducer)r^rors&& rJrpMixtureOfExperts.run_backward:s ~~~*Aq(++ vv) DOO4))44QVV4::4N +! s3C"8AC"" C3 c v<V^8dQhRS[PRS[RS[RS[S[S[3,R,RS[RS[/#)rSrr5rr6Nrrr)rXrYs"rJrZrGs`0J0J  0J$0J)0J#s(^d* 0J  0J " 0JrIc  Vf/pV\P8Xd\VRVVVR7#V\P8XdA\V3RRRVRVRV/VBpV\P 8XdVP \4pV#\RV 24h) a Initializes a :class:`MixtureOfExperts` instance. Args: fsdp_init_mode (FSDPInitMode): If ``NO_FSDP``, then does not wrap any modules with FSDP. If ``RECURSIVE``, then wraps some nested modules with FSDP, including the expert and shared layers, but not the top-level module. The model may later be wrapped with a top-level FSDP external to this method if desired. device_init_mode (DEVICEInitMode): Determines model movement to DEVICE. fsdp_kwargs (Optional[Dict[str, Any]]): Optional keyword arguments forwarded to the FSDP constructor. deterministic (bool): Whether to make the model deterministic across constructions. delay_before_free_ms (int): Delay before resharding expert parameters in the forward pass (in ms). F)rrrrrTrrrr:) r@rFrrGrLrNrrr?)rr5rr6rrrDs&&&&&& rJrxMixtureOfExperts.initFs4  K \11 1#!1%9+  |55 5)"2&:  ,   J >#>#>>']];7  77GHIIrI)rrrYrrr)NFr) rBrCrDrErr&rpr{rxrHr|rIrJs@@rJrrs=2 2 h0 O0J0J0J0JrIrcjaa]tRtRtoR RRRRR^/V3RlV3RlllltV3R lR ltR tR tVtV;t #)MLPizbiasT with_bufferFdim_multiplierc Z<V^8dQhRS[RS[PR,RS[RS[RS[/#)rSdimr_Nrrr)rrVr_r)rXrYs"rJrZMLP.__annotate__{sF  t#    rIc(<\SV`4\P!WV,W#R7Vn\P!WQ,WVR7VnV'd,VP R\P!V3VR74R#RVn R#))r_rrrN) rrrur in_projout_projr rVrandnr)r^rr_rrrrs&&&$$$rJr MLP.__init__{sf yys&:6U  ."6QUV   5;;vf+M NDKrIcN<V^8dQhRS[PRS[P/#rSr%rTre)rXrYs"rJrZrs#%,,rIcVPV4p\P!V4pVPV4p\P!V4pVPeW P,pV#r)rFrelurr)r^r%zs&& rJr& MLP.forwardsQ LLO FF1I MM!  FF1I ;; "KKArIcVPe6\PPP VP4R#R#r)rrVrurxnormal_rs&rJreset_parametersMLP.reset_parameterss+ ;; " HHMM ! !$++ . #rI)rrrr) rBrCrDrErr&rrHr|rIrJs@@rJrrzsA   "  "//rIrcXaa]tRtRtoRR/V3RlV3RllltV3RlRltRtVtV;t#) MLPStackiwith_seq_parallelFc&<V^8dQhRS[RS[/#)rSmlp_dimrrr)rXrYs"rJrZMLPStack.__annotate__s 3 3 34 3rIc<\V^R7\V4\V^R7.pV'd(VP\P!VRR74\SV`!V!W nR#))rFrN)rappendru LayerNormrrr)r^rrmodulesrs&&$ rJrMLPStack.__init__sV  * L  * $   NN2<<e< = '"!2rIc0<V^8dQhRS[RS[RS[RR/#)rStp_meshdp_meshuse_activation_checkpointingrTr)rr)rXrYs"rJrZrs3'+   rIc R\RR7R\RR7R\RR7R\RR7R\RR7RVP'd\\^4R7M \4/pVP'd\ ^R 7VR &\ WVR 7VFFp\ V\P4'dK%V'd \V4\V3R V/VBKH \V3R V/VBV#) z 0.in_projF)use_local_outputz 0.out_projz 1.in_projz 1.out_projz 2.in_projz 2.out_proj)output_layouts) sequence_dim3) device_meshparallelize_planmesh) r$r&rr#r'r%r;rur rr)r^rrrr6rrYs&&&&, rJ parallelizeMLPStack.parallelizes %@ /5A %@ /5A %@ %%%*qB "    ! ! !$4!$D S !4GWXF&",,//+6"  >X>rIc#"\Pp\P!4V\nRx\P!4V\nR# \P!4T\ni;i5ir)rr1rr/)r_ orig_backwards& rJ*patch_register_post_backward_hook_backwardrbQsS199MLLN,8 )>  0=$- 0=$-r2cHV^8dQhR\R\R\R\/#)rSrrrsrt)rr )rXs"rJrZrZ^s200!00  0  0rIc\V4^8d V^,pM"RV9d VR,pM\RV RV 24hV!V4V!V/VB#)rrhz,Cannot get reduce-scatter output from args: z kwargs: )lenr)clsrrrsrtrhs&&&*, rJreduce_scatter_with_assertrg^sa 4y1}a V !;D6F8 T  f  / //rIc~V^8dQhR\PR\PR\\R3,/#)rSreplicated_modulesharded_moduleprefixes_to_ignore.)rurvrUr)rXs"rJrZrZqs="T"Tyy"TII"Tc3h "TrIc\VP4VP4RR7EFwwrEwrgTpVFp VPV R4pK VPWH4VP V\ 4\ V\ 4'g \R4hVPVPr\V 4\^4\^438Xd \R4h\WZV 4p VPVP4V P44VPfVPVP4EKVP!VP4\VPW4p VP VP\ 4\ VP\ 4'g \R4hVPVPP4V P44EK R#)Trz&Expected sharded_param to be a DTensorzmFSDP's (Shard(0), Shard(0)) layout differs from distribute_tensor(), so we cannot check for equality using itNz+Expected sharded_param.grad to be a DTensor)rrreplace assertEqualassertIsInstancer"r;rr placementsrUr#r!to_localr assertIsNoneassertIsNotNone)rfrirjrkreplicated_namereplicated_param sharded_name sharded_paramclean_sharded_nameprefixrrqsharded_ref_paramsharded_ref_grads&&&& rJcheck_sharded_parityr}qs OR**,'')OOJ+-Jl *(F!3!;!;FB!G ) < ]G4-11 !IJ J(44m6N6Nj  q58 4 4 ; ..>jQ  ..02C2L2L2NO  (   ]// 0  M../,-=-B-BDU ]//9-,,g66 !NO O  **3357G7P7P7RS9OrIcTaa]tRtRto]R4tV3RltRtRtRt Rt Vt V;t #)FSDPTestMultiThreadic\#r DEVICE_COUNTrs&rJrFSDPTestMultiThread.world_sizerIcD<\SV`4VP4R#r)rsetUp_spawn_threadsr^rs&rJrFSDPTestMultiThread.setUps   rIc \V.VO5/VB#rr.rs&*,rJr. FSDPTestMultiThread.run_subtestsD242622rIcB\PP4R#rrV_dynamoresetrs&rJperThreadSetUp"FSDPTestMultiThread.perThreadSetUp rIcB\PP4R#rrrs&rJperThreadTearDown%FSDPTestMultiThread.perThreadTearDownrrIrA) rBrCrDrEpropertyrrr.rrrHr|rIrJs@@rJrrs0 3rIrc a]tRtRtoRtRtRtRtRt] R4t RV3R lR llt R^R ] !4RRRR R R R RR3 V3R lRllt RtVtR#) FSDPTestMixiniz Mixin class containing shared test utilities for FSDP tests. Provides common helper methods for both FSDPTest and FSDPTestContinuous. c<VPW!P4R#r)ror)r^rDrs&&&rJ_check_cpu_offload FSDPTestMixin._check_cpu_offloads &<&<=rIc<VPW!P4R#r)robackward_prefetch)r^rDrs&&&rJ_check_backward_prefetch&FSDPTestMixin._check_backward_prefetchs *,H,HIrIc<VPW!P4R#r)roforward_prefetch)r^rDrs&&&rJ_check_forward_prefetch%FSDPTestMixin._check_forward_prefetchs )+F+FGrIc \V.VO5/VB#rrrs&*,rJr.FSDPTestMixin.run_subtestsrrIc dV!V4pWnW6nVPRR4p\RVP RVP 24\ R8wdf\ PP4VP8d9\P!\RVP 2,P4V'db\ PPPP P#4p\$P&!RVPVVR7MF\$P&!VP(\*\-VP4VPR7R p VP\2,p \4'g \6'd \ PP9V 4V .p \$P:!V R 7\ P<P?4\A4VPCW$4\ P<P?4\$P:!V R 7\$PD!4R # \.dJp R T P0^,9d,\P!\R ,P4hR p ? ii;i) fake_pgF dist init r=, world=r= multi-gpu-fakebackendrrstore init_methodrrr recompilebackend_unavailableN device_ids)#r file_namegetprintrrrV accelerator device_countsysexitr/ exit_codetesting _internalrr FakeStorerinit_process_grouprDISTRIBUTED_BACKENDr RuntimeErrorrsrr3r6set_device_indexr/rrr2run_testdestroy_process_group rfr test_namerpipertr^rrer device_ids &&&&&, rJ_runFSDPTestMixin._runs9~ "**Y. TYYKx/@AB % E$5$5$B$B$Dt$V HHZ*T__,= >?II J  //;;CCMMO''"# '' $ 0 0/"4??3  II , 9    . .y 9[ +  i&  + ""$1 affQi'$9:DDE   s'=IA I&AI J/&AJ**J/NFc<V^8dQhRS[PRS[RS[RS[RS[R,RS[RS[R,R S[R S[R S[S[S[ 3,R,/ #) rSr num_stepsautocastlrfsdp_cpu_offloadN save_modelmixed_precisionenable_sharded_grad_scaler use_pure_fp16sharded_grad_scaler_kwargs) rurvrrfloatrrrrr )rXrYs"rJrZFSDPTestMixin.__annotate__sYYyyYY Y  Y %t+ YY($.Y%)YY%)cNT$9YrIc * T;'d VPp \VP44Pp V f/p \ R RV/V Bp \ P PVP4VRR7p\V4EFpVP4\ PP\VR7;_uu_4VPP\ P!\44pV 'gV'dt\V\ 4'g^\V\ P"4'dVP%4pM,\&;QJd.RV4FNK 5M !RV44pV!V!pV 'ds\V\ 4'd]VP(\*9dHVP4F3pVP-VP\ P!R44K5 VPP/VV4P1V 4pRRR4V P3X4pV'g4V 'g,VP4\ P68wd \9R4hMV 'd,VP-VP4\ P:4Mv\V\ 4'd7Vf \9R4hVP-VP4VP<4M*VP-VP4\ P64VPP?V4V 'd^\V\ 4'dHVP4F3pVP-VP\ P!R44K5 V PAV4V PC4V'gEKLVPE4PG4UUu/uFwppVVPI4bK ppp\KV4VPMV4EK \V\ 4'd VPO\PPR4XPU4# +'giELY;iuuppi) Nenabledg?)rmomentum)rc3@"TFqP4xK R#5ir)r)rr%s& rJr9FSDPTestMixin._train_for_several_steps..s%>1ffhhrr=zQloss data type should be float32, as the original parameter data type is float32.z'Expected mixed_precision to not be NonerA)+offload_paramsnextrr_rrVoptimSGDr zero_gradamprrrYr`r;rrWrrUr9rrorirscalerfloat32rfloat16 param_dtyperpstepupdaterrclonerload_state_dict _assert_staterIDLEr)r^rrrrrrrrrrcpu_offload_params model_devicesharded_grad_scalerrrrgrhrrokvrs&&&&&&&&&&& rJ_train_for_several_steps&FSDPTestMixin._train_for_several_stepssW.QQ2B2Q2QE,,./66 % -)+ &/ . 2L    0 0 2rCHy!A OO ##K#BB ..u||K/HI _Zt=T=T!%66 %  %%>%>%>%> >'"5$////>?#--/((5<<3FG0||,,UF;>>|L-C.',,T2D"=::.(:/ !$$TZZ?t,,&.,E$$TZZ1L1LM$$TZZ? LL % %d +!j&=&=))+A$$QXXu||E/BC, $ $U +  & & (z7<7G7G7I7O7O7QR7Qtq!al7Q RE"%%j1{"~ eT " "    2 2 3{{}CBBnSs' AQ; AQ;.Q;BQ;5R; R Tc" <V^8dQhRS[S[,RS[RS[RS[R,RS[RS[RS[R S[R,R S[ R,R S[ R,R S[R S[RS[RS[RS[ S[ S[ 3,R,RS[ S[ S[ 3,R,/#)rS model_classr5r ref_init_fnN num_itersrrrr9rruse_orig_paramsrr init_kwargsr)rrQr@rLrrrrrrrrrr )rXrYs"rJrZrMshh-(h%h) h _ h  hh h,d2h,d2h($.hhh%)hh #s(^d*!h"%)cNT$9#hrIc V\P8Xd \R4hVf/pRpVPP 4pVP !VP\P\ P3RR/VBpVfL\'d\V\.\R7pM/\R8Xd \V4pM\VV.VR7pMV!V4pV'dVP4pVPVVV RJVVV V VVR7 p\VP44pVPR VR VR V R V R V RV /4VP !VPVVV3RR/VBp\'T\(4'g\)TTP3/TBpT'dTP4pT\ P*8XdTP-\4pTRJ;'d TP.pT;'dT\ P*8HpT;'dT\ P*8gpT'dJ\0P2!R4pTP4FpTP5TP2T4K! T'dTP7\8R\ 24M \;4pT;_uu_4TPTTRTTTT T TTR7 p RRR4T'dR#T'd_\0P2!R4pTP4FpTP5TP2T4K! X P-\4p \=T4p!\0P>PATX RR7T f"T'gTP5TT!RRR7R#R#R# \ d!p\#RT R\%T4 24ThRp?ii;i +'giEL;i)a Tests FSDP training against a reference, which defaults to DDP but may be customized with ``ref_init_fn``. Args: model_class (Type[FSDPTestModel]): A model class that inherits from ``FSDPTestModel``, which defines the expected interface. fsdp_init_mode (FSDPInitMode): The mode to initialize the FSDP-wrapped model. This should not be ``NO_FSDP``. ref_init_fn (Optional[Callable]): A callable to invoke that wraps a non-wrapped model to construct the reference model, where this wrapper should provide data parallel semantics. If ``None``, then the callable defaults to the DDP constructor. z.Expects an FSDP init mode that wraps with FSDPN{Gz?rT)r output_devicer=)rrrrrrrrrr9rrrz Initializing z raised error zOAn FSDP-managed module with parameter CPU offloading enabled has parameters on F)rrrrrrrr) check_dtypezFSDP did not match DDP) exact_devicemsg)!r@rFrrrrxrLrMr4DDPrrrrrr Exceptionr?rr;rrNrrrVr_roassertRaisesRegexrrrr assert_close)"r^rr5rrrrrrr9rrrrrrrr6rrr ref_modelref_loss ddp_paramsrDrrexpects_device_errorexpects_cpu_device cpu_devicercontext fsdp_lossfsdp_unsharded_paramss"&&&&&&&&&&&&&&&&&, rJ_test_fsdp_parityFSDPTestMixin._test_fsdp_parityMsD \11 1 !QR R  K !!&&(       ( (       x{m; %J 4&M #E*I !(I00  $D0(+'A''A1  )..01 {#%6#%6!?"$4!?    Y$))""   #   J*d++j$*<*<L LJ #*J ~:: :#{3J$D0OO[5O5O  N N/>3N3NN   N N/>3N3NN  e,J#..0  z:1$  " "%%0M3   W55!,% /+E++E6 I  e,J#..0  z:1! [1I / ;  ""8YE"J  "=   %!,  ,9 "K Y}[MAxPQWX X YFWWs$"N/O O"N==O O rr)rNFNFFN)rBrCrDrErzrrrr. classmethodrrrrrHr|r}s@rJrrs >JH34%4%lYY@(,",,595915!& %+0#-1<@#hhhrIrcaa]tRtRtoV3Rlt]R4t]R4t]V3RlRl4t]R4t ] R4t R t Vt V;t#) FSDPTesticj<\SV`4R\PR&VP 4R#)0TORCH_NCCL_DESYNC_DEBUGN)rrosenviron_spawn_processesrs&rJrFSDPTest.setUps)  14 ,- rIc\#rrrs&rJrFSDPTest.world_sizerrIc>\PP4#r)rdistributed_c10d_get_default_grouprs&rJrFSDPTest.process_groups$$7799rIc <V^8dQhRS[/#rd)r)rXrYs"rJrZFSDPTest.__annotate__ sdrIcR#)FrArs&rJdestroy_pg_upon_exitFSDPTest.destroy_pg_upon_exit srIc*\ VP 2#r)r0rrs&rJrFSDPTest.init_methodst~~.//rIc NV!V4pWnW6nVPRR4p\RVP RVP 24\ P P4VP8d9\P!\RVP 2,P4V'db\ PPPPP!4p\"P$!RVPVVR7MF\"P$!VP&\(\+VP4VPR7R p VP\0,p \2'g \4'd \ P P7V 4V .p \"P8!V R 7\ P:P=4\?4VPAW$4\ P:P=4\"P8!V R 7\"PB!4R # \,dJp RT P.^,9d,\P!\R ,P4hR p ? ii;i) rFrrrrrrrrNr)"rrrrrrVrrrrr/rrrrrrrrrrrrrsrr3r6rr/rrr2rrrs &&&&&, rJr FSDPTest._runs9~ "**Y. TYYKx/@AB    ) ) +doo = HHZ*T__,= >?II J  //;;CCMMO''"# '' $ 0 0/"4??3  II , 9    . .y 9[ +  i&  + ""$1 affQi'$9:DDE   s'2I:A IAI J$AJJ$r)rBrCrDrErrrrr&rrrrHr|rIrJs@@rJrrss ::004%4%4%rIrcaa]tRtRtoRt]t]V3RlRl4t]V3RlRl4t ]V3Rl4t V3Rlt V3R lt ] R 4tV3R ltR tVtV;t#) FSDPTestContinuousiJz FSDP test base class using MultiProcContinuousTest for faster test execution. This class reuses worker processes across tests, reducing process spawn overhead. Use this for tests that don't require fresh process state between tests. c <V^8dQhRS[/#rdr)rXrYs"rJrZFSDPTestContinuous.__annotate__Ts##C#rIc\#r)rrfs&rJ backend_strFSDPTestContinuous.backend_strSs""rIc <V^8dQhRS[/#rdr/)rXrYs"rJrZr0XsCrIc\#r)rr2s&rJ device_typeFSDPTestContinuous.device_typeWsrIcx<R\PR&\PP 4V8d/\ P !\RV 2,P4V\,p\'g \'d \PPV4\SV`9WV4R#)rrrN)rrrVrrrrr/rrr3r6rr_init_pg)rfrr rdvz_filerrs&&&& rJr:FSDPTestContinuous._init_pg[s 14 ,-    ) ) +j 8 HHZ*ZL 9:DD E<' 9    . .y 9 95rIc<\SV`4VPVP8wd\P !4\ PP4\4R#r) rrrMAIN_PROCESS_RANKrr/rVrrr2rs&rJrFSDPTestContinuous.setUpks>   99.. . LLN rIc<VPVP8wd\P!4\SV`4\ PP4R#r) rr>rr/rtearDownrVrrrs&rJrAFSDPTestContinuous.tearDownus9 99.. . LLN  rIc.VPP#r)rr@rs&rJr FSDPTestContinuous.process_group|s~~   rIc&<V^8dQh/S[;R&#)rSrr)rXrYs"rJrZr0Js"rIrA)rBrCrDrErzrrrr3r7r:rrArr__annotate_func__rHr|rIrJs@@rJr-r-Jsq #J## 6 6!!grIr-c2V^8dQhR\R,/#)rScompile_compute_on_moduleN)r)rXs"rJrZrZs//$+/rIcDaaaV3Rlo!RR\4oVV3RlpV#)c<\PPP!V/VBSe\ V^,S4'dV^,P 4R#R#r)rVrrrr;r)rsrtrHs*,rJ!fully_shard_with_compiled_compute=compiled_fsdp_test..fully_shard_with_compiled_computesS **D;F; $ , G.1 1  GOO 1 rIc2]tRtRt]!4t]!4tRtR#)*compiled_fsdp_test..FullyShardModeirAN)rBrCrDrEr EAGERCOMPILED_COMPUTErHrArIrJFullyShardModerNs6rIrQc6<a\S4VVV3Rl4pV#)c<\PPPpSEFMpVSP8wd+\ 4'g\ P!R^R7K?\PPPp\PP4VSP8XdTpMAVSP8Xd#^\PPn SpM\RV 24hVSPVP&S!V/VB\PP4VSPVP&V\PPn EKP R#)z0Inductor on GPU needs Triton and recent GPU arch) stacklevelz!Need to implement FullyShardMode=N)rVrrrrOr7warningswarn _inductorconfigcompile_threadsr/rPNotImplementedError __globals__rB) rsrtoriginal_fully_shardmodeoriginal_compile_threadsfully_shard_patchrQrKfuncs *, rJwrapper6compiled_fsdp_test..decorator..wrappers,(-(9(9(>(>(J(J &>/// MMJWX+0??+A+A+Q+Q(!!))+>///(<%^<<<=>EOO**:(I%-;D6B CT  !5!>!>?d%f%!!))+BV  !5!>!>?9Q&&67'rIr )r`rarQrKsf rJ decorator%compiled_fsdp_test..decorators" t R  R>rI)r )rHrcrQrKsf @@rJcompiled_fsdp_testres"""!F rIcDaa]tRtRtoV3RlV3RlltRtRtVtV;t#) SkipModuleic<V^8dQhRR/#rlrA)rXrYs"rJrZSkipModule.__annotate__s11$1rIc`<\SV`4\P!^ ^ RR7VnR# Fr N)rrrur linrs&rJrSkipModule.__init__s" 99R%0rIc$VPV4#rrmras&&rJr&SkipModule.forwardsxx{rIrprrJs@@rJrgrgs11rIrgc8aa]tRtRtoV3RltRtRtVtV;t#) NestedLinearic<\SV`4V'd=\\P!^ ^ RR7P \ 44VnR#\P!^ ^ RR7P \ 4VnR#rk)rrr rur rr nested_linear)r^ fsdp_wraprs&&rJrNestedLinear.__init__sV  !%biiBU&C&F&F{&S!TD !#2r!>!A!A+!ND rIc$VPV4#rruras&&rJr&NestedLinear.forwards!!!$$rIryrrJs@@rJrsrssO%%rIrsc8aa]tRtRtoV3RltRtRtVtV;t#) SkipModelic <\SV`4\P!^ ^ RR7P \ 4Vn\4P \ 4Vn\\VR7\ R7Vn R#)rlFr )rv)rN) rrrur rrlinearrg linear_skipr rsru)r^ double_nestrs&&rJrSkipModel.__init__sW iiBU366{C %rs #$"++ 44 ?S IRRFF FH+q#a!8!8!:;D.Q"-Q"h[J-[J|$7D]J.]J@J mJ Z ? .,ryy,JJ*JJZ/"))/@*r}}*Z'299'6 11   "   " 99 ** .. .. :: >>0&"TJ/&LL^ O%}2O%d4!(?4!n/d %299 %  rI