+ j ^RIt^RIt^RIt^RIt^RIHtHtHtHt^RI H t H t H t ^RI t ^RIHt^RIHuHuHt^RIHuHuHt^RIHuHuHt^RIHt^RIHt^RIH t ^RI!H"t"^RI#H$t$^RI%H&t&H't'H(t(H)t)H*t*H+t+H,t,^R I-H.t.H/t/H0t0H1t1^R I2H3t3^R I4H5t5H6t6H7t7H8t8H9t9H:t:H;t;Ht>^R I?H@t@^RIAHBtB^RICHDtD] 'd^RIEHFtFRtG^RIHHItIHJtJRbtLRtM]N]P]P3,tP]P]P,R,tQ]:P]1P]:P]1P]:P]1P]:P]1P]:P]1P/tW]:P]:P.tX]:P]:P3tY] RcRRll4tZ] RRl4t[] RRl4t\] RRl4t]] RRl4t^] RR l4t_R!R"lt`] RcR#R$ll4taR%R&ltb] R'R(l4tc] R)R*l4td] R+R,l4te] R-R.l4tf] R/R0l4tg] RcR1R2ll4th] R3R4l4tiR5R6ltj] R7R8l4tk] R9R:l4tlR;R<ltmRcR=R>lltnR?R@ltoRARBltpRCRDltqRERFltrRGRHltsRIRJlttRKRLltuRMRNltvRORPltwRQRRltxRStyRTRUltzRVRWlt{RXRYlt|RZR[lt}R\R]lt~R^R_ltR`RaltR# ]KdRtGEL*i;i)dN)Callable GeneratorIterableIterator)Any no_type_check TYPE_CHECKING)CustomClassBase) default_hooks) DeviceMesh)_get_default_group)_FSDPDeviceHandle _FSDPState_get_module_fsdp_state_is_fsdp_flattened!_named_parameters_with_duplicatesclean_tensor_name TrainingState)_FSDP_USE_FULL_PREC_IN_EVAL FlatParameterFlatParamHandleHandleShardingStrategy)_FreeEventQueue)BackwardPrefetch CPUOffloadFullOptimStateDictConfigFullStateDictConfigMixedPrecisionShardingStrategyStateDictConfig StateDictType)_Policy)DTensorExtensions)_sync_params_and_buffers)is_traceable_wrapper_subclass)RemovableHandleT) deferred_initfakeF _fsdp_syncedc |V^8dQhR\R\R\R\R,R\R,R\/#)state process_groupsharding_strategypolicyN device_meshreturn)rProcessGroupTyperr"r )formats"c/home/luca/screen-checking/.venv/lib/python3.14/site-packages/torch/distributed/fsdp/_init_utils.py __annotate__r5[sM00 0#0(0 dN 0 d" 0  0cnVeVe \R4hV\9pV'd)VfVfVf\RV R24h\WV4pM;V'dW@nVP ^R7VnMVeTM \ 4VnVP P4VnVP P4Vn VPpV'd!W`PP4,p\PPV4VnW`P, VnV#)NzcCannot pass both process_group and device_mesh at the same time. Please just pass only one of them.zManual wrapping with zA requires explicit specification of process group or device_mesh.mesh_dim) ValueErrorHYBRID_SHARDING_STRATEGIES*_init_process_group_state_for_hybrid_shard _device_mesh get_groupr-r ranksize world_size_inter_node_pgr DefaultState_get_gradient_predivide_factor_gradient_predivide_factor_gradient_postdivide_factor)r,r-r.r/r0is_hybrid_strategydata_parallel_world_sizes&&&&& r4_init_process_group_staterIZsA [%< <  +.HH  V^ 8K'(9':;SS  ?kE !, "-"7"7"7"CE "/!: @R@T  $$))+EJ**//1E$// $8$8$=$=$?? ""AA $  $ !#C#CC % Lr6cHV^8dQhR\R\R\R\/#)r+r,r-r0r1)rr2r )r3s"r4r5r5s0(( (#(( (r6cV'd_\V4'd6W nVP^R7VnVP^R7VnM\ RVP 24hVf=\4p\W0PP44wrEW@nWPnM8\V4'dVwVnVnM\ R\V4 24h\VPR7VnV#)rr8z,Expected device_mesh to have ndim=2 but got zmExpected process_group to be passed in as either None or Tuple[dist.ProcessGroup, dist.ProcessGroup] but got r-)"_is_valid_hybrid_shard_device_meshr=r>rBr-r:ndimr !_init_intra_and_inter_node_groups_device_handle device_count_is_valid_hybrid_shard_pg_typetype_get_default_comm_hook_state_inter_node_state)r,r-r0 default_groupintra_node_groupinter_node_groups&&& r4r<r<s  -k : :!, $/#8#8!#8#DE "-"7"7"7"CE >{?O?O>PQ   *, -N //<<>. *// *- 8 89F 5E !5GGKMGZF[]  ;**E Lr6c0V^8dQhR\R\/#r+r-r1)rbool)r3s"r4r5r5s#$r6c\V\4;'dK\V4^8H;'d5\;QJdRV4F 'dK R# R#!RV44#)r+c3V"TFp\V\P4xK! R#5iN) isinstancedist ProcessGroup).0pgs& r4 1_is_valid_hybrid_shard_pg_type..s JMb 2t0011M')FT)r_tuplelenallrLs&r4rRrRsh =%( K K  ! # K K CJMJCC  JMJ Jr6c0V^8dQhR\R\/#)r+r0r1)r r[)r3s"r4r5r5sIIJI4Ir6cP\V\4;'dVP^8H#)r+)r_r rN)r0s&r4rMrMs" k: . H H;3C3Cq3HHr6cDV^8dQhR\R\P/#)r+num_devices_per_noder1)intr`ra)r3s"r4r5r5s   ARAR r6c6\P!V4wrV#)a5 Return a process group across the current node. For example, given each row is a distinct node: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 This API would return an intra-node subgroup across [0, 1, ..., 7] or [8, 9, ..., 15] depending on the process's rank. For example, rank 3 would get [0, 1, ..., 7]. )r` new_subgroups)rmintra_node_subgroup_s& r4_init_intra_node_process_grouprss"//0DE r6cdV^8dQhR\PR\R\P/#r+global_process_grouprmr1)r`rarn)r3s"r4r5r5s2""++"" "r6cRp\P!V4p\P!V4pWA,p\P!V4V,p\ V4FJp\ V4Uu.uFqW,,NK p p\P !WR7p Wv8XgKHT pKL Vf\ V R24hV#uupi)al Return an inter-node process group where each contained rank has the same local rank. For example, given each row is a distinct node: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 This API would return inter-node process group [0, 8], [1, 9], [2, 10], and so forth depending on the process's rank. For example, rank 1 would get [1, 9], rank 5 would get [5, 13]. N)ranksbackendz. expected to assign inter-node pg, but did not)r` get_backendget_world_sizeget_rankrange new_groupAssertionError) rvrm inter_node_pgsharding_backendrA num_nodes my_local_rank local_rankiranks_for_inter_groupgrps && r4_init_inter_node_process_grouprs M''(<=$$%9:J2IMM"67:NNM01 =B9=M! =M!2 3 3=M ! nn#8S  &M2oK L   ! s/CcV^8dQhR\PR\R\\P\P3,/#ru)r`rarnrg)r3s"r4r5r5sB++ 4  d// /0r6c.\V4\W43#)a Initialize intra and inter-node process groups and return the ones corresponding to this process's rank. This function can be used to initialize process groups for ``HYBRID_SHARD`` or ``_HYBRID_SHARD_ZERO2`` in FSDP. This function assumes each node has an equal number of CUDA-enabled devices. Returns: Tuple[dist.ProcessGroup, dist.ProcessGroup]: Intra and inter-node process group. )rsr)rvrms&&r4rOrOs  '';<&';R r6c TV^8dQhR\R\PR\\PP,R,R\\PP ,\\PP,,R,R\/#)r+r,moduleignored_modulesNignored_statesr1)rnnModulertorch Parameter)r3s"r4r5r5s|++ + II+ehhoo.5+UXX//0uxx   ++r6cVeVe \R4hRpVRJpV'd\V4p\VR4M.p\Ve \V4M.R4\V4^8d-\ V^,\ P 4'dTpMTp\W4Vn\VVPV4Vn \VVP4Vn V#)NzfCannot pass both ignored_modules and ignored_states at the same time. Please just pass ignored_states.TF) r:list_check_ignored_statesrhr_rr_get_ignored_modules_ignored_modules_get_ignored_params_ignored_params_get_ignored_buffer_names_ignored_buffer_names)r,rrrignored_parameterspassed_as_ignored_statesignored_states_lists&&&& r4_init_ignored_module_statesrs"~'A :  -T9">2148 %4%@D !b%  !# )!,bll ; ;!4 1O1&JE/ E #< #E Lr6cJV^8dQhR\\,R\RR/#)r+rrr1N)rrr[)r3s"r4r5r5@s(I9= r6c\V4^8XdR#V'd\;QJdRV4F 'dK RM RM !RV44p\;QJdRV4F 'dK RM RM !RV44pV'gCV'g9\VUu0uFp\V4kK up\R7p\ RV 24hR#R#\;QJdRV4F 'dK RM RM !RV44'g9\VUu0uFp\V4kK up\R7p\ R V 24hR#uupiuupi) z Check that the ignored states are uniformly parameters or uniformly modules. We may remove this check in the future if we permit mixing. Nc3V"TFp\V\P4xK! R#5ir^)r_rrrbr,s& r4rd(_check_ignored_states..KsUnUE2<<88nrfFTc3V"TFp\V\P4xK! R#5ir^r_rrrs& r4rdrLsSN5*UBII66Nrf)keyzUignored_states expects all nn.Parameter or all nn.Module list elements but got types c3V"TFp\V\P4xK! R#5ir^rrs& r4rdrUsL^E:eRYY//^rfz>ignored_modules expects nn.Module list elements but got types )rhrisortedrSreprr:)rr all_params all_modulesr, sorted_typess&& r4rr@s >aSUnUSSSUnUU cSNScccSNSS +!N"KN54;N"KQUVL**69 #.zsL^LsssL^LLL!N"KN54;N"KQUVL%( M #L#Ls EEc V^8dQhR\R\PR\\P,R\ \ P,R,R\/#)r+r,rignored_params device_idNr1)rrrsetrrnrdevice)r3s"r4r5r5^sU-- - II- %-U\\!D( -  -r6crRpVe9\V\P4'dTM\P!V4pVf\W4FpVPPR9dK VfVPpK2VPPVP8wgKY\ RVP RVPP 24h T;'g\P P4pVPR8Xd \ R4h\P!V4Vn V#)a Determine device handle used for initializing FSDP. If a device is specified by ``device_id``, then returns device handle corresponds to that device type. Otherwise, If the module is already on a non-CPU device, then the device type is that non-CPU device type. If the module is on CPU or meta, then the device type is the current accelerator device. See the :ref:`Accelerators` for details. This method will be called once ignored parameters was determined, as the device handle maybe needed for other initialization. NcpuzLFSDP does not support modules with different device types but got params on z and zOFSDP needs a non-CPU accelerator device, but no accelerator device is detected.>rmeta) r_rr_get_orig_paramsrS RuntimeError_C_get_acceleratorr from_devicerP)r,rrrdetermined_deviceparams&&&& r4_init_device_handler]s()U\\22 i(   %f=E||  O3 ($)LL!<<$$(9(>(>>&-->-C-C,DE%,,J[J[I\^>.LL1J1J1L  ! !U *a -889JKE Lr6cPV^8dQhR\R\PR\/#)r+r,rr1)rrr)r3s"r4r5r5s*  IIr6c\V4Vn/pVP4Fwr4\V4pVPW#&K W nV#r^)_get_buffer_names _buffer_names named_buffersrdtype_buffer_name_to_orig_dtype)r,rr buffer_namebuffers&& r4_init_buffer_statersT ,F3E :<%335 ' 4 28,,"/ 6(B$ Lr6cV^8dQhR\R\R,R\R,R\R,R\R\R\ R \ R \/ #) r+r,r.Nmixed_precision cpu_offloadlimit_all_gathersuse_orig_paramsbackward_prefetch_limitforward_prefetch_limitr1)rrrrr[rn)r3s"r4r5r5sq?? ?'$.?$d*?d" ?  ?  ?!? ??r6cVP^8Xd[V\P8wd5\P!RT;'g\P R2^R7\PpM2V\P8Xd\P!R\ ^R7T;'g\P VnT;'g \4Vn Ve6\PPR\VP4 24\PP!\"R4R8HVnT;'g \'4VnW@nWPn\.P0VnRVn\74Vn\:P<!4Vn\@PB!VP>VV4Vn"RVn#/pWn$Rp Wn%.p Wn&V#) z/FSDP is switching to use `NO_SHARD` instead of z since the world size is 1. stacklevelzoThe `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.Nz'torch.distributed.fsdp.mixed_precision.1)'rArNO_SHARDwarningswarn FULL_SHARD FutureWarningr.rrrr_log_api_usage_oncestrosenvirongetr_use_full_prec_in_evalrrr_use_orig_paramsrIDLEtraining_state_is_rootr_free_event_queuer`get_debug_level _debug_levelexec_order_utils_ExecOrderData_exec_order_data_unshard_event_fully_sharded_module_to_handle_handleparams) r,r.rrrrrrrrrs &&&&&&&& r4_init_core_staters 1  0 9 9 9 MMA$CC(8(C(CDE''   -55 .77 7  <   0NN3C3N3NE+??~/?E" $$5c%:O:O6P5Q R  2B73>  $33z|E/,(--EEN-/E--/E-<< E  EIK#,K)'+GM"$FL Lr6c0V^8dQhR\R\/#r+r,r1r)r3s"r4r5r5s     r6c`.pWn.pW n.pW0nRVnRVnRVnV#)TN)_root_pre_forward_handles_pre_forward_handles_post_forward_handles_sync_gradients _comm_hook_comm_hook_state)r,rrrs& r4_init_runtime_statersE8:&?#24!535"7 EE!E Lr6cHV^8dQhR\R\R\R\/#)r+r,backward_prefetchforward_prefetchr1)rrr[)r3s"r4r5r5s0    '   r6cWnW nV#r^)rr)r,rrs&&&r4_init_prefetching_staters 0- Lr6c<V^8dQhR\R\R\/#)r+r,r0r1)rr )r3s"r4r5r5s!  : J * r6cVeVP4MRpV'd-W P8wd\VP4VnV#RVnV#r^)_get_root_meshr=r#rP_fsdp_extension)r,r0 root_meshs&& r4_init_extensionrsR 1<0G **,TIy$6$66 1%2F2F G L!% Lr6c0V^8dQhR\R\/#rr)r3s"r4r5r5s*r6c~\PVn\4p\ 4VnWn/pW nV#r^)r!FULL_STATE_DICT_state_dict_typerr_optim_state_dict_config_state_dict_config_unshard_params_ctx)r,state_dict_configunshard_params_ctxs& r4_init_state_dict_staters;*::E)<)>%=%?E"057 2 Lr6crV^8dQhR\PR\\P,RR/#)r+rrr1N)rrrr)r3s"r4r5r5s.299d2<<6HTr6cVFcp\VP4^8XgKRpVP4FwrEW%JgK TpM V'g \R4h\ RV R24h R#)z Verify if the parameters are accepted by FSDP. The only restriction now is that the parameter cannot be a scalar tensor (param.shape == []). rzExpected param_name to be setz/FSDP doesn't support scalar parameters. Change z& to a 1D tensor with numel equal to 1.N)rhshapenamed_parametersrr:)rrr param_namenameparam_s&& r4_verify_managed_paramsrsw  u{{ q J & 7 7 9 ?!%J!:$%DEE$%KM r6c V^8dQhR\R\PR\\P ,R,R\ \P.R3,R,R\R\/#)r+r,fully_sharded_modulerN param_init_fnsync_module_statesr1)rrrrnrrrr[)r3s"r4r5r53ssNN N))NU\\!D(NRYYK-.5 N  N  Nr6c8a\VSPV4\VSPSP4p\ VSPSP 4wrgV'g V'dVe\WSP 4MOV'd$\VVSP SP4M$V'd\P!VV3RlR7SP UU u0uFpVP4Fp V kK K p pp \VSPV V4\VSPVSPSP4Sn\\!VSP44p \#W4V'dSP$\&9dW\)WSP*4VP4F,p \-V \.4'gK\1V \.R4K. \)WSP24\5SW4S#uup pi)zHInitialize a ``FlatParamHandle`` from a module ``fully_sharded_module``.cL<\V4RJ;'dVSP9#r^)rr) submoduler,s&r40_init_param_handle_from_module..Qs-'=i'HD'P(8(8!7!77(8r6)check_fnF)_check_single_device_moduler_get_device_from_device_idr?rP_need_to_materialize_moduler_materialize_with_param_init_fn_materialize_meta_moduler'materialize_modulebuffers_move_module_to_device_get_compute_devicecompute_devicerrrr.r;_sync_module_params_and_buffersrBhasattr FSDP_SYNCEDsetattrr-_init_param_handle_from_params) r,rrrrdevice_from_device_idis_meta_moduleis_torchdistX_deferred_initignored_modulerignored_buffersmanaged_paramss f&&&& r4_init_param_handle_from_moduler62s 4e6K6KYW65::u333Ne33U5K5K3/N 5=;T' 1G1G     " "   %(( 8 $444N$,,.F . 4    /    E*+?AVAVWXN/@  " "&@ @ ,$e6J6J /6686;//FK79 ( %2E2E #5.O LYs&"HczV^8dQhR\R\\P,R\P/#)r+r,rr)rrrrr)r3s"r4r5r5s3)) )  ))))r6c\V4^8XdR#\VVVP\VP,VP P VPPVPPVPPVPVPVPR7 pVP4VP'd \!R4hVP"P%VP&4W0nW0P(VP*&\,P.!R4pVP P 'd1VP&P.V8wdVP1V4R#R#R#)rN)fsdp_extensionz!Expected state._handle to be Noner)rhrr*SHARDING_STRATEGY_MAPr.roffload_paramsr param_dtype reduce_dtypekeep_low_precision_gradsr-rrshardrrrappend flat_paramr_fully_sharded_modulerr flat_param_to)r,rrhandle cpu_devices&&& r4r/r/s,  6{a  e556 (( )) ** 66  ,, F LLN }}}@AA LL))*MJP))&*F*FGe$J '''F,=,=,D,D ,RZ(-S'r6cV^8dQhR\PR\\PP,R,R\\P,/#)r+ root_modulerNr1)rrrrr)r3s"r4r5r5sD:::uxx/$6: ^:r6c RpVe \V4M \4pTFgp\T\P P 4'g\TR\T4 2,4h\T4'gK^\R4h TP4F2p\P!T4'dK!TPT4K4 TUUu0uF>pTP4F'p\T\P4'dK%TkK) K@ pppY9d\P !RX 2^R7TP4FLp\T4p T fK\#T R4'g \%R 4hTP'T P(4KN T# \d%p\TR\T4 2,4ThRp?ii;iuuppi) aP Check that ``_ignored_modules`` is an iterable of ``nn.Module`` s without any FSDP instances. Return the modules contained in their module subtrees as a :class:`set`. Nested FSDP instances are excluded, but their already-computed ignored modules are included. ``_ignored_modules`` represents the argument passed by the user to FSDP. z>`ignored_modules` should be an iterable of `torch.nn.Module`s Nzbut got zbut got an iterable with z1`ignored_modules` should not include FSDP moduleszTrying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: rrz?Expected optional_fsdp_state to have _ignored_modules attribute)r TypeErrorrSr_rrrrr:modulestraversal_utils _composableadd fsdp_fileFullyShardedDataParallelrrr,rupdater) rGr msg_prefixignored_root_moduleserchildrroptional_fsdp_states && r4rrsRJQ%5%AC !su  '&%((//22J+DT&\N)SST T !& ) )PQ Q '%%'**622 $ $V ,(+*F^^%E%!C!CD % * %  228 ;  !((* 4Y?  *.0BCC$U  " "#6#G#G H+ Q Q x5E0F/G%HHIqPQ$s#F5G G G*G  Gc 8V^8dQhR\PPR\\PP,R\\PP ,R,R\\PP ,/#)r+rGrrNr1)rrrrrr)r3s"r4r5r5sh""")"!!3!34t;"    "r6c\4pVUUu0uF.qDP4Fp\V4'dKVkK K0 pppVPV4Ve6VUu0uFp\V4'dKVkK ppVPV4VP 4FLp\ V4p V fK\ V R4'g \R4hVPV P4KN V#uuppiuupi)z Return the parameters of the modules in ``ignored_modules`` and the parameters in ``ignored_parameters``. :class:`FlatParameter` s are excluded from the result. rz>Expected optional_fsdp_state to have _ignored_params attribute) r parametersrrPrJrr,rr) rGrrall_ignored_paramsmpparams_in_ignored_modulesparams_in_ignored_parametersrrUs &&& r4rrs36%#!"aLLNqBTUVBWN?!78%)( )!1CA1FAA) %(  !!">?!((* 4Y?  *.0ABB$T  % %&9&I&I J+ -!( s%C3 C3 C98C9cV^8dQhR\PPR\\PP,R\\,/#r+rGrr1)rrrrr)r3s"r4r5r5s?$$$)$ X$r6c \4pVUUu0uFq3P4FqDkK K pppTPVP4UUu0uFwrdWE9gK \ V4kK upp4VP 4FLp\ V4pVfK\VR4'g \R4hVPVP4KN V#uuppiuuppi)z6Return the cleaned buffer FQNs in ``ignored_modules``.rzDExpected optional_fsdp_state to have _ignored_buffer_names attribute) rr'rPrrrJrr,rr) rGrall_ignored_buffer_namesrZrbuffers_in_ignored_modulesrrrUs && r4rrs *-("'1iikFk"##(3'@'@'B 'B# 3 + k *'B !((* 4Y?  *.0GHH$Z % + +,?,U,U V+ $#-"  s C C C cZV^8dQhR\PR\\,/#)r+rGr1)rrrr)r3s"r4r5r5&s"299Sr6chVP4UUu0uFwr\V4kK upp#uuppi)zrReturn the fully prefixed names of all buffers in the module hierarchy rooted at ``root_module`` as a class:`set`.)rr)rGrrrs& r4rr&s8>I=V=V=X =X>;+&=X  s.cV^8dQhR\PR\\P,R\\ P ,R,RR/#)r+rrrNr1)rrrrrnrr)r3s"r4r5r5-sK  II  % U\\!D(   r6c \W4Uu0uFq3PkK pp\V4^8Xd-\P!R4V9dVf \ R4hR#\V4^8d\ RV 24hR#uupi)z Raise an error if ``module`` has original parameters on multiple devices, ignoring the parameters in ``ignored_params``. Thus, after this method, the module must be either fully on the CPU or fully on a non-CPU device. rNzTTo support a module with both CPU and GPU params, please pass in device_id argument.z;FSDP only supports single device modules but got params on )rrrhrr)rrrrdevicess&&& r4r!r!-s*:&)QR)Q||)QGR 7|qU\\%0G;  5   W I' S   SsBc V^8dQhR\\P,R,R\R\R\PR,/#)r+rNr? device_handler1)rnrrr)r3s"r4r5r5KsGU\\!D( % \\D r6c VfR#\V\P4'dTM\P!V4pVPR8wdpVPfb\ P !RV RV RVP4 RVP R2 ^R7\P!VP44pV#) z Return a ``torch.device`` for the specified ``device_id``. Processes ``device_id`` and returns either the corresponding device or ``None`` if ``device_id`` is ``None``. Nrz"FSDP got the argument `device_id` z on rank zJ, which does not have an explicit index. FSDP will use the current device z6. If this is incorrect, please explicitly call `torch.zk.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.r)r_rrrSindexrrcurrent_device)rr?rirs&&& r4r"r"Ks 5<<88 ell9>U {{e 4 0 9f00=0L0L0N/OPCCI;;-P1 1  m::<= Mr6c V^8dQhR\PR\\P,R\\P,R\\ \ 3,/#)r+rrrr1)rrrrrgr[)r3s"r4r5r5isN77 II7 %7^7 4: 7r6c\\W44p\;QJdRV4F 'gK RM RM !RV44pVP4F6pWR9dK VP RR7FpWFP ,pK K8 V'*;'dC\ ;'d5\;QJdRV4F 'gK RM RM !RV44pWG3#)z Return if ``module`` has parameters on meta device and if ``module`` is using torchdistX deferred initialization. At most of the returned bools can be ``True``. If either is ``True``, then ``module`` needs to be materialized. c38"TFqPxK R#5ir^)is_metarbrs& r4rd._need_to_materialize_module..vsCN5NsTFrecursec3N"TFp\P!V4xK R#5ir^)r(is_fakerqs& r4rdrrs@ U##s#%)rranyrJr'rp_TORCHDISTX_AVAIL)rrrr5r1rbufr2s&&& r4r#r#is*6BCNSCNCSSSCNCCN^^%  ' $$U$3C kk )N4&  A A  A A C@@CCC@@ @  66r6cV^8dQhR\PR\\P.R3,R\\P,RR/#)r+rGrNrr1)rrrr)r3s"r4r5r5sL   RYYK-. ^  r6c\V4'g\RV R\V4 24h\W4pVF pV!V4K R#)z Expected z to be callable but got N)callabler:rS_get_modules_to_materialize)rGrrmodules_to_materializers&&& r4r$r$sR M " " &>tM?R>S T  9V(f)r6cV^8dQhR\PR\PR,R\\P,R\ /#)r+rGr0Nrri)rrrrrr)r3s"r4r5r5sE### <<$.#^#% #r6c T;'g%\P!VP44p\W4pRp\P!4;_uu_4VF~p\ P !VPRR7VPRR74p\\V44^8pV'gK[VPVRR7VP4K RRR4R# +'giR#;i \d9p \P!R\!T 4 R\#T4 R2^R7T hRp ? ii;i)NFrs)rrtzIUnable to call `reset_parameters()` for module on meta device with error z(. Please ensure that your module oftype z* implements a `reset_parameters()` method.r)rrrlr}no_grad itertoolschainrXr'rhrto_emptyreset_parameters BaseExceptionrrrrS) rGr0rrimaterialization_devicer~rmodule_state_iterhas_module_statesrSs &&&& r4r%r%s3ell$$&79V F]]__0%.OO%%e%4NN5N1%! %(->(?$@1$D!$$OO+A5OQ++-1___   !!$Q)L>!K M  s<C=AC)8'C)C=) C: 4C=:C== E3D;;EcV^8dQhR\PR\\P,R\\P,/#r_)rrrr)r3s"r4r5r5s8"""-0^" "))_"r6cD.p\P!V.4pV0pV'd|VP4pVPV4VP 4FDpWd9gK \ V4eKWa9gK"VP V4VPV4KF KV#r^) collectionsdequepopleftr@childrenrrM)rGrr~queuevisited_modulesr child_modules&& r4r}r}s /1   {m ,E'2mO %%f-"OO-L3*<8@ 7##L1 \*. "!r6c V^8dQhR\PR\\P,R\\P ,R\P R,RR/#)r+rrr4r0Nr1)rrrrrTensorr)r3s"r4r5r5sY33 II3 %3&3!<<$. 3  3r6c a \P!R4o VEe%\P!4pVP V4.p.pV'dVP 4pVP V 3RlVPRR744VP V 3RlVPRR744VP4F6p\V\P4'dK%VP V4K8 KVU u.uF qV9gK V NK p p VU u.uF qV9gK V NK p p \WV4R#\\W4R4p V e V PS 8Xd \!4R#R#R#uup iuup i)a Move ``module`` depending on ``device_from_device_id`` and its current device. This includes moving ignored modules' parameters. - If ``device_from_device_id`` is not ``None``, then this moves ``module`` to the device. - If ``device_from_device_id`` is ``None``, then this does not move ``module`` but warns the user if it is on CPU. Precondition: ``_check_single_device_module()``. rNc3N<"TFpVPS8XgKVxK R#5ir^r)rbrrEs& r4rd)_move_module_to_device..s&BE<<:-B% %Frsc3N<"TFpVPS8XgKVxK R#5ir^r)rbrrEs& r4rdrs&@F==J.@r)rrrrr@rextendrXr'rr_rNrO_move_states_to_devicenextr_warn_cpu_init)rrr4r0rrr' curr_modulerr[params_to_move bufs_to_moverrEs&&&& @r4r(r(sS$e$J(/:.?.?.A V%'&(--/K MM(33E3B  NN)11%1@  )113 !)Y-O-OPPLL+4&,GV/F!!VG#*G7a.F7 G~=RS !&94 @E U\\Z78 HGsE;E;F+FcV^8dQhR\\P,R\\P,R\P R,RR/#)r+rr'r0Nr1)rrrrrr)r3s"r4r5r5sK   %,, !<<$. r6c\V4^8Xd\V4^8XdR#\V4^8dV^,PpM#\V4^8dV^,Pp\P!R4pVeVFwp\P!4;_uu_4VP V4VnVP e+VP P V4VP nRRR4Ky VFpVP V4VnK R#XV8Xd \4R#R# +'giK;i)z Move states to the specified device. Precondition: ``_check_single_device_module()`` and module's parameters and buffers have been materialized if needed. Nr)rhrrrtodatagradr)rr'r0rlrErrs&&& r4rrs 6{aCLA- 6{Q)) W  **e$J(E"XX&;< ::)&+jjmm4I&JEJJO! F ))$9:FK : % & !s &AD77 E c6\P!R^R7R#)amThe passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.rN)rrr6r4rr's MM 1 r6c V^8dQhR\PR\\P,R\P R,R\ R\R\P /#)r+rrr0Nr?rir1)rrrrrrrnr)r3s"r4r5r53s^ II %!<<$.   %   \\ r6c \\W4R4pVe)VPPR8wdVPpM$\P!VP 44pVeWb8wd\ RV RV RV 24hV#)a Determine and return this FSDP instance's compute device. If the module is already on a non-CPU device, then the compute device is that non-CPU device. If the module is on CPU, then the compute device is the current device. Since this method should be called after materializing the module, any non-CPU device should not be meta device. For now, the compute device is always a CUDA or CUDA-like device with its explicit index. Precondition: ``_check_single_device_module()`` and ``_move_module_to_device()``. Nrz4Inconsistent compute device and `device_id` on rank z: z vs )rrrrSrrlr:)rrr0r?rirr*s&&&&& r4r)r)3s* !&94 @E U\\..%7m&B&B&DE(^-TB4&d#8"9 ;   r6cV^8dQhR\PR\\P,R\P RR/#)r+rrr-r1N)rrrrr`ra)r3s"r4r5r5VsC99 II9  9$$9 9r6c.pVP4Fp\V\R4'dK\V\R4VP 4p\ V4'dVP 4wrgVFep\WX4;;\PRc;ewp VPV 4K;;\Rc;ewKNp \R\V 4 24h KVPV4K VFp V P 4p \ V 4'dV P 4wrgVFep\W4;;\PRc;ewp VPV 4K;;\Rc;ewKNp \R\V 4 24h KVPV 4K \V4\VV\^R7R#)z Synchronize module states (i.e. parameters ``params`` and all not-yet-synced buffers) by broadcasting from rank 0 to all ranks. Precondition: ``sync_module_states == True`` and ``self.process_group`` has been set. FTrNz(expected Tensor or CustomClassBase, got )src)r'getattrr-r.detachr%__tensor_flatten__rrr@r rrS+_check_module_states_for_sync_module_statesr$PARAM_BROADCAST_BUCKET_SIZE) rrr- module_statesrdetached_bufferattrsrrattrv unexpectedrdetached_params &&& r4r+r+Vs)+M.."v{E22 FK .$mmoO,_==+==?!D!/80U\\^0)0031._. /'"0"J4PZK[J\ ]#"$$_5)#, ( 8 8%88:HEn3,,%,,Q/-**+#,FtJGWFXY   0"0 ># r6cRV^8dQhR\\P,RR/#)r+rr1N)rrr)r3s"r4r5r5s%    %     r6cV'dH\;QJdRV4F 'gK RM RM !RV44'd \R4hR#R#)c3f"TF'qP\P!R48HxK) R#5i)rN)rr)rbtensors& r4rd>_check_module_states_for_sync_module_states..s#;H e,,=s/1TFzThe module has CPU parameters or buffers when `sync_module_states=True`, which requires them to be on GPU. Please specify the `device_id` argument or move the module to GPU before passing it to FSDP.N)rwr:)rs&r4rrsQ;H;H C  }r6cV^8dQhR\PR\\P,R\\P,/#)r+rrr1)rrrrr)r3s"r4r5r5s<  II  % bll r6c#"VP4p\V4pW19gK\V4'dK'VxK- \dR#i;i5i)a, Return an iterator over the original parameters in ``module``. The iterator does not return the parameters in ``ignored_params``, any ``FlatParameter`` s (which may be present due to nested FSDP wrapping), or any original parameters already flattened (only relevant when ``use_orig_params=True``). N)rXrr StopIteration)rr param_genrs&& r4rrsM!!#I OE*3Ee3L3L    s,AAAA A AAAcRV^8dQhR\\P,RR/#)r+rr1N)rrr)r3s"r4r5r5s' % r6c \V4FJwr#W19gK \V4'dK \RV RVP4 RVP 24h R#)a Check that original parameters in ``fsdp_module`` have been flattened. The flattened parameters are made invisible to ``named_parameters()`` for the module hierarchy rooted at ``fsdp_module``. This should be called as a sanity check after flattening the wrapped module's parameters. z Found an unflattened parameter: z;  N)rrrr@ __class__) fsdp_modulerrrs&& r4_check_orig_params_flattenedrsY?{K  &/A%/H/H2:,b::<.%//!24 Lr6c$V^8dQhR\/#)r+r.)r)r3s"r4r5r5s.>r6clV\P8Xd\P#\P#r^)rrr allreduce_hookreduce_scatter_hook)r.s&r4_get_default_comm_hookrs3  0 9 9 9 $$ . .r6cXV^8dQhR\PR\P/#rZ)r`rar rC)r3s"r4r5r5s,CC$$CCr6c0\P!VR7#)rL)r rCrLs&r4rTrTs  % %M BBr6ir^)rrrrcollections.abcrrrrtypingrrr rtorch.distributed distributedr`(torch.distributed.fsdp._exec_order_utilsfsdp_exec_order_utilsr'torch.distributed.fsdp._traversal_utils_traversal_utilsrK2torch.distributed.fsdp.fully_sharded_data_parallelfully_sharded_data_parallelrNtorch.nnrtorch._custom_class_baser (torch.distributed.algorithms._comm_hooksr torch.distributed.device_meshr "torch.distributed.distributed_c10dr $torch.distributed.fsdp._common_utilsrrrrrrr"torch.distributed.fsdp._flat_paramrrrr%torch.distributed.fsdp._limiter_utilsrtorch.distributed.fsdp.apirrrrrrr r!torch.distributed.fsdp.wrapr"&torch.distributed.tensor.parallel.fsdpr#torch.distributed.utilsr$torch.utils._python_dispatchr%torch.utils.hooksr&rx torchdistxr'r( ImportErrorrr-rgraHybridShardProcessGroupTyper2rr SHARD_GRAD_OP HYBRID_SHARD_HYBRID_SHARD_ZERO2r:r;#NO_RESHARD_AFTER_FORWARD_STRATEGIESrIr<rRrMrsrrOrrrrrrrrrrr6r/rrrrr!r"r#r$r%r}r(rrr)r+rrrrrTrr6r4rsQ CC44 CCAAFF4B4A B   0D<F1.0 #D$5$5t7H7H$HI$$'BBTI 5>>!7!B!B""$:$H$H!!#9#F#F((*@*T*T !!(( ""(('#00f((VII  ""J&++\:--`"??D       (NNb))>:z"J$@ <<7< #L",3l@ F9x   ,(Cq$sK:: LL