+ j^RIt^RIt^RIt^RIt^RIt^RIt^RIHt^RIH t ^RI H t ^RI H t ^RIHt^RIHtHt^RIt^RIHt^R IHtHtHtHt^R I Ht^R IHtHt^R IH t H!t!H"t"^R I#H$t$]!]%4t&Rt'Rt(R.Rlt)R.Rlt*Rt+R/Rlt,R/Rlt-R0Rlt.R/Rlt/R1RRllt0R2RRllt1RRlt2R3RRllt3RR lt4R!R"lt5R#R$lt6R%R&lt7R'R(lt8R)t9R*R+lt:R,R-lt;R#)4N) defaultdict)Iterable) nullcontext)replace)Path)CallableUnion) get_logger)FSDP_MODEL_NAMEOPTIMIZER_NAMESAFE_WEIGHTS_NAME WEIGHTS_NAME)get_module_class_from_name)get_non_persistent_buffers is_peft_model)get_module_children_bottom_upis_compiled_modulesave)is_torch_versionc|R\P9dR\PR&R\PR&R#)zS Enables RAM efficient loading of Hugging Face models for FSDP in the environment. ACCELERATE_USE_FSDPTrueFSDP_CPU_RAM_EFFICIENT_LOADINGNosenviron\/home/luca/screen-checking/.venv/lib/python3.14/site-packages/accelerate/utils/fsdp_utils.py!enable_fsdp_ram_efficient_loadingr (s. BJJ.,2 ()39BJJ/0rc,R\PR&R#)zT Disables RAM efficient loading of Hugging Face models for FSDP in the environment. FalserNrrrr"disable_fsdp_ram_efficient_loadingr#2s4;BJJ/0rcV'd\\V4'dKVe.VP'd^RIHpV!V\ VRR7R7#^RIHpV!WPR7#Ve^RIHpV!WR7#VP4#)N)get_model_state_dictT)ignore_frozen_paramsoptions)get_peft_model_state_dict adapter_name) rfull_state_dict'torch.distributed.checkpoint.state_dictr%rpeftr)active_adapter state_dict)model adapter_only sd_optionsr%r)s&&& r_get_model_state_dictr49sq e,,  !j&@&@&@ T'wz`d7ef f2(=Q=QRRP#E>>!!rc V'd]\V4'dLVe.VP'd^RIHpV!W\ VRR7R7#^RIHpV!WVPR7#Ve^RIHpV!WVR7#VPV4#)N)set_model_state_dictF)strictr')set_peft_model_state_dictr*) rr,r-r6rr.r8r/load_state_dict)r1r0r2r3r6r8s&&&& r_set_model_state_dictr:Msw e,,  !j&@&@&@ T'7:^cCde e2(I]I]^^P#EzJJ$$Z00rc RpVP^8XdY^RIHp^RIHpV!VP VP 8H\VPRR4\VPRR4R7pV#)N)StateDictOptions StateDictTypeoffload_to_cpuF rank0_only)r, cpu_offloadbroadcast_from_rank0) fsdp_versionr-r<2torch.distributed.fsdp.fully_sharded_data_parallelr>state_dict_typeFULL_STATE_DICTgetattrstate_dict_config) fsdp_pluginr3r<r>s& r_prepare_sd_optionsrJasjJ1$LT%'77=;X;XX = =?OQVW!()F)F V[!\ rc`^RIHp^RIHp\P!VRR7VP VP 8Xd0VP^8p WPn WPn VP^8Xd2VP W P VPVP4M \4p \V4p T ;_uu_4\W%V R7p VP VP 8XdV^8Xd \ R2M \ RV R2p \P"P%W=4pVP&^8XdG\(P+RV 24\,P.!W4\(P+R V 24EMVP VP08XdV^8Xd\ R VP& R2M\ RV R VP& R2p \P"P%W=4p\(P+RV 24\,P.!W4\(P+R V 24EMMVP VP28XEd2V'd^R IHuHp^R IHp\P"P%V\ RV 24p\P!VRR7\(P+RV 24VP/R V /VP?V4V!4R7\(P+R V 24M~\ RV R VP& R2p \P"P%W=4p\(P+RV 24\,P.!W4\(P+R V 24R R R 4R # +'giR #;i)aZ Save an FSDP model checkpoint. When ``state_dict_type`` is ``SHARDED_STATE_DICT``, the ``use_dcp`` parameter controls the saving strategy: - ``use_dcp=True`` (default): Uses ``torch.distributed.checkpoint`` (DCP) to save sharded model state. This is the standard approach and works well for typical training setups. - ``use_dcp=False``: Uses per-rank ``torch.save`` instead of DCP. Each rank saves its own shard independently to a separate file, avoiding cross-rank communication entirely. This is significantly faster at large scale (e.g., 2800+ GPUs) where DCP's all-gather-based saving can cause timeouts due to TCP congestion from Gloo-based collective communication. .. note:: Checkpoints saved with ``use_dcp=False`` are saved as per-rank files (``{prefix}_rank{local_rank}.bin``) and must be loaded with ``use_dcp=False`` as well. FullyShardedDataParallelr=Texist_okr2r3.bin_zSaving model to zModel saved to _rankNDefaultSavePlannerr1r0storage_writerplanner) rDrMr>rmakedirsrErF num_processesrHr?r@rCoptim_state_dict_configrrJr4r pathjoin process_indexloggerinfotorchrLOCAL_STATE_DICTSHARDED_STATE_DICTtorch.distributed.checkpoint distributed checkpoint,torch.distributed.checkpoint.default_plannerrUFileSystemWriter)rI acceleratorr1 output_dir model_indexr2use_dcpFSDPr>is_multi_processctxr3r0 weights_nameoutput_model_filedist_cprUckpt_dirs&&&&&&& rsave_fsdp_modelrtrs$dPKK T*""m&C&CC'44q87G%%43C%%0  # #q (  .. 0M0M{OrOr ] %[1J *5Xbc  & &-*G*G G7Ba7Go.d3P_O``abmannrMsL " Z F ((A- ./@.ABC :9 o.?-@AB  ( (M,J,J J!###5)B)B(C4H'(+eK ? JJz 5 KK/*;)<= >  ( (M,L,L L>>[77<< 6Gq 4VW Ht4 .xj9: '4#*#;#;H#E.0  ohZ89"1!2!K=kF_F_E``de $&GGLL$J! ./@.ABC :9 o.?-@ABO sFN)D)N N- c ^RIHp^RIHpVP4VPVP 8Xd0VP ^8p WPnWPn VP^8Xd2VP W PVPVP4M \4p \V4p T ;_uu_4VPVP 8XEd!\V4VJd]VP^8wdLVP 'g:VP"'gVP^8Xd \%R4hRRR4R#V^8Xd \& R2M \& RV R2p \(P*P-W<4p \.P1RV 24VP '*;'g VP2pV'd\4P6!V RR 7pM/p\.P1R V 24EM'VPVP88XdV^8Xd\& R VP R2M\& RV R VP R2p \(P*P-W<4p \.P1RV 24\4P6!V RR 7p\.P1R V 24EMmVPVP:8XEdRV'd^RIHuH p^R I!H"p\GV4PH\& RV 28wd\GV4\& RV 2, M \GV4p\.P1RV 24R \KW%V R7/pVP7VVPMV4V!4R7VR ,p\.P1R V 24M\& RV R VP R2p \(P*P-W<4p \.P1RV 24\4P6!V RR 7p\.P1R V 24\OVXW[R7pRRR4VP4X# +'giL";i)au Load an FSDP model checkpoint. When ``state_dict_type`` is ``SHARDED_STATE_DICT``, the ``use_dcp`` parameter controls the loading strategy: - ``use_dcp=True`` (default): Uses ``torch.distributed.checkpoint`` (DCP) to load sharded model state. This is the standard approach and works well for typical training setups. - ``use_dcp=False``: Uses per-rank ``torch.load`` instead of DCP. Each rank loads its own shard independently from a separate file, avoiding cross-rank communication entirely. This is significantly faster at large scale (e.g., 2800+ GPUs) where DCP's broadcast-based loading can cause timeouts due to TCP congestion from Gloo-based collective communication. .. note:: Checkpoints saved with ``use_dcp=False`` must be loaded with ``use_dcp=False`` as well, since the file format differs (per-rank ``.bin`` files vs. DCP's sharded directory format). rLr=zzSet the `sync_module_states` flag to `True` so that model states are synced across processes when initializing FSDP objectNrQrRzLoading model from T weights_onlyzModel loaded from rSDefaultLoadPlannerr1rPr0storage_readerrX)(rDrMr>wait_for_everyonerErFrZrHr?r@rCr[rrJtyper^is_fsdp2sync_module_states ValueErrorr rr\r]r_r`is_main_processraloadrbrcrdrerfrgryrnamer4FileSystemReaderr:)rIrir1 input_dirrkr2rlrmr>rnror3rpinput_model_file load_modelr0rrryrs load_results&&&&&&& rload_fsdp_modelrs$dP!!#""m&C&CC'44q87G%%43C%%0  # #q (  .. 0M0M{OrOr ] %[1J   & &-*G*G GE{$&;+D+D+IR]RfRfRf"555+:R:RVW:W$3 8Ca7Go.d3P_O``abmannrMsL!ww||ID  KK-.>-?@ A(111PP[5P5PJ"ZZ(8tL  KK,-=,>? @  ( (M,J,J J!###5)B)B(C4H'(+eK-?@ A$44HJ KK,-=,>? @  ( (M,L,L L>>[I++/1B!K=/QQO(9;-&HHi  1(<=%'rrYrCrErHr[rrJr-roptim_state_dictrFr^r r\r]r_r`rarrdrerfrgrUrh)rIrirr1rjoptimizer_indexrlrmr>ror3r optim_stateoptim_state_nameoutput_optimizer_filerrrUrss&&&&&&& rsave_fsdp_optimizerr#sD dPKK T*  # #q (  .. 0M0M{OrOr ] %[1J   # #q ( X25ZXK//AK  & &-*G*G G((A-/>!/C~&d+NK[[\]l\mmqIr!)+ Z(R% 89N8OPQ ;> 78M7NOP>>[77<< ~6FaGX4YZ Ht4 8 CD +[9#*#;#;H#E.0  7zBC&4%5Q6Gu[MfMfLggk#l (* Z(R% 89N8OPQ ;> 78M7NOPI sC*J$1D)J$$ J5 c t^RIHp^RIHp VP4VP^8Xd2VP W0P VP VP4M \4p \V4p T ;_uu_4VP V P8XdRp VP^8XgVPP'gV^8Xd \ R2M \ RV R2p \PP!WM4p\"P%RV 24\&P(!VRR7p \"P%R V 24EMyV'd^RIHuHp^R IHp\5V4P6\ RV 28wd\5V4\ RV 2, M \5V4p\"P%R V 24VP^8Xd^R IHpV!W2V R 7p MVP=W24p RV /p VP)V VP?V4V!4R7V R,p \"P%RV 24M\ RV RVP R2p \PP!WM4p\"P%R V 24\&P(!VRR7p \"P%RV 24VP^8Xd&VPAW2V R7pVPCV4M^RIH"pV!W2WR 7RRR4VP4R# +'giL";i)a Load an FSDP optimizer state checkpoint. When ``state_dict_type`` is not ``FULL_STATE_DICT``, the ``use_dcp`` parameter controls the loading strategy: - ``use_dcp=True`` (default): Uses ``torch.distributed.checkpoint`` (DCP) to load the optimizer state. - ``use_dcp=False``: Uses per-rank ``torch.load`` instead of DCP, avoiding cross-rank communication. This is useful at large scale (e.g., 2800+ GPUs) where DCP can cause timeouts due to TCP congestion from Gloo-based collective communication. .. note:: Checkpoints saved with ``use_dcp=False`` must be loaded with ``use_dcp=False`` as well, since the file format differs (per-rank ``.bin`` files vs. DCP's sharded directory format). rLr=NrQrRzLoading Optimizer state from TrvzOptimizer state loaded from rxzLoading Optimizer from rr'rrzzOptimizer loaded from rS)r1optimr)set_optimizer_state_dict)#rDrMr>r|rCrErHr[rrJrFr^r@r rr\r]r_r`rarrdrerfrgryrrr-rrroptim_state_dict_to_loadr9r)rIrirr1rrr2rlrmr>ror3roptimizer_nameinput_optimizer_filerrryrsr flattened_osdrs&&&&&&&& rload_fsdp_optimizerris$dP!!#  # #q (  .. 0M0M{OrOr ] %[1J   & &-*G*G GK((A-[5X5X5c5c5c/>!/C~&d+NK[[\]l\mmqIr(*ww||I'N$ ;>[I++.1A?BS/TTO(8/9J&KKi  5hZ@A++q0`":5U_"`K"&"7"7"IK*K8  *#*#;#;H#E.0 *+6  4XJ?@$2#31_4EU;KdKdJeei!j')ww||I'N$ 56J5KLM#jj)=DQ  45I4JKL  # #q ( 99it9uM  % %m 4 X $U{ Wc f!!#g s>A L' B L'F7L'' L7 c<V^8dQhR\R\R\/#)checkpoint_dir save_pathsafe_serializationstrbool)formats"r __annotate__rs"ccgkrc^RIHuHp^RIHuHuHp/p\ V4pVP RR7VPVVPV4VP4RR7V'dV\, M V\, p\VP44^8XdV\V4^,,p\WQVR7V#)z Passthrough to `torch.distributed.checkpoint.format_utils.dcp_to_torch_save` Will save under `save_path` as either `model.safetensors` or `pytorch_model.bin`. NTrN)r{rXno_dist)r)rdrerf)torch.distributed.checkpoint.format_utils format_utilsrmkdir_load_state_dictr_EmptyStateDictLoadPlannerr rlenkeyslistr)rrrrrdist_cp_format_utilsr0s&&& r)_distributed_checkpoint_to_merged_weightsrs32LLJYI OOTO"))//?$??A * 2D --UaIaI :?? "Z 0 34 3EF rcHV^8dQhR\R\R\R\/#)rr output_pathrremove_checkpoint_dirr)rs"rrrs,444&)4?C4cg4rc"\V4p^RIHp\RR4'g \ R4hVP 4'gVR, P 4pVR, P 4pRV R2pV'd,V'd$VR , pVR V R V R 2, pVR , pM=V'dVR, pVRV R2, pMV'dVR, pVRV R2, p\ V4hV!4pVP 'ds\PRV 24\WV4p \PRV 24V'd/\PRV 24\P!V4VP4R#)a Merge the weights from sharded FSDP model checkpoints into a single combined checkpoint. Should be used if `SHARDED_STATE_DICT` was used for the model. Weights will be saved to `{output_path}/model.safetensors` if `safe_serialization` else `pytorch_model.bin`. Note: this is a CPU-bound process. Args: checkpoint_dir (`str`): The directory containing the FSDP checkpoints (can be either the model or optimizer). output_path (`str`): The path to save the merged checkpoint. safe_serialization (`bool`, *optional*, defaults to `True`): Whether to save the merged weights with safetensors (recommended). remove_checkpoint_dir (`bool`, *optional*, defaults to `False`): Whether to remove the checkpoint directory after merging. ) PartialState>=z2.3.0z/`merge_fsdp_weights` requires PyTorch >= 2.3.0`pytorch_model_fsdp_0 optimizer_0zTried to load from z) but couldn't find a valid metadata file.zE However, potential model and optimizer checkpoint directories exist.zPlease pass in either z/pytorch_model_fsdp_0 or z /optimizer_0zinstead.z8 However, a potential model checkpoint directory exists.zPlease try passing in z/pytorch_model_fsdp_0 instead.z< However, a potential optimizer checkpoint directory exists.z/optimizer_0 instead.zMerging FSDP weights from z.Successfully merged FSDP weights and saved to z"Removing old checkpoint directory N) raccelerate.staterrrexistsrr_r`rshutilrmtreer|) rrrrrmodel_path_existsoptimizer_path_existserrstaters &&&& rmerge_fsdp_weightsrs{(.)N- D' * *JKK  " "+.DDLLN!/-!? G G I#N#33\] !6 Z ZC +N+;;TUcTddpq qC : C  M MC +N+;;YZ ZC " Q QC +N+;;PQ QCo NE  00@AB=n[mn  DYKPQ KK<^.__annotate__"srch<\\4pVPRR7F8wr#\V4S9gKV\V4,P V4K: S!V4pVP 4F7wrEVF,pSV,pVf\ W4SV&K \WV4K. K9 V#)F)recurse)rrnamed_parametersidappenditemsrGsetattr) r params_to_tienparamid_key _param_names param_name _tied_params param_init_fns & rparam_init_fn_tied_param7ensure_weights_retied..param_init_fn_tied_param"s$D) ///>HA%yL(bi(//2? v&%2$7$7$9 F* $V,=,36+FL(F6+%: r)rGsplitr] get_submoduler) rr1r _tied_namesrrmodrrrs f&& @rensure_weights_retiedrs%!5ts/]]588??]T]`d]rc h^RIHp^RIHpHpVP 4p/pRp Rp VP 'EdVP4EFwrW9d(\RV R\V4 R\V4 R24hW+,p V PpV P4PVP4p \W4'dV P4p VP!V ^VP"P$R 7V!WV P&4pV !VV V 4wppV !VVV4pV'dVPR 4pWV &EK MVP4FwrV Pp\(P*!V P-4VPV P.R 7pVP!V^VP"P$R 7V!VWP&4pV !VV V4wppV !VVV4pV'dVPR 4pWV &K VP1VR R 7V#)au Loads the full state dict (could be only on rank 0) into the sharded model. This is done by broadcasting the parameters from rank 0 to all other ranks. This function modifies the model in-place. Args: accelerator (`Accelerator`): The accelerator instance model (`torch.nn.Module`): The model to load the state dict into, expected to be on meta device or a VRAM spike can occur full_sd (`dict`): The full state dict to load, can only be on rank 0 cpu_offload (`bool`, defaults to `False`): If True, move sharded parameters to CPU after distribution. Required when FSDP CPU offloading is enabled. N)DTensordistribute_tensorcVPV4p\ \ R4pRpT;'dVP\ P8Hp VPP'dV 'g VPpVRJ;'dVP4V3# \d4TPR^4wrETPT4p\ Ye4pLi;i)r float8_e4m3fnN) get_parameter_or_bufferAttributeErrorrsplitrrGhasattrradtyperis_floating_point is_contiguous) r1r empty_param old_parambase_param_namelocal_param_name submoduleis_torch_e4m3fn_available casting_dtypeis_param_float8_e4m3fns &&& r_infer_parameter_dtype:fsdp2_load_full_state_dict.._infer_parameter_dtypeSs =55jAI%,E?$C! !:!g!g{?P?PTYTgTg?g    . . .7M%OOM$BB)@)@)BMQQ =0:0A0A#q0I -O++O._cast_and_contiguouses.  YYUY+F &&(F rz Parameter 'zZ' found in sharded model state dict but missing from full state dict. Full state dict has z keys, sharded has z keys.)srcgroupcpu)rrT)assign)torch.distributedretorch.distributed.tensorrrr0rrKeyErrorr device_meshdetachr device_type isinstanceto_local broadcastrWORLD placementsraemptysizerr9)rir1rrAdistrrmeta_sharded_sd sharded_sdrrr sharded_param full_paramr sharded_tensorrr full_tensors&&&& rfsdp2_load_full_state_dictr>s'%C&&(OJR$""")8)>)>)@ %J(!*.++.w<.8KCP_L`Kaagi!,J'33K#**,// 0G0GHJ*..(002 NN:1DJJ4D4DN E.z H`H`aN+A, (M= 2.-Q^_N!/!2!25!9%3z "3*A8*9)>)>)@ %J'33K++m&8&8&:;CZCZbobubuvK NN;ATZZ5E5EN F.{KIaIabN+A, (M= 2.-Q^_N!/!2!25!9%3z "*A" *T2 LrcXV^8dQhR\PPR\/#)rrmapping)rar Optimizerr)rs"rrrs%  1F1F QU rc^RIHp/pRW2&VPF/pVR,Uu.uFqQVP,NK upVR&K1 R#uupi \d \ R4hi;i)a Switches the parameters of the optimizer to new ones (sharded parameters in usual case). This function modifies the optimizer in-place. Args: optimizer (`torch.optim.Optimizer`): Optimizer instance which contains the original model parameters mapping (`dict`): Mapping from the original parameter (specified by `data_ptr`) to the sharded parameter Raises: KeyError: If a parameter in the optimizer couldn't be switched to its sharded version. This should never happen and indicates a bug. If we kept the original params instead of raising, the training wouldn't be numerically correct and weights wouldn't get updated. )r _local_tensorparamszA parameter in the optimizer couldn't be switched to its sharded version. This breaks the training. Please raise an issue on GitHub.N)r r param_groupsdata_ptrr )rrraccessor_mapping param_groupps&& r!fsdp2_switch_optimizer_parametersr)s|1 / $11KBMhBW$XBWQQZZ%8%8BW$XK !2$X   S   sAA AAA+cLV^8dQhR\PP/#)rr1r)rs"rrrsuxxrc:^RIHpHp\VPP V4p\ VRR7RRF^wrVV!V4'gKV!VRR7pVPP P'd V!V4pVPWW4K` V#)a Applies the activation checkpointing to the model. Args: accelerator (`Accelerator`): The accelerator instance model (`torch.nn.Module`): The model to apply the activation checkpointing to Returns: `torch.nn.Module`: The model with the activation checkpointing applied )checkpoint_wrapperoffload_wrapperT) return_fqnsNF)preserve_rng_stater) ;torch.distributed.algorithms._checkpoint.checkpoint_wrapperr,r-fsdp2_prepare_auto_wrap_policyrrIr activation_checkpointing_offload set_submodule)rir1r,r-auto_wrap_policy_func layer_namelayerwrappeds&& rfsdp2_apply_acr8s ;;;L;L;X;XZ_`:5dSTWUWX  ' '(5IG  ,,MMM*'2    4Y LrcV^8dQhR\PPR\PPR,/#rr1returnNr)rs"rrrs/EHHOO$0Frc\VRR4pV'd \WR4MTp\V\PP4'gR#RpVP 4F#wrER\ V4P9gK!TpK% V#)aFind the final normalization layer before the output head. The final norm is conventionally a direct child of the base model (e.g. `model.norm` for Llama, `transformer.ln_f` for GPT-2), so we only scan the base model's direct children. Returns the last norm found there, or None. base_model_prefixNNorm)rGrrarrnamed_childrenr}__name__)r1 base_prefix base_model final_normrRrs& r_find_final_normrEst%!4b9K6AT2uJ j%((// 2 2J..0  T&\** *J1 rcV^8dQhR\PPR\PP/#)rr1r;r)rs"rrrs.ggEHHOOggrc  ^RIHpHpHp\ W4;'g*\ V4;'d\ VP V4pV'dV#VPPpVPV4\VRR4pRVPRVPRVP;'gV!4RVe'V\VPP 4,MR/p\#RR 4'd2VP$e$\'VP$WP(4VR &R p \+4p VP-4FywrV P.P0R 8XgK"R p V P2'dK8V P54'dKPV P74'dKhV P9V 4K{ V 'dv\#RR 4'dd\+VP;R \+444p W,VR &VP<'d$\>P@!R\CV 4 R24VPDR8wdV 'g.pVP-4FywrV P2'gKV PF\HPJ8wgK:VPMV 4V PNPQ\HPJ4V n'K{ VP<'d.V'd&\RPUR\CV4 RV R24VPW4pVPX'dV 'g\[VR R R7p\\P^!VPa4UUu/uFwppVV9gKVVbK upp4pVPQ\HP(!R44p\cVR4'dVPe4\gWa4pVeC\iV4RR F0pV!V4'gK\ VV4'dK'V!V3/VBK2 \VRR4!4p\VRR4!4p\VRR4p\VRR4pVRJ;'dVVJpV'g Ve\ VV4'g V!V3/VB\kV4pV'dVVV3pMVV3pVUu.uFpVfK \ VV4'dKVNK! ppV'dV!V3//VCRR /CB\ W4'g8VPm4UUu/uFwppVR8wgKVVbK pppV!V3/VBVPX'd1V 'g)^RIH7p \qWV\ VPV 4R7VPX'dV 'gXPm4Fgwp!p"V"PQVP(4p"RV!9d(V!PsR^4wp#p$VPuV#4p%MT!p$Tp%V%PwV$V"R R7Ki \cVR4'dVPe4V#uuppiuupiuuppi)!a Prepares the model for FSDP2 in-place. Also returns the model to avoid misuse of the original model. Args: accelerator (`Accelerator`): The accelerator instance model (`torch.nn.Module`): The model to prepare Returns: `torch.nn.Module`: Prepared model ) FSDPModuleMixedPrecisionPolicy fully_shardtorch_device_meshNreshard_after_forwardoffload_policy mp_policymeshrz2.7.0ignored_paramsF Params4bitTzFound z non-floating frozen Params4bit. Excluding from FSDP2 sharding to prevent quant_state corruption.To enable memory-efficient sharding of 4-bit weights, setbnb_4bit_quant_storage to a floating dtype (e.g. bf16).nozFSDP upcast of low precision parameters to fp32 (since mixed_precision != 'no') may affect the precision of model checkpoints. This effects z parameters: z...)rfqnsmeta tie_weightsget_input_embeddingscR#Nrrrr%fsdp2_prepare_model..[srget_output_embeddingscR#rXrrrrrYrZ\s4rweight)CPUOffloadPolicy)rAr) persistentr)<torch.distributed.fsdprHrIrJrr _orig_modrrIset_auto_wrap_policyrGrLrAmixed_precision_policytupleparallelism_configfsdp_dim_namesrignored_modulesget_parameters_from_modulesrsetr __class__rA requires_gradr is_complexaddgetrwarningswarnrmixed_precisionrrafloat32rdatarr_r`r0cpu_ram_efficient_loadingrcopydeepcopy named_buffersrrUr1rrErr^rrrregister_buffer)&rir1rHrIrJ is_type_fsdp fsdp2_pluginrO fsdp2_kwargsmodel_has_params4bitincompatible_params4bitrrignoredupcasted_params original_sdnon_persistent_buffer_fqnskvoriginal_non_persistent_buffersr4r input_embed output_embed input_weight output_weightis_weights_tiedrD tail_modulesmtail root_kwargsr^fqn buffer_tensor parent_fqnlocal_buffer_name parent_modules&&& rfsdp2_prepare_modelrsUTe05!MMj*&M $$00L%%e, ; 3T :D !C!C,22\88RR "L1 # W|!qAZ:=VAA|D W DML|L-DeLM e ( ()5(:(:(<](<1E\@\tq!t(< ]E)[)---6J <"  LD\D\^n9o ---6J"A"G"G"I C),,[-?-?@Mcz03 30B- - % 3 3J ? $'! %  ) )*;]W\ ) ]#J 5- ( (     L[ XH X^s* Y Y YY0Y3 YYcV^8dQhR\PPR\\PP.\3,R,/#r:)rarrrr)rs"rrrsC333HV[V^V^VeVeUfhlUlLmptLt3rcaa ^RIHpHpSPp\ V\ P 4'd VPpWCJd\VRR4pVf.p\V4pSPe SPp\4o VF2p\W4pVf\RV R24hS PV4K4 RV 3Rllp V #WBJd RV3R llp V #R#) aPrepares the auto wrap policy based on its type, done to mimic the behaviour of FSDP1 auto wrap policy. Args: fsdp2_plugin (`FullyShardedDataParallelPlugin`): Instance of `FullyShardedDataParallelPlugin` containing the configuration options model (`torch.nn.Module`): The model to wrap Returns: `Callable[[torch.nn.Module], bool] | None`: The auto wrap policy function to be applied to the model or `None` )size_based_auto_wrap_policytransformer_auto_wrap_policy_no_split_modulesNz+Could not find the transformer layer class z in the model.cXV^8dQhR\PPR\/#rrr;rarrr)rs"rr4fsdp2_prepare_auto_wrap_policy..__annotate__s& F F588?? Ft Frc\<S'gR#\VRV4p\V\S44#)F_checkpoint_wrapped_module)rGrrd)rtransformer_cls_to_wraps&rpolicy.fsdp2_prepare_auto_wrap_policy..policys-*V%A6JFfe,C&DE ErcXV^8dQhR\PPR\/#rr)rs"rrrs& C C588?? Ct Crcb<\RVP444pVSP8#)c3@"TFqP4xK R#5irX)numel).0r(s& r Afsdp2_prepare_auto_wrap_policy..policy..s#K7J!GGII7Js)sum parametersmin_num_params)rmodule_num_paramsrzs& rrrs. ##Kv7H7H7J#K K $|'B'BB Br)torch.distributed.fsdp.wraprrauto_wrap_policyr functoolspartialfuncrGrtransformer_cls_names_to_wraprirrrm) rzr1rrfnno_split_modulesr layer_classtransformer_clsrrs f& @rr1r1sf  & &B"i''(( WW )"5*=tD  #! (,-=(>%  5 5 A,8,V,V )"%%8K8LO& #N{m[i!jkk # ' ' 8 9  F F M * C C Mrc ^RIHpV!R/VB#)z Returns a `GradScaler` for FSDP2, as the current implementation of `get_grad_scaler` doesn't accept other args. We need this as current `get_grad_scaler` accepts only `distributed_type` as arg, which doesn't differentiate between FSDP1 and FSDP2 ) GradScalerr)torch.amp.grad_scalerr)kwargsrs, rget_fsdp2_grad_scalerrs 1   rc0V^8dQhR\R\/#)r named_paramsr;)r)rs"rrrs4DrcVP4UUu/uFwrVPRR4VbK pppVP4UUu/uF1wrVPR4'dVPRR4MTVbK3 pppVP4UUu/uFwrVPRR4VbK pppV#uuppiuuppiuuppi)aRemoves parameter name modifiers in order to map them back to their original names. See huggingface/accelerate#3554 for more context. Args: named_params (`dict`): The named parameters dictionary to canonicalize. Returns: `dict`: The canonicalized named parameters dictionary z._checkpoint_wrapped_moduler>z _orig_mod.z ._orig_mod)rr startswith)rrrs& rfsdp2_canonicalize_namesrsQ]PbPbPdePdAII;R@!CPdLeXdXjXjXlXlPTPQq||L'A'A ,#q!KXl@L?Q?Q?ST?StqAIIlB/2?SLT  fUsB87B>CcV^8dQhR\\\PP,\ 3,R\ \PP,/#)rmodulesr;)r rrarrrri Parameter)rs"rrrsB 8EHHOO,c1 2  rcVf \4#.p\V\4'do\P!V4p.pVP 4F@wrgVP V4'gKVPV4VPV4KB TpVF+pVP\VP444K- \V4#)zConverts modules to parameters where modules can be a string or list of torch.nn.Module Args: modules (`Union[Iterable[torch.nn.Module], str]`): List of modules Returns: `set[torch.nn.Parameter]`: List of parameters ) rirrrecompile named_modules fullmatchrrextendrr)rr1rrregmapped_modulesrrs&&& rrhrhsu J'3jj!!//1LD}}T"" &!%%f-2!$v00234 z?r)FN)rFT)rT)T)TF)F)rs  #$"" WW3?JJ& H :;"(1("MC`^BCQLQ$h84n+$\]@ >@$gT3l &r