+ j) ^RIt^RIt^RIHt^RIHt^RIt^RIHt ^RI H t ^RI H t Ht^RIHt^RIHtHt^RIHtHtHtHtHt^R IHtHtHt^R IHt^R I H!t!^R I"H#t#^R I$H%t%^RI&H't'.R)Ot(!RR]%4t)!RR]4t*RRlt+RRlt,]-R8XEd!RR]4t.]P^!4t0]0PcR]2R].Uu.uFqPfNK up].PhR7]0PcR]2RR7]0PcR ]2R!R7]0Pk4t6]7!R"]6Pp R#]6Pr R$]6Pt R%24R&]6Pp R'2t;]6Pt].PhPf8XdU]PxP{]6Pp4'd ],!]6Pp]6Pr4R#]7!];4R#]6Pt].P|Pf8XdU]PxP]6Pp4'd ]+!]6Pp]6Pr4R#]7!];4R#]@!R(]6Pt 24hR#uupi)*N)Enum)cast)narrow_tensor_by_index)FileSystemReaderFileSystemWriter)flatten_state_dict)_EmptyStateDictLoadPlannerDefaultLoadPlanner)MetadataSTATE_DICT_TYPE STORAGE_TYPESTensorPropertiesTensorStorageMetadata) LoadItemTypeLoadPlan LoadPlanner)_create_chunk_list)_load_state_dict)_save_state_dict) StorageReader)FutureBroadcastingTorchSaveReaderDynamicMetaLoadPlannerca]tRt^'toRtRV3RlRlltV3RlRltV3RlRltV3R lR ltV3R lR lt V3R lRlt RV3RlRllt ] V3RlRl4t RtVtR#)ra StorageReader for reading a Torch Save file. This reader will read the entire checkpoint on the coordinator rank, and then broadcast and shard each tensor to all ranks. . N.B. Intended to be used with DynamicMetaLoadPlanner .. warning:: Current implementation only supports loading Tensors. >>> # xdoctest: +SKIP("undefined vars") >>> sd = {"mode": model} >>> dcp.load( >>> sd, >>> storage_reader=BroadcastingTorchSaveReader(), >>> planner=DynamicMetaLoadPlanner(), >>> checkpoint_id="path_to_model.pt" >>> ) Nc\<V^8dQhRS[S[P,R,RS[RR/#) checkpoint_idNcoordinator_rankreturn)strosPathLikeint)format __classdict__s"j/home/luca/screen-checking/.venv/lib/python3.14/site-packages/torch/distributed/checkpoint/format_utils.py __annotate__(BroadcastingTorchSaveReader.__annotate__;s711R[[(4/11  1cWnW nR#N)rr)selfrrs&&&r&__init__$BroadcastingTorchSaveReader.__init__;s + 0r)c <V^8dQhRS[/#)rrr )r$r%s"r&r'r(Ds00x0r)c \/R7#)zGExtends the default StorageReader to support building the metadata filestate_dict_metadatar0)r,s&r& read_metadata)BroadcastingTorchSaveReader.read_metadataDsB//r)c:<V^8dQhRS[RS[RS[R,/#)rplanplannerrN)rrr)r$r%s"r&r'r(Js&44h444r)c  \\V4pVP'd]VPf \ R4h\ P !VPRRR7pVP'd\V4wr4MRpVPEFpVP\P8Xd:\RVPP R\V4P R24hVP'dK\ P"P%4pW5PP,P'V4pM;\ P(!VP*VPP,4p\ P,!WpP.RR 7\1WuP2VP44pVP7V4P94pVP;4VP;48Xg;\ R VP R VP;4 R VP;4 24hVP=V4VP?WX4EK \A4p V PCR4V #) z Reads torch save data on the coordinator rank, and broadcast afterwards this incurrs a communication cost, but avoids having to load the entire checkpoint on each rank, hopefully preventing OOM issues Nz-checkpoint_id must be set before reading datacpuF) map_location weights_onlyNon-tensor value identified at . At this time  only supports loading Tensors.)srcasync_opzreq z mismatch sizes, z vs )"rr is_coordinatorrAssertionErrortorchloadritemstyperBYTE_IO RuntimeError storage_indexfqn__name__distdistributed_c10d_get_pg_default_deviceto empty_like state_dict broadcastrrstorage_offsetslengthsresolve_tensordetachsizecopy_ commit_tensorr set_result) r,r7r8torch_state_dict_req pg_devicetensor target_tensorfuts &&& r& read_data%BroadcastingTorchSaveReader.read_dataJs )73    !!)$%TUU$zz""U  )))&89I&J# # ::Cxx<///"5c6G6G6K6K5LM$$(J$7$7#88WY """ 11HHJ )*;*;*?*?@CCIN))'*<*>@M %%'6;;=8$3,,-->$))+,DA    '  ! !# 536h t r)c*<V^8dQhRS[RS[RR/#)rmetadatarBrN)r bool)r$r%s"r&r'r(s#  h  QU r)c W nVP'dS\P!4VP8Xg/\ RVP R\P!4 24hVP f \ R4hR#)*Implementation of the StorageReader methodz$Coordinator rank mismatch: expected z, got Nz:checkpoint_id must be set before setting up storage reader)rBrMget_rankrrCr)r,rfrBs&&&r&set_up_storage_reader1BroadcastingTorchSaveReader.set_up_storage_readers},    ==?d&;&;;$:4;P;P:QR==?+-    % L  &r)c&<V^8dQhRS[RS[/#)rr7r)r)r$r%s"r&r'r(sxHr)c V#ri)r,r7s&&r&prepare_local_plan.BroadcastingTorchSaveReader.prepare_local_plans r)cF<V^8dQhRS[S[,RS[S[,/#)r global_planr)listr)r$r%s"r&r'r(s#tH~$x.r)c V#rorp)r,rts&&r&prepare_global_plan/BroadcastingTorchSaveReader.prepare_global_plansr)cV<V^8dQhRS[S[P,R,RR/#)rrNrr r!r")r$r%s"r&r'r(s)++3#4t#;+t+r)c WnR#)riN)r)r,rs&&r&reset!BroadcastingTorchSaveReader.resets*r)cJ<V^8dQhRS[S[P,RS[/#)rrr)r r!r"rg)r$r%s"r&r'r(s$--33D--r)c @\PPV4#ro)r!pathisfile)clsrs&&r&validate_checkpoint_id2BroadcastingTorchSaveReader.validate_checkpoint_idsww~~m,,r))rrrB)Nrr+)rL __module__ __qualname____firstlineno____doc__r-r4rcrkrqrwr| classmethodr__static_attributes____classdictcell__)r%s@r&rr'se&1100 44n  ++--r)cFaa]tRt^toRtRV3RlV3RllltRtVtV;t#)ra` Extension of DefaultLoadPlanner, which creates a new Metadata object based on the passed in state dict, avoiding the need to read metadata from disk. This is useful when reading formats which don't have a metadata file, like Torch Save files. . N.B. Intended to be used with BroadcastingTorchSaveReader .. warning:: Current implementation only supports loading Tensors. >>> # xdoctest: +SKIP("undefined vars") >>> sd = {"mode": model} >>> dcp.load( >>> sd, >>> storage_reader=BroadcastingTorchSaveReader(), >>> planner=DynamicMetaLoadPlanner(), >>> checkpoint_id="path_to_model.pt" >>> ) c><V^8dQhRS[RS[R,RS[RR/#)rrRrfNrBr)r r rg)r$r%s"r&r'#DynamicMetaLoadPlanner.__annotate__s=JJ#JT/J J  Jr)c  <\SV`WV4/pVPP4FwrV\P !V4'g&\ RV R\V4P R24h\\VPR7VP4\V44WE&K \VR7VnR#)zdSetups of the planner, extnding default behavior by creating the Metadata object from the state dictr=r>r?)dtyper2N)superset_up_plannerrRrFrD is_tensorrIrGrLrrrrXrr rf)r,rRrfrBr3keyr` __class__s&&&& r&r%DynamicMetaLoadPlanner.set_up_planners z^D8:??002KC??6**"5cU;$$(J$7$7#88WY (= v||4 "6*(  $3!5HI r))rf)NF) rLrrrrrrr __classcell__)rr%s@@r&rrs(JJJr)cV^8dQhR\\P,R\\P,/#)rdcp_checkpoint_dirtorch_save_pathrz)r$s"r&r'r's.$$bkk)$2;;&$r)cv/p\V\V4\4RR7\P!W!4R#)aQ Given a directory containing a DCP checkpoint, this function will convert it into a Torch save file. Args: dcp_checkpoint_dir: Directory containing the DCP checkpoint. torch_save_path: Filename to store the converted Torch save file. .. warning:: To avoid OOM, it's recommended to only run this function on a single rank. T)storage_readerr8no_distN)rrr rDsave)rrsds&& r&dcp_to_torch_savers4B '(:;*,   JJr#r)cV^8dQhR\\P,R\\P,/#)rrrrz)r$s"r&r'r's.2;;&bkk)r)cd\P!VRR7p\V\V4RR7R#)a& Given the location of a torch save file, converts it into a DCP checkpoint. Args: torch_save_path: Filename of the Torch save file. dcp_checkpoint_dir: Directory to store the DCP checkpoint. .. warning:: To avoid OOM, it's recommended to only run this function on a single rank. F)r<T)storage_writerrN)rDrErr)rrrRs&& r&torch_save_to_dcprs-O%@J#34F#GQUr)__main__c]tRtRtRtRtRtR#) FormatModei torch_to_dcp dcp_to_torchrpN)rLrrr TORCH_TO_DCP DCP_TO_TORCHrrpr)r&rrs % % r)rmodezConversion mode)rGhelpchoicesdefaultr@zPath to the source model)rGrdstzPath to the destination modelzConverting checkpoint from z to z using method: ''zNo checkpoint found at z. Skipping conversion.zUnknown conversion mode: )rrrr)Aargparser!enumrtypingrrDtorch.distributed distributedrMtorch.distributed._shard._utilsrtorch.distributed.checkpointrr)torch.distributed.checkpoint._nested_dictr,torch.distributed.checkpoint.default_plannerr r %torch.distributed.checkpoint.metadatar r r rr$torch.distributed.checkpoint.plannerrrr,torch.distributed.checkpoint.planner_helpersr.torch.distributed.checkpoint.state_dict_loaderr-torch.distributed.checkpoint.state_dict_saverr$torch.distributed.checkpoint.storager torch.futuresr__all__rrrrrLrArgumentParserparser add_argumentr valuer parse_argsargsprintr@rrcheckpoint_missing_warningrrrisdir ValueError)ms0r&rs  BKHUTKKJ>  x--x-v+J/+J\$2. z&T&  $ $ &F   ",-*Q*-''  C.HI C.MN    D %dhhZtDHH:=Mdii[XYZ "$((+AB yyJ++111 77>>$(( # # dhh 1 , - j--33 3 77== " " dhh 1 , -4TYYK@AAI.s I5