+ j<RaRt00t^RIt^RIt^RIt^RIt^RIt^RIHt^RIt^RI H t H t ^RI H t ^RIHtHtHtHtHtHtHt^RIHt^RIHtHtHtHtHtHt^RIH t H!t!H"t"H#t#H$t$H%t%^R I&H't'^R I(H)t)]PT!]+4t,]^kR R .t-!R R ] 4t.!RR ] 4t/R#)N)Any)FileSystemReaderFileSystemWriter)consolidate_safetensors_files)_gen_file_name_HFStorageInfo _metadata_fnCUSTOM_METADATA_KEYSAVED_OFFSETS_KEYSHARDED_DIR_NAMESUFFIX)SerializationFormat)ChunkStorageMetadataMetadata MetadataIndex StorageMetaTensorPropertiesTensorStorageMetadata)LoadPlan LoadPlannerReadItemSavePlan SavePlanner WriteItem) WriteResult)FutureHuggingFaceStorageWriterHuggingFaceStorageReadercaa]tRt^1toRtRV3RlV3RllltV3RlRltV3RlV3RlltV3RlR ltV3R lR lt ] V3R lR l4t Rt Vt V;t#)rzH A writer that writes to storage in the huggingface safetensors format. cf<V^8dQhRS[RS[S[S[3,R,RS[RS[RS[RS[RR/#) pathfqn_to_index_mappingN thread_countsave_distributedenable_consolidationthread_count_consolidationreturn)strdictintbool)format __classdict__s"h/home/luca/screen-checking/.venv/lib/python3.14/site-packages/torch/distributed/checkpoint/hf_storage.py __annotate__%HuggingFaceStorageWriter.__annotate__6sg'E'E'E#38nt3'E 'E  'E # 'E%('E 'Ec <<\SV`V\PVR7W nW@nWPnRVnVP 'dJ\VP4VnVPPVP\4Vn W`n R#)a Initialize the huggingface writer pointing to path. Args: path: directory where the checkpoint will be read from. fqn_to_index_mapping: A mapping from tensor FQN to the index of the file that the tensor should be written to. Indices are from 1 to N, where N is the number of files. If not provided, the tensors will be written to a single file. If none, then all the tensors on the same rank will be written to the same file. thread_count: Number of threads to use to write distributed checkpoint. Default to 1. save_distributed: If True, save the checkpoint using distributed APIs where every rank saves its own shard. Default is False which assumes rank-0 checkpointing of the full state_dict. enable_consolidation: If True, consolidate the sharded checkpoint after saving. The sharded tensors will be saved to path/sharded and the full tensors will be saved to path. Default to False. thread_count_consolidation: Number of threads to use for parallel processing of saving data to consolidated output files. Default to 1. )r"serialization_formatr$N)super__init__r SAFETENSORSr#r%r&consolidated_output_pathr)r"fs concat_pathr r')selfr"r#r$r%r&r' __class__s&&&&&&&r/r6!HuggingFaceStorageWriter.__init__6s6 !4!@!@%  !48%  $ $ $,/ ND )++DII7GHDI*D'r2cF<V^8dQhRS[S[,RS[S[,/#)r!plansr()listr)r-r.s"r/r0r1_s#  h DN r2c.p\V^R7F`wr4/pVPeVPVR&VP'dW5R&VP\P !WER74Kb V#))startr# shard_index) storage_data) enumerater#r%append dataclassesreplace)r;r? new_plansiplanrEs&& r/prepare_global_plan,HuggingFaceStorageWriter.prepare_global_plan_sq  a0GA+-L((47;7P7P 34$$$./]+   [00Q R1r2cL<V^8dQhRS[RS[RS[S[S[,,/#)r!rLplannerr()rrrr@r)r-r.s"r/r0r1ls38888 [! " 8r2cF<\VP4^8Xd\4pVP.4V#VPpRpRpRV9d VR,pRV9d VR,pVP WQP4pVe\ VP44M^p\P!4p VP4FHwr\WV4p V PVPPVPV 4W34KJ \S V`AW)4#)rNr#rD)lenitemsr set_resultrE_split_by_storage_planmaxvaluesqueueQueuerputr9r:r"r5 _write_data)r;rLrPfutrE storage_planrDbuckets highest_index file_queue file_index write_items file_namer<s&&& r/ write_data#HuggingFaceStorageWriter.write_datals tzz?a  (C NN2 J(,'8'8 .2 "& !\ 1'(>?L L (&}5K--lJJG6B6NL//12TU "'++- '.}} #J&z+NI NN$$TYY :IS (7 w"777r2cJ<V^8dQhRS[RS[S[S[,,RR/#)r!metadataresultsr(N)rr@r)r-r.s"r/r0r1s1%B%Bx%B$tK7H2I%Bd%Br2cVP'd*VP'g\PR4R#VP'dVPe VPM-\ P VPP4^4p\\VP4VPVPVR7#/p/p^pVFpTPVUu/uF-qPP VP"P$bK/ up4T\'VUu.uFqP"P(NK up4, pK RV/VR&WTR&VP*P-VP\. 4p VP*P1V R4;_uu_4p \2P4!WJ^R7RRR4R#uupiuupi +'giR#;i)z4Not consolidating sharded checkpoint in finish step.N) input_dir output_dir num_threadsr# total_sizerg weight_mapw)indent)r%r&loggerinfor#r*fromkeysstate_dict_metadatakeysrr)r"r8r'updateindexfqnrE relative_pathsumlengthr9r:r create_streamjsondump) r;rgrhr#metadata_to_write storage_mdrmwr_listwr metadata_path metadata_files &&& r/finishHuggingFaceStorageWriter.finishs  )B)B)B KKN O   ,,8))]]8#?#?#D#D#FJ ! 1dii.88 ;;%9    G   GNOwr<<y/ c4 ?" #r2cVf^V/#/pVFEpVPPpW,pWc9dV.W6&K.W6,PV4KG V#N)rwrxrG)r;r]rSr^itemkeyidxs&&& r/rU/HuggingFaceStorageWriter._split_by_storage_plans^  u: D**..C#C! $v  ##D)r2c <V^8dQhRS[/#r!r()r))r-r.s"r/r0r1ssr2c\#r)r )r;s&r/r&HuggingFaceStorageWriter.metadata_pathsr2)r8r&r#r"r%r')NrBFFrB)__name__ __module__ __qualname____firstlineno____doc__r6rMrdrrUpropertyr__static_attributes____classdictcell__ __classcell__r<r.s@@r/rr1s['E'ER  88>%B%BN&r2caa]tRt^toRtR V3RlV3RllltV3RlRltV3RlRltV3RlR ltV3R lR lt R t Vt V;t #)rzI A reader that reads a checkpoint in the huggingface safetensors format. c*<V^8dQhRS[RS[RR/#)r!r"r$r(N)r)r+)r-r.s"r/r0%HuggingFaceStorageReader.__annotate__s" ) )S ) )D )r2c 4<\SV`VR7W nR#)z Initialize the huggingface reader pointing to path. Args: path: directory where the checkpoint will be read from. thread_count: Number of threads to use to read distributed checkpoint. Default to 1. )r"N)r5r6r$)r;r"r$r<s&&&r/r6!HuggingFaceStorageReader.__init__s d#(r2c*<V^8dQhRS[RS[RR/#)r!reqrPr(N)rr)r-r.s"r/r0rs"22H2{2t2r2c  j\;QJd2.R\VPVP44FNK 5M+!R\VPVP444pVP VP P 4V,pVPV4P4pVP4VP48wd;\RVP RVP4 RVP4 24hVPV4VPW&4R#)z1Helper function to process a single read request.c3J"TFwr\WV,4xK R#5ir)slice).0offsetr{s& r/ AHuggingFaceStorageReader._process_read_request..s% "G &6/ * *"Gs!#zreq z mismatch sizes z vs N) tuplezipstorage_offsetslengths get_slice storage_indexrxresolve_tensordetachsizeAssertionErrorcopy_ commit_tensor)r;frrPslicestensor target_tensors&&&& r/_process_read_request.HuggingFaceStorageReader._process_read_requests "%c&9&93;;"G  "%c&9&93;;"G  S..223F;..s3::<    6;;= 0 s(())9-:L:L:N9OtTZT_T_TaSbc  F#c1r2cX<V^8dQhRS[PRS[PRS[RR/#)r!r` result_queuerPr(N)rXrYr)r-r.s"r/r0rs;KKkk   r2c&^RIHpVP4wrVV!VRR7;_uu_4pVFpVPWxV4K RRR4VP R4K\ +'giL#;i \ P dR#i;i)r safe_openTptfilename frameworkN) safetensorsr get_nowaitrrZrXEmpty) r;r`rrPrrcreqsrrs &&&& r/_read_files_from_queue/HuggingFaceStorageReader._read_files_from_queues * ","7"7"9  TBBa#2217C $C  &CB{{   s(%A7A$ A7$ A4 /A77BBc:<V^8dQhRS[RS[RS[R,/#)r!rLrPr(N)rrr)r-r.s"r/r0rs&77h777r2c ^RIHp/pVPFMpVPVP,pVP pVP V.4PV4KO VP^8:g\V4^8:dRVP4F;wrxV!VRR7;_uu_4p VFp VPWV4K RRR4K= EM\P!4p \P!4p VP4FwrxV PWx34K .p \VP\V44p\V4FHp\ P"!VP$WV3R7pVP'4V PV4KJ V FpVP)4K ^pV P+4V^, pK\14pVP3R4V# +'giEKu;i \P,dMi;iT\T48wgK_\/RT R\T4 24h)rrrrN)targetargszNot all files were processed: z out of )rrrSrErry setdefaultrGr$rRrrXrYrZminrange threadingThreadrrCjoinrrrrrT)r;rLrPrper_file read_itemitem_mdrcrrrr`rthreadsrl_tprocessed_countr\s&&& r/ read_data"HuggingFaceStorageReader.read_datas).0I&*&7&7 8O8O&PG--I    2 . 5 5i @$    !S]a%7#+>>#3  TBBa#2217C $CB$4 ',kkmJ(- L$,>>#3  01$4Gd//X?K;'$$66$G< q! ( O  ++-#q(Oh t WCBBB;;  #h-/$4_4EXcRZm_]s.G:H: H H'&H'c <V^8dQhRS[/#r)r)r-r.s"r/r0r9sNNxNr2c\^RIHp^RIHp/p/p.pVPP VP 4F1pVP\4'gK VPV4K3 VEFpV!VRR7;_uu_4pVP4p VP4p Rp V 'dEV P\4'd*\P!V P\44p V EFCp VP!V 4P#4p VP!V 4P%4pV eW,\&,pM^.\)V 4,pW9d\+\-V!V4R7\.P0!\3W4UUu.uFwppVV,NK upp4\5\.P0!X4\.P0!V 4R7.R7W<&MW<,P6P\5\.P0!V4\.P0!V 4R 74\9W<,P:4p\=\)V44F.p\?VV,V V,VV,,4VV&K0 \.P0!V4W<,nV e \AWV ,\&,R 7pM\AV ^.\)V 4,R 7p\CV\.P0!V 4V!V4R 7VV&EKF RRR4EK \EVVR 7p\GVR R4f\I4Vn%VPLVPJn&V#uuppi +'giEK<;i)rr) _getdtyper)rN)dtype)offsetssizes) propertiesrchunks)r)rxr)ryshaper)rtrE storage_meta)'rrsafetensors.torchrr9lsr"endswithr rGrurggetr r}loadsr get_shape get_dtyper rRrrtorchSizerrrr@rrrVrrrgetattrrrload_id)r;rrrtrEsafetensors_filesfilesafetensor_filerruextra_metadatadcp_sharding_inforrrrsavedrrKmetadata_indexrgs& r/ read_metadata&HuggingFaceStorageReader.read_metadata9s)/@B<> GGJJtyy)D}}V$$!((.* 1O?d;;qvvx!"$(!!n&8&89L&M&M(, &**+>?)% CKK,668EKK,668E(4!2!78I!J"#s5z!153H'7i>N'O!&EHEW XEWME6EW X"!5,1JJv,>*/**U*;!"$ 4+0,077>>0 % 6 2%**U:K $$7$<$A$AB!&s4y!1A&)$q'58fQi3G&HDG"28= 48H+05)4)6 #c,BCT,U**73sSQVZGW)X3A&5#jj/'.4L0Q <; 1r 3%  8^T 2 :$/MH !(, %U!Y/<;;s'AN CNN2F NN N+ )r$)rB) rrrrrr6rrrrrrrrs@@r/rrsE ) )22$$77tNNNr2cTV^8dQh/^\9d\P;R&#)r!rq)__conditional_annotations__loggingLogger)r-s"r/r0r0s T544Ur2)1rrHr}rrXrtypingrrtorch.distributed.checkpointrr8torch.distributed.checkpoint._consolidate_hf_safetensorsr&torch.distributed.checkpoint._hf_utilsrrr r r r r 'torch.distributed.checkpoint.filesystemr%torch.distributed.checkpoint.metadatarrrrrr$torch.distributed.checkpoint.plannerrrrrrr$torch.distributed.checkpoint.storager torch.futuresr getLoggerrrq__all__rrr0)rs@r/r s   KH= !**844 %'A BV/Vr}/}r2