+ j[z^RIt^RIt^RIHt^RIHtHt^RIt^RIH t ^RI H t ^RI H t ^RIHtHtHt.ROt]P&!]4tR t!R R 4t!R R ]4t]!]P2!R4]P44t^t!RR4t!RR4tRRltRRlt Rt!RRRllt"RRlt#R#)N)Sequence)Anycast)DTensor map_aggregate) BlockMask) tree_flattentree_maptree_unflattenTensorChunkSpecFc*a]tRt^toRtRtRtVtR#)_CustomReducera Custom reducer class that can be used to specify a custom operation that reduces losses of multiple microbatches into one value. Example: >>> # xdoctest: +SKIP >>> sum_reducer = _CustomReducer( >>> torch.tensor(0.0), >>> lambda a, b: a + b >>> ) cWnW nR#N init_value reduce_fn)selfrrs&&&h/home/luca/screen-checking/.venv/lib/python3.14/site-packages/torch/distributed/pipelining/microbatch.py__init___CustomReducer.__init__,s $"rN)__name__ __module__ __qualname____firstlineno____doc__r__static_attributes____classdictcell__ __classdict__s@rrrs ##rrc]tRt^1tRtR#) _LossReducerNrrrrrr%rrr$r$1rr$gcza]tRt^>+B+B*C1T^^DTTU V rc"RVP R2#)zTensorChunkSpec(r0r*r2s&r__str__TensorChunkSpec.__str__Ks!$..!133rc4<V^8dQhRS[S[R3,/#) chunk_dims.)tupleint)formatr"s"r __annotate__TensorChunkSpec.__annotate__Os#s(Orc \VR4pV#)aJ A helper for creating a tuple of `TensorChunkSpec` from a tuple of chunk dimensions (int's). Example: >>> # xdoctest: +SKIP >>> # There are three positional arguments to the model, and >>> # we are chunking them along dimension 0, 0 and 1, respectively >>> args_chunk_spec = TensorChunkSpec.from_tuple((0, 0, 1)) c\V4#rr dims&r,TensorChunkSpec.from_tuple..] ,rr)r:args_chunk_specs& r from_tupleTensorChunkSpec.from_tupleNs(  , rc6<V^8dQhRS[S[S[3,/#)r9r:)dictstrr<)r=r"s"rr>r?bs!!cN!rc \VR4pV#)a$ A helper for creating a dictionary of `TensorChunkSpec` from a dictionary of chunk dimensions (int's). Example: >>> # xdoctest: +SKIP >>> # Chunk dimension 0 for the "id" argument, 1 for the "mask" argument >>> kwargs_chunk_spec = TensorChunkSpec.from_dict({"id": 0, "mask": 1}) c\V4#rrBrCs&rrE+TensorChunkSpec.from_dict..orGrr)r:kwargs_chunk_specs& r from_dictTensorChunkSpec.from_dictas*  , ! rc&<V^8dQh/S[;R&#)r9r+)r<)r=r"s"rr>r?<sNrr*N)rrrrrrr3r6 staticmethodrIrR__annotate_func__rr r!s@rr r <sI#  4$!!Mrc]tRt^utRtR#) _Replicater%Nr&r%rrrXrXur'rrXcRV^8dQhR\R\R\\,/#)r9 block_mask num_chunksreturn)r r<list)r=s"rr>r>ys*>>>> )_>rc NaSPP^4^8Xd S.V,#SPP^4V8g \R4h^p\P!SPW4p\P!SP W4pSP e"\P!SP W4M R.V,pSPe"\P!SPW4M R.V,p.p^p\V4Fp V3Rlp VP\P!W9,WI,WY,Wi,SPV !V4SPR74WV ,P^4, pK V#)zGiven a block mask, split the block mask along the batch dimension (dim0). Args: block_mask: Block mask to split num_chunks: Number of chunks to split the block mask into Returns: chunk_block_masks: List of chunked block masks z;Block mask has fewer batch size than the number of chunks. Nc<aVV3RlpV#)cd<\P!VS4pSPW,WV4#r)torch full_likemask_mod)bhq_idxkv_idxb_offsetrZidxs&&&& rbatch_offset_mask_modI_split_block_mask..create_mask_mod..batch_offset_mask_mods* ??1c2!**1<6JJrr%)rirjrZsf rcreate_mask_mod*_split_block_mask..create_mask_mods K) (r) kv_num_blocks kv_indicesfull_kv_num_blocksfull_kv_indices BLOCK_SIZErc seq_lengths)rnsizeAssertionErrorra tensor_splitrorprqrangeappendr from_kv_blocksrrrs) rZr[ batch_dimkv_num_blocks_chunkskv_indices_chunksfull_kv_num_blocks_chunksfull_kv_indices_chunkschunk_block_masks batch_offset chunk_idxrls f& r_split_block_maskrys$$Q'1,|j((  # # ( ( +z 9 I  I --  ***:+@+@*X  ( ( 4 :88*PVj   % % 1 :55zMVj  L:&  )    $ $2=,7#<#G 6 A%00(6&22   Y7<r>sCe<e< LLe< e<e<ell e#_split_tensor..__annotate__s:  ll %*\\  u||S ! rcx<.p^pVFp\P!V4pW4PSP4,p\ R4.VP ,p\ W64VSP&WEV&VP V4W4PSP4, pK \V4#)rN)ra zeros_likertr+slicendimrxr;) rrexpandedrichunknew_valupperslicesrs &* r_expand_chunks%_split_tensor.._expand_chunkssE&&t,G**T^^44E#(;-',,">F%*3%6F4>> "#FO OOG $ ::dnn- -CXrc3^"TF#p\\V4P4xK% R#5ir)rrto_local).0cs& r _split_tensor..s# A=ad7A''))=s+-)rtr+ru isinstancer placements device_meshrrarvrdivmodr enumerater]rx from_localSize requires_gradis_leaf retain_grad_debug_mask_minibatches)rrr[ _is_dtensorrmesh local_tensor local_chunksglobal_split_sizequotient remainder global_stridechunk_tensors_listi local_chunk chunk_shape chunk_tensorsrr global_shapelocal_expandedts&f& r _split_tensorrsA ;;t~~ &* 46;;t~~677R S  VW-K{ && !!( )),DNNS "LL8$%6C  13' 5NAv||,K*2y=aa*PK '  % %""**[1(#  61C **6t~~N "E    # # "  && !!||   ' OO  A= A $  $   "$  $  N6:M:;;  s%Jc V'g\V4Uu.uFp/NK up#\V4\V48Xg@\R\VP 44 R\VP 44 24hVf \R4h\ VRR7wrE\ VRR7wrc.p\ WFRR7EFwrV \Jg\V \4'dVPV4K9\V\P4'dZ\V \4'g\R \V 4 24hVPVPV P44K\V\ 4'd\V \4'g\R \V 4 24hV P^8Xg \R 4hVP"P^4^8XdVPV4EKEVPVP"P^44EKr\%R V R V R 24h \'.VOVN5!p \V 4Uu.uFp.NK p p\ WFRR7Fwr.p V \Jg\V \4'd V.V ,p Ma\V\P4'd\)WV 4p M4\V\ 4'd \+W4p M\%R V R V R 24h\ WRR7FwrV PV4K K V U u.uFp \-W4NK up #uupiuupiuup i)a3 Given a dictionary of args, and a dictionary of chunking specs, shard the args according to the chunking specs. Args: args_dict: Dictionary of args args_chunk_spec: Dictionary of chunking specs num_chunks: Number of chunks to shard the args into Returns: args_split: List of sharded args zargs_dict.keys() = z args_chunk_spec.keys() = z.args_chunk_spec should have been set by callerc"\V\4#rrr xs&rrE%_shard_dict_of_args..@s Z9%=rrc"\V\4#rrrs&rrErCs :a+CrTstrictzExpected TensorChunkSpec, got z#BlockMask only supports split_dim=0zUnsupported chunk spec: z and value: z combination.)rwlenrur]keysr ziprXrrxrarr typertr+r rn ValueErrorminrrr ) args_dictrHr[_values tree_spec chunk_specs split_sizesvrresult_num_chunksflat_split_resultsv_splits_flat_split_result_v_splits&&& r_shard_dict_of_argsr"s$ !*-.-q-.. y>S1 1!$y~~'7"8!9:((,_-A-A-C(D'E G  MNN$=F"!CNK Kv488 : D*!=!=   z * 5<< ( (dO44$'Ed4j\%RSS   qvvdnn5 6 9 % %dO44$'Ed4j\%RSS>>Q&$%JKK##A&!+"":.""1??#7#7#:;*4& QC}M )9.5[5*5167H1I$J1IAR1I$Jv48"$ : D*!=!=s..H 5<< ( ($Q.?@H 9 % %(>H*4& QC}M -0 - (   % %h /- 9(#5 "4  )5"4 /X%K& s M M7McPV^8dQhR\\R3,R\\\3,R,R\R\\ R3,R,R\\\ 3,R,R\\ \,\ \,3,/#) r9args.kwargsNrrHrQr\)r;rrLrMr<r r])r=s"rr>r>zsp$p$ S/p$ cNT !p$ p$?C/047 p$ C01D8 p$  4;T "# p$rc a Vf/pRpVf\WPRR7pVf\WQRR7p\\\V44\\V44V4p\ V4p\VVV4p\ V4V8d=\ V4p\\\V44\\V44V4p\ V4\ V48wd$\ R\ V4 R\ V4 24hVU a u.uFYo \ ;QJd).V 3Rl\\ S 444FNK 5M"!V 3Rl\\ S 4444NK[ p p W3#uup i)a Given a sequence of args and kwargs, split them into a number of chunks according to their respective chunking specs. Args: args: Tuple of args kwargs: Dict of kwargs chunks: Number of chunks to split the args and kwargs into args_chunk_spec: chunking specs for args, in same shape as args kwargs_chunk_spec: chunking specs for kwargs, in same shape as kwargs Returns: args_split: List of sharded args kwargs_split: List of sharded kwargs c\V\P\,4'd\ \ 4#\ 4#r)rrarr r DEFAULT_CHUNK_DIMrXrs&r default_spec3split_args_kwargs_into_chunks..default_specs, a 1 2 2"#45 5< rc"\V\4#rrrs&rrE/split_args_kwargs_into_chunks..s *Q 2Jrrc"\V\4#rrrs&rrErs Jq)4Lrz;args and kwargs are split into different number of chunks: z, c36<"TFpSV,xK R#5irr%)rr chunk_argss& rr0split_args_kwargs_into_chunks..s <%;jmm%;s)r rrLrr RuntimeErrorr;rw) rrrrHrQrargs_split_dictreal_num_chunks kwargs_splitr args_splits &&&&& ` rsplit_args_kwargs_into_chunksrzskp~ " (J  $ *L * Yt_ Y '(O /*O&L  <?*l+- 4 ! ?+ ,    ?s<00 I?#$Bs<'8&9 ;  *)J  r>sj8j8 Ij8rc X a!Ve\V4wr#M4\V^,4wrC\\4.\V4,p.pVFKp\V4wrx\V4\V48wd\ RV RV 24hVP V4KM .p \ V4EFewp o!\S!\4'Edi\\V44U u.uFp W[,V ,NK p p \'Ed^V ^,Pp V R,F/pVPV 8XdK\RV RVP 24h \P!\P!V RR/\V 4S!PR 7p.p^p\V 4\V48Xg$\R \V 4 R \V4 24h\!WR R 7F|wppVVP#S!P4,p\%RRR4.VP&,p\%VV4VS!P&VV,pVP V4TpK~ MT pVUu.uFp\V\(4NK pp\+V4'Edh\-V4'g \R4hV^,P.p\ VR,^4F5wppVP.V8wgK\RV RV RVP. 24h V^,P0p\P2!VUu.uFpVP54NK upS!PR7p\7V^,P4p\9V!3RlV44VS!P&\P:!V4pV^,P=4pV P \(P>!VVVVVRR74EKRV P \P2!VS!PR74EK\S!\@4'd[S!PBp\\V44F"p S!PEVW[,V ,4pK$ V P V4EKV^,V ,p \^\V44F5p W[,V ,V 8XdK\RV RW[,V , 24h V P V 4EKh \GW4#uup iuupiuupi)z Given a list of chunks, merge them into a single value according to the chunk spec. Args: chunks: list of chunks chunk_spec: Chunking spec for the chunks Returns: value: Merged value NzChunk z did not match chunk spec :NNzExpected shape z, got devicemeta)sectionsrDz6Expected len(partial_values) == len(meta_chunks), got z != TrzRmerge_chunks: expected all values to be DTensors or none to be DTensors, got a mixz*merge_chunks: placement mismatch at chunk z : expected rCc3\<"TF!qPSP,xK# R#5ir)rr+)rrargs& rrmerge_chunks..ss"/4AqGGCMM**Ms),Frz Expected )$r r rrrrxrrrwrrrurarvemptyr+rrtrrranyallrrcatrr]sumrrrrrrr )"r chunk_specspec_flattened flatten_spec chunk0_flatchunks_flattenedrchunk_flattenedrargs_flattenedarg_idxrpartial_values overall_shapeval meta_chunks values_to_catchunk_start_idx partial_value meta_chunk chunk_end_idx slice_indicesslicedr dtensor_flagsrrr local_cat cat_shape cat_stride reduced_valvaluers"&& @r merge_chunksrsZ'3J'?$ %1$;! )*;<=K@PP)%0  3~#6 6veW,FzlST T0 !#N!.1  c? + +"'s+;'v> 0  !# "#>*c+.>>(PQTUcQdPeeijmnyjzi{|25"2-M:%4joocmm6T$TM%*4t%<$= @R@R$RM38-3XM#--0*=9F!((0&3O2!/ >KK]Z73]MK=!!=))(9 +1-88 %mB&7;DAq||z1,HL((2|6!,,I< %Q'33!II+89=aQZZ\=9  !q!1!7!78 +./4A/, #--("JJy1 *1-446 %%&&!"')"'  %%eii 3==&QR ^ , ,..K"3'7#89 !mm!1!r$s$ ,'7FF    8 $  ##$ > 5<<,hll; 5!5!r  >Be