+ j<^RIt^RIt^RIt^RIHt^RIHt^RIt^RIH uH t ^RI H t HtHt^RIHt^RIHt^RIHt^RIHtHt]!4'd^R IHtHt^R IHt^R IHt^R I H!t!^R I"H#t#^RI$H%t%^RI&H't'H(t(^RI)H*t*^RI+H,t,^RI-H.t.^RI/H0t0H1t1^RI2H3t3^RI4H5t5H6t6H7t7H8t8^RI9H:t:H;t;Ht>^RI?H@t@HAtAHBtB^RICHDtD^RIEHFtF^RIGHHtHHItIHJtJHKtKHLtLHMtM^RINHOtO^RIPHQtQHRtRHStSHTtTHUtUHVtV^RIWHXtXHYtYHZtZR:Rlt[Rt\!R R!4t]R"t^R#t_!R$R%]4t`R&ta!R'R(4tb!R)R*]4tcR+td!R,R-]4te!R.R/]e4tf!R0R1]e4tg!R2R3]e4thR4tiR;R5ltj!R6R7]PP4tlR8tmR9tnR#)<N)ABC)partial)BCEWithLogitsLossCrossEntropyLossMSELoss)AcceleratedOptimizer)AcceleratedScheduler)is_megatron_lm_available)recursively_applysend_to_device)mputensor_parallel)DistributedDataParallel)finalize_model_grads) ModelType)get_num_microbatches)get_megatron_optimizer)get_tensor_model_parallel_group"get_tensor_model_parallel_src_rank)get_forward_backward_func)get_model_config)build_train_valid_test_datasets) BertModelT5Model)Classification)get_argsget_tensorboard_writer get_tokenizerprint_rank_last)_add_data_args_add_validation_args!core_transformer_config_from_args parse_args validate_args)load_args_from_checkpointload_checkpointsave_checkpoint)set_global_variables) gpt_builder)_compile_dependencies_init_autoresume_initialize_distributed_set_random_seedset_jit_fusion_optionswrite_args_to_tensorboard)_vocab_size_with_padding)%build_train_valid_test_data_iteratorsget_optimizer_param_schedulernum_floating_point_operationssetup_model_and_optimizer train_step training_log))average_losses_across_data_parallel_groupcalc_params_l2_normget_ltor_masks_and_position_idsc z\4pVP'dRMRpVP^8Xd(\RVP RV R24\R4\ V4pVPR8XdcVP'd4VP 'd^M^p\VVVP RVVR 7pV#\VVP^VVR 7pV#VPR 8XdR Vn \W@VR R R7pV#VPR8Xd\V^RVVVVR7pV#\RVP 24h)zBuild the model.z pre-trainingz fine-tuningz Building z model in the z mode.zThe Megatron LM model weights are initialized at random in `accelerator.prepare`. Please use `accelerator.load_checkpoint` to load a pre-trained checkpoint matching the distributed setup.bertT)confignum_tokentypesadd_binary_headparallel_output pre_process post_process)r< num_classesr=r@rAgptFN)vp_stager<t5)r<r=r?r@rA add_encoder add_decoderUnsupported model type: )rpretraining_flagrankprintmodel_type_namer"bert_binary_headrr num_labelsuse_legacy_modelsr)r ValueError) r@rArFrGargsmoder<r=models &&&& ]/home/luca/screen-checking/.venv/lib/python3.14/site-packages/accelerate/utils/megatron_lm.pymodel_provider_funcrUUs\ :D!222> D yyA~ $../~dV6JK  x /t 4F v%  "&"7"7"7QQN- $ 5 5 $') E@ L/# OO ') E. L!    &!&D|dSWX L    % #%##  L3D4H4H3IJKKcVPR4\4pVPPPeVPPP f \ R4hVPPP pVPPP V4p\W4p\WRR7pM\PpVPR8Xd\Pp\pVPPP e!VPPP p\VV4wr4p\V4VnW4V3#)z#Preparing model optimizer schedulerNzaYou must provide a `custom_model_provider_function` when using a `custom_prepare_model_function`.) schedulerrE)rKrstatemegatron_lm_plugincustom_prepare_model_functioncustom_model_provider_functionrPprepare_optimizerprepare_schedulerrencoder_or_decoderrLencoder_and_decoderrUr4len model_len) acceleratorrQcustom_model_provider_funcrS optimizerrX model_typemodel_provider_func_s& rT!prepare_model_optimizer_schedulerrhs,;< :D++IIU    / / N N Vs &1%6%6%I%I%h%h"!!44RRSmn%k9 %kM 11   4 '"66J2    / / N N Z#.#4#4#G#G#f#f (A ) %9ZDN Y &&rVc<a]tRt^toRtRtRtRtRtRt Vt R#)MegatronLMDummyDataLoaderz Dummy dataloader presents model parameters or param groups, this is primarily used to follow conventional training Args: **dataset_kwargs: Megatron data arguments. c \P!4p\V4p\V4pVP 4p\ V^,4VnVP PV4RVP R&R#)rTmegatron_dataset_flagN)argparseArgumentParserr r!parse_known_argsvars dataset_argsupdate)selfdataset_kwargsparser data_argss&, rT__init__"MegatronLMDummyDataLoader.__init__sh((*'%f-++-  1.   05912rVc \4pVPP4F:wr#\WR4pWC8wd\ RV RV RV RV 24\ WV4K< R#)z.train_valid_test_datasets_providers,:DDNNTSXM1Z1Zt~~aeaoao`p.0J L ##v-##($//%t'<'< %%.##($// %%-##($*A*A,d.E.E&!#;D.DummyMegatronDataloadericV#Nrrss&rT__iter__H_handle_megatron_data_iterator..DummyMegatronDataloader.__iter__sKrVc/#rrrs&rT__next__H_handle_megatron_data_iterator..DummyMegatronDataloader.__next__sIrVrN)rrrrrrrrrs@rTDummyMegatronDataloaderrs   rVrNdtypedevicegroup)torchtensorboolr distributed broadcastrr)rc data_iteratorris_data_iterator_emptyis_src_data_iterator_emptys&& rT_handle_megatron_data_iteratorrsw+d2!&.DEJJ_j_q_q!r "$F$HPoPq  &*@&(( rVc VPR4\4pVP'EgP^RIHpHpVP VP,pVUu/uFqf\WW6,4bK ppVR,fa\VR,\PPP4'dWWR,nMVRVRVRWWR,nMVRWWR&\PPP!VP 3/VBpV!VVP"\$P&!4\$P(!4RRVP*P-4VP.R7#VP0e!VP0wVnVnVnM^^^uVnVnVnVP VP,VnVP9V4wpp p VP VP,Vn\;WR 7p\;W R 7p \;W R 7p WV 3#uupi) zPreparing dataloader)_PYTORCH_DATALOADER_KWARGSprepare_data_loader batch_sizesamplershuffle batch_samplerFT) num_processes process_index split_batches put_on_device rng_typesdispatch_batches)rcr)rKrrl data_loaderrrmicro_batch_sizenum_micro_batchesr~rrutilsdata BatchSamplerr DataLoaderdatasetrr get_data_parallel_world_sizeget_data_parallel_rankrcopyrconsumed_samplesconsumed_train_samplesconsumed_valid_samplesconsumed_test_samplesr1r) rc dataloaderrQrrrkkwargsrrrs && rTrr!s#,- :D  % % %Q0043I3IITnoTnqWZ,F,IJJTno ,  '&+U[[-=-=-J-JKK/?y!,9%9%<(5E'2'#3< [[%%001C1CNvN #    ::<446!++002(99    , %%  ++*defgij `D ')DdF` $ 5 58N8N N  < <[ I     $ 5 59O9O O<# =# < v"9KKKopsIcRaa]tRtRtoV3RltRRltRt]R4tRt Vt V;t #)MegatronLMOptimizerWrappericc,<\SV`VRRR7R#)FN)device_placementscalersuperrw)rsre __class__s&&rTrw#MegatronLMOptimizerWrapper.__init__ds U4HrVcR#rr)rs set_to_nones&&rT zero_grad$MegatronLMOptimizerWrapper.zero_gradg rVcR#rrrs&rTstepMegatronLMOptimizerWrapper.stepjrrVc .VPP#)zTWhether or not the optimizer step was done, or skipped because of gradient overflow.)re skipped_iterrs&rTstep_was_skipped+MegatronLMOptimizerWrapper.step_was_skippedms~~***rVrr) rrrrrwrrpropertyrrr __classcell__rrs@@rTrrcs.I  +++rVrcVPR4\4p\WPVPVP 4#)zPreparing optimizer)rKrrno_wd_decay_cond scale_lr_condlr_mult)rcrSrQs&& rTr]r]ss:+, :D !%)>)>@R@RTXT`T` aarVc.a]tRtRtoRtRRltRtVtR#)MegatronLMDummyScheduleriza Dummy scheduler presents model parameters or param groups, this is primarily used to follow conventional training loop when scheduler config is specified in the deepspeed config file. Args: optimizer (`torch.optim.optimizer.Optimizer`): The optimizer to wrap. total_num_steps (int): Total number of steps. warmup_num_steps (int): Number of steps for warmup. **kwargs (additional keyword arguments, *optional*): Other arguments. Nc 6WnW nW0nW@nR#r)retotal_num_stepswarmup_num_stepsr)rsrerrrs&&&&,rTrw!MegatronLMDummyScheduler.__init__s". 0 rV)rrerr)Nr)rrrrrrwrrrs@rTrrzs rVrc8aa]tRtRtoV3RltRtRtVtV;t#)MegatronLMSchedulerWrapperic&<\SV`W4R#rr)rsrX optimizersrs&&&rTrw#MegatronLMSchedulerWrapper.__init__s /rVcR#rr)rsrQrs&*,rTrMegatronLMSchedulerWrapper.stepsrVr) rrrrrwrrrr r s@@rTrrs0rVrc>VPR4\V4pV#)zPreparing scheduler)rKr2)rcrerXs&&&rTr^r^s!+,-i8I rVcHaa]tRtRtoRtV3RltRtRtRtRt Vt V;t #)AbstractTrainStepiz;Abstract class for batching, forward pass and loss handler.c0<\SV`4WnR#r)rrwname)rsr!rs&&rTrwAbstractTrainStep.__init__s  rVcR#rr)rsrcrls&&&rTget_batch_func AbstractTrainStep.get_batch_funcrrVcR#rrrs&rTget_forward_step_func'AbstractTrainStep.get_forward_step_funcrrVcR#rr)rsrcs&&rT get_loss_funcAbstractTrainStep.get_loss_funcrrV)r!) rrrrrrwr$r'r*rrr r s@@rTrrs!E    rVrcHaa]tRtRtoRtV3RltRtRtRtRt Vt V;t #) BertTrainStepizW Bert train step class. Args: args (`argparse.Namespace`): Megatron-LM arguments. c`<\SV`R4VPWP4VnVP WP VP4VnVPVP VP4Vn VP'g RVn R#^RIHpW0n R#)r-N)SequenceClassifierOutput)rrwr$rl get_batchr*rIrN loss_funcr'rM forward_stepmodel_return_dictmodel_output_classtransformers.modeling_outputsr/)rsrcrQr/rs&&& rTrwBertTrainStep.__init__s ),,[:T:TU++K9N9NPTP_P_` 66t7L7LdNcNcd%%%&*D # N&> #rVcRpRpVPPPe!VPPP#V'd ^RIHpV#V# \ dT#i;i)c.ROp\PpVe \V4pMRp\P!WV4pVR,P 4pVR,P 4pVR,P 4pVR,P 4pVR,P 4p VR,P 4p WVWxW3#) Build the batch.texttypeslabels is_random loss_mask padding_maskN)r:r;r<r=r>r?rint64nextrbroadcast_datalongfloat) rkeysdatatyperdata_btokensr;sentence_orderr> lm_labelsr?s & rTget_batch_megatron8BertTrainStep.get_batch_func..get_batch_megatronsYD{{H(M*$33DIFF^((*F7O((*E#K0557N{+113Ix(--/I!.1668L.YT TrVc\V4p\V\PP 44pVR,P 4pVR,P 4pRV9dVR,P 4pMRpRV9dBVR,P 4pVR,R8gP \P4pMRpRpRV9dVR,P 4pMRpW$WvWS3#)r9 input_idsattention_masktoken_type_idsNr<next_sentence_label)rBr rcudacurrent_devicerDtorE)rrrIr?r;rKr>rJs& rTget_batch_transformer;BertTrainStep.get_batch_func..get_batch_transformers &D!$ (A(A(CDD+&++-F 01668L4'-.3354 N//1 !(^t377 D   $,!%&;!rJ output_tensorlm_loss_ sop_logitslm_losssop_losslossaveraged_lossess &&& rTloss_func_pretrain7BertTrainStep.get_loss_func..loss_func_pretrains#0 H~~'H!)Iii b 1I4E4Eb4I IJY]]_\G%??:??2q+A+G+G+I>K^K^_aKbqst#>>+)"KWL_"`i);ZYZI[\\\"KWI"Vi);<<.loss_func_finetunesQ"9 BRA1$&,,5::uyy:Q*Q+- B ;V[[_M,./GOO&/!"455 5rVrYrZcustom_loss_function)rsrcrIrNrprvsf&&f rTr*BertTrainStep.get_loss_funcsN =& 6    / / D D P$$77LL L % %% %rVcaaaVVV3RlpV#)c<S PV4wr#rErgS 'gRpS 'd#V!W'W6R7pV\S PWT43#V!W'VR7p V \S PV43#) Forward step.N tokentype_idsrK)rr0rr1) rrSrIr;rJr>r<r?rirtrMrIrss && rTr29BertTrainStep.get_forward_step_func..forward_step.shMQ^^\iMj JF>f# %f% b $gdnni&XXXv5Iwt~~v>>>rVr)rsrIrMr2sfff rTr'#BertTrainStep.get_forward_step_func-s ?rVr2r0r1r4 rrrrrrwr$r*r'rrr r s@@rTr-r-s% ?>)@'&RrVr-cHaa]tRtRtoRtV3RltRtRtRtRt Vt V;t #) GPTTrainStepi>zV GPT train step class. Args: args (`argparse.Namespace`): Megatron-LM arguments. c<\SV`R4VPWP4VnVP V4VnVP4VnVPe\4pVPVn VPVn VPVnVPVnVP VnVP"VnVP$'g RVnR#^RIHpW@nR#)rN)!CausalLMOutputWithCrossAttentions)rrwr$rlr0r*r1r'r2 vocab_filereod eod_token eos_token_id pad_tokenreset_position_idsreset_attention_mask eod_mask_lossr3r4r5r)rsrcrQ tokenizerrrs&&& rTrwGPTTrainStep.__init__Fs (,,[:T:TU++K8 668 ?? &%I&]]DN****"&"9"9$($=$=!!//%%%&*D # W&G #rVcaV3RlpV3RlpVPPPe!VPPP#V'd ^RIHpV#V# \ dT#i;i)c <R.p\PpVe \V4pMRp\P!WV4pVR,P 4pVR,P 4pVRRR13,P 4p\VS PS PS PS PS PRR7wrp WvWV 3#)zGenerate a batchr:NNNNTrrrrr pad_mask_lossrr`NNrc) rrArBrrCrD contiguousr9rrrr) rrFrGrrHtokens_r<rIrPr> position_idsrss & rTrL7GPTTrainStep.get_batch_func..get_batch_megatron[s8D{{H(M*$33DIFVn))+GU^..0FQV_//1F7V....#'#:#:%)%>%>"00"7 3N|9lJ JrVc ~<\V4pRVR,/p\V\PP 44pVR,P 4p\P !VP^,^3VPVPR7S P,p\P!W#.^R7pVR,P4pVRRR13,P4p\VS PS PS PS PS P RR7wrgpWTWvV3#) rOrdimrNTrrrc)rBr rrTrUrDzerosshaperrrconcatrr9rrr) rrrpaddingr<rIrPr>rrss & rTrW:GPTTrainStep.get_batch_func..get_batch_transformerys  &Dk!23D!$ (A(A(CDD;',,.Gkk7==#3Q"7w}}U\UcUcdgkguguuGllG#51=GU^..0FQV_//1F6U....#'#:#:%)%>%>"00"7 3N|9lJ JrVrY)rYrZrZrr0rr[sf&& rTr$GPTTrainStep.get_batch_funcZst K< K,    / / I I U$$77QQ Q  2  ) (  % % sA## A21A2ca\4oV3RlpVPPPe!VPPP#V#)cB<SP'dVwr#MTpVP4pVPR4P4pSP^8d\P !\P !VPR4V,4P^4VP 4P^4.4p\PPV\P!4R7V^,V^,, pMA\P !VPR4V,4VP 4, pSP'd{\PP4pVP4'dGQRV R\PP4 R\ P"!4^, 24h\%V.4pRV^,/pSP'dVP'RX/4WG3#)r`rzRank z7: found NaN in local forward loss calculation. Device: z, node: rbrtrc) return_logitsrErecontext_parallel_sizercatrdr all_reducer get_context_parallel_groupcheck_for_nan_in_loss_and_gradget_rankisnanrTrUosunamer7rr) r>rilossesrtrn global_rank averaged_loss output_dictrQs && rTr1-GPTTrainStep.get_loss_func..loss_funcs!!!!.&\\^F!r*002I))A-yy%))FKKOi,G"H"M"Ma"PR[R_R_RaRfRfghRi!jk!!,,T9W9W9Y,ZAwa(yyR9!<= O222#//88: ::<<K=)$zz88:;8BHHJqM?T' FtfMM$mA&67K!!!""Hf#56$ $rV)rrYrZry)rsrcr1rQs&& @rTr*GPTTrainStep.get_loss_funcsGz %<    / / D D P$$77LL LrVcaV3RlpV#)cr<SPV4wr#rEpV!W&WSR7pV\SPV43#)r})r<r) rrSrIr<r>rPrrirss && rTr28GPTTrainStep.get_forward_step_func..forward_steps?GKnnUbFc CFI|!&VM '$..)"DD DrVrrsr2sf rTr'"GPTTrainStep.get_forward_step_funcs ErV) rrr2r0r1r4rrrrr s@@rTrr>s'H(A)F#J  rVrcxaa]tRtRtoRtV3Rlt]R4t]R4t]R4t Rt Rt R t R t VtV;t#) T5TrainStepizU T5 train step class. Args: args (`argparse.Namespace`): Megatron-LM arguments. c <\SV`R4VPWP4VnVP V4VnVP4VnVP'g RVn R#^RI H pW0n R#)rN)Seq2SeqLMOutput) rrwr$rlr0r*r1r'r2r3r4r5r)rsrcrQrrs&&& rTrwT5TrainStep.__init__sh ',,[:T:TU++K8 668%%%&*D # E&5 #rVcdVP^4pVP^4pW,pVR8pV#)r`?) unsqueeze)rPattention_mask_b1sattention_mask_bs1attention_mask_bssextended_attention_masks& rTattn_mask_postprocess!T5TrainStep.attn_mask_postprocesss@,55a8+55a8/D"4s":&&rVcl\P!\P!^W3VR74pVR8pV#r`rr)rtrilones)rrrPs&& rTget_decoder_maskT5TrainStep.get_decoder_masks1EJJ:/JSY$Z['#-rVcVPwr4VP^4p\P!W1^3VR7pWe,pVR8pV#r)rrrr) rPdec_seq_lengthrr_rrrrs &&& rTget_enc_dec_maskT5TrainStep.get_enc_dec_masksS&,, ,55a8"ZZQ(GPVW/D"4s":&&rVcRpRpVPPPe!VPPP#V'd ^RIHpV#V# \ dT#i;i)c.R Op\PpVe \V4pMRp\P!WV4pVR,P 4pVR,P 4pVR,P 4pVR,P 4pVR,R 8p VR,R 8p VR,R 8p WVWWV 3#) r9text_enctext_decr<r>enc_maskdec_mask enc_dec_maskNr)rrr<r>rrrr@) rrFrGrrH tokens_enc tokens_decr<r>rrrs & rTrL6T5TrainStep.get_batch_func..get_batch_megatronskD{{H(M*$33DIF +002J +002JH%**,F{+113Ij)C/Hj)C/H!.1C7L9hR^^ ^rVcr\V4p\V\PP 44pVR,P 4pVR,P 4pVR8gP \P4pRV9dVR,P 4pMnVPVPVP\P R7pVRRR 13,P4VR &^VR &VPVR8H^4\PVR,P 44p\PVP^,VP4p\P!VR,P 4VP^,VP4pW%WCWgV3#) r9rOr<decoder_input_ids)rr.NrPrSrc).r).r)rBr rrTrUrDrVrE new_zerosrrclone masked_fill_rrrr) rrrr<r>rrrrs & rTrW9T5TrainStep.get_batch_func..get_batch_transformersn &D!$ (A(A(CDDk*//1J(^((*F4++EKK8I"d*!"56;;= #--fll6==X]XbXb-c &,S#2#X&6&<&<&> 7#%& 6"'' d(:A>"88>N9O9T9T9VWH"33J4D4DQ4GIZIZ[H&77%&++-z/?/?/BJDUDUL9hR^^ ^rVrY)rYrZrZrr0rr[s&&& rTr$T5TrainStep.get_batch_funcss _2 _.    / / I I U$$77QQ Q  1  ) (  % % r]cRpVPPPe!VPPP#V#)cVP4p\P!VPR4VP R4,4VP4, pTp\ V.4pVRV^,/3#)r`rbrc)rErrdrerfr7)r>rirjrlrnros&& rTr1,T5TrainStep.get_loss_func..loss_funcAsh$**,Hii b 1I4E4Eb4I IJY]]_\GDG RO)_Q%788 8rVrx)rsrcr1s&& rTr*T5TrainStep.get_loss_func@s? 9    / / D D P$$77LL LrVcaV3RlpV#)c z<S PV4wr#rErgpV!W#WgVRVR7p V \S PV43#)r}Nr~r) rrSrrr>rKrrrrirss && rTr27T5TrainStep.get_forward_step_func..forward_stepOsU^b]k]k^ ZJI(l"LX\hqM!'$..)"DD DrVrrsf rTr'!T5TrainStep.get_forward_step_funcNs ErVr)rrrrrrw staticmethodrrrr$r*r'rrr r s@@rTrrs` 6 ' '  ' '=)~ rVrc\4p\RRR4VP^8Xd\RVP R24\ VPVP 4R#)Nz> setting random seeds to z ...)rr,rJrKrr-data_parallel_random_init)rQs rTfinish_mpu_initr_sL :DD$- yyA~ *499+T:;TYY > >?rVc Vf/pVPR4\PP4'gQR4h\ VRR7pVP 4FRwrE\ W4R4e4VP^8Xd#\RV R\ W44 RV RV 2RR7\W4V4KT VP'gVPR R 4'd!VPfQR 4h\V4\V4\VR R 7\4\!4\#4\%4\'4p\ VR R4f\)VP*V4VnVP.R8Xd,VP0'dVP2^8Xd RVnMR Vn^VnR#)NzInitializing Megatron-LMzMegatron requires CUDA.T)ignore_unknown_argsz*WARNING: overriding default arguments for r{r|)flushuse_checkpoint_argsFz/--use-checkpoints-args requires --load argument)build_tokenizerpadded_vocab_sizer;)rKrrT is_availabler#r}r~rJrrgetloadr%r$r(rr+r*r.rr0orig_vocab_sizerrLrIrNrM iteration)rcextra_args_provider args_defaultsrQrrs&&& rT initializerls 01 :: " " $ $?&?? $ )t DD$))+  4d # /yyA~@QwtGYFZZ`ad`eefglfmn 5!, =#4#45JE#R#Ryy$W&WW$!$'$u5 :Dt($/7!9$:N:NPT!U v%$*?*?*?DOOWXDX $ %DNrVcraa]tRtRtoRtV3RltRtRtRtRt Rt R t R t R t R tR tRtVtV;t#)MegatronEngineiz Megatron-LM model wrapper Args: accelerator (:class:`~accelerate.Accelerator`): The accelerator object to use. model: Megatron-LM model optimizer: Megatron-LM optimizer lr_scheduler: Megatron-LM lr scheduler c<\SV`4W nV^,VnW0nW@n\ 4pVPPPeLVPPP!V3/VPPPBVn M~VPR8Xd\W4Vn M\VPR8Xd\W4Vn M:VPR8Xd\W4Vn M\!RVP 24hRVPn/Vn/Vn^VnRVn^VnRVnVP0e \34R#R#)rNr;rCrErHFT)rrwmodule base_modelrerXrrYrZcustom_train_step_classcustom_train_step_kwargstrain_step_handlerrLr-rrrPrtotal_loss_dicteval_total_loss_dictrreport_memory_flag$num_floating_point_operations_so_far module_configtensorboard_dirr/)rsrcrSrerXrQrs&&&&& rTrwMegatronEngine.__init__sD  (""z    / / G G S&1&7&7&J&J&b&b'#))<<UU'D # ! !V +&3K&FD #  ! !U *&2;&ED #  ! !T )&1+&DD #78L8L7MNO O&+# "$&!"&451!    + % ' ,rVcaa\4p\SP^,4pSPPVn\ SP^,\4'dVP'dVPeQR4hSPUu.uFq3PNK upVn \SP4^8XdVP^,Vn VP'd]SPUu.uFq3PNK upVn\SP4^8XdVP^,VnVP'd~VP 'dl\#\SP44Uau.uF oVV3RlNK upVn\SP4^8XdVP$^,Vn\&VnV#uupiuupiuupi)rzWhen overlap_grad_reduce is True, config.no_sync_func must be None; a custom no_sync_func is not supported when overlapping grad-reducec<<SPPSV4#r)refinish_param_sync)x model_indexrss&rT2MegatronEngine.get_module_config..s$..::;JrV)rrrre scale_lossgrad_scale_funcrLocalDDPoverlap_grad_reduce no_sync_funcno_syncradelay_grad_reducestart_grad_syncgrad_sync_funcoverlap_param_gatherdelay_param_gatherrparam_sync_funcrfinalize_model_grads_func)rsrQr< model_chunkrsf `rTget_module_config MegatronEngine.get_module_configsz!$++a.1!%!:!: dkk!nh / /D4L4L4L&&. V .KO++"V+;#6#6+"VF 4;;1$&,&9&9!&<#%%%X\XcXc(dXc)D)DXc(d%t{{#q(,2,A,A!,DF)  $ $ $)@)@)@^cdghlhshsdt^u&^u{JJ^u&F "4;;1$)/)?)?)B&+?( #W)e&s G8G=!HcVPFpVP4K VPfVP4VnVP 4R#r)rtrainrr'log_eval_resultsrs model_modules& rTr*MegatronEngine.trainsG KKL    (    %!%!7!7!9D  rVcVPFpVP4K VPfVP4VnR#R#r)revalrr'r,s& rTr0MegatronEngine.evals@ KKL    (    %!%!7!7!9D  &rVc \4p.p\V4^8dVP^8d}\^VP4FapTP VP 4UUu/uF2wrVWVWBP ,V^,VP ,bK4 upp4Kc MV.p\VP4^8di\V4^8d:\\VP44Uu.uFp\V4NK uppV#R.\VP4,pV#\V4^8d \V4MRpV#uuppiuupirN) rrarrrr}rriter) rs batch_datarQ data_chunksrrvrbatch_data_iterators && rTget_batch_data_iterator&MegatronEngine.get_batch_data_iteratorsCz z?Q %%)q$"8"89A&&)3(8(8(:(:%:%:!:a!etG\G\=\]](:: *l t{{ a z?Q&-2#dkk2B,CD,Cqk",CD #" Vc$++.. #"8;:7J${"3PT ""!Es %8E 7Ec VPV4p\VPPVVPVP VP VP\4R7wr4rVpV^8HVP n W4Wg3#)z` Training step for Megatron-LM Args: batch_data (:obj:`dict`): The batch data to train on. )forward_step_funcrrSreopt_param_schedulerr<forward_backward_func) r9r5r r2rrerXrrr)rsr5r8 loss_reducedrr grad_normnum_zeros_in_grads&, rTr5MegatronEngine.train_step s#:::FLV"55BB-++nn $%%";"=M I Aq!8I'3a&7#9GGrVc "\4pVPV4p\4pV!VPPVVP \ 4VPVPRR7pVP^8d\PP4V;P\P!4VP,\ 4,, un \P !RR7'd/pV^,FrpVUu.uF qV,NK p p\#V ^,P$4^8Xd \'V 4\#V 4, Wg&KZ\P(!V 4Wg&Kt V#/#uupi)ze Evaluation step for Megatron-LM Args: batch_data (:obj:`dict`): The batch data to evaluate on. T)r<rrSnum_microbatchesrr forward_only)ignore_virtual)rr9rr r2rrrrempty_unused_memory_levelrrT empty_cacherr ris_pipeline_last_stagerarrdr) rsr5rQr8r> loss_dictsr?rrlosses_reduced_for_keys &, rT eval_stepMegatronEngine.eval_step#sJz":::F 9 ;*"55BB-++13!22   ) )Q . JJ " " $ ##  , , .1F1F FI]I_ _ #  % %T : :L!!}}:D)E*QC&&*&)E-a06671<(+,B(CcJ`Fa(aL%(- 5K(LL% %   *FsF c f\4pVP^,P'EdXVP!R /VBwr4rVV;P^, un\ P !4VP,\4,pV;PV, un V;P\W'4, un VPeVPP4P4pRp VP 'd\#VP$4p \'VVP(VPP*^,R,VPVVP,VVV V4 VnMVP.!R /VBpVPeVFp VP0P3V \4P6P9R.44W:,,VP0V &VP0P3V R,\4P6P9R.44\4P6P9R.4,VP0V R,&K \4P:!R\4P6P=4R7p VF4p \?W:,P@4^8XgK%WV ,, p K6 Rp RV9d VR,p VPBPDeVPBPEWR7#V #) rNlrg _num_itersg?rrt)rnrtr)#rrtrainingr5rr rrrrrr3rreget_loss_scaleitemlog_params_normr8rSr6r  param_groupsr rLr rrrT FloatTensorrrUrarr r4) rsr5rQ loss_dictrr@rAr loss_scale params_normrrnrts &, rTforwardMegatronEngine.forwardKsz ;;q> " " "DHOODaV`Da AIY NNa N99;d>S>SSVjVllJ  ' ': 5 '  5 59VW[9h h 5##/!^^::<AAC " '''"5djj"AK*6((NN//248NN++ % +'44I##/$C1155c5::;Q;QSVRW;XY\e\jj--c2EID]D]DaDal*EJJ,B,BC5,IE ..u5E6D--cL.@A %||C (A(A(CDC9>''(A-#& y x(F  " " 5 5 A**==4=W W rVc\4pVPeVP^8XdR#\4p\4pRVP R2pVPEFpVP R4'dKVPV,VPVR,,, pW4 RV R2, p\ P!\^VP444pVP'dW4 RV R2, pV'gKVPV R2VP4VP4VP'gKVPV R2W`P4EK \V4^,p\RV,4\V4\RV,4/VnR#) Nzvalidation loss at iteration z | rPz value: z PPL: z validationz validation ppl-)rrrrr endswithmathexpminrSrI add_scalarrar)rsrQwriterstringrrppllengths& rTr+MegatronEngine.log_eval_resultssdz    '4>>Q+> z')00@D,,C||L))--c2T5N5NsUaOa5bbE XeWC0 0F((3r5::<01C$$$EuC00v!!SE"5uzz|T^^T(((%%_&=sNNS-Vqf %f %$&!rVcBVP4\4pWn\PP 4\ VPVPVPVPVPR7\PP 4R#))rN) r+rsaverrbarrierr'rrrerXr)rs output_dirrQs&& rTr'MegatronEngine.save_checkpointsm z  !!# NN KK NN NN151Z1Z   !!#rVc\4pWn^Vn^Vn\P P 4\VPVPVP4wr4\P P 4W0n W@n VP'd0VP^8XdVPP4R#R#R#r3)rrrrrrrjr&rrerXrrfp16reload_model_params)rs input_dirrQrrs&& rTr&MegatronEngine.load_checkpointsz &'#&'# !!#:I$++W[WeWegkgugu:v7  !!#"4X1 9991, NN . . 0-9rV) rr rrrrrer rXr r )rrrrrrwr'r*r0r9r5rLrZr+r'r&rrr r s@@rTrrsI(>4 :#2H0&P=~'4 $ 1 1rVrc\V4#)z Average losses across data parallel group. Args: losses (List[Tensor]): List of losses to average across data parallel group. )r7)rs&rT%avg_losses_across_data_parallel_grouprss 5V <._gpu_gather_ones ;;! \\^D)F5,,;;#B]B]B_;`a a   V $a   $$^3C^C^C`$ayyQ//  s'CT)error_on_other_type)r )rr|s& rT"gather_across_data_parallel_groupsrs0 _$ OOrV)TTTT)NN)ormr_rabcr functoolsrrtorch.nn.functionalnn functionalrgtorch.nnrrrrerrXr importsr operationsr r megatron.corer rmegatron.core.distributedrrrmegatron.core.enumsr)megatron.core.num_microbatches_calculatorrmegatron.core.optimizerrmegatron.core.parallel_staterrmegatron.core.pipeline_parallelrmegatron.core.utilsr"megatron.legacy.data.dataset_utilsrmegatron.legacy.modelrr$megatron.legacy.model.classificationrmegatron.trainingrrrrmegatron.training.argumentsr r!r"r#r$megatron.training.checkpointingr%r&r'megatron.training.global_varsr(megatron.training.gpt_buildersr)megatron.training.initializer*r+r,r-r.r/%megatron.training.tokenizer.tokenizerr0megatron.training.trainingr1r2r3r4r5r6megatron.training.utilsr7r8r9rUrhrjrrrr]rrr^rr-rrrrModulerrsrrrVrTrsf  AA,,-92M>-N>pI4R8C lkB:O.b'8jLjLZ$>LD +!5 + b.!5   "K%K\M$M`N#Nb @/d_1UXX___1F =PrV