+ jd,,^RIt^RIt^RIHt^RIHtHt^RIHt^RI H t H t H t H t HtHtHtHtHtHt]P,!]4t]P3]P44R RRllt!R R ]4tR R ltR# ]dt]!R4]hRtAii;i)N)IntEnum)Graph is_submodule)SACStats) lpDot LpInteger LpMaximize LpMinimize LpProblemLpStatuslpSum LpVariable PULP_CBC_CMDvaluezBPlease install pulp package. See: https://github.com/coin-or/pulp.cV^8dQhR\R\R\R\\,R,R\\,R,R\ \ \\3,\\3,/#)graph memory_budget world_sizeac_unitsN fsdp_unitsreturn)rfloatintliststrtupledict)formats"a/home/luca/screen-checking/.venv/lib/python3.14/site-packages/torch/distributed/_tools/sac_ilp.py __annotate__r!"ssy6y6 y6y6y63i$ y6 S D y6  4U UC '( y6caa&\SP4p\V3Rl\V44^R7p\^d^V,4pR%p\ R\ 4p \ P!R\\V44^^\4p \ P!R\\V44^^4p \ P!R\\V44^4p \ P!R\\V44^4p \ P!R\\V44^4p\ P!R \\V44^4p\ P!R \\V44^4p\ R ^4pV'dQ\V4p\V4F6o&SPS&,R ,V9gK$WS&,^8H, p K8 V'di\V4FYo&\;QJd VV&3R lV4F 'gK RM RM!VV&3R lV44'gKGWS&,^8H, p K[ \V4F^o&\S&^,V4FDpSPS&,V,^8XgK$WS&,V V,,^8*, p KF K` \V4F7o&SPS&,R,'gK%WS&,^8H, p K9 \V4EFo&SPS&,R,'EgSPS&,R,SPS&,R,8d\PRSPS&,R ,4\PRSPS&,R,4\PRSPS&,R,4\PR4\PR4SPS&,R,SPS&,R&SPS&,R,V, pSPS&,R,V, pWS&,VV S&,,VV, V S&,,, 8H, p EK \V4Fo&WS&,V S&,8, p SPS&,R,'dK>SPS&,R,V, pSPS&,R,V, pWS&,VV, V, V S&,,8, p K \V4Fo&SPS&,R,V, pSPS&,R,V, pSPS&,R,p^.V,p\V4F3pSPSP V,,R,p^VV&K5 WS&,VV,\#VV 4, 8H, p K SP^,R,V, p\V4FRo&SPS&,R,V, pWS&,V S&,VV,V, ,8H, p KT \V4Fo&V VVS&,8, p K \V4Fo&\SPS&,R,4FlpSPS&,R,V,pSPS&,R ,V,pWS&,VV S&,,V,8, p Kn K \V4Fo&SPS&,R!,p V VS&,WzS&,,8*, p V VS&,V VS&,,8*, p V VS&,V VS&,,V^V S&,, ,, 8, p K V VV8*, p V \%V4, p \'R"^^R#7p!V P)V!4p"V"^8wd'\P+R$\,V",4/^R&3#/p#\V4F^o&\/V S&,P04^8XgK&\/V S&,P0^4V#SPS&,R ,&K` \/\3V P44^4p$\/VP0V,4p%V#V$V%3#)'a MILP to decide which modules to AC and how much memory to discard. The objective is to minimize recomputation time. The constraint is to ensure peak memory is under budget. Args: graph: graph representation of the model as a module submodule tree where each node is a submodule with memory & runtime stats memory_budget: memory budget in GiB world_size: number of GPUs. In the case of FSDP, world_size will be used to compute the amount of parameter and gradient memory on each rank ac_units: a list of user-specified AC units. fsdp_units: a list of FSDP units. AC units cannot be supermodules of FSDP units. Returns: Dict[str, float]: the optimal SAC solution, mapping from module fqn to the percentage of activation memory to **discard** float: the recomputation time of the optimal SAC solution int: upper bound on the peak memory of the optimal SAC solution. note that value of -1 means that the ILP solver failed to find a solution. c3X<"TFpSPV,R,xK! R#5i) sac_runtimeN)nodes).0irs& r sac_milp..As#A0@1Q & &0@s'*)defaultSACyrdamrcprctmax_mfqnc3l<"TF)p\VSPS,R,4xK+ R#5i)r5N)rr&)r' fsdp_unitrr(s& r r)r*fs/!+IY Au(=>>!+s14TFis_leaf sac_memoryact_fw_per_modulezFor module {%s}: z.activation memory from memory tracker is {%d},z-activation memory from SAC estimator is {%d}.z!Something is wrong. Please check!z&Overriding the latter with the former.act_grad_per_module act_totalpos_fw_post_orderindexparam_per_module grad_total n_segmentsslopes interceptsr%皙?gapRel timeLimitmsg$Solver failed to find a solution: %si@)lenr&maxranger r rmatrixrrsetany ad_matrixloggerwarning name2node fw_post_orderrr rsolveerrorr roundvarValuer objective)'rrrrr num_nodes max_runtimeMMEM_MULTIPLIERprobr-r.r/r0r1r2r3r4 ac_units_setjACM_iIA_iAG_iTA_iposcoeffpP_1TG_isslope interceptACT_isolverstatus ac_decisionsrecomputation_timepeak_memr(s'f&&&& @r sac_milprt"sP:EKK IAi0@A1K E1{?#AN UJ 'D #tE)$45q!YGA#tE)$45q!) $ $ aDAI D 9   Ay))u{{1~ 0 KKN. /00 NN. Au0E F NN@ A23  NN? A|,  NN> ? NNC D+0;;q>:M+NEKKN< ( A|,~={{1~12^C !!  !'<<<</<9  !!  ;;q>) $ $  A|,~={{1~12^C !.1555 9 {{1~34~E{{1~k*^;kk!n01isA 3 3A 67@AE!H !t eE1o555 ++a.+ ,~ =C 9 {{1~l+n< !!d j8888 9  1 9 u{{1~l34AKKN8,Q/E A|4Q7I FeadlY66 6D59  A}- A!d("" A%#a&.(( A%#a&.1AaD>999  E] ""D E#JD! 4NK up#uupi) a This is adapted from -- https://github.com/facebookresearch/xformers/blob/c6c0ac31f1b08542a0bc27278c6ed10f825f6963/xformers/checkpoint.py#L375 Given the SACStats of a module, including list of operators, their memory, runtimes, and metadata, decide via MILP an optimal set of operators to checkpoint under a given ``memory_budget``. Args: sac_stats: the SACStats object of the module memory_budget: a float between zero and one Returns: List[int]: the decision whether each operator should be saved (1) or recomptued (0). z5`memory_budget` must be a float between 0 and 1. Got .zSAC-per-modulexN:NNrDrErIrJ) ValueErrorrK func_namesr r rrNrrMrforce_store_randomrand_opsrvr}rzip view_like_opsr| inplace_opsmathceilsummemoryrruntimesrrVrRrWr rXrY) rrnum_opsr_rr(i1i2op op_parent max_memoryrorps && r +get_optimal_checkpointing_policy_per_modulers"  #! #CM?RS T  )&&'G %z 2D #tE'N3Q9EA #####A aDK,,222 2D$),,Sb193E3Eb3IJFB bEQUN "DK $ $ ! --3333%#..  ? bEQ\) )D bE[--333 3D /=3y/?/?+@@AJE!%% &* 44D  E!'' ((D ;F ZZ F{ ;Xf=MN +0. 9.QE!$-- . 99 9s.$J)rNN)loggingrenumr"torch.distributed._tools.ilp_utilsrr&torch.distributed._tools.sac_estimatorrpulprrr r r r r rrr ImportErrorerr getLoggerrxrRsetLevelINFOrtrvrrwr"r rs B;   $   8 $ y6x ' F:[ L  sA>>B BB