+ jqQRt^RIt^RIHt^RIHt^RIt^RIHt^RI H t ^RI H t ^RI HtHtHtHt^RIHtR R lt!R R 4tR#) uCentralized offloading logic for continuous batching. Handles two offloading strategies when the GPU KV cache is full: 1. CPU offloading: copy the KV cache to a pre-allocated pinned CPU buffer, preserving exact request state. 2. Soft reset: discard the KV cache and re-prefill from scratch when the request is re-scheduled. This incurs no data transfer overhead, but we need to re-run prefill over all initial + generated tokens (so more compute overhead). The CPU swap pool is a static set of pinned tensors allocated once at init (like vLLM/SGLang). Blocks are tracked with a simple free set — no dynamic allocation or deallocation of tensors ever happens at runtime. N) nullcontext)chain)is_psutil_available)PagedAttentionCache)DistributedHelper)FutureRequestState RequestState RequestStatuslogger) SchedulercV^8dQhR\\,R\\\\\3,,/#)indicesreturn)listinttuple)formats"/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/generation/continuous_batching/offloading_manager.py __annotate__r&s.T#Y4c3m0D+Ec0.pV'gV#V^,;r#^p\^\V44FApW,V^,8wd#VPW$WT, 34W,TrBW,pKC VPW$\V4V, 34V#)zGroups an index list into (start_index, offset, length) runs of consecutive values, so scattered block copies can be performed as a few slice copies.)rangelenappend)rrunsstartprevoffsetis& rcontiguous_runsr &s D  1:E F 1c'l # : ! KK 3 4#J6z $  KKG v 567 Krca]tRt^7toRtV3RlRltV3RlRltRtV3RlRltV3R lR lt RV3R lR llt V3R lRlt V3RlRlt V3RlRlt V3RlRltRtVtR#)OffloadingManageru#Manages request offloading and restoration for continuous batching. Owns a static CPU swap pool (pre-allocated pinned tensors mirroring the GPU cache layout), performs GPU↔CPU block copies, decides between CPU offloading and soft reset, and ensures cleanup on cancellation/failure/reset. c<V^8dQhRS[RS[RS[R,RS[RS[PP R,RS[RR/#) r cache schedulercpu_offload_space_gibNsafety_thresholdcompute_streamdistributed_helperr)rr floattorchcudaStreamr)r __classdict__s"rrOffloadingManager.__annotate__>sd9 9 "9 9  %t| 9  9  ))D0 9 .9  9 rc WnW nWPn.Vn.Vn.Vn.Vn.Vn/Vn/Vn VPW44p\P!V\PRR7p\VPVRR7P!44VnVRJ;'dV^8pVP"^8Xd&V'd\$P&!RVR R24R#VP"VP(VP*VP,3p VP.Fyp VPP1\P2!WP4RR74VPP1\P2!WP4RR74K{ R VP(VP*VP,3p \7VP.VP84FWwrVP P1V P:!V !4VP P1V P:!V !4KY \=\?VP"44VnVP^,p^VPA4,VPC4,\EVP.4,p\$PF!R VP" R VR, R R 24R#)cpudtypedeviceT)on_cpuNcpu_offload_space=.1fz8 GiB is too small for even one block. No CPU offloading.)r3 pin_memoryzCPU swap pool initialized: z blocks (.2fz GiB pinned)@)$r$r%_compute_stream_cpu_key_cache_cpu_value_cache_gpu_key_views_gpu_value_views_free_cpu_blocks_request_id_to_cpu_blocks!_request_id_to_group_block_counts_compute_num_cpu_blocksr+tensorint32rtp_all_reduce_minitem_num_cpu_blocksr warning block_sizenum_key_value_headshead_dim key_cacheremptyr3zip value_cacheviewrrnumel element_sizerinfo)selfr$r%r&r'r(r)num_cpu_blocksoffloading_enabledcpu_cache_shape_ block_shapek_cachev_cache cache_tensor size_in_bytess&&&&&&& r__init__OffloadingManager.__init__>sy "-35462446+-?A&GI.556K^nEKKPUV"#5#G#G_c#G#d#i#i#kl2$>\\CX[\C\   1 $!()>s(CD)) //1A1A5C\C\^c^l^lmA    & &u{{?++bf'g h  ! ! ( (_KKdh)i j! 5++U-F-FW #EOOU5F5F G G    & &w||['A B  ! ! ( ({)C D!H !%U4+?+?%@ A**1- L..00<3L3L3NNQTUZUdUdQee  )$*>*>)?yZaIbcfHggs t rc:<V^8dQhRS[R,RS[RS[/#)r r&Nr'r)r*r)rr.s"rrr/ys(3030UT\30]b30gj30rc  PVe\VR,4MRp\4'd1^RIpVP4Pp\WR,4pMRpVeEVeAW68d:VR, p\ P !RVR RVR RXR, R RVR R2 4TpMOVe\ P !R 4M4Ve&Tp\ P !R VR, R R24M \R 4h^\VPP4,VPP,VPP,VPP,VPPP,pV^8Xd \!R 4hW8,#)z?Returns the number of blocks that can fit in the CPU swap pool.Nr6r7z GiB exceeds z.0%z of total RAM (z GiB). Clamping to z GiB.u{psutil is not available — cpu_offload_space_safety_threshold cannot be enforced. Install psutil to enable the safety cap.z1Auto-sizing CPU swap pool from safety threshold: r9ztcpu_offload_space=None requires psutil to auto-size the CPU swap pool. Install psutil or pass an explicit GiB value.z9The number of bytes per block is 0. This is not possible.r;)rrpsutilvirtual_memory availabler rJ ImportErrorrr$rNrKrLrMr3itemsize ValueError) rVr&r' offload_bytesrd total_ram max_bytes clamped_gibbytes_per_blocks &&& rrD)OffloadingManager._compute_num_cpu_blocksysCXBc1W=>im   --/99II89II  $)>('73 ()>s(C=QabePfg!W-c22EkRUEVV[]!*  & NN;   "%M NNNy\cOdehNiino p&  $**&&' (jj## $jj,, -jj!!  " jj''  (  a XY Y//rc VPe*\PPVP4#\ 4#)zdReturns a context manager that runs enclosed ops on the compute stream, or a no-op when none is set.)r<r+r,streamrrVs&r _stream_ctxOffloadingManager._stream_ctxs1:>:N:N:Zuzz  !5!56m`k`mmrc <V^8dQhRS[/#)r r)r)rr.s"rrr/s>>#>rc  VPpVPpV'g^#VPP4p\ RV44p\ VP 4p.pWC8dxV'dpV\ V4, ^8dYVP4wrxVPV4WH,pW0PPVP4, pK}V'g^#VP4VPV4p VEF:pVPp W9d^Vn VP\P 8XdVP"\ VP$4\ VP&4,8XdJV;P"^,unVP&;'g VP$R ,p V .VnMVP*R,Vn\P,VnTp M%VP/4p \P0VnVP3V 4VP5V 4EK= RVn\8P:!\<P>4'duVPP4p\8P@!R\ V4 R\ V 4 R\ V4\ V 4, R\ V4 RV R 2 4\ V4#) aEvict enough active requests that, at the next batch, every remaining starved request can allocate the blocks it needs. Victims are taken from the starved requests reported by the scheduler, newest first, so the batch that was just scheduled is never touched and the demand directly bounds the amount of offloading. Tries CPU offloading first; victims that do not fit in the pool are soft reset. Returns the number of evicted requests.c3*"TF wrVxK R#5i)N).0rZ blocks_neededs& r 5OffloadingManager.offload_requests..sC7'7q]7s:NNNTz Offloaded z requests (z to CPU, z soft reset): z starved requests remain for z free blocks.r:)!r%starved_requestsr$get_num_free_blockssumractive_requestspopr blocks_in_use request_idreverse_offload_to_cpuallocated_blocks_statusr DECODINGposition_offsetinitial_tokensgenerated_tokensremaining_prefill_tokenstokens_to_processPENDING!create_equivalent_initial_requestFINISHEDfinish_requestadd_waiting_requestblock_new_requestsr isEnabledForloggingINFOrU) rVr%starved free_blocksdemand num_activevictimsstaterz cpu_offloadedrlast_true_token new_states & roffload_requests"OffloadingManager.offload_requestssf NN ,,jj446 C7CC223 &("w:G 3Lq3P#*;;= E NN5 !  #F ::33E4D4DE EK,,W5 E))J*)*&==M$:$::,,E4H4H0ICPUPfPfLg0gg--2-+0+A+A+Y+YUEYEY[]*^:I9J6:?9P9PQR9S6!. 5 5 ! !CCE - 6 6  $ $Z 0  ) )) 414(, $   w|| , ,**88:K KKS\N+c-6H5ISVW^S_befsbtStRuv #G ~-J;-Wdf 7|rc4<V^8dQhRS[S[,RR/#)r requests_in_batchrN)rr)rr.s"rrr/s&8P8PDAS3p\2P@!WPBVP:R 7p\EVPFVPH4p\EVPJVPL4p\OVV4FMwppV F.wpppVVVV,PQVVVV,RR 7K0 VPS^VV4KO RRR4\UVPVV,4Vn+R# +'giL3;i) zRestore KV caches from CPU for any CPU-offloaded requests in the scheduled batch. Indices are accumulated per group across all requests, then copied in one batched operation per layer.NFzRestored CPU-offloaded request z with z prefill tokens and z generated tokens.r3T non_blockingr2),r$ris_cpu_offloadedrBrrrCextend enumerategroup_cache_managers block_tablegetmaxrrallow_block_sharingcomplete_blocksrrKr debugrrr rsr+ as_tensorlongtor4rLrMrOr3rr=r>r?r@rPcopy_ index_copy_sortedrA)rVrr$all_cpu_indicesall_gpu_indices future_stater cpu_indices group_countsmax_allocated_blocks group_idxn gpu_blocksrgpu_ids staging_shapestaging cpu_caches gpu_views cpu_cachegpu_viewrrlengths&& rrestore_scheduled_requests,OffloadingManager.restore_scheduled_requestss %'%'-L &&E)))88<  /      oooUZZHKKELLgkKlG 0 0%2K2KU^^\Mkk-{{5<<XGt22D4I4IJJd1143H3HII'*:y'A# 8-1)E66FVf_5;;IeeV\n<]lp;q.2$$Q9(B  !'t'<'<'N O s 4D.M  M c*<V^8dQhRS[RS[RR/#)r r keep_unsortedrN)rbool)rr.s"rrr/,s#--L--Z^-rc VP'dIVPVP4RVnV'gVPP 4R#R#R#)z=Free CPU blocks for a single request (e.g., on cancellation).FN)r_return_cpu_blocksrrAsort)rVrrs&&&rfree_request_cpu_cache(OffloadingManager.free_request_cpu_cache,sI  ! ! !  # #E$4$4 5%*E " %%**,! "rc<V^8dQhRR/#r rNrx)rr.s"rrr/4s%%T%rc VPPP4FpVPVRR7K VPP 4R#)zPFree all CPU-offloaded caches in the waiting queue (e.g., on fail_all or reset).T)rN)r%waiting_requestsvaluesrrAr)rVrs& rfree_all_waiting_cpu_caches-OffloadingManager.free_all_waiting_cpu_caches4sD^^44;;=E  ' 'T ' B> ""$rc<V^8dQhRR/#rrx)rr.s"rrr/:sBBtBrc VP4VPP4VPP4\ \ VP 44VnR#)z8Reset CPU offloading state for a new generation session.N)rrBclearrCrrrIrArrs&rresetOffloadingManager.reset:sJ ((* &&,,. ..446 $U4+?+?%@ ArcF<V^8dQhRS[S[,RS[S[,/#)r rr)rrsetstr)rr.s"rrr/As$4?4?tL'94?c#h4?rc  .p.p\VP4pVFp.p.pVPPFTpVPP VP .4p VPV 4VP\V 44KV V'gK\V4\V4,V8:gKVPWVV34VPV4K V'g \4#\V4p VPRV p VPV RVn\V 4p VP4;_uu_4\P!V\PR7PVPP RR7p \#VP$VP&4p\#VP(VP*4p\-W4FLwppVP/^V 4pV F.wpppVVVV,P1VVVV,RR7K0 KN RRR4^pVFcwrVpV VV\V4,VP2VP &WpP4VP &RVnV\V4, pKe VUUu0uFwppVP kK upp# +'giL;iuuppi)aCopy the KV cache blocks of as many victims as fit in the CPU swap pool from GPU to the pool, in one batched, non-blocking copy per layer. Returns the request ids that were offloaded. All transfers are enqueued on the compute stream with pinned destinations, so the host never waits on them: correctness is guaranteed by stream ordering, since restores and cache writes go through the same stream. NrTr)rrAr$rrrrrrrr rsr+rrrr4rr?r@r=r>rP index_selectrrBrCr)rVr offloadedrfree_pool_blocksr gpu_indicesgroup_block_countscmblocksrrrrrrrrgathered_blocksrrrrZs&& rr!OffloadingManager._offload_to_cpuAsFH %'t445EK!# jj55++E,<,>I[UAq  I>>% $?sC'K'K K c\<V^8dQhRS[RS[S[S[,S[S[,3,/#)r rr)rrrr)rr.s"rrr/ws-%%S%U49d3i;O5P%rc VPPV4pVPPV4pVPP V4W#3#)zrAr?r@rIrBrCr$r%N)F)__name__ __module__ __qualname____firstlineno____doc__r`rDrsrrrrrrr__static_attributes____classdictcell__)r.s@rr"r"7ss 9 9 v3030jn>>@8P8Pt--%% BB4?4?l%%rr")rr contextlibr itertoolsrr+utilsrr$r distributedrrequestsrrr r r%r r r"rxrrrs; " (&*MM "E%E%r