+ "ݳj^RIHt^RIt^RIt^RIt^RIt^RIHtHtH t H t ^RI t RR.t ]P!RR7!RR44t!R R4tR#) ) annotationsN)AnyCallableListOptionalActivationPacketCudaEventActivationBufferT)slotsc]tRt^t$RtR]R&R]R&RtR]R&]P!] PR 7t R ]R &R t R#) rz Tracks a single async device->host transfer triggered from a forward hook. The event is recorded on the dedicated copy stream so the consumer can decide when to block. The `host_tensor` already points at pinned memory. ztorch.cuda.Eventevent torch.Tensor host_tensorNz Optional[Any]meta)default_factoryfloat created_at) __name__ __module__ __qualname____firstlineno____doc____annotations__r dataclassesfieldtime perf_counterr__static_attributes__rg/home/luca/screen-checking/.venv/lib/python3.14/site-packages/gptqmodel/utils/cuda_activation_buffer.pyrrs: D-#))$:K:KLJLrc]tRt^!tRtRRRlltRRRR/RR lltR RR R/R R lltRRltRRlt RRlt RRlt RRlt RRlt RRltRRltRtR#) r a Schedules non-blocking GPU->CPU copies using a dedicated CUDA stream + event. Typical usage inside a forward hook:: buffer = CudaEventActivationBuffer(device="cuda:6") def hook(module, inputs, output): tensor = output[0] if isinstance(output, (tuple, list)) else output buffer.capture_async(tensor, meta=module.__class__.__name__) # elsewhere in consumer thread for packet in buffer.drain(): packet.event.synchronize() process(packet.host_tensor, packet.meta) The hook thread returns immediately after enqueuing the async copy which allows the caller to release activation VRAM without waiting on D2H traffic. NTc 0V^8dQhRRRRRRRRR R R R /#) deviceztorch.device | str | intstreamzOptional[torch.cuda.Stream] pin_memoryboolhost_allocatorzIOptional[Callable[[torch.Size, torch.dtype, torch.layout], torch.Tensor]]host_reclaimerz(Optional[Callable[[torch.Tensor], None]]returnNoner)formats"r __annotate__&CudaEventActivationBuffer.__annotate__6sH!!(!,! ! b ! A ! !rc \PP4'g \R4h\P!V4pVP R8wd\ RV R24hVPf4\P!R\PP44pW`n W0n W@n WPn \PP VP4;_uu_4T;'g\PP4VnRRR4\P !4Vn\$P&!4Vn^VnR# +'giLM;i)z4CUDA is not available for CudaEventActivationBuffer.cudaz6CudaEventActivationBuffer requires a CUDA device, got .N)torchr0 is_available RuntimeErrorr$type ValueErrorindexcurrent_device_device _pin_memory_host_allocator_host_reclaimerStream _copy_streamqueue SimpleQueue_pending threadingLock_lock_approx_pending)selfr$r%r&r(r)devs&&&&&& r __init__"CudaEventActivationBuffer.__init__6szz&&((UV Vll6" 88v UVYUZZ[\] ] 99 ,,vuzz'@'@'BCC %-- ZZ  t|| , , & = =%***;*;*=D -@E?P?P?R ^^%   - ,s 0.E## E3 renqueuec(V^8dQhRRRRRRRR/#) r# activationr rrrJr'r*rr)r,s"r r-r.Ts2$$ $ $  $  $rc VPVP8wd'\RVP RVP R24hVP4pVP 4'gVP 4pVP V4p\PPRRR7p\PPVP4pVPpVPV4\PPV4;_uu_4VPVRR7VPV4RRR4\!WTVR 7pV'dVP#V4V# +'giL8;i) z Enqueue an async D2H copy of ``activation`` onto the buffer stream. Returns an ActivationPacket which is also available later via drain(). zActivation tensor is on z , expected r1F)blocking interprocessT) non_blockingN)r rr)r$r9r6detach is_contiguous contiguous_allocate_hostr2r0Eventcurrent_streamr> wait_streamr%copy_recordr _pending_put) rFrLrrJhostr current copy_streampackets &&$$ r capture_async'CudaEventActivationBuffer.capture_asyncTs+    ,*:+<+<*=[VWX  &&( ''))#..0J"":.   %e D**++DLL9'' ( ZZ  { + + JJzJ 5 LL %,"dK    f % , +s %E,, E< wait max_itemsc$V^8dQhRRRRRR/#)r#rar'rbz Optional[int]r*zList[ActivationPacket]r)r,s"r r-r.zs"T]Ndrc .p^pVe WB8dV#VP4pT^, pT'dTPP 4TP T4K] \PdT#i;i)z Collect all queued packets (or up to max_items) in FIFO order. When ``wait`` is True we synchronize each packet's event before returning. ) _pending_getr?Emptyr synchronizeappend)rFrarbpacketspulledr^s&$$ r drainCudaEventActivationBuffer.drainzs +-$)< **, aKF ((* NN6 " ;;  sA""A;:A;c V^8dQhRRRR/#r#r^rr*r+r)r,s"r r-r.s55.545rc \VPeVPVP4R#R#)zD Return a packet's host buffer to the allocator pool (if provided). N)r<rrFr^s&&r recycle!CudaEventActivationBuffer.recycles)    +  !3!3 4 ,rcV^8dQhRR/#r#r*intr)r,s"r r-r.ssrc VP;_uu_4\VR^4pRRR4V# +'giX#;i)z Non-blocking length check. The SimpleQueue does not expose qsize() reliably on all platforms, so we track with a lock-protected counter. rEN)rDgetattr)rFcounts& r pending_count'CudaEventActivationBuffer.pending_counts3 ZZZD"3Q7E Z s- > cV^8dQhRR/#rtr)r,s"r r-r.s$$$rc "VP4#N)ryrFs&r __len__!CudaEventActivationBuffer.__len__s!!##rcV^8dQhRR/#)r#r*z'CudaEventActivationBuffer'r)r,s"r r-r.s6rc V#r}rr~s&r __enter__#CudaEventActivationBuffer.__enter__s rcV^8dQhRR/#)r#r*r+r)r,s"r r-r.sTrc *VPRR7R#)T)raN)rk)rFexc_typeexctbs&&&&r __exit__"CudaEventActivationBuffer.__exit__s  rc V^8dQhRRRR/#rnr)r,s"r r-r.s""#3""rc VP;_uu_4\VR^4^,VnRRR4VPP V4R# +'giL-;irEN)rDrwrErAputrps&&r rZ&CudaEventActivationBuffer._pending_putsA ZZZ#*41BA#F#JD  &!Zs A A$ cV^8dQhRR/#)r#r*rr)r,s"r r-r.s.rc  VPP4pVP;_uu_4\\ VR^4^, ^4VnRRR4V# +'giT#;ir)rA get_nowaitrDmaxrwrErps& r re&CudaEventActivationBuffer._pending_getsS))+ ZZZ#&wt5F'JQ'NPQ#RD  Z s $A A. c V^8dQhRRRR/#)r#rLr r*r)r,s"r r-r.s     ,  rc LVPeUVPVPVPVP4pVP 4'g \ R4hV#\ P!VPVPVPRVPR7#)Nz5Custom host allocator must return pinned CPU tensors.cpu)dtypelayoutr$r&) r;shaperr is_pinnedr6r2emptyr:)rFrLr[s&& r rT(CudaEventActivationBuffer._allocate_hosts    +'' (8(8*:J:JJL]L]^D>>## !XYYK{{   ""$$''   r)rEr>r9r;r<rDrAr:)NTNN)rrrrrrHr_rkrqryrrrrZrerTrrrr r r !sf(!<$ $  $LDT25$"    r) __future__rrr?rBrtypingrrrrr2__all__ dataclassrr rrr rs_#  00  : ; T" M M# Mb b r