+ j^RIt^RIHt^RIHt^RIHtHt^RIH t ^RI t ^RI H t ^RI HtHtHt!RR 4t]!R R 44tR R ltRRltRRRlltRRltRRRlltRRltRRltRRlt]R4tR#)N)contextmanager) dataclass)ceillog2)Any)PretrainedConfig)FutureRequestState RequestState RequestStatuscla]tRt^toRtV3RlRltV3RlRltV3RlRltV3RlR ltR t Vt R #) CudaGraphBufferz`A dict for CUDA graphs with a special __del__ method to make sure the graphs are properly reset.c<V^8dQhRR/#returnN)format __classdict__s"r/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/generation/continuous_batching/utils.py __annotate__CudaGraphBuffer.__annotate__sHH$Hc/VnR#N_storage)selfs&r__init__CudaGraphBuffer.__init__s EG rc<V^8dQhRR/#rr)rrs"rrr!srcVP'd/VPP4wrVP4K@R#r)rpopitemreset)r_graphs& r__del__CudaGraphBuffer.__del__!s-mmm}},,.HA KKMrcp<V^8dQhRS[S[R3,RS[PPR,/#)rkey.rNtupleinttorchcuda CUDAGraph)rrs"rrr&s2&&U38_&1E1E1L&rc8VPPV4#r)rget)rr)s&&r get_graphCudaGraphBuffer.get_graph&s}}  %%rcf<V^8dQhRS[S[R3,RS[PPRR/#)rr).r%rNr*)rrs"rrr)s4##U38_#UZZ5I5I#d#rc"W PV&R#rr)rr)r%s&&&r set_graphCudaGraphBuffer.set_graph)s" crrN) __name__ __module__ __qualname____firstlineno____doc__rr&r2r6__static_attributes____classdictcell__rs@rr r s2jHH &&##rr c<a]tRt^-toRt^t^t^tV3RltRt Vt R#) WorkloadHintszRA tiny dataclass containing hints to help choose good continuous batching defaultsc><V^8dQh/S[;R&S[;R&S[;R&#)rmax_prompt_lengthmax_generated_length num_requestsr,)rrs"rrWorkloadHints.__annotate__-s,  !   rrN) r8r9r:r;r<rCrDrE__annotate_func__r=r>r?s@rrArA-s \ !L rrAc0V^8dQhR\R\/#)rconfigr)rbool)rs"rrr6sHH 0HTHrc VPR9#)z:Checks if attention mask is needed for the given (config).)z paged|eagerz paged|sdpa)_attn_implementation)rJs&rattn_mask_is_neededrN6s  & &*G GGrcHV^8dQhR\R\R\R\/#)rsize interval_size max_valuerrF)rs"rrr;s(""#"c"c"c"rclV^8:dV#V^8d\W, 4V,MTp\W24#)zQReturn the smallest multiple of (interval_size) >= (size), capped at (max_value).)rmin)rPrQrRpaddeds&&& rpad_to_intervalrV;s5;?!8T$& '- 7F v !!rcHV^8dQhR\R\R\R\/#)rvaluerR min_valuerrF)rs"rrrCs(""s"s"s"3"rc\V\^V44p^\\\V444,p\ W14#)zReturn the smallest power of 2 >= (value), capped at (max_value). If a minimum value is provided, the value is at least padded to that value.)maxr,rrrT)rXrRrYrUs&&& r pad_to_pow2r\Cs: s1i( )E #d4;'( (F v !!rcHV^8dQhR\R\R\R\/#)rx divide_byalign_torrF)rs"rrrKs(  c c S S rc\\W, 44pW,'dWW,, , pV#r)r,r)r^r_r`s&&&raligned_dividerbKs. D  A||  && Hrc V^8dQhR\PR\\,R\\,R\RR/#)rattention_maskcumulative_seqlens_qcumulative_seqlens_ksliding_windowrN)r-Tensorlistr,)rs"rrrRsMQ=Q=LLQ=s)Q=s)Q= Q=  Q=rc\P!VP4Pp\ \ V4^, 4EF+pW^,,W,, pW%^,,W%,, pWg8dV^8dWv, ^,pM^p\ W,W^,,4p \ W%,W%^,,4p \P!VRW3,PVVPVPR7p \P!WR7p V^8d.Wv, V, p V \P!WR7, p WRW3&EK. R#)u~Builds an attention mask inplace using the cumulative seqlens of the query and key. If given a sliding window, it will also apply a sliding window mask on top. The attention mask is not boolean, it uses zeroes and -inf (or its equivalent) so it's more of an attention score bias tensor. The attention mask is a block-diagonal matrix, with each block an attention mask for a single query-key pair. Each of those block is built from a causal mask and, if there is a sliding window, a sliding window mask. An example is represented below, with seqlen_k = 8, seqlen_q = 4 and sliding_window = 6: CAUSAL MASK: █ █ █ █ █ ░ ░ ░ █ █ █ █ █ █ ░ ░ █ █ █ █ █ █ █ ░ █ █ █ █ █ █ █ █ SLIDING WINDOW MASK: ┌──────────────────────── seqlen_k - seqlen_q - sliding_window = 8 - 4 - 6 = -2 offset to the left <─┴─> ░ █ | █ █ █ █ █ █ █ █ ░ ░ | █ █ █ █ █ █ █ █ ░ ░ | ░ █ █ █ █ █ █ █ ░ ░ | ░ ░ █ █ █ █ █ █ ATTENTION MASK (sum of causal and sliding window masks): █ █ █ █ █ ░ ░ ░ █ █ █ █ █ █ ░ ░ ░ █ █ █ █ █ █ ░ ░ ░ █ █ █ █ █ █ Another example with seqlen_k = 5, seqlen_q = 3 and sliding_window = 2: CAUSAL MASK: █ █ █ ░ ░ █ █ █ █ ░ █ █ █ █ █ SLIDING WINDOW MASK: ┌──────────────────────── seqlen_k - seqlen_q - sliding_window = 5 - 3 - 2 = 0 offset to the left <┴> | ░ █ █ █ █ | ░ ░ █ █ █ | ░ ░ ░ █ █ ATTENTION MASK (sum of causal and sliding window masks): ░ █ █ ░ ░ ░ ░ █ █ ░ ░ ░ ░ █ █ .)dtypedevice)diagonalN) r-finforkrTrangelenslicefullshaperltriutril)rdrerfrgrYiseqlen_qseqlen_kcausal_diagonal query_range key_range minus_infmaskedsliding_diagonals&&&& rbuild_attention_maskrRs-t N00155I 3+,q0 1'A.1E1HH'A.1E1HH  8q=&1A5OO035Ia%5PQ .13GA3NO JJ 3 6 7 = =  &&!((  I@ A '2^C  ejjF FF6K@V# \PdT#i;i)z3Drains a queue and returns a list of RequestStates.)empty get_nowaitrrEmpty)r new_statesrs& r drain_queuersh%'J!!## !,,.E !!%(! {{    s%AAAc$V^8dQhR\/#)rr)r+)rs"rrrs###rc\\PP4pVPpW3#)z=Returns a tuple of (mem_pool, graph_pool_id) for CUDA graphs.)r-r.MemPoolid)mem_pool graph_pool_ids rget_cuda_poolsrs&zz!!#HKKM  ""rc#"\PPV4;_uu_4RxRRR4R# +'giR#;i5i)z)A context manager to use a CUDA mem pool.N)r-r. use_mem_pool)rs&r mem_pool_ctxrs-   * *  + * * *s(A 9 A  A  A )r))r contextlibr dataclassesrmathrrtypingrr- transformers.configuration_utilsrrequestsr r r r rArNrVr\rbrrrrrrrrrs %! =EE##$  H "" Q=h: #r