+ j,Rt^RIt^RIt^RIt^RIt^RIt^RIHtHt^RI H t ^RI H t ^RI Ht^RIHt^RIHt^RIHt^R IHt]'d3^RIt^RIt^RIt^R IHtHtHtHtHt^R I H!t!^R I"H#t#^R I$H%t%^RI&H't']PP!])4t*Rt+RRlt,!RR]PZ4t.!RR4t/!RR]04t1!RR]24t3!RR]44t5RoRRR/RR R!R"R#R$R%R&R'R(/R)R*R+R%R,R-R.R'R/R0R1R2R2R3//R'R4R5R'R(///RpRRR/RR R!R"R#R$R%R&R'R(/R)R*R6R4R7R-R.R'R8R9R:R;/R0R1R2R2RR3///R'R4R?R'R(///RqRRR/RR@RA.R!R"R#RBR%RCR'R(/R)R*RDR%RER-R.R'R/R9RFR.RGRHRH../R0R1R2R2RR3///R'R%.RrOR'R(////t6]!RIRJ44t7RKRLlt8RMRNlt9RORPlt:RQRRlt;RsRSRTlltRYRZlt?!R[R\4t@!R]R^4tAR_R`ltBRaRbltC!RcRd4tD!ReRf]4tE!RgRh]E4tF!RiRj]E4tG!RkRl4tH!RmRn4tIR#)tz? Shared types, constants, and utilities for the serving layer. N)ABCabstractmethod)CallableFuture) dataclass)Queue) TYPE_CHECKING)logging)ResponseParser)ContinuousBatchingConfigGenerationConfigPreTrainedModelPreTrainedTokenizerFastProcessorMixin)ContinuousBatchingManager)GenerationOutput) Scheduler) ModelManagerz x-request-idc`V^8dQhR\R\\\R,3,/#)model_idreturnN)strtuple)formats"_/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/cli/serving/utils.py __annotate__r;s'  S U3d ?%; cVPR4wrpV'dVPR4'gVR3#Y;'gR3#)af`[@revision][:.gguf]` -> the id without the file, and the file if one was named. `:.gguf` is how GGUF clients name a file inside a repository: the repository says which model, the file which weights, and both are answered for rather than one of them dropped. A colon that is not a `.gguf` file is left alone -- it is part of the id. :z.ggufN) partitionendswith)rhead_nameds& rsplit_model_idr&;sB'',NDU U^^G,,~ $ rc*]tRt^HtRtRtRtRtRtRt R#)ModalityLLMVLM MULTIMODALSTTTTSN) __name__ __module__ __qualname____firstlineno__r)r*r+r,r-__static_attributes__r.rrr(r(Hs C CJ C Crr(c6a]tRt^PtoRtV3RlRltRtVtR#) _StreamErrorz5Sentinel to signal an error from the generate thread.c <V^8dQhRS[/#)rmsgr)r __classdict__s"rr_StreamError.__annotate__SsCrcWnR#Nr7)selfr7s&&r__init___StreamError.__init__Ssrr=N)r/r0r1r2__doc__r?r3__classdictcell__r9s@rr5r5Ps?rr5c]tRt^WtRtRtR#)_GenerationCancelledzERaised inside ``DirectStreamer.put()`` to abort ``model.generate()``.r.Nr/r0r1r2rAr3r.rrrErEWsOrrEc]tRt^[tRtRtR#) ReasoningTextzTagged str subclass: text chunk belonging to a thinking/reasoning block. Streamers wrap reasoning text with this so handlers can route it to ``reasoning_content`` deltas instead of ``content``. r.NrFr.rrrHrH[rrHc]tRt^ctRtRtR#)CBWorkerDeadErrorzRaised when a request is submitted to a CB worker that has died. Surfaced as 503 by the FastAPI exception handler. Carries the original error message that killed the worker so the client knows why the server is in this state. r.NrFr.rrrKrKcrIrrKdefaultsrole assistant start_anchorz<|im_start|>assistant fieldsthinkingopenzclosezcontenttext tool_calls open_patternz\s*z repeatsTjson transformtypefunctionz {content} close_patternz0\s*(?:<\|im_end\|>|<\|endoftext\|>|<\|eot_id\|>)z-\s*\s*[^>\n]+)>z\s*z xml-inline content_args tag_patternz<[^>\n]+)>\s*(?P.*?)\s*namez{name} argumentsz#\s*(?:<\|im_end\|>|<\|endoftext\|>)z <|turn>model zz<|channel>thought z z<\|tool_call>call:(?P\w+)z unquoted_keys string_delimsz<|"|>c0a]tRt^toRtV3RltRtVtR#)ToolCallzA parsed tool call surfaced through the stream queue when the parser closes a `tool_calls` region. ``arguments`` is always a JSON string.c2<V^8dQh/S[;R&S[;R&#)rr`rar8)rr9s"rrToolCall.__annotate__s I  N rr.N)r/r0r1r2rA__annotate_func__r3rBrCs@rreresLrrec6V^8dQhRRR\R,/#)rmodelrrNdict)rs"rrrs# g g,= g$+ grca\VRV4p\VRR4pVeV#VPPo\V3Rl\P 44R4#)a?Return the response template (new declarative format) for this model. Resolution order: 1. ``tokenizer.response_template`` if set on the tokenizer. 2. Fallback in :data:`_RESPONSE_TEMPLATE_FALLBACKS` matched by ``model.config.model_type``. 3. ``None`` (no parsing performed; raw text is streamed through). tokenizerresponse_templateNc3><"TFwrSV9gKVxK R#5ir<r.).0typesv model_types& r (get_response_template..s_#Gxu:Y^K^#Gs  )getattrconfigrtnext_RESPONSE_TEMPLATE_FALLBACKSitems) processorrjrntmplrts&& @rget_response_templater~sZ ; :I 914 8D  ((J _#?#E#E#G_ae ffrc V^8dQhRRRR/#)rrjrrResponseParser | Noner.)rs"rrrs"QQ Q Qrcj\W4pVfR#\VRV4p\V\WB4R7#)uBuild a streaming :class:`ResponseParser` for the current generation. Returns ``None`` when no template is available for the model (then text streams through unparsed). `input_ids` is decoded and passed to the parser as `prefix=` so templates that declare a `start_anchor` get prefill-aware parsing (e.g. for a future prefilled `\n` opener). Templates without `start_anchor` fail closed and ignore the prefix — safe for chat templates where the anchor isn't at the actual generation point (e.g. Gemma 4 tool-result followups). Nrnprefix)r~rwr _decode_prefix)r|rj input_idstemplaterns&&& rbuild_response_parserrs7 %Y6H ; :I (>)+O PPrc0V^8dQhR\R\/#)rvaluer)rlre)rs"rrrs5555rcVR,pVR,p\V\4'g\P!V4p\ VR,VR7#)uConvert one parsed ``tool_calls`` entry into a :class:`ToolCall`. Every template in this codebase shapes a tool call as ``{"type": "function", "function": {"name": ..., "arguments": ...}}`` — the parser already extracted ``name`` and ``arguments`` for us. The only work left is re-serializing ``arguments`` to a JSON string, since the field uses ``"content": "json"`` so the parser hands us a parsed dict while the OpenAI API expects a string. r\rar`)r`ra) isinstancerrYdumpsre)rfnargss& r_normalize_tool_callrsC z B k?D dC zz$ Ft 44rc$V^8dQhR\/#rrr8)rs"rrrs ' 'C 'rc\VR4'dVP4pV'd'\V^,\4'd V^,pVP V4#)anDecode ``input_ids`` to the chat prompt string the parser expects as ``prefix=``. Serve always handles one request at a time, but the upstream shape varies: ``apply_chat_template(return_tensors="pt")`` hands us a 2D ``[1, N]`` tensor while continuous batching hands us a flat ``list[int]``. Both forms hold a single sequence; this helper unwraps it before decoding.tolist)hasattrrrlistdecode)rnrs&&rrrsLy(##$$& Z ! d33aL   I &&rc V^8dQhRRR\R\\\R,\\,R,3,/#)rrjrcleaned_contentrN)rrrre)rs"rrr!sA** *  *  3d DNT1 12 *rcT\W4pVfVRR3#\VRV4pVPW%\Wc4R7pVP RR4pVP R4p VP R4;'g.p V U u.uFp \ V 4NK up ;'gRp WV 3#uup i)u8Parse a full assistant message into ``(content, reasoning, tool_calls)``. Uses :meth:`tokenizer.parse_response` with the resolved response template (see :func:`get_response_template`). When no template is available, returns ``(cleaned_content, None, None)`` — i.e., surface the raw model decode. ``cleaned_content`` is the ``skip_special_tokens=True`` decode of ``generated_ids``. The parser itself runs on the raw decode (keeping special tokens so it can match delimiters like Gemma 4's ``<|tool_call>``); ``cleaned_content`` is only used on the no-template path.NrnrrTrQrV)r~rwparse_responsergetr) r|rj generated_idsrrrrnparsedrT reasoningrawrsrVs &&&&& rparse_assistant_messager!s"%Y6Hd** ; :I  % %mnU^Fj % kFjjB'G :&I **\ " ( (bC3673a&q)37??4J z ))8sB%cFV^8dQhR\\,R\/#)reventsr)rrl)rs"rrr@s!!d4j!T!rc.pVFpVR,pVPR4pVR8Xd~VPR4'dK=VPR4;'gRpV'gKbVR8XdVP\V44KVR8XdVPV4KKVR 8XgKVR 8XgK\VPR 44pVfKVPV4K V#) uTranslate :class:`ResponseParser` events into stream-queue items. Output items are one of: - ``str`` — plain content delta (also raw text when no parser is active) - :class:`ReasoningText` — reasoning / thinking delta - :class:`ToolCall` — completed tool call on region close Other events (``region_open``, raw dirty chunks of structured fields like ``json`` tool calls, ``region_close`` for non-tool fields) carry no user-visible payload and are dropped. r[field region_chunkdirtyrUrrQrT region_closerVr)rappendrHr)routevkindrrUtcs& rresponse_events_to_chunksr@sC&zw > !vvg66&>''RD " =./)# 4 $^ #(=%bffWo6B~ 2'( Jrc~a]tRtRtoRtV3RlRltV3RlRltV3RlRltV3R lR ltV3R lR lt R t Vt R#)DownloadAggregatoridzAggregates byte-progress across multiple concurrent download tqdm bars. huggingface_hub opens one tqdm bar per file shard. This class tracks them all and emits a single aggregate ``{"stage": "download", "progress": {...}}`` event whenever any updates. c&<V^8dQhRS[RS[/#)renqueuer)rr)rr9s"rrDownloadAggregator.__annotate__ks555C5rc:WnW n/VnRVnR#r<)rrjbarslast_emitted_current)r>rrs&&&rr?DownloadAggregator.__init__ks  79 04!rc8<V^8dQhRS[RS[R,RR/#)rbar_idtotalNrint)rr9s"rrrqs&s3:$rc H^V3VPV&VP4R#)z6Register a new download bar with its total byte count.Nr_emit)r>rrs&&&rregisterDownloadAggregator.registerqsJ & rc><V^8dQhRS[RS[RS[R,RR/#)rrcurrentrNrr)rr9s"rrrvs-S3sTzdrc FW#3VPV&VP4R#)z>Update a bar's current byte count and emit aggregate progress.Nr)r>rrrs&&&&rupdateDownloadAggregator.updatevs$, & rc$<V^8dQhRS[RR/#)rrrNr)rr9s"rrr{s  C D rcR#r<r.)r>rs&&rrSDownloadAggregator.close{ rc<V^8dQhRR/#rrNr.)rr9s"rrr~s  t rc t\RVPP444pWP8XdR#WnVPP4UUu.uF wr#VfK VNK pppV'd \V4MRpVP RRRVP RRRRVR V//4R#uuppi) c3*"TF wrVxK R#5ir<r.)rqcr$s& rru+DownloadAggregator._emit..s;(:!(:sNstatusloadingrjstagedownloadprogressrr)sumrvaluesrrrj)r> agg_currentr$ttotals agg_totals& rrDownloadAggregator._emit~s; (8(8(:;; 33 3 $/! $ 0 0 2D 2a!! 2D#)CKt  )Y WiH   Es B4-B4)rrrrjN) r/r0r1r2rAr?rrrSrr3rBrCs@rrrds< 55       rrc<V^8dQhR\R\R\/#)rcallbackrr)rrr[)rs"rrrs&DDxD3D4DrcPaaa^RIHp\SS4o!VVV3RlRV4pV#)uCreate a tqdm subclass that routes progress to a callback. Bars with ``unit="B"`` are download bars — aggregated via ``DownloadAggregator``. Other bars (e.g. "Loading weights") emit ``weights`` stage events. Args: callback (`callable`): Called with a dict payload ``{"status": "loading", "model": ..., "stage": ..., "progress": ...}``. model_id (`str`): The model ID (included in progress payloads). Returns: A tqdm subclass that forwards progress to *callback*. )tqdmch<aa]tRtRtoVV3RltRVVV3RlltVVV3RltVV3RltRtVt V;t #).make_progress_tqdm_class..ProgressTqdmic <VPR4;'gRVnRVR&\SV` !V/VB^VnRVnVPR8Xd9\ V4VnSPVPVP4R#R#)unititTdisableBN) rsse_unitsuperr?n last_emittedid_bar_idrr)r>rkwargs __class__download_aggregators&*,rr?7make_progress_tqdm_class..ProgressTqdm.__init__sz"JJv.66$DM $F9  G d -f -DF "D }}#!$x #,,T\\4::F$rc z<Vf^pV;PV, unVPR8Xd4SPVPVPVP4R#VPVP 8wd<VPVnS!RRRSRRRRVPR VP//4R#R#) Nrrrrjrweightsrrr)rrrrrr)r>rrrrs&&rr5make_progress_tqdm_class..ProgressTqdm.updatesy FFaKF}}##**4<<L4,,,$(FF! )"Y$L -rc3<"VPFpV;P^, unVPR8Xd3SPVPVPVP 4MTVPVP 8wd:VPVnS!RRRSRRRRVPR VP //4VxK R #5i) rrrrjrrrrrN)iterablerrrrrr)r>itemrrrs& r__iter__7make_progress_tqdm_class..ProgressTqdm.__iter__s ! ==C''..t||TVVTZZPVVt000(,D%$i#X#Y&DFFGTZZ(P  &sC C c|<VPR8XdSPVP4\SV`4R#)rN)rrSrr)r>rrs&rrS4make_progress_tqdm_class..ProgressTqdm.closes*}}##))$,,7 GMOr)rrrr)r) r/r0r1r2r?rrrSr3rB __classcell__)rr9rrrs@@r ProgressTqdmrs$ G  " "  rr) tqdm.autorr)rr base_tqdmrrsff @rmake_progress_tqdm_classrs/,,Xx@00y0d rca]tRtRtoRtRV3RlRlltV3RlRltV3RlR ltV3R lR ltV3R lR lt Rt Vt R#)DirectStreameriarStreamer for ``model.generate()`` (used by :class:`GenerateManager`). Implements the ``put``/``end`` protocol that ``model.generate()`` expects: generate calls ``put(token_tensor)`` after each decode step, and ``end()`` when generation is complete. Tokens are decoded incrementally via the Rust ``DecodeStream`` (O(1) per token) and pushed as text to an asyncio.Queue. NcV<V^8dQhRRRS[PRS[PRR/#)rrntokenizers.Tokenizerloopqueueresponse_parserr)asyncioAbstractEventLoopr)rr9s"rrDirectStreamer.__annotate__sCNN)N''N}} N 1 Nrc ^RIHpWnW nW0nVRJpV!.VR7VnW@nRVn\P!4Vn ^Vn VP eY\VP P4F3pVPPVPPV4K5 R#R#)aE Args: tokenizer: The Rust tokenizer (``tokenizer._tokenizer``). loop (`asyncio.AbstractEventLoop`): The event loop to push decoded text to. queue (`asyncio.Queue`): The queue that receives decoded text chunks. response_parser (`ResponseParser`, *optional*): When provided, every decoded chunk is fed through this parser; the resulting events are translated into typed queue items (plain ``str`` for content, :class:`ReasoningText` for thinking, :class:`ToolCall` on tool call close). Without a parser, raw decoded text is streamed.  DecodeStreamNskip_special_tokensT)tokenizers.decodersr _tokenizer_loop_queue_decode_stream_parser_first threadingEvent _cancelled total_tokensrinitial_eventscall_soon_threadsafe put_nowait)r>rnrrr rrrs&&&&& rr?DirectStreamer.__init__s$ 5#  -5*2CVW&  #//+ << #1$,,2M2MN // 0F0FMO $rc<V^8dQhRR/#rr.)rr9s"rrr sFFFrcfVPPVPPV4R#r<)rrrr)r>rs&&r_enqueueDirectStreamer._enqueues  '' (>(>Erc"<V^8dQhRRRR/#)rr torch.TensorrNr.)rr9s"rrr s(((D(rc VPP4'd \4hVP'd RVnR#VP 4FpV;P ^, unVP PVPV4pVfKHVPfVPV4Ki\VPPV44FpVPV4K K R#)zHCalled by ``model.generate()`` after each decode step with new token(s).FN) ris_setrErrrrsteprrr#rfeed)r>rtoken_idrUrs&& rputDirectStreamer.puts ?? ! ! # #&( ( ;;;DK  H    " &&++DOOXFD|||# d#5dll6G6G6MNDMM$'O'rc<V^8dQhRR/#rr.)rr9s"rrr s  T rc 4VPeAVPP4wr\ V4FpVPV4K VPR4R# \\3d%p\P RT 24.pRp?LfRp?ii;iz:Flush any final parser events, then push the EOS sentinel.Nz>ResponseParser.finalize() failed; dropping final tail events: )rfinalize ValueError RuntimeErrorloggerwarningrr#r>_msg final_eventsers& rendDirectStreamer.ends << # "%)\\%:%:%<" 2,? d#@ d - "!_`a_bcd!  "sA""B3BBc<V^8dQhRR/#rr.)rr9s"rrr $src :VPP4R#)zWSignal cancellation. The next ``put()`` call will raise and abort ``model.generate()``.N)rsetr>s&rcancelDirectStreamer.cancel$s r)rrrrrrrrr<) r/r0r1r2rAr?r#r,r:r@r3rBrCs@rrrsANNBFF((&  rrcpa]tRtRtoRtR V3RlRlltV3RlRltV3RlR ltV3R lR ltR t Vt R#) CBStreameri)apStreamer for continuous batching (used by :class:`CBGenerateManager`). Same ``put``/``end`` protocol as :class:`DirectStreamer`, but called manually by :class:`CBGenerateManager` instead of by ``model.generate()``: ``put(output)`` receives a CB ``GenerationOutput``, decodes new tokens, and pushes text to the asyncio.Queue. ``end()`` signals the stream is complete. Nc `<V^8dQhRRRS[RRRS[PRS[PRR /#) r cb_managerr request_idrnrrrr r)rr r r)rr9s"rrCBStreamer.__annotate__2sQ!-!-/!-!-* !- '' !- }} !-1!-rc >^RIHpWnW nW@nWPnW0nVRJpV!.VR7VnW`n^Vn ^Vn VPeD\VPP4Fp VP PV 4K R#R#)a Args: cb_manager (`ContinuousBatchingManager`): The CB manager instance. request_id (`str`): The request ID to track in the CB scheduler. tokenizer: The Rust tokenizer (``tokenizer._tokenizer``). loop (`asyncio.AbstractEventLoop`): The event loop to push decoded text to. queue (`asyncio.Queue`): The queue that receives decoded text chunks. response_parser (`ResponseParser`, *optional*): See :class:`DirectStreamer`. rNr)rr_cb _request_idrrrrr _prev_lenrrrr) r>rErFrnrrr rrrs &&&&&&& rr?CBStreamer.__init__2s$ 5%  #-5*2CVW&  << #1$,,2M2MN &&t,O $rc"<V^8dQhRRRR/#)routputrrNr.)rr9s"rrrGUs 1 1, 1 1rc VPVPRp\VP4VnVFpV;P^, unVPP VP V4pVfKHVPfVPPV4Ks\VPPV44FpVPPV4K K R#)zLDecode new tokens from a CB ``GenerationOutput`` and push text to the queue.N) generated_tokensrKlenrrr)rrrrrr*)r>rN new_tokensr+rUrs&& rr,CBStreamer.putUs,,T^^-=> V445"H    " &&++DOOXFD|||# &&t,5dll6G6G6MNDKK**40O#rc<V^8dQhRR/#rr.)rr9s"rrrGds % %T %rc \VPeKVPP4wr\ V4FpVPPV4K VPPR4R# \\3d%p\P RT 24.pRp?LzRp?ii;ir0) rr1r2r3r4r5rrrr6s& rr:CBStreamer.endds << # "%)\\%:%:%<" 2,? &&t,@ t$ - "!_`a_bcd!  "sA66B+B&&B+c<V^8dQhRR/#rr.)rr9s"rrrGrs222rc PVPPVP4R#)zCancel the CB request.N)rIcancel_requestrJr?s&rr@CBStreamer.cancelrs  0 01r) rIrrrrKrrJrrr<) r/r0r1r2rAr?r,r:r@r3rBrCs@rrCrC)s3!-!-F 1 1 % %22rrCc(V^8dQhR\RR/#)rseedrNr)rs"rrrwsrc2^RIpVP!V4R#)z8Set the PyTorch random seed for reproducible generation.N)torch manual_seed)r\r^s& rset_torch_seedr`ws drcV^8dQhRR/#rr.)rs"rrr~s!!4!rc^RIpVPP4'dVPP4R#R#)z+Empty the CUDA cache if a GPU is available.N)r^cuda is_available empty_cache)r^s rreset_torch_cacherf~s- zz    !rc`a]tRtRtoRtRtV3RlRltV3RlRltV3RlR ltR t Vt R #) InferenceThreadizPersistent thread for ``model.generate()`` calls. ``torch.compile`` with CUDA graphs stores state in thread-local storage. All inference must run on the same thread to avoid corrupted graph state. c\4Vn\P!VPRR7VnVP P 4R#)T)targetdaemonN)rrrThread_run_threadstartr?s&rr?InferenceThread.__init__s3"W  ''tyyF  rc<V^8dQhRR/#rr.)rr9s"rrInferenceThread.__annotate__s**d*rcvVPP4wrr4pV!V/VBpVeVPVPV4MVPV4R;p;r#Kb \dApTe"TPTP T4Rp?L7TP T4Rp?LMRp?ii;i R;p;r#i;i)TN)rrr set_result Exception set_exception)r>rrrfuturerresultr9s& rrmInferenceThread._runs-1[[__-> *Bfd *T,V,#--f.?.?H%%f-&*))TF ,#--f.B.BAFF((++  ,&*))TFsG(A# B1 A#B1# B.. B)B1B)$B1)B..B11B8c <V^8dQhRS[/#rr)rr9s"rrrrsVrc V\4pVPPWW4R34V#)ESubmit a callable to the inference thread. Returns a blocking Future.N)rrr,)r>rrrrws&&*, rsubmitInferenceThread.submits% 6489 rc4<V^8dQhRS[P/#r)r r)rr9s"rrrrs7>>rc \P!4pVP4pVPP WW5V34V#zOSubmit a callable to the inference thread. Returns an awaitable asyncio.Future.)r get_running_loop create_futurerr,)r>rrrrrws&&*, r async_submitInferenceThread.async_submits:'')##% 6489 r)rrnN) r/r0r1r2rAr?rmr}rr3rBrCs@rrhrhs-  **& rrhca]tRtRtoRtV3RlRlt]R V3RlRll4t]V3RlR l4t]V3R lR l4t R t Vt R#)BaseGenerateManageriuBase class for generation managers. Subclasses: - :class:`GenerateManager` — sequential ``model.generate()`` on a persistent thread. - :class:`CBGenerateManager` — continuous batching with paged attention. c&<V^8dQhRRRRRR/#rrjr gen_configr rNr.)rr9s"rr BaseGenerateManager.__annotate__s*II.Irjrs&&&rinit_cbBaseGenerateManager.init_cbrNcd<V^8dQhRRRRRS[RRRS[R R R S[S[PR 3,/#) rrjrr|(ProcessorMixin | PreTrainedTokenizerFastinputsrr rFr rrzDirectStreamer | CBStreamer)rlrrr r)rr9s"rrrs`    >   '    1  w}};; < rc R#)a>Start streaming generation. Args: model (`PreTrainedModel`): The loaded model. processor: The processor or tokenizer for decoding. inputs (`dict`): Tokenized inputs (tensors for sequential, lists for CB). gen_config (`GenerationConfig`): Generation parameters. request_id (`str`): Unique request identifier. response_parser (`ResponseParser`, *optional*): When provided, decoded text is fed through this parser and typed items (``str`` for content, :class:`ReasoningText`, :class:`ToolCall`) are pushed onto the queue instead of raw text. Returns: `tuple[asyncio.Queue, DirectStreamer | CBStreamer]`: A ``(queue, streamer)`` pair where *queue* yields ``str | ReasoningText | ToolCall | _StreamError | None`` and *streamer* exposes ``.total_tokens`` and ``.cancel()``. Nr.)r>rjr|rrrFr s&&&&&&&rgenerate_streaming&BaseGenerateManager.generate_streamingrrcb<V^8dQhRRRRRS[RRRS[R S[S[S[S[S[,3,/# rrjrr|rrrr rFrrlrrrr)rr9s"rrrsW    >   '     sCc" # rc "R#5i)aRun generation to completion. Args: model (`PreTrainedModel`): The loaded model. processor: The processor or tokenizer for decoding. inputs (`dict`): Tokenized inputs (tensors for sequential, lists for CB). gen_config (`GenerationConfig`): Generation parameters. request_id (`str`): Unique request identifier. Returns: `tuple[str, int, list[int]]`: ``(text, input_len, generated_ids)``. Nr.)r>rjr|rrrFs&&&&&&rgenerate_non_streaming*BaseGenerateManager.generate_non_streamingssc<V^8dQhRR/#rr.)rr9s"rrrs>>d>rc R#)z/Stop the generation manager and free resources.Nr.r?s&rstopBaseGenerateManager.stoprrr.r<) r/r0r1r2rArrrrrr3rBrCs@rrrsWII   8  *>>rrca]tRtRtoRtRtRV3RlRlltV3RlRltV3R lR ltV3R lR lt V3R lRlt Rt Vt R#)GenerateManagerizFSequential generation via ``model.generate()`` on a persistent thread.c$\4VnR#r<)rhrnr?s&rr?GenerateManager.__init__s &( rNcf<V^8dQhRRRRRS[RRRS[R R R S[S[PS[3,/# rrjrr|rrrr rFr rr)rlrrr rr)rr9s"rrGenerateManager.__annotate__s_ >  '   1 w}}n, -rc 2aa a a \P!4o \P!4o \VRV4Pp\ VS S VR7p/VCRVRVRV/Co \ SR4'dRS R&RV V VV 3R llp VPV 4S V3#) zLStart streaming generation via ``model.generate()`` on the inference thread.rnr streamergeneration_config has_talkerrUgeneration_modecV^8dQhRR/#rr.)rs"rr8GenerateManager.generate_streaming..__annotate__s R Rd Rrc <SP!R/SBR# \d!SPSPR4R#\d:pSPSP\ \ T444Rp?R#Rp?ii;i)Nr.)generaterErrrur5r)r9 gen_kwargsrrjrs rrm0GenerateManager.generate_streaming.._runsk R,,' B))%*:*:DA R))%*:*:LQrjr|rrrFr rust_tokenizerrrmrrrs &f&&&&& @@@rr"GenerateManager.generate_streamings'')&}} KCNN!.$_nn H6I:Wbdmn 5, ' ',2J( ) R R DhrcP<V^8dQhRRRRRS[RRRS[R S[S[S[R 3,/#) rrjrr|rrrr rFrr&)rlrrr)rr9s"rrrsS.. .>. . ' .  . sC' (.rc "/VCRVRV/Cp\VR4'dRVR&VP!VP3/VBGRjxL pVR,PR ,pV^VR13,p VP V RR 7p WV 3#LB5i) zNRun generation to completion via ``model.generate()`` on the inference thread.rrnrrUrNrTrr)rrrshaper) r>rjr|rrrFgenerate_kwargs sequences input_lenrrUs &&&&&& rr&GenerateManager.generate_non_streamings^V]%8*kS\] 5, ' '17O- .++ENNNoNN ;'--b1 !!YZ-0  4H -- OsAB BAB c&<V^8dQhRS[RS[/#rrr)rr)rr9s"rrr.s888v8rc BVPP!V.VO5/VB#)r|)rnr}r>rrrs&&*,rr}GenerateManager.submit.s!||""27777rc:<V^8dQhRS[RS[P/#r)rr r)rr9s"rrr2s>>x>W^^>rc BVPP!V.VO5/VB#r)rnrrs&&*,rrGenerateManager.async_submit2s!||((=d=f==rc<V^8dQhRR/#rr.)rr9s"rrr6s  d rcR#r<r.r?s&rrGenerateManager.stop6rr)rnr<) r/r0r1r2rAr?rrr}rrr3rBrCs@rrrs?P)<..(88>>  rrca]tRtRtoRtRV3RlRlltV3RlRltV3RlR ltV3R lR ltRV3R lR llt V3RlRlt ] V3RlRl4t V3RlRlt RtVtR#)CBGenerateManageri:aQContinuous batching generation via paged attention. Translates between the handler's text-level asyncio.Queue and CB's token-level interface. Per-request: ``max_new_tokens``, ``eos_token_id``. The CB manager is initialized lazily on the first request via :meth:`ensure_initialized`, using that request's ``gen_config`` for shared sampling params (temperature, top_p, do_sample). .. todo:: Remove :meth:`init_cb` when CB supports per-request generation config. At that point, ``gen_config`` can be passed directly to ``add_request`` and the CB manager no longer needs a shared config. Nc<V^8dQhRR/#)r cb_configContinuousBatchingConfig | Noner.)rr9s"rrCBGenerateManager.__annotate__Is$$"C$rc RVnWnR#r<rI _cb_config)r>rs&&rr?CBGenerateManager.__init__Is59#rc&<V^8dQhRRRRRR/#rr.)rr9s"rrrMs%.rFrs&& r _check_aliveCBGenerateManager._check_alivebsQ 88 ((;;GGK&'*VWaVbbdepdq(rss' rcf<V^8dQhRRRRRS[RRRS[R R R S[S[PS[3,/#r)rlrrr rrC)rr9s"rrrms_6$6$ 6$>6$ 6$ ' 6$  6$16$ w}}j( )6$rc  a a VPpVf \R4hVPV4\P!4p\P !4o VR,p VP V VRVPVPR7p\VRV4Pp \VPVV VS VR7o V V 3Rlp VPW[4S S 3#)zFStart streaming CB generation. Registers a per-request output handler.3CB manager not initialized. Call `init_cb()` first.rT)rF streamingmax_new_tokens eos_token_idrnrc|<SPV4VPe7SP\VP44SP 4R#VP 4'dSP 4R#R# \ d/pSP\\T444Rp?R#Rp?ii;ir<)r,errorrr5r: is_finishedrur)rNr9r text_queues& r _on_output8CBGenerateManager.generate_streaming.._on_outputs < V$<<+)),v||*DELLN''))LLN* <%%l3q6&:;; rjr|rrrFr cbrrrrrrs&&&&&&& @@rr$CBGenerateManager.generate_streamingmsXX :TU U *%'')$+MMO ;' ^^ !%44#00 $ !KCNN HH    +   < ""::8##rcb<V^8dQhRRRRRS[RRRS[R S[S[S[S[S[,3,/#rr)rr9s"rrrsW/./. /.>/. /. ' /.  /. sCc" #/.rc na"VPpVf \R4hVPV4VR,p\V4p\P !4p V P 4oV3Rlp VPWZ4VPVVVPRVPR7SGRjxL p V PeOVPPe\RV RV P 24h\R V RV P 24hV Pp VP!V R R 7p WV 3#L5i) zcRun non-streaming CB generation. Registers a handler that resolves an asyncio.Future on completion.NrrcZ<SP4'gSPV4R#R#r<)donert)rxrws&r _on_result._on_results!;;==!!&)!rF)rFrrrzCB worker died during request rzCB generation failed for Tr)rIr3rrQr rrrrrrrrrrKrPr)r>rjr|rrrFrrrrrrxrrUrws&&&&&& @rr(CBGenerateManager.generate_non_streamings8XX :TU U *%;'  N '')##% * ""::  !%44#00   << #**66B'*H TVW]WcWcVd(eff!::,bWX X//  4H --sB,D5/D30BD5c<V^8dQhRR/#)rrrr.)rr9s"rrrs22;2rc VPeVPPf \R4hVPPP#)z*The CB scheduler (for testing/monitoring).z.Continuous batching processor not initialized.)rIbatch_processorr3 schedulerr?s&rrCBGenerateManager.schedulers? 88 txx77?OP Pxx''111rc<V^8dQhRR/#rr.)rr9s"rrrs11d1rc`VPe VPPR^R7R#R#)NT)blocktimeout)rIrr?s&rrCBGenerateManager.stops% 88  HHMMaM 0 rrr<)r/r0r1r2rAr?rrrrrpropertyrrr3rBrCs@rrr:sj $$"YY t t6$6$p/./.b22 11rrca]tRtRtoRtRV3RlRlltV3RlRltRV3RlR lltV3R lR ltV3R lR lt Rt Vt R#)GenerationStateiaShared generation state across all handlers. Manages per-model :class:`GenerateManager` instances (each with its own :class:`InferenceThread` so different models can run concurrently while ``torch.compile`` / CUDA graphs require same-model-same-thread) and a single :class:`CBGenerateManager` for continuous batching. Args: continuous_batching (`bool`, *optional*, defaults to `False`): Whether to use continuous batching with paged attention instead of sequential ``model.generate()`` calls. Nc*<V^8dQhRS[RS[RR/#)rcontinuous_batchingcompilerrr)rr9s"rrGenerationState.__annotate__s) - -! - -5 -rcTWnW nW0n/VnRVnRVnR#r<)_continuous_batching_compiler_generate_managers _cb_manager _cb_model_id)r>r r rs&&&&rr?GenerationState.__init__s, %8! #>@59(,rc*<V^8dQhRRRS[RS[/#)rrjrmodalityr)r(r)rr9s"rrr s$->(W[rc VP'gR#\VR4;'dV\P8HpV'g-\P VP P R24V#)a'Check if continuous batching can be used for this model and modality. Args: model (`PreTrainedModel`): The loaded model. modality (`Modality`): The detected model modality (LLM, VLM, etc.). Returns: `bool`: ``True`` if CB is enabled and the model supports it, ``False`` otherwise. FrzM does not support continuous batching. Falling back to sequential generation.)r rr(r)r4 warning_oncerr/)r>rjrcans&&& ruse_continuous_batching'GenerationState.use_continuous_batchingsc(((e78UUX=U   ??++,-99  rc,<V^8dQhRS[RS[RS[/#)rruse_cbr)rrr)rr9s"rrr s#11C11BU1rc |V'd|VPV8wd0VPe"VPP4RVnVPf"\VPR7VnWnVP#WP 9d\ 4VP V&VP V,#)a6Return a per-model generation manager, lazily created on first request. Args: model_id (`str`): The model ID in ``'model_id@revision'`` format. use_cb (`bool`): Whether to return a CB manager or a sequential one. Returns: `BaseGenerateManager`: Either a `GenerateManager` or `CBGenerateManager`. N)r)rrrrrrr)r>rrs&&&r get_managerGenerationState.get_managers   H,##/$$))+'+D$'#4t#O $,!## # 22 20?0AD # #H -&&x00rc<V^8dQhRR/#rr.)rr9s"rrr (s$$$$rc hVPe$VPP4RVnR#R#)z$Stop any active generation managers.N)rrr?s&rshutdownGenerationState.shutdown(s-    '    ! ! ##D  (rc <V^8dQhRS[/#rr)rr9s"rrr .sGGTGrc bVPRJ;'gVPP4#)zTWhether the CB worker is healthy. ``True`` if CB is disabled or not yet initialized.N)rrr?s&r is_cb_aliveGenerationState.is_cb_alive.s*4'FF4+;+;+D+D+FFr)rrrrr r)FFNF) r/r0r1r2rAr?rrr!r%r3rBrCs@rrrs>  - -(11.$$ GGrrca]tRtRtoRtRt]!4tRV3RlRlltV3RlRlt ] V3RlR l4t V3R lR lt RV3R lR llt ] V3RlRl4tV3RltRtVtR#) BaseHandleri3aShared logic for chat completion and responses handlers. Provides model resolution, generation config building, and SSE formatting. Generation is delegated to the shared :class:`GenerationState`. Args: model_manager (`ModelManager`): Handles model loading, caching, and lifecycle. generation_state (`GenerationState`): Shared state managing per-model generation managers. Nc8<V^8dQhRRRS[RS[R,/#)r model_managerrgeneration_statechat_template_kwargsN)rrl)rr9s"rrBaseHandler.__annotate__Cs-??%?*?#Tk ?rc@WnW nT;'g/VnR#r<)r+r,r-)r>r+r,r-s&&&&rr?BaseHandler.__init__Cs + 0$8$>$>B!rc$<V^8dQhRS[RR/#)rbodyrNrk)rr9s"rrr.Ms Y Yd Yt Yrc >^RIHp\VP44pVPe<V\ VPR\44, pV'dV!RRV 2R7hW0P ,pV'd\PRV 24R#R#)zMValidate request fields against the handler's params class and unused fields. HTTPExceptionN__mutable_keys__iz"Unexpected fields in the request:  status_codedetailz,Ignoring unsupported fields in the request: ) fastapir5r>keys_valid_params_classrw_unused_fieldsr4r)r>r2r5 input_keys unexpectedunuseds&& r_validate_requestBaseHandler._validate_requestMs)%  # # /#gd.F.FHZ\_\a&bbJ#>`ak`lr2r5 requestedrjrMrr|s&& r_resolve_modelBaseHandler._resolve_modelas *HHW%    ) ) 5$6H6H6T6T)T# #243E3E3Q3Q2RR`aj`kkmn**66I*)4W %%88?--FFxFe ))rc.<V^8dQhRS[RRRS[RR/#)rr2model_generation_configr rr)rlr)rr9s"rrr.zs-0!0!0!3E0!OS0! 0!rc ^RIHpVPR4e%V!R /\P!VR,4BpM<\ P !V4pVPeVPR8dRVnVPR4e6\VR,4Vn \VR,4R8XdRVn VPR4e\VR,4Vn VPR4e\VR,4VPP'dVPfR VnV'dRVnV#) aBuild a GenerationConfig from shared params (temperature, top_p, seed, generation_config JSON). Subclasses should call ``super()._build_generation_config(...)`` then apply endpoint-specific params (``max_tokens``, ``max_output_tokens``, etc.). Args: body (`dict`): The raw request body. model_generation_config (`GenerationConfig`): The model's default generation config (will be deep-copied). use_cb (`bool`, *optional*, defaults to `False`): Whether continuous batching is active. If ``True``, disables the model's internal KV cache (CB manages its own paged cache). Returns: `GenerationConfig`: A new config with request-specific overrides applied. )r ri temperaturegFtop_pr\staticr.) transformersr rrYloadscopydeepcopyrfloatrW do_samplerXr`r,rcache_implementation use_cache)r>r2rUrr rs&&&& r_build_generation_config$BaseHandler._build_generation_configzs ( 2 88' ( 4 0 Y4::dCV>W3X Y  $ .E F  //7;L;[;[^b;b37!0 88M " .,1$}2E,F  )T-()S0.3!+ 88G  (&+DM&:  # 88F  ' 4< (  ) ) ).?.T.T.\5=  2 */  '! rcL<V^8dQhRS[S[,RS[RS[S[,/#)rmessagesrr)rrlr()rr9s"rrr.s2H H T$ZH 8H X\]aXbH rc  j.pVEF)pRVR,R./pRV9d.pVR,Fp\P!V4pVPR4;'gTp\VR,\4'd!\ P !VR,4VR&VPV4K WTR&RV9d VR,VR&RV9d.MVPR4;'g.p\V\4'dRRRV/.pVEFp V R,p V R9d&VR,PRRRV R,/4K9V R9dlV\P\P39dGV R ,p \V \4'd V R ,p VR,PRR R V /4KV R 8XdV\P8XdnV R ,p \V \4'd.V R ,p V PR4pV'd RV RV 2MT p MT p VR,PRRR V /4EK4V R8XdSV\P\P39d.VR,PRRR V R,R ,/4EKV R8XgEKV\P8XgEKVR,PRRR V R,R ,/4EK V\P8Xd#RPRVR,44VR&VPV4EK, V#)aConvert OpenAI-format messages to the format expected by HF processors. All modalities extract text. VLM additionally handles ``image_url`` and ``video_url``. MULTIMODAL handles all of the above plus ``input_audio`` and ``audio_url``. For LLMs, the content parts are collapsed into a plain text string. Args: messages (`list[dict]`): OpenAI-format chat messages. modality (`Modality`): The model modality (LLM, VLM, or MULTIMODAL). Returns: `list[dict]`: Processor-compatible messages. rMrTrVr\ra tool_call_idr[rU image_urlurlimage input_audiodatarz data:audio/z;base64,audio video_urlvideo audio_url c32"TF qR,xK R#5i)rUNr.)rqrs& rruABaseHandler.get_processor_inputs_from_messages..s,R@Q1vYY@Qs)rU input_text output_text)rh input_image)r\r]rrrrYr[rr(r*r+rlr)join)rerprocessor_inputsmessagerrVrr raw_contentrT content_typerirk audio_b64fmts&& r"get_processor_inputs_from_messages.BaseHandler.get_processor_inputs_from_messagessGgfoy"=Fw& !,//Br*B +11rB!"[/377*.**R _*E;%%b) 0 (2|$()0)@~&!- 7"gkk)>T>Z>ZXZK+s++ & DE &&v #HH9%,,fffgfo-VW!%AAhS[S_S_aiatatRuFu!+.C!#t,,!%j9%,,fguc-JK"]2x8CVCV7V")-"8K!+t44$/$7 )ooh7HK C5 DQZ)9%,,fguc-JK![0X(,,PXPcPcAd5d9%,,fgugkFZ[`Fa-bc![0XATAT5T9%,,fgugkFZ[`Fa-bc7'<8<<'$'HH,Ry@Q,R$Ry!  # #F +o p rcP<V^8dQh/S[R,;R&S[S[,;R&#)rNr<r=)r[r>r)rr9s"rrr.3s'+H$r)r-r,r+r<r')r/r0r1r2rAr<r>r=r?rA staticmethodrIrRrbr~rhr3rBrCs@rr)r)3sx (,"uN?? Y YGG **20!0!dH H urr)) qwen2 qwen2_moeqwen2_vl qwen2_5_vlqwen3 qwen3_moe qwen3_nextqwen3_vl qwen3_vl_moe)qwen3_5 qwen3_5_moe)gemma4)zz<|tool_response>z)r)JrAr r\enumrYrabcrrcollections.abcrconcurrent.futuresr dataclassesrrrtypingr transformers.utilsr transformers.utils.chat_parsingr pydantic tokenizersr^rZr r rrr:transformers.generation.continuous_batching.continuous_apir4transformers.generation.continuous_batching.requestsr5transformers.generation.continuous_batching.schedulerrr+r get_loggerr/r4 X_REQUEST_IDr&Enumr(r5rurErrHr3rKrzrer~rrrrrrrrrCr`rfrhrrrrr)r.rrrsN #$%! &:eUO+   H %  tyyP9PC   V[)1  6  146fj*kJ   !T6 ! @V[)1  6  P!>4<!#b!fj*vxQ\^i>jk   !G6 ' !<V[))+=> -6  B46#T#w&8%9!fj*vxQ\^i>jk  A6# Sc L   g Q.5" '*>!H( ( VDNPPfK2K2\!**Z@>#@>FA )A He1+e1PNGNGbB B r