+ j!^RIHt^RIt^RIHt^RIHt^RIH t H t ^RI H t ^RI Ht^RIHtHt^RIHt^R IHt^R IHtHtHtHt^R IHt^R IHt^R IH t ^RI!H"t"^RI#H$t$H%t%H&t&H't'H(t(H)t)H*t*H+t+^RI,H-t-H.t.^RI/H0t0]Pb!]24t3]!RR7]!RR]444t4!RR])4t5!RR]*4t6!RR]'4t7!RR]$4t8!RR]%4t9]!R R7]!R!R"]444t:]!R#R$](]:44t;!R%R&]Px4t=]!R'R7!R(R)]&] 44t>!R*R+]Px4t?]!R,R-](44t@]!R.R7!R/R0]:]044tA.R1OtBR#)2) dataclassN)initialization)Cache DynamicCache)GenerationMixin)create_causal_mask)BaseModelOutputWithPastCausalLMOutputWithPast)PreTrainedModel)Unpack) ModelOutputauto_docstringcan_return_tuplelogging)merge_with_config_defaults)is_torchdynamo_compiling)capture_outputs) AutoModel)LlamaAttentionLlamaDecoderLayerLlamaForCausalLMLlamaMLP LlamaModel LlamaRMSNormLlamaRotaryEmbeddingTransformersKwargs) CsmConfigCsmDepthDecoderConfig)CsmGenerationMixinz: Base class for the model autoregressive outputs. ) custom_introc\a]tRt^1toRtRtRtRtRtRt Rt Rt Rt Rt RtRtV3RltRtVtR#)CsmOutputWithPasta= loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. depth_decoder_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the depth decoder model. depth_decoder_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the depth decoder (scores for each vocabulary token before SoftMax). depth_decoder_past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). depth_decoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. depth_decoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. backbone_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the backbone model. Nc<<V^8dQh/S[PR,;R&S[PR,;R&S[R,;R&S[S[PR3,R,;R&S[S[PR3,R,;R&S[PR,;R&S[PR,;R &S[R,;R &S[S[PR3,R,;R &S[S[PR3,R,;R &S[PR,;R &#)Nlosslogitspast_key_values. hidden_states attentionsdepth_decoder_lossdepth_decoder_logitsdepth_decoder_past_key_valuesdepth_decoder_hidden_statesdepth_decoder_attentions backbone_loss)torch FloatTensorrtuple)format __classdict__s"d/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/csm/modular_csm.py __annotate__CsmOutputWithPast.__annotate__1s=F   d ")GH    $+IJT\(KL**C/047>MNe'',-4;OP))D07QR ++d29ST$)4<6UV"'u'8'8#'=!>!ELWX$E$5$5s$:;dBIYZ$$t+2[)__name__ __module__ __qualname____firstlineno____doc__r%r&r'r(r)r*r+r,r-r.r/__annotate_func____static_attributes____classdictcell__)r4s@r5r"r"1sP8&*D'+F$(O:>M7;J375926!HLEI.2M[r8r"c]tRt^btRtR#) CsmRMSNormr9Nr:r;r<r=r@r9r8r5rCrCbr8rCc]tRt^ftRtR#)CsmRotaryEmbeddingr9NrDr9r8r5rGrGfrEr8rGc]tRt^jtRtR#)CsmMLPr9NrDr9r8r5rIrIjrEr8rIc]tRt^ntRtR#) CsmAttentionr9NrDr9r8r5rKrKnrEr8rKc]tRt^rtRtR#)CsmDecoderLayerr9NrDr9r8r5rMrMrrEr8rMz[ The bare Csm Model outputting raw hidden-states without any specific head on top. caa]tRt^vtoRtR tRtR.tR.tRt Rt Rt Rt R] R]/t]P"!4V3Rl4tV3RltR tVtV;t#) CsmPreTrainedModelmodelTrMr'r(r)c<\SV`V4\V\4'd_VPp\ V^, 4F:p\ P!VPRVPPR7K< R#\V\4'dg\ P!VP\P!VPP4VPP ,4R#R#)g)meanstdN)super _init_weights isinstanceCsmCodebooksHead num_codebooksrangeinitnormal_weightconfiginitializer_rangeCsmBackboneModelEmbeddingscopy_audio_tokens_offsetsr0arange vocab_size)selfmodulerYi __class__s&& r5rV CsmPreTrainedModel._init_weightss f% f. / /"00M=1,- V]]$++:W:WX.  : ; ; JJv22ELLAZAZ4[^b^i^i^t^t4t u*>ARAR*RU[UpUpq')yy1L1LfN`N`gl'm$r8c6<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[R,RS[PR,RS[R,R S[S[,R S[S[ ,/ #) r$ input_idsNbackbone_last_hidden_stateattention_mask position_idsr' inputs_embeds use_cachekwargsreturn) r0 LongTensorr1Tensorrboolr rr2r )r3r4s"r5r6!CsmDepthDecoderModel.__annotate__sN N ##d*N %*$5$5$<N  t+ N &&- N  N ((4/N $;N +,N  ( (N r8c `Ve(\4'g\PR4RpVRJVRJ, 'd \R4hV'dVf\ VP R7pVeVP 4M^p VeVP^,MVP^,p Ve VPM VPp \P!WV ,V R7pVf\P!V^, ^R7p WP,p VPW,4pV^,^8HpVeW&R&M-\4'gV'd\PR4VPV4p\!VP VVVVR7pTpVP#^4pVP%VVR 7pVP&RVP P(FpV!V3R VR VR VR VRV/VBpK VP+V4p\-TV'dVR7#RR7#)a* backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. NzCustom `position_ids` were provided but will be ignored. CSM depth decoder automatically determines position_ids and as it requires them to be identical across the batch, the provided position_ids will be ignored.z;You must specify exactly one of input_ids or inputs_embeds.)r^device)minzvWhen the first codebook token is provided, `backbone_last_hidden_state` should also be provided for correct inference.)r^rrr'r)rrrr'rposition_embeddings)last_hidden_stater')NNNr)rlogger warning_once ValueErrorrr^get_seq_lengthshaperr0rcclamprdrwarningrr unsqueeze rotary_emblayersnum_hidden_layersnormr )rerrrrr'rrrpast_seen_tokensinputs_seq_lengthr codebook_idxsoffsetinput_ids_are_first_codebook causal_maskr(r decoder_layers&&&&&&&&, r5forwardCsmDepthDecoderModel.forwards;&  #,D,F,F   w  L  -t"; < <Z[ [ 0*$++>O?N?Z?99;`a6C6OM//2U^UdUdefUg)6)B%% HXHX||$4IZ6Zcij  !KK q(8a@M"__4F --i.@AM+7?a+? ()5&@d#/116RNNQ44]C (;;')+%  & $--a0 "oom,oW![[)H4;;+H+HIM)**!0  $  %8 MJ -0 &+/8O  >B  r8c&<V^8dQh/S[;R&#rk)r)r3r4s"r5r6rs !!r8)rr)NNNNNNN) r:r;r<r=rrrrrr?r@rArzr{s@@r5r}r}s>n  N N  N r8r}c<aa]tRt^toV3RltRRltRtVtV;t#)rXc<\SV`4W n\P!\ P !VP^, W44VnR#)rRN)rUrrYr Parameterr0emptyr])rerrYrdrhs&&&&r5rCsmCodebooksHead.__init__s: *ll5;;t/A/AA/E{#_` r8c >V^, pVPV,p\VP^,4Uu.uF=p\PP VRVR3,W4,P 4NK? pp\P!V^R7pV#uupi)rRrdim) r]rZrr functionallinearTr0stack)rer(codebook_indicescodebook_weight codebook_idxs&&& r5rCsmCodebooksHead.forwards+a/++&67!&o&;&;A&> ? ?  MM q,/A!BODaDcDc d ?   Mq9   sAB)rYr]N r:r;r<r=rrr@rArzr{s@@r5rXrXsa  r8rXa$ The CsmDepthDecoder Model transformer, with a [`CsmCodebooksHead`] on top, which can be seen a position-specific language modeling head, allowing to use a different linear layer for each codebook (e.g. position 0 is the first codebook and uses the first codebook head, etc.) caa]tRtRtoRtRtRtV3RltR V3RlV3Rlllt] ] R V3RlRll44t Rt Vt V;t#) CsmDepthDecoderForCausalLMi Nc<\SV`V4V=\VPVP VP 4Vn\V4Vn R#r) rUrlm_headrXrrYrdcodebooks_headr}rPrs&&r5r#CsmDepthDecoderForCausalLM.__init__sE  L.v/A/A6CWCWY_YjYjk)&1 r8c <V^8dQhRS[PRS[R,RS[R,RS[PR,RS[PR,RS[R,/#)r$rnext_sequence_lengthNr'rris_first_iteration)r0rintrr1r)r3r4s"r5r6'CsmDepthDecoderForCausalLM.__annotate__so##"Dj  ((4/  ((4/ !4Kr8c <\S V`!WW4V3/VBpV'gVPR4VPR4V#)rr)rUprepare_inputs_for_generationpop) rerrr'rrrr model_inputsrhs &&&&&&&, r5r8CsmDepthDecoderForCausalLM.prepare_inputs_for_generationsMw< _m _e "   9 : (r8c<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[R,RS[PR,RS[PR,R S[R,R S[S[P,R S[S[,R S[ S[ ,/ #) r$rNrrrr'rlabelsrlogits_to_keeprr) r0rr1rrrrr rr2r )r3r4s"r5r6r5sA A ##d*A %*$5$5$<A  t+ A &&- A  A ((4/A   4'A $;A ell*A +,A  ' 'A r8c  VeVP4M^p VeVP^,MVP^,p Ve VPM VPp \P!WR7V ,pVP !RRVRVRVRVRVRVR V/V BpV^,p\ V \4'd#V ^8Xd\^R4pM\V )R4pMT pVPVR VR 3,VV,4pVP4pRpVeFVR,P4pVP!RR VR RR VPPRV/V Bp\VVVPVP VP"R7#)a backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Nrrrrrr'rrrr&rrd shift_labels)r%r&r'r(r)r9).rRNN)rrrr0rcrPrWrslicer contiguous loss_functionr^rdr r'r(r))rerrrrr'rrrrrrseq_lenrroutputsr( slice_indicesr&r%rs&&&&&&&&&&, r5r"CsmDepthDecoderForCausalLM.forward3s0@O?Z?99;`a,9,E-%%a(9??[\K])6)B%% HXHX <<?BRR**    'A  *  &  ,  (         nc * *" %a %~ot < *M$$]1mQ3F%GIYZgIhi""$  !'?557L%%&*7;{{7M7M\hlrD&#33!//))   r8)rrP)NNNNF) NNNNNNNNr)r:r;r<r=_tied_weights_keys_tp_plan_pp_planrrrrrr@rArzr{s@@r5rr sPHH2 ,A A A A r8rc8aa]tRtRtoV3RltRtRtVtV;t#)r`iyc><\SV`4\P!VPVP ,VP 4Vn\P!\P!VP4VP ,RR7Vn R#)F) persistentN) rUrrrrY codebook_sizerembed_audio_tokensBufferr0rcrbrs&&r5r#CsmBackboneModelEmbeddings.__init__zsj "$,,0D0DvG[G[0[^d^p^p"q$&II LL-- .1E1E ERW% !r8clVPWP,4pVP^R7pV#)r$r)rrbsum)rerrs&& r5r"CsmBackboneModelEmbeddings.forwards4// r8c $<\SV`!R/VB#)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks) or (batch_size, sequence_length)`): 1. (batch_size, sequence_length): corresponds to the input sequence prepared with the processor from the text prompt. Such input requires `input_values` to be provided so that audio can be encoded in codebook tokens and then merged with the text tokens. 2. (batch_size, sequence_length, num_codebooks): codebook tokens generated during the autoregressive decoding. Such input is not meant to be used by end users. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) r9)rUr)re super_kwargsrhs&,r5rCsmBackboneModel.forwards w...r8)r) r:r;r<r=rrrrrr@rArzr{s@@r5rrs3?  /  / /r8rz The Csm model consists of two llama-like auto-regressive transformer models: a backbone model that predicts the first codebook token and a depth decoder that predicts the other codebook tokens. caa]tRtRtoRR/tV3RltRtRt]V3Rl4t V3Rlt RV3R lR llt RV3R lV3R lllt ] ]RV3R lRll44tRtVtV;t#)CsmForConditionalGenerationiz5backbone_model.embed_tokens.embed_audio_tokens.weightz'depth_decoder.model.embed_tokens.weightc<\SV`V4VPVn\P!VP VPRR7Vn\P!VPVP 4Vn \PV4Vn \PVP4Vn\ P"!VP$4VnVP)4R#r)rUrrdrrrrrtext_vocab_sizeembed_text_tokensr _from_configbackbone_modelrdepth_decoder_config depth_decoderr from_config codec_config codec_model post_initrs&&r5r$CsmForConditionalGeneration.__init__s   ++yy!3!3V5F5FUS !#f.D.DfFXFX!Y.;;FC7DDVE`E`a$001D1DE r8c.VPP#rrr)res&r5get_input_embeddings0CsmForConditionalGeneration.get_input_embeddingss""///r8c&WPnR#rr)revalues&&r5set_input_embeddings0CsmForConditionalGeneration.set_input_embeddingss+0(r8c <VPRR4'd\S V`!V/VBwr4M\S V`!V/VBpRp\V4p\ VP 4P 4UUu/uF#wrxVPV4'gKWvRVbK% p pp\ VPP 4PRR/V C4V Fp\VP WW,4K! RV9dVX3#V#uuppi)output_loading_infoFdepth_decoder_N_from_model_config) getrUfrom_pretrainedlenvarsgeneration_configitems startswithrupdatedelattr) clsargsrrP loading_infoprefix prefix_lenattrr depth_decoder_attrsrhs &*, r5r+CsmForConditionalGeneration.from_pretraineds  ::+U 3 3"''"94"J6"J E<G+TQCsmForConditionalGeneration._merge_input_ids_with_input_values..s&O=Nrxx{{=Ns!)rdtypeTas_tuplerrrrRri)$rrrpaddiffr0rcmaxrexpandrrryziprZrrencode audio_codes transposeappendrget_audio_codes_maskr^audio_token_idrronesrYlongcodebook_eos_token_idsqueezeaudio_eos_token_idrepeatrnonzero)rerr*r+rr audio_lengthsinput_values_maskaudio_tokens_listbatch_input_valuesbatch_input_values_cutoffsrg start_idxend_idx audio_batch codec_outputs codebook_idsmax_audio_framesr0batched_audio_token_idsaudio_codes_maskrCaudio_token_mask audio_embedsaudio_eos_frame_idsaudio_eos_embedsaudio_eos_token_masklabels_expanded depth_decoder_ignore_frames_idxss&&&&& r5"_merge_input_ids_with_input_values>CsmForConditionalGeneration._merge_input_ids_with_input_valuess|*..y9  ##%==#4#45I6#R 01JKPPRM)!*;$A "<U"C&8i>O9O&P (,(8(8(?(? @U@UVW@X(Y '4'@'@'J'J1b'Q )00aA LGn$'&O=N&O#O */++`qr`qZ\R]]&&rAq!5EQR 5S+TU`qr+'$(#3#3#H#HIZ#[ !!$"[[77N(N: ..;; "O>O O c <V^8dQhRS[PRS[R,RS[R,RS[PR,RS[PR,/#)r$rrNr'rr)r0rrrr1)r3r4s"r5r6r,1sa##"Dj  ((4/  ((4/ r8c Z<\S V`!R RVRVRVRVRV/VBpVeVP^8Xd{VPR4fhVP VVPR4VPR4VPR4R 7pVP RVR,RVR,RR/4V#) rrr'rrNr*r+r)rr*r+rr9)rUrndimrr_r) rerrr'rrrr merged_inputsrhs &&&&&&, r5r9CsmForConditionalGeneration.prepare_inputs_for_generation1sw<  !5 , *  (      Y^^q%8\=M=Mo=^=f CC##ZZ7%+ZZ0F%Gzz(+ DM    -"@(MZbLceprvw r8c<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[R,RS[PR,R S[PR,R S[R,R S[S[P,R S[S[,R S[ S[ ,/ #)r$rNr*rr+rr'rrrrrr) r0rrrr1rrr rr2r")r3r4s"r5r6r,RsY Y ##d*Y llT)Y  t+ Y $llT1 Y &&- Y Y ((4/Y   4'Y $;Y ell*Y +,Y  " "Y r8c  rVe8VP^8Xd'VPWWH4p V R,pV R,pRpVP!RRVRVRVRVRVRV /V Bp V ^,p\V \4'd\ V )R4MT pVP VR VR 3,4pRpRpRpRpVEeVR,pVP!RR VRVR VPP/V BpVR,R8HPRR 7(pVV,R RVPP^, 13,p\PPVR^R7pVPRR7pVV^,V^,^, R 3,pVV,pVP !RRVRVRV RRRV/V BpVP"pVV,p\%TTTTV P&V P(V P*Ve VP,MRVe VP&MRVe VP(MRVeVP*R7 #RR7 #)a` input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks) or (batch_size, sequence_length)`): 1. (batch_size, sequence_length): corresponds to the input sequence prepared with the processor from the text prompt. Such input requires `input_values` to be provided so that audio can be encoded in codebook tokens and then merged with the text tokens. 2. (batch_size, sequence_length, num_codebooks): codebook tokens generated during the autoregressive decoding. Such input is not meant to be used by end users. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) input_values_cutoffs (`torch.Tensor` of shape `(batch_size, max_num_audio)`, *optional*): Specify the end positions of audio segments within each batch entry, relative to the concatenated audio input. If a batch entry has fewer segments than the maximum, it is padded with -1. For example, in a batch of 2 sequences where the first contains 2 audio segments of length l1, and the second contains 1 audio segment of length l2, the input_values_cutoffs would be: [[l1, 2 * l1], [l2, -1]]. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[config.audio_token_id, -100, -101]`. Requires targeted `input_values` to be provided as audio tokens will be inferred from it using the `codec_model`. - `config.audio_token_id` indicates an audio frames (considering sequence length elements as frames) - `-100` will be ignored in the loss computation - `-101` indicates the audio frame will be used only for the backbone model (using the first codebook token as labels) Such labels can be prepared using `output_labels=True` when calling [`CsmProcessor`]. logits_to_keep (`int` or `torch.Tensor`, *optional*): Kept for compatibility. Does not support another value than: 1. `0`, which is equivalent to keeping all logits, used in the training regime 2. `1`, which is equivalent to keeping only the last logit, used in the generation regime Example: ```python >>> import torch >>> from transformers import CsmForConditionalGeneration, AutoProcessor >>> from datasets import load_dataset, Audio >>> model_id = "sesame/csm-1b" >>> torch_device = "cuda" if torch.cuda.is_available() else "cpu" >>> processor = AutoProcessor.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/dailytalk-dummy", split="train") >>> # ensure the audio is 24kHz >>> ds = ds.cast_column("audio", Audio(sampling_rate=24000)) >>> conversation = [] >>> # prepare a conversation with text and corresponding audio >>> for text, audio, speaker_id in zip(ds[:4]["text"], ds[:4]["audio"], ds[:4]["speaker_id"]): ... conversation.append( ... { ... "role": f"{speaker_id}", ... "content": [{"type": "text", "text": text}, {"type": "audio", "path": audio["array"]}], ... } ... ) >>> inputs = processor.apply_chat_template( ... conversation, ... tokenize=True, ... return_dict=True, ... output_labels=True, ... ).to(torch_device) >>> model = CsmForConditionalGeneration.from_pretrained(model_id, device_map=torch_device) >>> output = model(**inputs) >>> output.loss.backward() ```Nrrrrrr'rrr&rdr.)r Tr4r return_dict) r%r/r*r&r'r(r)r+r,r-r.r9)rrr)rrrr8r7r6)rcr_rrWrrrrr^rdallrYrrr9rJrr%r"r'r(r)r&)rerr*rr+rr'rrrrrrdbackbone_outputsbackbone_hidden_statesrbackbone_logitsr%r/r*depth_decoder_outputsbackbone_labels train_maskdepth_decoder_input_ids train_idxsbackbone_last_hidden_statesdepth_decoder_labelss&&&&&&&&&&&, r5r#CsmForConditionalGeneration.forwardPsd  Y^^q%8 CC)=M*/:M"8,FI..  ) & ,  (     "2!!48B>SV8W8W~ot4]k ,,'=aPQ>Q'RS ! $  $WoO ..&/>KO;;KaKaekM "(+t388R8@@J&,Z&8>] @Y@Y\]@]>]9]&^ #&(mm&7&78OQW_`&7&a ##++T+:J*@APZ[\P]`aPacdAd*e '#)*#5 $($6$6%1%+F%$%! % , %  % !"7!;!;  #55D '1",<<*88'22AVAb!6!=!=hl$0+@*O*O$0)>(K(KI^Ij%:%E%E  qu  r8)rrrrrrd)NNNN) NNNNNNNNNr)r:r;r<r=rrrr  classmethodrr'r_rrrrr@rArzr{s@@r5rrs @Aj0141PBPBd>Y Y Y Y r8r)rOrr}rr)C dataclassesrr0torch.nnrrr[ cache_utilsrr generationr masking_utilsrmodeling_outputsr r modeling_utilsr processing_utilsr utilsr rrr utils.genericrutils.import_utilsrutils.output_capturingrautorllama.modeling_llamarrrrrrrrconfiguration_csmrrgeneration_csmr get_loggerr:rr"rCrGrIrKrMrOr}ModulerXrr`rr__all__r9r8r5rs" &.)/O-&KK7:5   @.   H %  '3 '3  '3V   -  X  >  '  vv  v<Y :'9Y Y xryy(d !1?d d N  /z//0 F "46HF  F R  r8