+ ja&Rt^RIt^RIHt^RIHt^RIHt^RIt^RI H t ^RI H t ^RI H t^RIHt^RIHtHt^R IHtHt^R IHt^R IHt^R IHtHtHtHt^R I H!t!H"t"^RI#H$t$H%t%^RI&H't'^RI(H)t)H*t*H+t+^RI,H-t-^RI.H/t/^RI0H1t1^RI2H3t3H4t4]+Pj!]64t7]*!RR7]!RR]444t8]*!RR7]!RR]444t9]*!RR7]!RR]444t:]*]!RR ]444t;!R!R"] Px4t=!R#R$] Px4t>!R%R&] Px4t?!R'R(] Px4t@R)tARAR*ltB!R+R,] Px4tCR-R.ltDRBR/R0lltE!R1R2] Px4tF!R3R4]4tG]*!R5R6]%44tH!R7R8]H]4tI]*!R9R:]H44tJ]*!R;R7!R<R=]H]44tK]*!R>R7!R?R@]H]44tL.RCOtMR#)DzPyTorch Moshi model.N)Callable) dataclass)Any)CrossEntropyLoss)initialization)ACT2FN)Cache DynamicCache)GenerationConfigGenerationMixin)create_causal_mask)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast ModelOutputSeq2SeqLMOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging)deprecate_kwarg)maybe_autocast) AutoModel) MoshiConfigMoshiDepthConfigzK Outputs of [`MoshiForConditionalConditionalGeneration.generate`]. ) custom_introcXa]tRt^-toRtRtRtRtRtRt Rt Rt Rt Rt RtV3RltRtVtR#)(MoshiConditionalGenerationGenerateOutputab audio_sequences (`torch.LongTensor` of shape `(batch_size*num_return_sequences, 1, sequence_length)`, *optional*): The generated audio waveforms. sequences (`torch.LongTensor` of shape `(batch_size*num_return_sequences, sequence_length)`): The generated text sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. sequences_scores (`torch.FloatTensor` of shape `(batch_size*num_return_sequences)`, *optional*, returned when `output_scores=True`): Final beam scores of the generated `sequences`. scores (`tuple(torch.FloatTensor)` *optional*, returned when `output_scores=True`): Beam transition scores for each vocabulary token at each generation step. Beam transition scores consisting of log probabilities of tokens conditioned on log softmax of previously generated tokens in this beam. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size*num_beams, config.vocab_size)`. logits (`tuple(torch.FloatTensor)` *optional*, returned when `output_logits=True`): Unprocessed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size, config.vocab_size)`. beam_indices (`torch.LongTensor`, *optional*, returned when `output_scores=True`): Beam indices of generated token id at each generation step. `torch.LongTensor` of shape `(batch_size*num_return_sequences, sequence_length)`. attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size*num_beams, num_heads, generated_length, sequence_length)`. hidden_states (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size*num_beams*num_return_sequences, generated_length, hidden_size)`. past_key_values (`Cache`, *optional*, returned when `use_cache=True`): Contains the model cache, used to speed up decoding. Different models have a different cache format, check the model's documentation. Usually, a [`~cache_utils.Cache`] instance. audio_codes (`torch.LongTensor` of shape `(batch_size*num_return_sequences, num_codeooks, sequence_length)`, *optional*): The generated audio codes. Returned if `return_audio_codes=True`. Intermediate audio "tokens" which transforms to `audio_sequences` once passed through the audio decoder. Nc2<V^8dQh/S[PR,;R&S[PR,;R&S[PR,;R&S[S[P,R,;R&S[S[P,R,;R&S[PR,;R&S[S[S[P,,R,;R&S[S[S[P,,R,;R &S[R,;R &S[PR,;R &#) Naudio_sequences sequencessequences_scoresscoreslogits beam_indices attentions hidden_statespast_key_values audio_codes)torchTensor LongTensor FloatTensortupler)format __classdict__s"i/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/moshi/modeling_moshi.py __annotate__5MoshiConditionalGenerationGenerateOutput.__annotate__-sP\\D(/QR$&-ST''$.5UV %## $t +2WX %## $t +2YZ""T)0[\eE--./$6=]^u0012T9@_`T\(ab!!D(/c)__name__ __module__ __qualname____firstlineno____doc__r$r%r&r'r(r)r*r+r,r-__annotate_func____static_attributes____classdictcell__r4s@r5r!r!-sHB,0O)-I15.2F.2F,0L9=J<@M$(O+/Kcr8r!z% `MoshiForCausalLM` outputs. cHa]tRt^atoRtRtRtRtRtRt Rt V3Rlt Rt Vt R#)MoshiCausalLMOutputWithPasta loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. NcB<V^8dQh/S[PR,;R&S[PR,;R&S[PR,;R&S[R,;R&S[S[PR3,R,;R&S[S[PR3,R,;R&#) r#Nlossr(last_hidden_stater,.r+r*r.r1rr2)r3r4s"r5r6(MoshiCausalLMOutputWithPast.__annotate__as&   d ")'(    $+)*((4/6+,T\(-.**C/047>/0e'',-4;1r8r9)r:r;r<r=r>rFr(rGr,r+r*r?r@rArBs@r5rDrDas2 &*D'+F26$(O:>M7;J1r8rDz2 `MoshiForConditionalGeneration` outputs. c\a]tRt^|toRtRtRtRtRtRt Rt Rt Rt Rt RtRtV3RltRtVtR#)(MoshiConditionalGenerationOutputWithPasta loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `text_labels` is provided): Text language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the text language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. depth_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `audio_labels` is provided): Audio language modeling loss (for next-token prediction). audio_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the audio language modeling heads. depth_past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Past key-values of the depth decoder. depth_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Hidden states of the depth decoder depth_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Depth decoder's Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nc<<V^8dQh/S[PR,;R&S[PR,;R&S[PR,;R&S[R,;R&S[S[PR3,R,;R&S[S[PR3,R,;R&S[PR,;R &S[PR,;R &S[R,;R &S[S[PR3,R,;R &S[S[PR3,R,;R &#)r#NrFr(rGr,.r+r* depth_loss audio_logitsdepth_past_key_valuesdepth_hidden_statesdepth_attentionsrH)r3r4s"r5r65MoshiConditionalGenerationOutputWithPast.__annotate__|s9<   d ")=>    $+?@((4/6ABT\(CD**C/047>EFe'',-4;GH!!D(/IJ##d*1KL!4<.MNu00#56=DOPE--s23d:AQr8r9)r:r;r<r=r>rFr(rGr,r+r*rMrNrOrPrQr?r@rArBs@r5rKrK|sO.&*D'+F26$(O:>M7;J+/J-1L*.@D=AQr8rKc@a]tRt^toRtRtRtRtRtV3Rlt Rt Vt R#)MoshiUnconditionalInputa input_ids (`torch.Tensor `of shape `(batch_size, sequence_length), *optional*): The sequence used as a text prompt for the generation. user_audio_codes (`torch.Tensor `of shape `(batch_size, num_codebooks, sequence_length), *optional*): The audio codes used as audio user prompt for the generation. Has priority over `user_input_values` and represents the audio "tokens" of `user_input_values` once passed through the audio encoder. moshi_audio_codes (`torch.Tensor `of shape `(batch_size, num_codebooks, sequence_length), *optional*): The audio codes used as audio Moshi prompt for the generation. Has priority over `moshi_input_values` and represents the audio "tokens" of `moshi_input_values` once passed through the audio encoder. attention_mask (`torch.LongTensor`) of shape `(batch_size, sequence_length)`, *optional*): Attention mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: 1 for tokens that are **not masked**, 0 for tokens that are **masked**. Nc<V^8dQh/S[PR,;R&S[PR,;R&S[PR,;R&S[PR,;R&#)r#N input_idsuser_audio_codesmoshi_audio_codesattention_mask)r.r0r/)r3r4s"r5r6$MoshiUnconditionalInput.__annotate__sa$&- llT)0!"||d*1#$$$t+2%r8r9) r:r;r<r=r>rVrWrXrYr?r@rArBs@r5rTrTs) *.I,0-1.2N%r8rTcTaa]tRt^toRV3RlV3RllltRtRtRtRtVt V;t #) MoshiRMSNormc&<V^8dQhRS[RS[/#)r#dimeps)intfloat)r3r4s"r5r6MoshiRMSNorm.__annotate__s44C4e4r8c<\SV`4W n\P!\ P !V44VnR#N)super__init__r_nn Parameterr.onesweight)selfr^r_ __class__s&&&r5rfMoshiRMSNorm.__init__s+ ll5::c?3 r8cV\P!VP^4PRRR7VP,4,#)r#T)keepdim)r.rsqrtpowmeanr_)rkxs&&r5_normMoshiRMSNorm._norms45;;quuQx}}R}>IJJJr8cVPVP44pW PP4,pVPV4#rd)rurarjtype_as)rkrtoutputs&& r5forwardMoshiRMSNorm.forwards:AGGI&++++--~~a  r8c^\VPP4 RVP 2#)z, eps=)r2rjshaper_rks&r5 extra_reprMoshiRMSNorm.extra_reprs' ))*+6$((<) r:r;r<r=rfrurzrr@rA __classcell__rlr4s@@r5r\r\s$44 K! ==r8r\c<aa]tRt^toV3RltRRltRtVtV;t#)MoshiFlexibleLinearc<\SV`4\P!\P !W2V44VnR#rd)rerfrgrhr.randnrj)rk input_size output_size num_layersrls&&&&r5rfMoshiFlexibleLinear.__init__s) ll5;;z #ST r8c Ve#\P!VP^V4M VPpVP^^4R,p\P!VR,V4pVP ^4#)a `MoshiFlexibleLinear` creates one linear layer per codebook. There's multiple ways to use it. In the default case, `sequence_length=num_layers`, so each element of the sequence will be matmul to the weights corresponding to its index on the sequence. For more advanced cases, one can specify which codebook's layer(s) to use with `layer_idx`. If `layer_idx` indicates a single integer, all of the element of the sequence will be matmul to this single codebook's layer. But if `layer_idx` is a tensor of shape `(seq_length,)`, it will matmul each i-th element of the input sequence to the corresponding layer `weight[i]`. Args: x (`torch.FloatTensor): input to the layer of shape `(batch, num_layers, embed_dim)` or of shape `(batch, seq_length, embed_dim)` layer_idx (`torch.Tensor`, *optional*): Can be used to specify which codebook's layers(s) to use. If it's a tensor of shape `(seq_length,)`, will matmul each element of the sequence to the corresponding weights. But if `layer_idx` is a tensor of shape `(seq_length,)`, it will matmul each i-th element of the input sequence to the corresponding layer `weight[i]`. )NNNNrr)rrNr)r. index_selectrj transposematmulsqueeze)rkrt layer_idxselected_weightss&&& r5rzMoshiFlexibleLinear.forwardso(MVLa5--dkk1iHgkgrgr,55a;MJ LL=)+; <yy|r8)rjrd r:r;r<r=rfrzr@rArrs@@r5rrsU r8rc@aa]tRt^toRV3RlltRRltRtVtV;t#) MoshiLinearc<\SV`4W@nV'g \P!WRR7VnR#\ WVR7VnR#)Fbias)rN)rerfuse_flexible_linearrgLinearlinearr)rk input_dim output_dim num_codebooksrrls&&&&&r5rfMoshiLinear.__init__s8 #6 "))IFDK-iP]^DKr8cjVP'dVPW4#VPV4#rd)rr)rkrtrs&&&r5rzMoshiLinear.forwards+  # # #;;q, ,;;q> !r8)rrFrdrrs@@r5rrs_""r8rcaa]tRtRto]!RRR7R V3RlV3Rlll4t]]!RRR7R V3RlRll44t]P!4] R 44t R t Vt V;t#) MoshiRotaryEmbeddingi devicez5.18)versionc <V^8dQhRS[/#r#configr)r3r4s"r5r6!MoshiRotaryEmbedding.__annotate__ sOO{Or8c<\SV`4VPVnVPVnWnVP P R,VnVPpVPR8wd\VP,pV!VP V4wq@n \P!VRR7Vn \P!VP4RR7VnR#) rope_typedefaultF) persistentN)rerfmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrrope_parametersrcompute_default_rope_parametersrattention_scalingrgBufferinv_freqcloneoriginal_inv_freq)rkrr rope_init_fnrrls&&& r5rfMoshiRotaryEmbedding.__init__ s "("@"@$*$B$B! 44[A!%!E!E >>Y &.t~~>L+7 V+L(( (u= !#8>>+;!Nr8cP<V^8dQhRS[RS[S[PS[3,/#)r#rreturn)rr2r.r/ra)r3r4s"r5r6rs.55 5W\]b]i]ikp]pWq5r8c 0VPR,p\VRR4;'gVPVP,pRpRV\P !^V^\P R7V, ,, pVPV4V3#)a~ Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimNg?dtype)rgetattr hidden_sizenum_attention_headsr.arangerato)rrkwargsbaser^attention_factorrs&&, r5r4MoshiRotaryEmbedding.compute_default_rope_parameterss~%%l3fj$/cc63E3EIcIc3c$5<<3#MPS#STU{{6"$444r8c^VPR,PVP^,R ^4P\P VP R7pVR ,P 4p\VP P\4'd2VP PR8wdVP PMRp\VRR7;_uu_4W4,P^^4p\P!Wf3R R7pVP4VP,pVP4VP,p RRR4XPVP R7X PVP R73# +'giLH;i) NrrmpscpuF) device_typeenabledr^r)NrNrp)rNr)rexpandr}rr.rar isinstancetypestrrrcatcosrsinr) rkrt position_idsinv_freq_expandedposition_ids_expandedrfreqsembrrs &&& r5rzMoshiRotaryEmbedding.forward0s2 MM- ( / / 0B0B10Er1 M P PW\WbWbklksks P t !-Z 8 > > @'1!((--'E'E!((--[`J`ahhmmfk  U C C&>II!QOE))UN3C'')d444C'')d444C D vvAGGv$cff177f&;;; D Cs (A4F F, )rrrrrrrrd)r:r;r<r=rrf staticmethodrr.no_gradrrzr@rArrs@@r5rr suXv.OO/O Xv.55/5$ ]]_<<