+ jYRt^RIHt^RIHt^RIt^RIHt^RIHtH t ^RI H t ^RI H t HtHtHtHt^RIHt^R IHt^R IHt^R IHtHt^R IHt^R IHt]P>!] 4t!]!RR7]!RR]444t"]]!RR]444t#]!RR7]!RR]444t$]!RR]$44t%]!RR7!RR]$44t&]!RR7!RR]$]44t'.ROt(R#) zRAG model implementation.)Callable) dataclassN)nn)CacheEncoderDecoderCache)PreTrainedConfig)GenerationConfigGenerationMixinGenerationModeLogitsProcessorListStoppingCriteriaList)GENERATION_MODES_MAPPING) ModelOutputPreTrainedModel)auto_docstringlogging RagConfig RagRetrieverzI Base class for retriever augmented marginalized models outputs. ) custom_introcta]tRt^$toRtRtRtRtRtRt Rt Rt Rt Rt RtRtRtRtRtRtRtRtV3RltRtVtR#)RetrievAugLMMarginOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head. The score is possibly marginalized over all documents for each vocabulary token. doc_scores (`torch.FloatTensor` of shape `(batch_size, config.n_docs)`): Score between each retrieved document embeddings (see `retrieved_doc_embeds`) and `question_encoder_last_hidden_state`. past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains precomputed hidden-states (key and values in the attention blocks) of the decoder that can be used (see `past_key_values` input) to speed up sequential decoding. retrieved_doc_embeds (`torch.FloatTensor` of shape `(batch_size, config.n_docs, hidden_size)`, *optional*, returned when *output_retrieved=True*): Embedded documents retrieved by the retriever. Is used with `question_encoder_last_hidden_state` to compute the `doc_scores`. retrieved_doc_ids (`torch.LongTensor` of shape `(batch_size, config.n_docs)`, *optional*, returned when *output_retrieved=True*): The indexes of the embedded documents retrieved by the retriever. context_input_ids (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Input ids post-processed from the retrieved documents and the question encoder input_ids by the retriever. context_attention_mask (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Attention mask post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. question_encoder_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden states at the output of the last layer of the question encoder pooled output of the model. question_enc_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the question encoder at the output of each layer plus the initial embedding outputs. question_enc_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the question encoder, after the attention softmax, used to compute the weighted average in the self-attention heads. generator_enc_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the generator encoder of the model. generator_enc_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the generator encoder at the output of each layer plus the initial embedding outputs. generator_enc_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the generator encoder, after the attention softmax, used to compute the weighted average in the self-attention heads. generator_dec_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the generator decoder at the output of each layer plus the initial embedding outputs. generator_dec_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the generator decoder, after the attention softmax, used to compute the weighted average in the self-attention heads. generator_cross_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Cross-attentions weights of the generator decoder, after the attention softmax, used to compute the weighted average in the cross-attention heads. Nc<V^8dQh/S[PR,;R&S[PR,;R&S[PR,;R&S[R,;R&S[PR,;R&S[PR,;R&S[PR,;R&S[PR,;R &S[PR,;R &S[S[PR 3,R,;R &S[S[PR 3,R,;R &S[PR,;R&S[S[PR 3,R,;R&S[S[PR 3,R,;R&S[S[PR 3,R,;R&S[S[PR 3,R,;R&S[S[PR 3,R,;R&#)Nlosslogits doc_scorespast_key_valuesretrieved_doc_embedsretrieved_doc_idscontext_input_idscontext_attention_mask"question_encoder_last_hidden_state.question_enc_hidden_statesquestion_enc_attentionsgenerator_enc_last_hidden_stategenerator_enc_hidden_statesgenerator_enc_attentionsgenerator_dec_hidden_statesgenerator_dec_attentionsgenerator_cross_attentionstorch FloatTensorr LongTensortuple)format __classdict__s"e/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/rag/modeling_rag.py __annotate__%RetrievAugLMMarginOutput.__annotate__$sZ   d ")[\    $+]^!!D(/_`T\(ab ++d29cd''$.5ef''$.5gh",,t3:ij).(9(9D(@Gkl!&e&7&7&< = DKmn#5#4#4c#9:TAHop&+%6%6%=Dqr"'u'8'8#'=!>!ELst$E$5$5s$:;dBIuv"'u'8'8#'=!>!ELwx$E$5$5s$:;dBIyz!&e&7&7&< = DK{)__name__ __module__ __qualname____firstlineno____doc__rrrrr r!r"r#r$r%r&r'r(r)r*r+r,__annotate_func____static_attributes____classdictcell__r3s@r4rr$sxDL&*D'+F+/J$(O5915156:CG&GKDH@D#HLEIHLEIGK{r7rcpa]tRt^toRtRtRtRtRtRt Rt Rt Rt Rt RtRtRtRtRtRtRtV3RltRtVtR#)RetrievAugLMOutputa: logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head. The score is possibly marginalized over all documents for each vocabulary token. doc_scores (`torch.FloatTensor` of shape `(batch_size, config.n_docs)`): Score between each retrieved document embeddings (see `retrieved_doc_embeds`) and `question_encoder_last_hidden_state`. past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains precomputed hidden-states (key and values in the attention blocks) of the decoder that can be used (see `past_key_values` input) to speed up sequential decoding. retrieved_doc_embeds (`torch.FloatTensor` of shape `(batch_size, config.n_docs, hidden_size)`, *optional*, returned when *output_retrieved=True*): Embedded documents retrieved by the retriever. Is used with `question_encoder_last_hidden_state` to compute the `doc_scores`. retrieved_doc_ids (`torch.LongTensor` of shape `(batch_size, config.n_docs)`, *optional*, returned when *output_retrieved=True*): The indexes of the embedded documents retrieved by the retriever. context_input_ids (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Input ids post-processed from the retrieved documents and the question encoder input_ids by the retriever. context_attention_mask (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Attention mask post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. question_encoder_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden states at the output of the last layer of the question encoder pooled output of the model. question_enc_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the question encoder at the output of each layer plus the initial embedding outputs. question_enc_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the question encoder, after the attention softmax, used to compute the weighted average in the self-attention heads. generator_enc_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the generator encoder of the model. generator_enc_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the generator encoder at the output of each layer plus the initial embedding outputs. generator_enc_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the generator encoder, after the attention softmax, used to compute the weighted average in the self-attention heads. generator_dec_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the generator decoder at the output of each layer plus the initial embedding outputs. generator_dec_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the generator decoder, after the attention softmax, used to compute the weighted average in the self-attention heads. generator_cross_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Cross-attentions weights of the generator decoder, after the attention softmax, used to compute the weighted average in the cross-attention heads. Ncr<V^8dQh/S[PR,;R&S[PR,;R&S[R,;R&S[PR,;R&S[PR,;R&S[PR,;R&S[PR,;R&S[PR,;R &S[S[PR 3,R,;R &S[S[PR 3,R,;R &S[PR,;R &S[S[PR 3,R,;R&S[S[PR 3,R,;R&S[S[PR 3,R,;R&S[S[PR 3,R,;R&S[S[PR 3,R,;R&#)rNrrrr r!r"r#r$.r%r&r'r(r)r*r+r,r-)r2r3s"r4r5RetrievAugLMOutput.__annotate__sN    $+OP!!D(/QRT\(ST ++d29UV''$.5WX''$.5YZ",,t3:[\).(9(9D(@G]^!&e&7&7&< = DK_`#5#4#4c#9:TAHab&+%6%6%=Dcd"'u'8'8#'=!>!ELef$E$5$5s$:;dBIgh"'u'8'8#'=!>!ELij$E$5$5s$:;dBIkl!&e&7&7&< = DKmr7r8)r9r:r;r<r=rrrr r!r"r#r$r%r&r'r(r)r*r+r,r>r?r@rAs@r4rCrCssBH(,F+/J$(O5915156:CG&GKDH@D#HLEIHLEIGKmr7rCa RAG models were released with the paper [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://huggingface.co/papers/2005.11401) by Patrick Lewis, Ethan Perez, Aleksandra Piktus et al. RAG is a retriever augmented model and encapsulate three components: a question encoder, a dataset retriever and a generator, the encoder and generator are trainable while the retriever is just an indexed dataset. cXa]tRt^toRtRtRt]RV3RlRll4tV3Rlt Rt Vt R#) RagPreTrainedModelragTNc\<V^8dQhRS[R,RS[R,RS[R,RS[/#)r.question_encoder_pretrained_model_name_or_pathN'generator_pretrained_model_name_or_path retrieverreturn)strrr)r2r3s"r4r5RagPreTrainedModel.__annotate__sLBoBo8;d Bo25tBo $& Bo  Bor7c VP4UUu/uF-wrVVPR4'gKV\R4RVbK/ pppVP4UUu/uF-wrVVPR4'gKV\R4RVbK/ pppVF p VRV ,K VF p VRV ,K VPRR4p V fOVfQR4h^RIHp RV9d$^RIHp V P!V3/VBR R /BwrWR&V P!V3/VBp VPRR4pVfQVfQR 4h^R IH pRV9d&^RIHp V P!V3/VBR R /BwppVVR&VP!V3/VBpVPR4pVf-\P!V PVP3/VBpV!WVVR 7#uuppiuuppi)a Instantiates an question encoder and a generator from one or two base classes of the library from pretrained model checkpoints. The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated). To train the model, you need to first set it back in training mode with `model.train()`. Params: question_encoder_pretrained_model_name_or_path (`str`, *optional*, defaults to `None`): Information necessary to initiate the question encoder. Can be either: - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co. - A path to a *directory* containing model weights saved using [`~PreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`. generator_pretrained_model_name_or_path (`str`, *optional*, defaults to `None`): Information necessary to initiate the generator. Can be either: - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co. - A path to a *directory* containing model weights saved using [`~PreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`. model_args (remaining positional arguments, *optional*): All remaining positional arguments will be passed to the underlying model's `__init__` method. retriever ([`RagRetriever`], *optional*): The retriever to use. kwwargs (remaining dictionary of keyword arguments, *optional*): Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., `output_attentions=True`). - To update the question_encoder configuration, use the prefix *question_encoder_* for each configuration parameter. - To update the generator configuration, use the prefix *generator_* for each configuration parameter. - To update the parent model configuration, do not use a prefix for each configuration parameter. Behaves differently depending on whether a `config` is provided or automatically loaded. Example: ```python >>> from transformers import RagModel >>> # initialize a RAG from two pretrained models. >>> model = RagModel.from_pretrained_question_encoder_generator( ... "facebook/dpr-question_encoder-single-nq-base", "google-t5/t5-small" ... ) >>> # saving model after fine-tuning >>> model.save_pretrained("./rag") >>> # load fine-tuned model >>> model = RagModel.from_pretrained("./rag") ```question_encoder_N generator_modelznIf `model` is not defined as an argument, a `question_encoder_pretrained_model_name_or_path` has to be defined AutoModelconfig) AutoConfigreturn_unused_kwargsTzqIf `generator_model` is not defined as an argument, a `generator_pretrained_model_name_or_path` has to be definedAutoModelForSeq2SeqLM)question_encoder generatorrVrL)items startswithlenpopauto.modeling_autorUauto.configuration_autorWfrom_pretrainedrZgetr'from_question_encoder_generator_configsrV)clsrJrKrLkwargsargumentvaluekwargs_question_encoderkwargs_generatorkeyr[rUrWquestion_encoder_configr\rZgenerator_configrVs&&&&, r4*from_pretrained_question_encoder_generator=RagPreTrainedModel.from_pretrained_question_encoder_generatorsX|$*<<># #1""#67 8HS,-/ 0% 7#1 # $*<<> #1""<0 1HS&( )5 0#1  +C*S01+#C|c)*$ 366wE  #AM  M 766@CMC]C]BD-D*.D@' 5L1(88> BY  %(($7  :F ! F C//@5?5O5O;6?O6fj62 "2.> *-==7;KI H% >FF '')9)9=CF$4RXdmnnO#  sGGG=Gc&<V^8dQh/S[;R&#)rrVr)r2r3s"r4r5rOs r7r8)NNN) r9r:r;r<base_model_prefix_supports_flash_attn_supports_sdpa classmethodror>r?r@rAs@r4rGrGs<NBoBoBo#r7rGcbaa]tRtRtoRV3RlV3Rlllt]RV3RlRll4tRtVtV;t #) RagModelisc j<V^8dQhRS[R,RS[R,RS[R,RS[R,/#rrVNr[r\rLrrr)r2r3s"r4r5RagModel.__annotate__usG22 4'2*D02#T) 2  $& 2r7c <VfVeVfQR4hVf.\P!VPVP3/VBpM2\WP4'gQRV RVP 24h\ SV`V4Vf"^RIHpVPVP4pVf"^RIH pVPVP4pW@n VPe;\V\4'gQR\VP4 R24hW@n W n W0n RVnR VnVP%4R#)  question_encoder (`PreTrainedModel`, *optional*): The model responsible for encoding the question into hidden states for retrieval. generator (`PreTrainedModel`, *optional*): The model responsible for generating text based on retrieved documents. retriever (`RagRetriever`, *optional*): The component responsible for retrieving documents from a knowledge base given the encoded question. NzQEither a configuration or an question_encoder and a generator has to be provided.zconfig: z has to be of type rTrYz`self.retriever` is of type z&, but should be of type `RagRetriever`F)rrerV isinstance config_classsuper__init__rarU from_configr[rZr\rLrtype ctx_encodercontext_encoder_training post_init) selfrVr[r\rLrgrUrZ __class__s &&&&&, r4rRagModel.__init__usB !&6&ByG\ _ ] >FF '')9)9=CFf&7&788 sHVHL_`d`q`q_r:s s8   # 6(44V5L5LM    B-99&:J:JKI" >> %i66 .tDNN/C.DDjk 6'N 0"(-% r7c,<V^8dQhRS[PR,RS[PR,RS[S[S[P,,R,RS[PR,RS[P R,RS[R,RS[PR,R S[PR,R S[PR,R S[R,R S[R,R S[R,RS[R,RS[R,RS[S[P,S[ ,/#)r input_idsNattention_maskencoder_outputsdecoder_input_idsdecoder_attention_maskrrr"r# use_cacheoutput_attentionsoutput_hidden_statesoutput_retrievedn_docsrM) r.r0Tensorr1r/ BoolTensorrboolintrC)r2r3s"r4r5r{s9e e ##d*e  t+e uU%6%6784? e !++d2 e !& 0 04 7 e e %%,e !++d2e !& 0 04 7e $;e  $;e #Tke +e d e " u|| 1 1#e r7c 4 VeTMVPPpV eT MVPPp V eT MVPPp V eT MVPPp V eT MVPP p VP RJ;'d)VRJ;'gV RJ;'g VRJ;'dVRJpVEfnV'EdDVPWRR7pV^,pVP VVP4PR\PR7P4\VPPRR4VRR7pVP'EdVR ,VR ,VR ,VR ,VR ,VR,3wpp ppppVPV4pV PV4p VPV4pVPV4pVP!VVRR7P"pVP%R%VVP&^,4p\P(!VP+^4VP-^^44P/^4pMVR ,VR ,VR ,VR,3wrppVPV4pVPV4pV PV4p \P(!VP+^4VP-^^44P/^4pM!VfQR4hV fQR4hVfQR4hVfQR4hVP&^,V,^8Xg QRV RVP&^, R24hVeVP1V^R7pVeVP1V^R7pVPVV VVVVV V RR7 pV'g RpRpRpRpRpMXP2pVP4pV'd V 'g R&pRp RpRp\7R'/RVP8bRVbRVP:bR VbR V bR XbRXbRXbRVbRVbRVP<bR VP>bR!VP@bR"VPBbR#VPDbR$VPFb#)(a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [`RagConfig`], used to initialize the model, specifies which generator to use, it also specifies a compatible generator tokenizer. Use that tokenizer class to obtain the indices. [What are input IDs?](../glossary#input-ids) encoder_outputs (`tuple(tuple(torch.FloatTensor)`, *optional*) Tuple consists of (`generator_enc_last_hidden_state`, *optional*: `generator_enc_hidden_states`, *optional*: `generator_enc_attentions`). `generator_enc_last_hidden_state` of shape `(batch_size, n_docs * sequence_length, hidden_size)` is a sequence of hidden-states at the output of the last layer of the generator's encoder. Used by the ([`RagModel`]) model during decoding. decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Provide for generation tasks. `None` by default, construct as per instructions for the generator model you're using with your RAG instance. decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. doc_scores (`torch.FloatTensor` of shape `(batch_size, config.n_docs)`): Score between each retrieved document embeddings (see `retrieved_doc_embeds`) and `question_encoder_last_hidden_state`. If the model has is not initialized with a `retriever` `doc_scores` has to be provided to the forward pass. `doc_scores` can be computed via `question_encoder_last_hidden_state` and `retrieved_doc_embeds`, see examples for more information. context_input_ids (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Input IDs post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model was not initialized with a `retriever` ``context_input_ids` has to be provided to the forward pass. `context_input_ids` are returned by [`~RagRetriever.__call__`]. context_attention_mask (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`,*optional*, returned when *output_retrieved=True*): Attention mask post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model has is not initialized with a `retriever` `context_attention_mask` has to be provided to the forward pass. `context_attention_mask` are returned by [`~RagRetriever.__call__`]. output_retrieved (`bool`, *optional*): Whether or not to return the `retrieved_doc_embeds`, `retrieved_doc_ids`, `context_input_ids` and `context_attention_mask`. See returned tensors for more detail. n_docs (`int`, *optional*): The number of documents to retrieve. Example: ```python >>> from transformers import AutoTokenizer, RagRetriever, RagModel >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("facebook/rag-token-base") >>> retriever = RagRetriever.from_pretrained( ... "facebook/rag-token-base", index_name="exact", use_dummy_dataset=True ... ) >>> # initialize with RagRetriever to do everything in one forward call >>> model = RagModel.from_pretrained("facebook/rag-token-base", retriever=retriever) >>> inputs = tokenizer("How many people live in Paris?", return_tensors="pt") >>> outputs = model(input_ids=inputs["input_ids"]) ```NT)r return_dictcpudevicedtypeprefixptrrreturn_tensorsr"r#r tokenized_doc_idstokenized_doc_attention_maskdoc_idszMake sure that `context_input_ids` are passed, if no `retriever` is set. Alternatively, you can set a retriever using the `set_retriever(...)` function.zMake sure that `context_attention_mask` are passed, if no `retriever` is set. Alternatively, you can set a retriever using the `set_retriever(...)` function.zMake sure that `doc_scores` are passed, if no `retriever` is set. Alternatively, you can set a retriever using the `set_retriever(...)` function.z^Make sure that `doc_scores` are passed when passing `encoder_outputs` to the forward function.M The first dimension of `context_input_ids` should be a multiple of `n_docs`= , but is .dim) rrrrrrrrrrrrr!r$r%r&r'r(r)r*r+r,Nr8)$rVrrrrrrLr[detachtor.float32numpygetattrr\rr pooler_outputviewshapebmm unsqueeze transposesqueezerepeat_interleave hidden_states attentionsrCrrencoder_last_hidden_stateencoder_hidden_statesencoder_attentionsdecoder_hidden_statesdecoder_attentionscross_attentions)rrrrrrrrr"r#rrrrrrghas_to_retrievequestion_enc_outputsr$retriever_outputsr retrieved_doc_input_idsretrieved_doc_attention_maskr! gen_outputsr%r&s&&&&&&&&&&&&&&&, r4forwardRagModel.forwardsT"-4;;3E3E!*!6IDKK>#8#8(DI!#' %3%!000**=>)*BC)*@A)*=>)*HI))4 ).,/4)):(<()*BC)*@A))4 jf%?SUf,@+B+BCe+f((9(<(FF '')9)9=CF  6Xaw r7c <V^8dQhRS[/#rrLr)r2r3s"r4r5r''|'r7c&WPnR#rrHrLrrLs&&r4 set_retriever&RagSequenceForGeneration.set_retriever &r7c <V^8dQhRS[/#rrr)r2r3s"r4r5r++O+r7cHRVPnWPnR#TNrHrrrrs&&r4 set_context_encoder_for_training9RagSequenceForGeneration.set_context_encoder_for_training,0)*r7c$H<V^8dQhRS[PR,RS[PR,RS[S[S[P,,R,RS[PR,RS[PR,RS[R,RS[PR,R S[PR,R S[P R,R S[R,R S[R,R S[R,RS[R,RS[R,RS[R,RS[PR,RS[R,RS[ /#)rrNrrrrrr"r#rrrrrexclude_bos_score reduce_losslabelsrrM) r.r0rr1rrr/rrr)r2r3s"r4r5rs\^ ^ ##d*^  t+^ uU\\23d: ^ !++d2 ^ !& 0 04 7 ^ ^ !++d2^ !& 0 04 7^ %%,^ $;^  $;^ #Tk^ +^  $;^ D[!^ "  4'#^ $d %^ ( ")^ r7c DVeTMVPPpVeTMVPPpVeTMVPPpVe VfTpRp VP VVVVVVVV VV V V V VR7pRpVeAVP VP VPVVVPPVVR7p\R/RVbRVP bRVPbRVPbR VPbR VPbR VPbR VPbR VPbRVP bRVP"bRVP$bRVP&bRVP(bRVP*bRVP,bRVP.b#)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [`RagConfig`], used to initialize the model, specifies which generator to use, it also specifies a compatible generator tokenizer. Use that tokenizer class to obtain the indices. [What are input IDs?](../glossary#input-ids) encoder_outputs (`tuple(tuple(torch.FloatTensor)`, *optional*) Tuple consists of (`generator_enc_last_hidden_state`, *optional*: `generator_enc_hidden_states`, *optional*: `generator_enc_attentions`). `generator_enc_last_hidden_state` of shape `(batch_size, n_docs * sequence_length, hidden_size)` is a sequence of hidden-states at the output of the last layer of the generator's encoder. Used by the ([`RagModel`]) model during decoding. decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Provide for generation tasks. `None` by default, construct as per instructions for the generator model you're using with your RAG instance. decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. context_input_ids (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Input IDs post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model was not initialized with a `retriever` ``context_input_ids` has to be provided to the forward pass. `context_input_ids` are returned by [`~RagRetriever.__call__`]. context_attention_mask (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`,*optional*, returned when *output_retrieved=True*): Attention mask post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model has is not initialized with a `retriever` `context_attention_mask` has to be provided to the forward pass. `context_attention_mask` are returned by [`~RagRetriever.__call__`]. doc_scores (`torch.FloatTensor` of shape `(batch_size, config.n_docs)`): Score between each retrieved document embeddings (see `retrieved_doc_embeds`) and `question_encoder_last_hidden_state`. If the model has is not initialized with a `retriever` `doc_scores` has to be provided to the forward pass. `doc_scores` can be computed via `question_encoder_last_hidden_state` and `retrieved_doc_embeds`, see examples for more information. output_retrieved (`bool`, *optional*): Whether or not to return the `retrieved_doc_embeds`, `retrieved_doc_ids`, `context_input_ids` and `context_attention_mask`. See returned tensors for more detail. exclude_bos_score (`bool`, *optional*): Only relevant if `labels` is passed. If `True`, the score of the BOS token is disregarded when computing the loss. reduce_loss (`bool`, *optional*): Only relevant if `labels` is passed. If `True`, the NLL loss is reduced using the `torch.Tensor.sum` operation. n_docs (`int`, *optional*): The number of documents to retrieve. Example: ```python >>> from transformers import AutoTokenizer, RagRetriever, RagSequenceForGeneration >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("facebook/rag-sequence-nq") >>> retriever = RagRetriever.from_pretrained( ... "facebook/rag-sequence-nq", index_name="exact", use_dummy_dataset=True ... ) >>> # initialize with RagRetriever to do everything in one forward call >>> model = RagSequenceForGeneration.from_pretrained("facebook/rag-token-nq", retriever=retriever) >>> inputs = tokenizer("How many people live in Paris?", return_tensors="pt") >>> targets = tokenizer(text_target="In Paris, there are 10 million people.", return_tensors="pt") >>> input_ids = inputs["input_ids"] >>> labels = targets["input_ids"] >>> outputs = model(input_ids=input_ids, labels=labels) >>> # or use retriever separately >>> model = RagSequenceForGeneration.from_pretrained("facebook/rag-sequence-nq", use_dummy_dataset=True) >>> # 1. Encode >>> question_hidden_states = model.question_encoder(input_ids)[0] >>> # 2. Retrieve >>> docs_dict = retriever(input_ids.numpy(), question_hidden_states.detach().numpy(), return_tensors="pt") >>> doc_scores = torch.bmm( ... question_hidden_states.unsqueeze(1), docs_dict["retrieved_doc_embeds"].float().transpose(1, 2) ... ).squeeze(1) >>> # 3. Forward to generator >>> outputs = model( ... context_input_ids=docs_dict["context_input_ids"], ... context_attention_mask=docs_dict["context_attention_mask"], ... doc_scores=doc_scores, ... decoder_input_ids=labels, ... ) ```NFrrrrrr"r#rrrrrrr)repsilonrrrrrrr"r#r r!r$r%r&r'r(r)r*r+r,r8)rVrrrrHget_nllrrlabel_smoothingrrr"r#r r!r$r%r&r'r(r)r*r+r,)rrrrrrrr"r#rrrrrrrrrrgoutputsrs&&&&&&&&&&&&&&&&&&, r4r RagSequenceForGeneration.forwardsN"-4;;3E3E1B1N-TXT_T_TqTq%0%> )) $33  &77  $+#A#A  ")!=!= &77 07/Y/Y (/'I'I %,$C$C -4,S,S )0(K(K &-%E%E )0(K(K &-%E%E! "(/'I'I#  r7c.VPP#rrrs&r4rL"RagSequenceForGeneration.retriever_xx!!!r7c.VPP#rrHr\rs&r4r\"RagSequenceForGeneration.generatorcrr7c.VPP#rrHr[rs&r4r[)RagSequenceForGeneration.question_encodergxx(((r7cL<V^8dQhRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[R,RS[R,R S[R,R S[R,R S[P/ #) rrNrr"r#rdo_deduplicationnum_return_sequences num_beamsrrM)r.r0r/rr)r2r3s"r4r5rlsVYVY##d*VY((4/VY!++d2 VY !& 0 04 7 VY %%, VY+VY"DjVY:VYd VY   VYr7c V eT MVPPp VeTMVPPpVeTM\VPR^4p VeTM\VPR^4pVf VfQR4hVPeVfVP WR7^,p VP VV P 4PR\PR7P4\VPPRR4V R R 7R ,pVPV4p.p WR&WR&RV R &VeVP^,MVP^,V ,p\V4EFpW?V ,V^,V ,pVPP!V3/V BpV'dV\P!\!VUu/uFp\#VP%44VbK upP'444pVP^,pVe(WV^,P)V^4pV!VVR R7pMVfQR4hVfQR4hVP)V^4pWOV ,V^,V ,pVP)V^4pW_V^,1R3,pVP)V^4pV!VVVVR R7pVR,)P+V 4^,pV P-VV,4EK VP/WPPP0R7#uupi)a Implements RAG sequence "thorough" decoding. Read the [`~generation.GenerationMixin.generate`]` documentation for more information on how to set other generate input parameters. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): The sequence used as a prompt for the generation. If `input_ids` is not passed, then `context_input_ids` has to be provided. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) context_input_ids (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Input IDs post-processed from the retrieved documents and the question encoder input_ids by the retriever. context_attention_mask (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Attention mask post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model is not initialized with a `retriever` or `input_ids` is not given, `context_input_ids` and `context_attention_mask` have to be provided to the forward pass. They are returned by [`~RagRetriever.__call__`]. doc_scores (`torch.FloatTensor` of shape `(batch_size, config.n_docs)`): Score between each retrieved document embeddings (see `retrieved_doc_embeds`) and `question_encoder_last_hidden_state`. If the model is not initialized with a `retriever` or `input_ids` is not given, `doc_scores` has to be provided to the forward pass. `doc_scores` are returned by [`~RagRetriever.__call__`]. do_deduplication (`bool`, *optional*): Whether or not to deduplicate the generations from different context documents for a given input. Has to be set to `False` if used while training with distributed backend. num_return_sequences(`int`, *optional*, defaults to 1): The number of independently computed returned sequences for each element in the batch. Note that this is not the value we pass to the `generator`'s `[`~generation.GenerationMixin.generate`]` function, where we set `num_return_sequences` to `num_beams`. num_beams (`int`, *optional*, defaults to 1): Number of beams for beam search. 1 means no beam search. n_docs (`int`, *optional*, defaults to `config.n_docs`) Number of documents to retrieve and/or number of documents for which to generate an answer. kwargs (`dict[str, Any]`, *optional*): Additional kwargs will be passed to [`~generation.GenerationMixin.generate`]. Return: `torch.LongTensor` of shape `(batch_size * num_return_sequences, sequence_length)`: The generated sequences. The second dimension (sequence length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. Nrrz= At least one of input_ids or context_input_ids must be givenrrrrrrr"rT)rrzMake sure that `context_attention_mask` are passed, if no `input_ids` is set. Alternatively, you can set a retriever using the `set_retriever(...)` function.zMake sure that `doc_scores` are passed, if no `input_ids` is set. Alternatively, you can set a retriever using the `set_retriever(...)` function.NNN)r"r#rrrr) pad_token_id)rVrrrrLr[rrr.rrr\rrangegeneratestacklistrNtolistvaluesrepeattopkappend _cat_and_padr)rrrr"r#rrrrr model_kwargsnum_doc_return_sequencesquestion_hidden_stateshypos batch_sizeindexgenerator_input_idsoutput_sequencesknum_candidates new_input_idsrindividual_input_idsindividual_attention_maskindividual_doc_scores top_cand_indss&&&&&&&&&&, r4r!RagSequenceForGeneration.generatekswB"-4;;3E3E/?/K+QUQ\Q\QmQm$/ !&> %*;*C%)%:%:9%:%def%g " $&--/22%u}}2U[[]t~~44hE# !/! " !# !2 4 4Y ? $-[!/8+,)- %&+4+@Y__Q'FWF]F]^_F`djFj :&E"3FNeaiSYEY"Z #~~66#    #(;;tQa4bQaAS_a5GQa4b4i4i4k/l#m -33N $ )%!) < C CNTU V }5EY]^-9T9"-J- (;'A'A"A($-C6>UZ]^U^bhTh,i),E,L,L^]^,_)(2EAI3F3I(J%(=(D(D^UV(W%&:+D4+&* &fo-334LMaPM LL)-8 9g'j  [[5J5J5W5W XXW5cs#Mchaa\P!SR,SPSP^,^4P SP P P4.^4oVeTMSP PpSP P;'g!SP P PpVRJ;'d'SR ,PV4P4p VV3Rlp \PPVR R7PVP^,V,VR VP!R 44p \PPV^R7P#R 4P#R 4p V R ,p V R ,pV R ,p\P!WV ,V.^R7pSP#^4P#R 4P%^V^^4oSP'4VP'48XgQhVP)R SR7pVP+R RR7pV !VV4wppV'd!V 'dVR,P+^4MVP+^4pVP+^4pVP-^4pVP-^4pV)pV)pV'd!VP+4pVP+4pVVP!R 4, pRV, V,VV,,pV#)rNc<SPSPPP4pVP 4'd%VP VR4VP VR4VP R4VP R43#reqrVr\rany masked_fill_rll smooth_objpad_maskrtargets&& r4 _mask_pads4RagSequenceForGeneration.get_nll.._mask_padshyy!6!6!C!CDH||~~#.''#6::b>:#5#5b#99 9r7rrrTrkeepdim?rNNrr)rr:Nr7Nr)rr:r7rNr)rrrNNr)rrr6)r.catnewrfill_rVr\rr bos_token_idr&allr functional log_softmaxrsizerr rgathersum logsumexp)r seq_logitsrr-rrrrr>use_bosr. seq_logprobs doc_logprobsfirst_token_scoressecond_token_scores remainder rag_logprobsr*r+nll_loss smooth_losseps_irsf&&f&&&& r4r RagSequenceForGeneration.get_nlls E]FJJv||A:@@AVAVAcAcd egh "-4;;3E3E{{//UU4;;3H3H3U3U d*RRvd||/L/P/P/R :}}000DII   Q 6 )62zr7J }}000CMMbQ[[\^_ *+6*<8 - yy"4L6XZc!djkl !!!$..r299!VQJzz||//1111  Rv 6!%%"d%; #B 3J%6'R\  a rvvay^^A& \\!_))!, 3!k ||~H%//+K,++B//g )EK,?? r7cBV^,P\RV44\RV444PV4p^pVFNpWBW3VP^,,1RVP^,13&W4P^,, pKP V#)r9c3F"TFqP^,xK R#5i)r9Nr.0ts& r4 8RagSequenceForGeneration._cat_and_pad..As#@1GGAJJ!c3F"TFqP^,xK R#5i)r7NrTrUs& r4rXrYAsEbZaUVggajjZarZN)r<rDmaxr=r)tensorsroutputindrWs&& r4r%RagSequenceForGeneration._cat_and_pad?s#@#@ @#EbZaEbBbciijvwA;<3qwwqz))''++^ ^ ^ @"""")) ]]_VYVYVYp9vr7rzo A RAG-token model implementation. It performs RAG-token specific marginalization in the forward pass. c laa]tRtRtoRV3RlV3RllltV3RlRltV3RlRltRR lt]R 4t ]R 4t ]R 4t ] R 4t RRlt]RV3RlRll4t]P$!4RRRRRRRR]!4]!43 V3RlRll4tRtRtRtRtRRltRRltRtVtV;t#)RagTokenForGenerationiINc j<V^8dQhRS[R,RS[R,RS[R,RS[R,/#ryrz)r2r3s"r4r5"RagTokenForGeneration.__annotate__OsG 4'*D0#T)   $& r7c <VfVeVfQR4hVf-\P!VPVP3/VBp\SV`V4\ WW4R7VnVP4R#rrrs&&&&&,r4rRagTokenForGeneration.__init__Os !&6&ByG\ V ] >FF '')9)9=CF  6Xaw r7c <V^8dQhRS[/#rr)r2r3s"r4r5riorr7c&WPnR#rrrs&&r4r#RagTokenForGeneration.set_retrieverorr7c <V^8dQhRS[/#rr)r2r3s"r4r5rirrr7cHRVPnWPnR#rrrs&&r4r6RagTokenForGeneration.set_context_encoder_for_trainingrrr7c LVeVRR R13,pRRRVRVRVRVRVRVR R R V/ #) Nrrrrr#rrrdo_marginalizeTrrr8) rrrrrrrrrgs &&&&&&&&,r4prepare_inputs_for_generation3RagTokenForGeneration.prepare_inputs_for_generationvsU  & 1!RS& 9    * $n !2   d f  r7c.VPP#rrrs&r4rLRagTokenForGeneration.retrieverrr7c.VPP#rrrs&r4r\RagTokenForGeneration.generatorrr7c.VPP#rrrs&r4r[&RagTokenForGeneration.question_encoderrr7c aa Ro Rp\\V44EFp\V\4'dV V3RlVPP V,P VPP V,PVPP V,P VPP V,P34wrErgWEWg3pMJV V3RlVP V,P VP V,P34wrEWE3pW(3, pEK \V4!V4#)zeReorders cache for generation. BART-inspired but we need to take care of the extra dimension for docscVP^,VP^,,pVP!RV.VPR,O5!pVP^V4pVP!R.VPR,O5!pV#)r9r6r:r)rr index_select)r new_orderrresults&& r4_reorder_stacked>RagTokenForGeneration._reorder_cache.._reorder_stackedsz"((+yq/AAF)..r6TM.s4\%Q AHH(=>>/2c3h<"TF'pS!VSPVP44xK) R#5irrrs& r4rXrs/6c%Q AHH(=>>crr8) rr_r~rself_attention_cachelayerskeysr cross_attention_cacher) rrreordered_pastidxself_attention_kself_attention_vcross_attention_kcross_attention_v new_tuplers &f @r4_reorder_cache$RagTokenForGeneration._reorder_caches# _-.C/+>??\(<<CCCHMM'<<CCCHOO'==DDSINN'==DDSIPP \X 4E.ARf 6-44S9>>@V@VWZ@[@b@bc62 .@ l *N%/&O$^44r7cVeTMVPPp\PP VRR7P VP ^,V,VRVPR44p\P!V^R7pWEPR4PR4,p\P!V^R7#)Nrr) rVrrr@rArrrBr.rrE)rrFrrrHrI log_prob_sums&&&& r4 marginalize!RagTokenForGeneration.marginalizes!-4;;3E3E}}000DII   Q 6 )62zr7J ((; #&<&>> from transformers import AutoTokenizer, RagRetriever, RagTokenForGeneration >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("facebook/rag-token-nq") >>> retriever = RagRetriever.from_pretrained( ... "facebook/rag-token-nq", index_name="exact", use_dummy_dataset=True ... ) >>> # initialize with RagRetriever to do everything in one forward call >>> model = RagTokenForGeneration.from_pretrained("facebook/rag-token-nq", retriever=retriever) >>> inputs = tokenizer("How many people live in Paris?", return_tensors="pt") >>> targets = tokenizer(text_target="In Paris, there are 10 million people.", return_tensors="pt") >>> input_ids = inputs["input_ids"] >>> labels = targets["input_ids"] >>> outputs = model(input_ids=input_ids, labels=labels) >>> # or use retriever separately >>> model = RagTokenForGeneration.from_pretrained("facebook/rag-token-nq", use_dummy_dataset=True) >>> # 1. Encode >>> question_hidden_states = model.question_encoder(input_ids)[0] >>> # 2. Retrieve >>> docs_dict = retriever(input_ids.numpy(), question_hidden_states.detach().numpy(), return_tensors="pt") >>> doc_scores = torch.bmm( ... question_hidden_states.unsqueeze(1), docs_dict["retrieved_doc_embeds"].float().transpose(1, 2) ... ).squeeze(1) >>> # 3. Forward to generator >>> outputs = model( ... context_input_ids=docs_dict["context_input_ids"], ... context_attention_mask=docs_dict["context_attention_mask"], ... doc_scores=doc_scores, ... decoder_input_ids=labels, ... ) >>> # or directly generate >>> generated = model.generate( ... context_input_ids=docs_dict["context_input_ids"], ... context_attention_mask=docs_dict["context_attention_mask"], ... doc_scores=doc_scores, ... ) >>> generated_string = tokenizer.batch_decode(generated, skip_special_tokens=True) ```NFr)rrrrrrrr"r#r r!r$r%r&r'r(r)r*r+r,r8)rVrrsrrHrrrrrrrr"r#r r!r$r%r&r'r(r)r*r+r,)rrrrrrrr"r#rrrrrrsrrrrgrrrs&&&&&&&&&&&&&&&&&&, r4rRagTokenForGeneration.forwards"^"-4;;3E3E+9+E4;;KeKe%0%S +I"JT"QI .4I 0$6I    I r7c aaVPRV RRR4p VP!V3/V Bwr}VP4pV\P\P \P \P39d\RV R24h\\V4\V,4pVPV P44VPWV 4V PRR4RJpVP!VV4SeSMVP"P$oVP&Ee!VEfVP)WR7^,pVP'VVP+4P-R\.P0R7P34\VP4P"R R4SR R 7pVR ,VR ,VR,prCVP-V4pVP-V4pVP-V4p\.P6!VP9^4VP;^^44P=^4pVP>^,S,^8Xg QRS RVP>^, R24hVP>^,S,oVP@P4PC4pV!W4RR7p\.PD!SVPF,^3VPH\.PJ\MVPO44PPR7pVP>R#,pVR,pR$VV3RllpV!WGPFR7pV!VVPFR7VR&VPSVPF^R7pW]R&VV R&WMR&SV R&VPTV R&VPWVVVVV VPPR7pVPYWzR7pVP[VV RVP>^,VP\^, R7V!VV3R VR!VR"V/V BV B#)%a Implements RAG token decoding. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): The sequence used as a prompt for the generation. If `input_ids` is not passed, then `context_input_ids` has to be provided. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) context_input_ids (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Input IDs post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model is not initialized with a `retriever`, `context_input_ids` has to be provided to the forward pass. `context_input_ids` are returned by [`~RagRetriever.__call__`]. context_attention_mask (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Attention mask post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model is not initialized with a `retriever`, `context_input_ids` has to be provided to the forward pass. `context_input_ids` are returned by [`~RagRetriever.__call__`]. doc_scores (`torch.FloatTensor` of shape `(batch_size, config.n_docs)`): Score between each retrieved document embeddings (see `retrieved_doc_embeds`) and `question_encoder_last_hidden_state`. If the model is not initialized with a `retriever`, `context_input_ids` has to be provided to the forward pass. `context_input_ids` are returned by [`~RagRetriever.__call__`]. n_docs (`int`, *optional*, defaults to `config.n_docs`) Number of documents to retrieve and/or number of documents for which to generate an answer. generation_config (`~generation.GenerationConfig`, *optional*): The generation configuration to be used as base parametrization for the generation call. `**kwargs` passed to generate matching the attributes of `generation_config` will override them. If `generation_config` is not provided, the default will be used, which has the following loading priority: 1) from the `generation_config.json` model file, if it exists; 2) from the model configuration. Please note that unspecified parameters will inherit [`~generation.GenerationConfig`]'s default values, whose documentation should be checked to parameterize generation. prefix_allowed_tokens_fn (`Callable[[int, torch.Tensor], list[int]]`, *optional*): If provided, this function constraints the beam search to allowed tokens only at each step. If not provided no constraint is applied. This function takes 2 arguments `inputs_ids` and the batch ID `batch_id`. It has to return a list with the allowed tokens for the next generation step conditioned on the previously generated tokens `inputs_ids` and the batch ID `batch_id`. This argument is useful for constrained generation conditioned on the prefix, as described in [Autoregressive Entity Retrieval](https://huggingface.co/papers/2010.00904). logits_processor (`LogitsProcessorList`, *optional*): Custom logits processors that complement the default logits processors built from arguments and a model's config. If a logit processor is passed that is already created with the arguments or a model's config an error is thrown. stopping_criteria (`StoppingCriteriaList`, *optional*): Custom stopping criteria that complement the default stopping criteria built from arguments and a model's config. If a stopping criteria is passed that is already created with the arguments or a model's config an error is thrown. kwargs (`dict[str, Any]`, *optional*): Ad hoc parametrization of `generate_config` and/or additional model-specific kwargs that will be forwarded to the `forward` function of the model. Return: `torch.LongTensor` of shape `(batch_size * num_return_sequences, sequence_length)`: The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. NFz!RAG model is not compatible with z5 generation. Please check your generation parameters.rrrrrrrr"r#r rrrT)rrr)rrlast_hidden_statec2<VR,PS^S3VPR,,4pVPSVS3VPR,,4pVPSV,S,3VPR,,4#)Nr6:NN)NNr)reshaperexpand)tensorrrrs&&r4extend_enc_output9RagTokenForGeneration.generate..extend_enc_outputs}M*22J63JV\\Z\M]3]^F]]J 6#BV\\RTEU#UVF>>: #9F#B"Dv||TVGW"WX Xr7)rrrrrr)rinput_ids_seq_lengthencoder_input_idsrrr)rr)generation_modermax_cache_lengthrrrrr)/_extract_generation_mode_kwargs_prepare_generation_configget_generation_moder SAMPLE GREEDY_SEARCH BEAM_SEARCH BEAM_SAMPLE ValueErrorrrr _validate_model_kwargscopy_validate_generation_moderd_prepare_special_tokensrVrrLr[rrr.rrr\rrrrrrH get_encoderfullrdecoder_start_token_idlongnext parametersrrr_get_logits_processor_get_stopping_criteria_prepare_cache_for_generation max_length)rrrr"r#rrrrrrrggeneration_mode_kwargsrrdecoding_methodkwargs_has_attention_maskroutr encoderrrrrprepared_logits_processorprepared_stopping_criteriars&&&&&&f&&&&, @r4rRagTokenForGeneration.generatexsb"&!E!EdFTY[_ae!f*.*I*IJ[*f_e*f'+??A   ! !  ( (  & &  & & #  3O3DDyz "$t*.F.WX ##L$5$5$78 &&Kab$0$4$45Et$LTX$X! $$%68QR"-4;;3E3E >> %*;*C%)%:%:9%:%def%g "..&--/22%u}}2U[[]t~~44hE# !C'(,-*+8L5$8#:#:;Q#R 1 4 4Y ? %;%>%>y%I "#9#C#CA#FH\HfHfghjkHlmuuJ"''*V39 [\b[cd!''*+1 . 9 ',,Q/69 (($$002!,=rvwJJ +55 5q 9  4 4**)*11  )r2+,?@ Y Y"33IUpUp!q/@ ):)D)D0 +, 112C2M2MST1U &0\"*9 &')?%&!' X$5$?$? [!$($>$>/!5/%=-## %?% !&*%@%@/&A& " **    q).99A= +    7 9  0  %    r7c(VPW4pV#r)r)rrrs&&&r4_temporary_reorder_cache.RagTokenForGeneration._temporary_reorder_cacheEs--oHr7cJVPPP4#r)rHr\get_input_embeddingsrs&r4r*RagTokenForGeneration.get_input_embeddingsLsxx!!6688r7cJVPPP4#r)rHr\get_output_embeddingsrs&r4r+RagTokenForGeneration.get_output_embeddingsOsxx!!7799r7cLVPPPV4#r)rHr\set_output_embeddings)rnew_embeddingss&&r4r+RagTokenForGeneration.set_output_embeddingsRsxx!!77GGr7c VfVPPpVPVP4pVRRR13,P 4VR&W#R&V#)zCShift input ids one token to the right, and pad with start_token_idNrrr5r8)rVr new_zerosrclone)rrstart_token_idshifted_input_idss&&& r4shift_tokens_right(RagTokenForGeneration.shift_tokens_rightUs[  !![[??N%// @#,QV#4#:#:#<% "0$  r7c$aaVeTMSPPp\P!SR,SP SP ^,^4P SPPP4.^4oVV3RlpSPWV4pSPR4oSP4VP48XgQhVPRSR7p VPRRR7p V!W4wrV P^4p V P^4p V )p V )p V'd!V P4p V P4p WXPR4, p RV, V ,W,,pV#)Nc<SPSPPP4pVP 4'd%VP VR4VP VR4VP R4VP R43#r#r%r)s&& r4r.1RagTokenForGeneration.get_nll.._mask_padser0r7r1Tr2r4r5r)rVrr.r;r<rr=r\rrrrrCrDrB)rrFrr-rrrr.rMr*r+rNrOrPrsf&&f&&& r4rRagTokenForGeneration.get_nll^sQ!-4;;3E3E E]FJJv||A:@@AVAVAcAcd egh  :'' G !!"%zz||//1111  Rv 6!%%"d%; #B3 VVAY^^A& 3!k ||~H%//+K++B//g )E,?? r7rar)NNNNNNrrb)Fr$N)r9r:r;r<rrrrtrcrLr\r[rerrrrr.rdr r rrrrrrrr?r@rrs@@r4rgrgIs  @''++ :""""))55@ 4j j j X ]]_.22659:>/3!59TX7J7L9M9OI I I X9:H!""r7rg)rwrGrrg))r=collections.abcr dataclassesrr.r cache_utilsrrconfiguration_utilsr generationrr r r r generation.utilsr modeling_outputsrmodeling_utilsrutilsrrconfiguration_ragr retrieval_ragr get_loggerr9loggerrrCrGrwrrg__all__r8r7r4rsZ $! 53vv8+-,('   H %  WL{WL  WLt TLTL TLnIoIoIoX[ ![ [ | o1o  od  r.r  rj br7