+ jhSRt^RIt^RIt^RIHt^RIt^RIHtHt^RIH t ^RI H t ^RI H t HtHt^RIHt^RIHt^R IHt^R IHt^R IHtHtHtHt^R IHt]P>!] 4t!R;R lt"Rt#R;Rlt$Rt%]!RR7]!RR]444t&]!RR7]!RR]444t']!RR7]!RR]444t(]!RR7]!RR]444t)]!RR]44t*!RR ]PV4t,!R!R"]PZ4t.!R#R$]PZ4t/!R%R&]PZ4t0!R'R(]4t1!R)R*]4t2]!R+R7!R,R-]*44t3]!R.R7!R/R0]*44t4]!R1R2]*44t5]!R3R7!R4R5]*]44t6]!R6R7!R7R8]*]44t7!R9R:]*4t8.R10!Q& hh-u/?/?@V/WX\\^all m "'+A!B!&+A5CSCSTjCk!lq Ixx.:Huyy)?)E)E)G))STW[W_W_ X    ""L99\5??<+HKZ[O+\]aacL/%++hHbHbHdfr2ss r!cVP^4P^VPR4^4pW2PR4, p\P!V^, V3RR7P^4pVP^VPR4^4pWBPR4, p\ WVRR7p\ WVRR7pWV3#)ze This function computes both main and predict relative position buckets. For more detail, see paper. rF)rH) unsqueezerepeatsizerr.rN)rErF position_idsmain_stream_relative_positions$predicting_stream_relative_positionsmain_relative_position_buckets!predict_relative_position_bucketss&&& r #compute_all_stream_relative_bucketsrY]s &2%;%;A%>%E%EaIZIZ[]I^`a%b"%CF\F\]_F`%`",199lQ6F 5U[]+^+h+hij+k(+O+V+VWXZfZkZklnZoqr+s(+ORhRhikRl+l(&>#ATY&")A#GZ_)% * LLr!zF Base class for sequence-to-sequence language models outputs. ) custom_introc`a]tRt^ttoRtRtRtRtRtRt Rt Rt Rt Rt RtRtRtV3RltRtVtR#)ProphetNetSeq2SeqLMOutputar loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss. logits (`torch.FloatTensor` of shape `(batch_size, decoder_sequence_length, config.vocab_size)`): Prediction scores of the main stream language modeling head (scores for each vocabulary token before SoftMax). logits_ngram (`torch.FloatTensor` of shape `(batch_size, ngram * decoder_sequence_length, config.vocab_size)`): Prediction scores of the predict stream language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the attention blocks) of the decoder that can be used (see `past_key_values` input) to speed up sequential decoding. decoder_ngram_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, ngram * decoder_sequence_length, hidden_size)`. Hidden-states of the predict stream of the decoder at the output of each layer plus the initial embedding outputs. decoder_ngram_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_attn_heads, decoder_sequence_length, decoder_sequence_length)`. Attentions weights of the predict stream of the decoder, after the attention softmax, used to compute the weighted average in the self-attention heads. encoder_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder of the model. Nc<V^8dQh/S[PR,;R&S[PR,;R&S[PR,;R&S[R,;R&S[S[P,R,;R&S[S[P,R,;R&S[S[P,R,;R&S[S[P,R,;R &S[S[P,R,;R &S[PR,;R &S[S[P,R,;R &S[S[P,R,;R &#)Nlosslogits logits_ngrampast_key_valuesdecoder_hidden_statesdecoder_ngram_hidden_statesdecoder_attentionsdecoder_ngram_attentionscross_attentionsencoder_last_hidden_stateencoder_hidden_statesencoder_attentionsr FloatTensorrtuple)format __classdict__s"r __annotate__&ProphetNetSeq2SeqLMOutput.__annotate__tsYJ   d ")KL    $+MN##d*1OPT\(QR!!2!23d:AST"'u'8'8!9D!@GUVe//047>WX$E$5$56=DYZE--.5<[\ %0047>]^!!2!23d:A_`e//047>ar!r%)__name__ __module__ __qualname____firstlineno____doc__r_r`rarbrcrdrerfrgrhrirj__annotate_func____static_attributes____classdictcell__ros@r r\r\tsX<&*D'+F-1L$(O=ACG:>@D8<:>=A:>ar!r\z Base class for model encoder's outputs that also contains : pre-computed hidden states that can speed up sequential decoding. cXa]tRt^toRtRtRtRtRtRt Rt Rt Rt Rt RtV3RltRtVtR#)ProphetNetSeq2SeqModelOutputad last_hidden_state (`torch.FloatTensor` of shape `(batch_size, decoder_sequence_length, hidden_size)`): Sequence of main stream hidden-states at the output of the last layer of the decoder of the model. If `past_key_values` is used only the last hidden-state of the sequences of shape `(batch_size, 1, hidden_size)` is output. last_hidden_state_ngram (`torch.FloatTensor` of shape `(batch_size,ngram * decoder_sequence_length, config.vocab_size)`, *optional*): Sequence of predict stream hidden-states at the output of the last layer of the decoder of the model. past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the attention blocks) of the decoder that can be used (see `past_key_values` input) to speed up sequential decoding. decoder_ngram_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, ngram * decoder_sequence_length, hidden_size)`. Hidden-states of the predict stream of the decoder at the output of each layer plus the initial embedding outputs. decoder_ngram_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_attn_heads, decoder_sequence_length, decoder_sequence_length)`. Attentions weights of the predict stream of the decoder, after the attention softmax, used to compute the weighted average in the encoder_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder of the model. Ncb<V^8dQh/S[P;R&S[PR,;R&S[R,;R&S[S[P,R,;R&S[S[P,R,;R&S[S[P,R,;R&S[S[P,R,;R&S[S[P,R,;R &S[PR,;R &S[S[P,R,;R &S[S[P,R,;R &#) r^last_hidden_stateNlast_hidden_state_ngramrbrcrdrerfrgrhrirjrk)rnros"r rp)ProphetNetSeq2SeqModelOutput.__annotate__s:J(((KL#..5UV$E$5$56=DWXE--.5[\!!2!23d:A]^e//047>_r!r%)rrrsrtrurvrrbrcrdrerfrgrhrirjrwrxryrzs@r r|r|sO<9=$(O=ACG:>@D8<:>=A:>_r!r|zs Base class for model's outputs that may also contain a past key/values (to speed up sequential decoding). cLa]tRt^toRtRtRtRtRtRt Rt Rt V3Rlt Rt VtR#)ProphetNetDecoderModelOutputas last_hidden_state (`torch.FloatTensor` of shape `(batch_size, decoder_sequence_length, hidden_size)`): Sequence of main stream hidden-states at the output of the last layer of the decoder of the model. If `past_key_values` is used only the last hidden-state of the sequences of shape `(batch_size, 1, hidden_size)` is output. last_hidden_state_ngram (`torch.FloatTensor` of shape `(batch_size, ngram * decoder_sequence_length, config.vocab_size)`): Sequence of predict stream hidden-states at the output of the last layer of the decoder of the model. past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the attention blocks) of the decoder that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states_ngram (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, ngram * decoder_sequence_length, hidden_size)`. Hidden-states of the predict stream of the decoder at the output of each layer plus the initial embedding outputs. ngram_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_attn_heads, decoder_sequence_length, decoder_sequence_length)`. Attentions weights of the predict stream of the decoder, after the attention softmax, used to compute the weighted average in the Nc<V^8dQh/S[P;R&S[PR,;R&S[R,;R&S[S[P,R,;R&S[S[P,R,;R&S[S[P,R,;R&S[S[P,R,;R&S[S[P,R,;R &#) r^r~Nrrb hidden_stateshidden_states_ngram attentionsngram_attentionsrgrk)rnros"r rp)ProphetNetDecoderModelOutput.__annotate__sD(((EF#..5%>%@'"$:&!'\)\\'&&'8D'@,Oyy*a/05577s9ss7  r!c&<V^8dQh/S[;R&#r^rr)rnros"r rp&ProphetNetPreTrainedModel.__annotate__9s r!r%N) rrrsrtrubase_model_prefixsupports_gradient_checkpointingrrwrxryrzs@r rr9s%&*#! r!rc^aa]tRtRtoRtV3RlV3RlltRV3RlltV3RltRtVt V;t #) ProphetNetPositionalEmbeddingsiVz This module learns positional embeddings up to a fixed maximum size. Padding ids are ignored by either offsetting based on padding_idx or by setting padding_idx to None and ensuring that the appropriate position ids are passed to the forward function. c$<V^8dQhRS[RR/#)r^rreturnNr)rnros"r rp+ProphetNetPositionalEmbeddings.__annotate__]s bb/bDbr!c<VPVn\SV` VPVPVP 4R#N)max_position_embeddings max_lengthsuper__init__ hidden_sizerrr __class__s&&r r'ProphetNetPositionalEmbeddings.__init__]s3 88 779K9KVM`M`ar!c<VeVPeQR4hVEfVeVP4^8wdjVP4pV^,V,p\P!R\PVR7\ VPV,4,pMVf(\P!V\PVR7p\P !V^R7PV4V,P 4VP,pVP^VP^, 4p\SV`-V4V3#)NzCIf position_ids is pre-computed then padding_idx should not be set.r$r)rr) padding_idxget_seq_lengthroneslongr?cumsumtype_asclamprrforward) r inputs_shaper#attention_maskrbrTprev_num_input_ids num_input_idsrs &&&&&& r r&ProphetNetPositionalEmbeddings.forwardas$$*:*:*B Q C  */M/M/OST/T&5%C%C%E" ,Q2D D $zz& 6R((=89  ")%*ZZ EJJW]%^NLLQ7??OR``$&4++ , ,11!T__q5HI w|,l::r!c"<\SV`V4#r)rr)rrTrs&&r _forward'ProphetNetPositionalEmbeddings._forward}sw|,,r!)r)NNN) rrrsrtrurvrrrrxry __classcell__rros@@r rrVs% bb;8--r!rc\aa]tRtRtoRtRV3RlV3RllltR V3RlRlltRtVtV;t #) ProphetNetAttentioniz=Multi-headed attention from 'Attention Is All You Need' paperc:<V^8dQhRS[RS[RS[R,/#)r^rnum_attn_heads layer_idxNrr?)rnros"r rp ProphetNetAttention.__annotate__s*<</<