+ j%&Rt^RIt^RIt^RIt^RIHt^RIt^RIH t H t H t H t H t ^RIHt^RIHt^RIHt^RIHtHt]P.!]4tRR R R R R RR/t/RRbRRbRRbRRbRRbRRbRRbRRbR R!bR"R#bR$R%bR&R'bR(R)bR*R+bR,R-bR.R/bR0R1b/R2R3bR4R5bR6R7bR8R9bR:R;bRR?bR@RAbRBRCbRDREbRFRGbRHRIbRJRKbRLRMbRNRObRPRQbRRRSbC/RTRUbRVRWbRXRYbRZR[bR\R]bR^R_bR`RabRbRcbRdRebRfRgbRhRibRjRkbRlRmbRnRobRpRqbRrRsbRtRubC/RvRwbRxRybRzR{bR|R}bR~RbRRbRRbRRbRRbRRbRRbRRbRRbRRbRRbRRbRRbC/RRbRRbRRbRRbRRbRRbRRbRRbRRbRRbRRbRRbRRbRRbRRbRRbRRbCRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR/Ct/]P94UUu/uFwrWbK uppCRRRR&RR(RRRRRRRRRRBRRBRRRRRR/ CtRR.t!RR]4tRRRllt RRlt!RR/Rlt"RRlt!Rt#RRRllt RRlt$RRlt%Rt&R.t'R#uuppi)z!Tokenization classes for Whisper.N) lru_cache) AddedToken Tokenizerdecoderspre_tokenizers processors)BPE)TokenizersBackend)logging)BasicTextNormalizerEnglishTextNormalizer vocab_filez vocab.jsontokenizer_fileztokenizer.json merges_filez merges.txtnormalizer_fileznormalizer.jsonenenglishzhchinesedegermanesspanishrurussiankokoreanfrfrenchjajapanesept portuguesetrturkishplpolishcacatalannldutchararabicsvswedishititalianid indonesianhihindififinnishvi vietnamesehehebrewuk ukrainianelgreekmsmalaycsczechroromaniandadanishhu hungariantatamilno norwegianththaiururduhrcroatianbg bulgarianlt lithuanianlalatinmimaoriml malayalamcywelshskslovaktetelugufapersianlvlatvianbnbengalisrserbianaz azerbaijanisl slovenianknkannadaetestonianmk macedonianbrbretoneubasqueis icelandichyarmeniannenepalimn mongolianbsbosniankkkazakhsqalbanianswswahiliglgalicianmrmarathipapunjabisisinhalakmkhmersnshonayoyorubasosomaliaf afrikaansococcitankageorgianbe belarusiantgtajiksdsindhigugujaratiamamharicyiyiddishlolaouzuzbekfofaroesehtzhaitian creolepspashtotkturkmennnnynorskmtmaltesesasanskritlb luxembourgishmymyanmarbotibetantltagalogmgmalagasyasassamesetttatarhawhawaiianlnlingalahahausababashkirjwjavanesesu sundaneseyue cantoneseburmese valencianflemishhaitian letzeburgeschpushtopanjabi moldavianmoldovan sinhalese castilianmandarin translate transcribecaa]tRt^toRt]tRR.t]t R)V3RlV3Rlllt R*V3RlV3Rlllt R+R lt ] R,R l4tR-V3R lR lltR tR.V3RlV3RllltRRRRRR/V3RlV3RllltRt]R-Rl4tR/V3RlRlltR0V3RlRllt]V3RlRl4tR/V3RlRlltR1V3RlV3R llltR2R!ltR"tR3V3R#lR$lltV3R%lR&lt]R'4t R(t!Vt"V;t##)4WhisperTokenizera Construct a "fast" Whisper tokenizer (backed by HuggingFace's *tokenizers* library). This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`, *optional*): Path to the vocabulary file. merges_file (`str`, *optional*): Path to the merges file. normalizer_file (`str`, *optional*): Path to the normalizer_file file. tokenizer_file (`str`, *optional*): Path to [tokenizers](https://github.com/huggingface/tokenizers) file (generally has a .json extension) that contains everything needed to load the tokenizer. unk_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The beginning of sequence token. The `decoder_start_token_id` is used to set the first token as `"<|startoftranscript|>"` when generating. eos_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The end of sequence token. add_prefix_space (`bool`, *optional*, defaults to `False`): Whether or not to add an initial space to the input. This allows to treat the leading word just as any other word. (Whisper tokenizer detect beginning of words by the preceding space). language (`str`, *optional*): The language of the transcription text. The corresponding language id token is appended to the start of the sequence for multilingual speech recognition and speech translation tasks, e.g. for Spanish the token `"<|es|>"` is appended to the start of sequence. This should be used for multilingual fine-tuning only. task (`str`, *optional*): Task identifier to append at the start of sequence (if any). This should be used for mulitlingual fine-tuning, with `"transcribe"` for speech recognition and `"translate"` for speech translation. predict_timestamps (`bool`, *optional*, defaults to `False`): Whether to omit the `<|notimestamps|>` token at the start of the sequence. input_idsattention_maskFcT<V^8dQhRS[S[S[S[3,,R,/#)vocabN)strdictint)format __classdict__s"q/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/whisper/tokenization_whisper.py __annotate__WhisperTokenizer.__annotate__s*F!F!T#s(^#d*F!c  <\V\4'd\VRRRRR7MTp\V\4'd\VRRRRR7MTp\V\4'd\VRRRRR7MTpVeTM/VnVeTM.Vn\ \ VPVPRRRRR74Vn\P!VR7VPn \P!4VPn \S V`8!RRVRVR VR VR VR VR V RV /V BVe;\VRR7;_uu_4p \ P"!V 4VnRRR4MRVn\&P(!R4VnWnWnWnVP34R# +'giLO;i)FT)lstriprstrip normalizedspecialN)rmergesdropoutcontinuing_subword_prefixend_of_word_suffixfuse_unk)add_prefix_space unk_token bos_token eos_tokenrrlanguagetaskpredict_timestampsutf-8encodingz<\|(\d+\.\d+)\|>) isinstancerr_vocab_mergesrr _tokenizerr ByteLevel pre_tokenizerrdecodersuper__init__openjsonloadenglish_spelling_normalizerrecompile timestamp_patrrrset_prefix_tokens)selfrrrrrrrrrrkwargs vocab_handle __class__s&&&&&&&&&&&, rrWhisperTokenizer.__init__s )S)) yuX\ ] )S)) yuX\ ] )S)) yuX\ ]  %0eb !'!3v # kk||*,#%    )7(@(@Rb(c%"*"4"4"6      . ,    2    &o88L3799\3J09804D ,ZZ(;<  "4  98s F55 G c <V^8dQhRS[/#rreturnr)rrs"rrrs.(.( .(rc <VPR,^,p..pRpRpRp \V4FwrW8d\W, V,4p W8dfV ^8;'d2W^, ,V8;'dW^, ,V8'*p V 'dWV,, pMT pW, pVRR pTp T pVPRW,R R24VP.4KVR,PV 4K .pVFfp\ V\ 4'dVPV4K,V'd"VP\ SV`WR74KUVPR4Kh RPV4#) z Timestamp tokens are above the special tokens' id range and are ignored by `decode()`. This method decodes given tokens with timestamps tokens annotated, e.g. "<|1.08|>". Nz<|z.2fz|>skip_special_tokensr) all_special_ids enumeratefloatappendr rrdecodejoin)r token_idsr)time_precision segment_sizetimestamp_beginoutputscur_max_timestampprev_segments_lenpenultimate_timestampitoken timestamplast_was_single_endingdecoded_outputssr s&&&&& r_decode_with_timestamps(WhisperTokenizer._decode_with_timestampssx..r2Q6$ #!),HA'!5#:n"LM 0-.!V..!a%(O;cc a%@PTc@c9*.)l-JJ),A))B)")#2,(9%$-!Y%BC#HKLr" ""5)--0A!S!!&&q)&&uw~a~'ab&&r* ww''rc  .pR\\V449d>\VR4'd,\VP4'dVP 4p\ P !V4pVP^,^8d&\VP4^8d \R4hVPR,^,pW8p\ P!VRRVR,,4^,^,pVP^,^8XdVP4^8:d.#\ P!V4^,R,^,V9dA\ P!V\ P!V4^,R,^,4p\ P!V4^,^,p^p ^p VEF&p WV p \V 4^8Ed V ^,P4V, p V R,P4V, pW8dXV^8;'d2W^, ,V8;'dW^, ,V8'*pV'd W, p MW, p Tp VPV 4p VP!V 4pVP#V4pVPRVRW,W,,W,W,,3/4T pEK) V#) a Compute offsets for a given tokenized input Args: token_ids (`Union[int, list[int], np.ndarray, torch.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. time_precision (`float`, *optional*, defaults to 0.02): The time ratio to convert from token to time. segment_size (`int`, *optional*, defaults to 1500): The number of features in the input mel spectrogram. torchcpuz)Can only process a single input at a timeNNNtextr<r*)rtypehasattrcallablerDnparrayshapelen ValueErrorr,wheresumr/item_preprocess_token_ids_decode_filter_timestamp_ids)rr2r3r4offsetsr5timestamp_tokens consecutive last_slicer7r8 current_slice sliced_tokensstart_timestamp_positionend_timestamp_positionis_single_endingrGs&&&& r_compute_offsets!WhisperTokenizer._compute_offsetsHs c$y/* * 50I0IhW`WdWdNeNe! IHHY' ??1  !c)//&:Q&>HI I..r2Q6$7hh/47G7KKLQORSS   Q 1 $)9)=)=)?1)DI XX& ' *2 . 2+ E))K:J1KA1Nr1RUV1VWKXX./215 (M%?M=!A%+8+;+@+@+B_+T()6r):)?)?)AO)S&+?'1Q((!q.1_Duu`aSaIbfuIu<$()9))>)$:!!% : := I ||M211$7#4EHYHjj2CFWFhh&'J?)Brc zTP\R4Uu.uFpRW!,,NK up4#uupi)z Compute the timestamp token ids for a given precision and save to least-recently used (LRU) cache. Args: time_precision (`float`, *optional*, defaults to 0.02): The time ratio to convert from token to time. z<|%.2f|>i)convert_tokens_to_idsrange)rr3r:s&& r timestamp_idsWhisperTokenizer.timestamp_idss<))X]^fXg*hXgSTJ!:L,M,MXg*hii*hs8c <V^8dQhRS[/#)rr))bool)rrs"rrrsDrc ~V'd5VPR4pVPR4pVPWV4pV#)a Pre-process the token ids for decoding by removing the prompt tokens ids and timestamp token ids. Args: token_ids (`Union[int, list[int], np.ndarray, torch.Tensor]`): List of tokenized input ids. Typically, obtained using the `__call__` method of the tokenizer. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens from the token ids. If `True`, the prompt token ids will be removed. <|startofprev|><|startoftranscript|>)rb _strip_prompt)rr2r)prompt_token_iddecoder_start_token_ids&&& rrS&WhisperTokenizer._preprocess_token_idssC "889JKO%)%?%?@W%X "**9G]^IrcF\P!VPRV4#)r)rsubr)rrGs&&rrU&WhisperTokenizer._filter_timestamp_idssvvd(("d33rc^<V^8dQhRS[RS[R,RS[RS[RS[RS[RS[R S[R S[/ #) rr)clean_up_tokenization_spacesNoutput_offsetsr3decode_with_timestamps normalizebasic_normalizeremove_diacriticsr$)rgr.r)rrs"rrrsvYY"Y'+Tk Y  Y  Y!%YYY Y Yrc :<VPVVR7p \SV`!V 3RVRVRVRVRV /V Bp V'ds\V \4'dIV 'dA\V ^,\4'd$V U u.uFp VP WVR7NK p p M[VP WVR7p MG\V \4'd!V Uu.uFqP V4NK p pMVP V 4p V'dVPWR7pR V R V/#V #uup iuupi) a Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokenization spaces. Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`. Args: token_ids (`Union[int, list[int], np.ndarray, torch.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. Will remove the previous tokens (pre-prompt) if present. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces` (available in the `tokenizer_config`). output_offsets (`bool`, *optional*, defaults to `False`): Whether or not to output the offsets of the tokens. This should only be set if the model predicted timestamps. If there are previous tokens (pre-prompt) to decode, they will only appear in the decoded text if they contain timestamp tokens. time_precision (`float`, *optional*, defaults to 0.02): The time ratio to convert from token to time. decode_with_timestamps (`bool`, *optional*, defaults to `False`): Whether or not to decode with timestamps included in the raw text. normalize (`bool`, *optional*, defaults to `False`): Whether or not to apply the English text normalizer to the decoded text. Only applicable when the target text is in English. Otherwise, the basic text normalizer should be applied. basic_normalize (`bool`, *optional*, defaults to `False`): Whether or not to apply the Basic text normalizer to the decoded text. Applicable to multilingual target text. remove_diacritics (`bool`, *optional*, defaults to `False`): Whether or not to remove diacritics when applying the Basic text normalizer. Removing diacritics may destroy information in the decoded text, hence it should be used with caution. kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `str`: The decoded sentence. r(r)rsrvrwrx)r3r))r3rGrV)rSrr0r listr@rUr_)rr2r)rsrtr3rurvrwrxr filtered_idsrGseqtrVr s&&&&&&&&&&, rr0WhisperTokenizer.decodesed11  32 w~  3 *F   ,  0    ",--,:l[\o_cCdCd ,  ,00Pc1 , 33 Uh4 $%%?CDt!2215tD11$7 ++I+UGD)W5 5 +Es 7DDrvrwrxc2<V^8dQhRS[RS[RS[RS[/#)rrvrwrxr$)rgr)rrs"rrr s-   $ ?C `d   rc<\SV`!V/VBpV'dVPV4pV#V'dVPWcR7pV#V#)rx)rrTrvrw) rrvrwrxargsrrG clean_textr s &$$$*, rrTWhisperTokenizer._decode sPw// -J  --d-XJ Krc <\VP4pV!V4#)zz Normalize a given string using the `EnglishTextNormalizer` class, which performs commons transformation on english text. )r r)rrG normalizers&& rrvWhisperTokenizer.normalizes +4+K+KL $rc *\VR7pV!V4#)z} Normalize a given string using the `BasicTextNormalizer` class, which performs commons transformation on multilingual text. r)r )rGrxrs&& rrw WhisperTokenizer.basic_normalize!s );LM $rcJ<V^8dQhRS[RS[R,RS[S[,/#)rsave_directoryfilename_prefixNr$)rtuple)rrs"rrr*s-99c9C$J9Z_`cZd9rc \PPV4'g\P RV R24R#\PP Y'd VR,MR\ R,,4p\PP Y'd VR,MR\ R,,4p\PP Y'd VR,MR\ R,,4p\VRR R 7;_uu_4pVP\P!VP^R R R 7R,4RRR4\VRR R 7;_uu_4pVPR4VPRVP44RRR4VPeZ\VRR R 7;_uu_4pVP\P!VP^R R R 7R,4RRR4W4V3# +'giL;i +'giL;i +'giL:;i)zVocabulary path (z) should be a directoryN-rr rrwrr TF)indent sort_keys ensure_ascii z#version: 0.2 c3R"TFpRPV4R,xK R#5i) rN)r1).0 merge_pairs& r 3WhisperTokenizer.save_vocabulary..>s!YLjchhz2T99Ls%')ospathisdirloggererrorr1VOCAB_FILES_NAMESrwriterdumpsr  writelinesrr)rrrr merge_filerfwriters&&& rsave_vocabulary WhisperTokenizer.save_vocabulary*sww}}^,, LL,^,< 885 45 4 >=s$;H64H0;I H- 0 I  I cV<V^8dQhRS[R,RS[R,RS[R,/#)rrNrr)rrg)rrs"rrrHs3& & d & 14t& X\_cXc& rc  VeTM VPVnVeTM VPVnVeTM VPVnVPpVP V4pVP pVP pRPVUu.uFq R2NK up4p \P!V RV R2V RV R2Wg3.\WT4OR7VPn R#uupi)aF Override the prefix tokens appended to the start of the label sequence. This method can be used standalone to update the prefix tokens as required when fine-tuning. Example: ```python >>> # instantiate the tokenizer and set the prefix token to Spanish >>> tokenizer = WhisperTokenizerFast.from_pretrained("openai/whisper-tiny", language="spanish") >>> # now switch the prefix token from Spanish to French >>> tokenizer.set_prefix_tokens(language="french") ``` Args: language (`str`, *optional*, defaults to `None`): The language of the transcription text. task (`str`, *optional*, defaults to `None`): Task identifier to append at the start of sequence (if any). predict_timestamps (`bool`, *optional*, defaults to `None`): Whether to omit the `<|notimestamps|>` token at the start of the sequence. Nrz:0z $A:0 z $A:0 $B:1 z:1)singlepairspecial_tokens) rrr prefix_tokensconvert_ids_to_tokensr eos_token_idr1rTemplateProcessingzipbackend_tokenizerpost_processor) rrrrprefix_token_idsprefixeseosrr;prefix_templates &&&& rr"WhisperTokenizer.set_prefix_tokensHs,%-$8dmm  ,D$)) 8J8V"4\`\s\s----.>?nn(( ((h#GhUgRLh#GH0:0M0M%&fSE4#$KuB7#X01 -$Hs C!c0<V^8dQhRS[S[,/#r#rzr)rrs"rrrrstCyrc pVPR4pVPR4pVPR4pVPR4p\\P44pVPeVPP 4VnVP\ 9d\ VP,pMVP\ P49dVPpMt\VP4^8Hp\RVP RV'd\\ P44M\\ P44 R24hVPe5VP\9d \RVP R\ 24hV.pVPe/VPV^,VPX4,4VPe%TPVPR 8XdTMT4VP'gVPV4V#) rjz <|translate|>z<|transcribe|><|notimestamps|>zUnsupported language: z. Language should be one of: .zUnsupported task: z. Task should be in: r)rbr LANGUAGESkeysrlowerTO_LANGUAGE_CODEvaluesrNrOrzrTASK_IDSr/indexr) r bos_token_idtranslate_token_idtranscribe_token_idnotimestamps_token_idlangs language_idis_language_code bos_sequences & rrWhisperTokenizer.prefix_tokensps112IJ !77H"889IJ $ : :;M Ninn&' == $ MM//1DM}} 00.t}}= "2"9"9";;"mm #&t}}#5#:  ,T]]O<;K-4467QUVfVkVkVmQnoopr 99 yy( #5dii[@UV^U_!`aa$~ == $    q 05;;{3K K L 99   tyyL7P 3Vh i&&&    5 6rc0<V^8dQhRS[S[,/#r#r)rrs"rrrsTTQUVYQZTrc Vf&VPV,VP.,#VPV,V,VP.,#)z=Build model inputs from a sequence by appending eos_token_id.)rr)r token_ids_0 token_ids_1s&&&r build_inputs_with_special_tokens1WhisperTokenizer.build_inputs_with_special_tokenssK  %% 3t7H7H6II I!!K/+=ARAR@SSSrc p<V^8dQhRS[S[,RS[S[,R,RS[RS[S[,/#)rrrNalready_has_special_tokensr$)rzrrg)rrs"rrrsD__9_379t3C_hl_ c_rc 2<V'd\SV`WRR7#^.\VP4,p^.pVf"V^.\V4,,V,#V^.\V4,,^.\V4,,V,#)ad Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`list[int]`): List of IDs. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `list[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrr)rget_special_tokens_maskrNr)rrrr prefix_ones suffix_onesr s&&&& rr(WhisperTokenizer.get_special_tokens_masks& &72']a3 cC 2 233 c  1#K(8"89KG GqcC $445!s;?O9OPS^^^rcVPWV'*R7VPR,p\V4UUu.uFwrVV^,V3NK pppV#uuppi))rrrrE)rrr-)rrr no_timestamps forced_tokensrankr;forced_decoder_idss&&&& rget_decoder_prompt_ids'WhisperTokenizer.get_decoder_prompt_idss^ DTaPab **2. CL]C[\C[KDtax/C[\!!]sAc"\VVVVVR7#))return_timestampsreturn_languager3) _decode_asr)r model_outputsrrr3s&&$$$rrWhisperTokenizer._decode_asrs  /+)   rc <V^8dQhRS[/#)rrGr%)rrs"rrrs + +3 +rc aS!RRVP4,RR7pVR,R,p\V3RlV4R4pVe!SPV4p\R V R 24hVP VR 7VR,#) z`Converts prompt text to IDs that can be passed to [`~WhisperForConditionalGeneration.generate`].rirF)add_special_tokensrrEc3Z<"TF qSP^,8gKVxK" R#5iN)r,)rxrs& rr2WhisperTokenizer.get_prompt_ids..s$ \OqDDXDXYZD[?[Os+ +NzJEncountered text in the prompt corresponding to disallowed special token: r) tensor_type)stripnextrrOconvert_to_tensors)rrGreturn_tensorsbatch_encodingprompt_text_idsspecial_token_idr;sf&& rget_prompt_idsWhisperTokenizer.get_prompt_idss/tzz|1CX]^)5b9 \O \^bc  '../?@Eijoippqrs s))n)Ek**rc<<V^8dQhRS[S[,RS[RS[/#)rr2rlrmr)rrs"rrrs'tCy3`crc\V\4'gVPV4pV'gV#V^,V8HpV'dW19dWPV4R#.#V#r)r rz_convert_to_listr)rr2rlrm has_prompts&&&& rrkWhisperTokenizer._strip_promptsb)T**--i8I q\_4 %2 1G!H!JKK rc\VR4'dVP4P4p\V\P 4'dVP 4pV#)numpy)rIrDrr rKndarraytolist)r2s&rr!WhisperTokenizer._convert_to_listsJ 9g & &! --/I i , ,!((*Ir)rrr rrrrr) NNN <|endoftext|>rrFNNF)F{Gz?)rr)r)F)FNFrFFFFN)NNN)NF)NNT)rK)$__name__ __module__ __qualname____firstlineno____doc__rvocab_files_namesmodel_input_namesrmodelrr@r_rrdrSrUr0rTrv staticmethodrwrrpropertyrrrrrrrkr__static_attributes____classdictcell__ __classcell__)r rs@@rrrs)$L*$&67 EF!F!R.(.(bBHjj&4YYv ', FK gl     99<& & PDTT__>"  + +$rrclV^8dQhR\\,R\R,R\R\/#rtokensrNprepend_punctuationsappend_punctuationsrzrr)rs"rrr:-- I-Dj- -  -rcVf VPpVfRpVR9d\W4wrVpM\W4wrVp\WVWsV4WVV3#z Groups tokens by word. Returns a tuple containing a list of strings with the words, and a list of `token_id` sequences with the tokens making up each word. r>rrNrrr rr_split_tokens_on_unicode_split_tokens_on_spaces_merge_punctuations tokenizerrrrrwords word_tokens token_indicess&&&&& r_combine_tokens_into_wordsr(_%%QQ,DY,W)M,CI,V)MMQde } ,,rc aaaaaaV^,p\V4p.pS'd S^,o.p\VR,4EFwopRpW3^^3p\V4p \^W9,4EFp V R, p \^W:, 4o\ W3V ,V , 4p \ P !VSV 4p \^W, 4o\ W4p\ P !VSV4o\V 4\S48wd \R4hS'd%\VVVVVV3Rl\V 444pM\ P!V S8H4pW, V ,pV^8gEK VV8gEKTpSV SV3pEK Vwop opV S,^,pVS,^,pVPVRV4VVRp\V4pS'gEKXPSRV4SS^,,VRoEK VPV4SfV#\S4^8dXPS4WE3#V.3#)rrEr'@iThere is a bug within whisper `decode_asr` function, please report it. Dropping to prevent bad inference.c3<"TFLwrVSV,8XgKSSV,,SS^,,SV,,8:gKH^xKN R#5irFNr  ridxelem left_startleft_token_timestamp_sequenceright right_startseq_idxtoken_timestamp_sequencess & rr0_find_longest_common_sequence.._Y%4 c *:*s:JK4Wq[A+PSBSTU A%4A.A AN rNr-rcmaxminrKrL RuntimeErrorrQextend sequencesr7 left_sequence left_lengthtotal_sequencetotal_token_timestamp_sequenceright_sequencemax_ max_indices right_lengthr:eps left_stopleft right_stopmatchesmatchingleft_mid right_midr2r3r4r5r6s&f @@@@@r_find_longest_common_sequencerRVaLMm$KN (A!(D%)+&#,Yr]#;"A6 <>* q+45Ag+C Q 0JK|)Ca)GHI88M*Y?@Da1K\-JHH^K CDE4yCJ&")%.t_&&/{S(H{x$)9k:N Q6T.new_chunksM;t fbQQrr'rFrirjrwordtoken_timestampsstriderGrTr<r%c3$"TFqxK R#5irr )rps& rr_decode_asr..Zs1AaszWhisper did not predict an ending timestamp, which can happen if audio is cut off in the middle of a word. Also make sure WhisperTimeStampLogitsProcessor was used during generation.rc32"TF qR,xK R#5i)rGNr )rchunks& rrr^ts:6%f 6schunksr+)rbsetr,r-rrkreversedr0rgetr/rRr.round_collate_word_timestampsanyrwarningr1poprr?)0r$rrrr3r4rWrar` time_offsetr5previous_tokensprevious_token_timestampsskipright_stride_startr,rlrmchunk_idoutputr2rZlast_timestampfirst_timestampr7r8r9 chunk_len stride_left stride_rightr;current_tokenscurrent_token_timestampsr:rGrresolved_tokens resolved_textr<r=timeresolved_token_timestamps start_timeend_time full_text new_chunksoptionalrVs0&&$$$$ @rrrsd$MRF KEK556HIAMOO " D)334O556GHO&<<=TU%m4&8$Q'..0 ++IH^_  &%&89!<CCE  )  # v 39(3C 0I{L ; &K!*\!9  "-">"P%i0E/+6!&!8N JM_ _!).1#% "),HAu ' ''0Abz$==.'%])BK\'..~>*G*X(1(8(8(I (5f  e,+-)+ ) (0E*%$,M/)"5?#:n"LM 00-.!V..!!a%(O;cc !a%@PTc@c9*.)^-JJ),A))-BB)(9%$-!/>AKORccT1~!e~&=  Do%/2I D;'*2,0E+&q)u[1!4404k*1-'..~>,65<<=UVEb+FB)B)2(8(8(I (5f ,6-E )? 2 F*)001IJ+#11###1111KE ON(* %') $Y5\  NN]  6S 6 22"((9 %f  &5?,E}VeE'N  e:6::IOE$ +&%*5+=%>k""? *%   &J!!%.1  *-H h !&)H h  h rc aaaaaaV^,p\V4p.pS'd S^,o.p\VR,4EFwopRpW3^^3p\V4p \^W9,4EFp V R, p \^W:, 4o\ W3V ,V , 4p \ P !VSV 4p \^W, 4o\ W4p\ P !VSV4o\V 4\S48wd \R4hS'd%\VVVVVV3Rl\V 444pM\ P!V S8H4pW, V ,pV^8gEK VV8gEKTpSV SV3pEK Vwop opV S,^,pVS,^,pVPVRV4VVRp\V4pS'gEKXPSRV4SS^,,VRoEK VPV4SfV#\S4^8dXPS4WE3#V.3#)rrEr'r+r,c3<"TFLwrVSV,8XgKSSV,,SS^,,SV,,8:gKH^xKN R#5ir.r r/s & rrr8r9r:Nr;r@s&f @@@@@rrRrRrSrc \WV4wrVpV'dRV/M/p\WW4U U u.uF7wrRV RW*^,,^,W*R,,^,3/VCNK9 p p p V #uup p i)rrGr<r*)r(r) r$rrZrrr%_r'optional_language_fieldrYindicestimingss &&&&& rrfrfs8HUEm8Gz84R!6  7MD D *1:6q9;KTVK;XYZ;[\ & 7   Ns=A.clV^8dQhR\\,R\R,R\R\/#rr)rs"rrrrrcVf VPpVfRpVR9d\W4wrVpM\W4wrVp\WVWsV4WVV3#rrr#s&&&&& rr(r(r)rc:V^8dQhR\\,/#rrr)rs"rrr,s--S -rcVPVRR7pRp.p.p.p.p.p^p \V4FwrVPV 4VPV 4VPVRR7p W<9gKWPV4,\ V48g&W)V PV4,,V8XgKVPV 4VPV4VPV4.p.pV \ V 4, p K WEV3#)zlCombine tokens into words by splitting at any position where the tokens are decoded as valid unicode points.T)ruu�)r0r-r/rrN) r$r decoded_fullreplacement_charr%r&r'rvcurrent_indicesunicode_offset token_idxr;decodeds && rr r ,s##F4#HL EKMNON%f- e$y)"">$"O  + .> ??3|CTTW]];K-LLMQaa LL !   ~ .   1N O c'l *N." } ,,rc:V^8dQhR\\,/#rr)rs"rrrLs--tCy-rc\W4wr#p.p.p.p\W#V4Fwrp V ^,VP8p VPR4p VP 4R9p V 'g!V 'gV 'g\ V4^8Xd6VP V4VP V 4VP V 4KVR,V,VR&VR,PV 4VR,PV 4K WVV3#)zLCombine tokens into words by splitting at whitespace and punctuation tokens.rz !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~r*)r rr startswithrrNr/r?)r$rsubwordssubword_tokens_listsubword_indices_listr%r&r'subwordsubword_tokenssubword_indicesr with_space punctuations&& rr!r!Ls:RS\:e7H#7 EKM47Wk4l0 #y'='=='', mmo)MM jK3u:? LL !   ~ .   1b G+E"I O " "> 2 "  $ $_ 55m } ,,rc\V4^, p\V4^, pV^8dW,PR4'dnW,P4V9dSW,W,,W&W,W,,W&W%,W&,,W&&RW&.W&.W%&MTpV^,pK^p^pV\V48dW,PR4'giW,V9d\W;;,W,, uu&W;;,W,, uu&W%;;,W&,, uu&RW&.W&.W&&MTpV^, pKVUu.uFqw'gK VNK upVR&VUu.uFq'gK VNK upVR&VU u.uFq'gK V NK up VR&R#uupiuupiuup i)z1Merges punctuation tokens with neighboring words.rr:NNNN)rNrrendswith) r%rr prependedappendedr:jrYr;r0s &&&&& rr"r"ds| E QA E QA q& 8  s # #(8I(Ex%(*EH FI-FI gj0GJEHFIGJA Q A A c%j.x  %%%(h*> H H I "I J'* $JEHFIGJA Q"'/$/E!H$*4F5eF4F1I!(0#C##0GAJ040s$- G;G G G ) G7G)Nu "'“¡¿([{-u"'.。,，!！?？:：”)]}、r)(r rrr functoolsrrrK tokenizersrrrrrtokenizers.modelsrtokenization_utils_tokenizersr utilsr english_normalizerr r get_loggerr rrritemsrrrr(rRrrfr r!r"__all__)coders00rrsL( RR!>J   H %,&<( e )e )e  (e  ) e   ) e   ( e  (e  *e  ,e  )e  (e  )e  'e  (e  )e   )!e " ,#e $ '%e & )'e ( ,)e * (+e , +-e . '/e 0 '1e 2 '3e 4 *5e 6 (7e 8 +9e : ';e < +=e > &?e @ &Ae B *Ce D +Ee F ,Ge H 'Ie J 'Ke L +Me N 'Oe P (Qe R (Se T )Ue V )We X )Ye Z )[e \ -]e ^ +_e ` )ae b *ce d ,ee f (ge h (ie j +ke l *me n (oe p +qe r )se t (ue v *we x )ye z *{e | )}e ~ )e @ )Ae B 'Ce D 'Ee F (Ge H (Ie J +Ke L )Me N *Oe P ,Qe R 'Se T (Ue V *We X )Ye Z )[e \ %]e ^ '_e ` )ae b ce d (ee f )ge h )ie j )ke l */)))**' :)')*+ ;Ie  P,5OO,=>,=.$x~,=> t t tT d t  &V (V r-4u"pynryxu"p  -4-@-0#1L  s'?s H