+ jh$Rt^RIt^RIt^RIt^RIHt^RIHt^RIH t H t H t H t ] !4'd^RI t ] !4'd^RIt] P !]4tRR/tRt!RR ]4tR .tR#) zTokenization class for VITS.N)Any)PreTrainedTokenizer)is_phonemizer_availableis_uroman_availableloggingrequires_backends vocab_filez vocab.jsonc\\P!R4pVPV4pVRJpV#)z [^\x00-\x7F]N)recompilesearch) input_stringnon_roman_patternmatch has_non_romans& k/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/vits/tokenization_vits.pyhas_non_roman_charactersr$s3 ?3  $ $\ 2E%M caa]tRt^.toRt]tRR.tRV3RlV3Rlllt] R4t Rt Rt R t RV3R lR lltV3R lR ltV3RlRltRtRtRV3RlRlltRtVtV;t#) VitsTokenizera| Construct a VITS tokenizer. Also supports MMS-TTS. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. language (`str`, *optional*): Language identifier. add_blank (`bool`, *optional*, defaults to `True`): Whether to insert token id 0 in between the other tokens. normalize (`bool`, *optional*, defaults to `True`): Whether to normalize the input text by removing all casing and punctuation. phonemize (`bool`, *optional*, defaults to `True`): Whether to convert the input text into phonemes. is_uroman (`bool`, *optional*, defaults to `False`): Whether the `uroman` Romanizer needs to be applied to the input text prior to tokenizing. input_idsattention_maskc<V^8dQhRR/#)returnN)format __classdict__s"r __annotate__VitsTokenizer.__annotate__Gs! !  ! rc  <\VRR7;_uu_4p \P!V 4VnRRR4VPP 4U U u/uFwrWbK up p VnW@nWPnW`nWpn Wn \S V`0!R RVRVRVRVRVRVR VR R /V BR# +'giL;iuup p i) utf-8encodingN pad_token unk_tokenlanguage add_blank normalize phonemize is_uromanspecial_tokens_patternnoner) openjsonloadencoderitemsdecoderr&r'r(r)r*super__init__)selfrr$r%r&r'r(r)r*kwargs vocab_handlekv __class__s&&&&&&&&&, rr4VitsTokenizer.__init__Gs*w / /<99\2DL0*.););)=>)=)=>  """"          $*  0 /?sB. C. B> c,\VP4#N)lenr0)r5s&r vocab_sizeVitsTokenizer.vocab_sizejs4<<  rc\VP4Uu/uFqPV4VbK ppVPVP4V#uupir=)ranger?convert_ids_to_tokensupdateadded_tokens_encoder)r5ivocabs& r get_vocabVitsTokenizer.get_vocabnsL;@;QR;Qa++A.1;QR T../ SsAc \VPP44\VPP44,pRp^pV\ V48duRpVF:pWV\ V4,V8XgKW6, pV\ V4, pRpM V'dK\W1V,P 4, pV^, pKV#)zfLowercase the input string, respecting any special token ids that may be part or entirely upper-cased.FT)listr0keysrEr>lower)r5r all_vocabulary filtered_textrF found_matchwords&& rnormalize_textVitsTokenizer.normalize_textssdll//12T$:S:S:X:X:Z5[[ #l##K&AD M2d:!)MTNA"&K ';a!6!6!88 Qrc LVPR8XdVPRR4pV#)z4Special treatment of characters in certain languagesronuțuţ)r&replace)r5texts&&r_preprocess_charVitsTokenizer._preprocess_chars# ==E !<<d+D rc l<V^8dQhRS[RS[RS[R,RS[S[S[S[S[3,3,/#)rrXis_split_into_wordsr(Nr)strbooltupledictr)rrs"rrrsC>%>%>%.2>%GKd{>% sDcN" #>%rc aVeTM SPpV'dSPV4pSPV4p\V4'd_SP'dM\ 4'g\ PR4M&\P!4pVPV4pSP'dE\SR.4\P!VRRRRRR7p\P!RRV4pWT3#V'd7R P!\#\%V3R lV444P'4pWT3#) am Performs any necessary transformations before tokenization. This method should pop the arguments from kwargs and return the remaining `kwargs` as well. We test the `kwargs` at the end of the encoding process to be sure all the arguments have been used. Args: text (`str`): The text to prepare. is_split_into_words (`bool`, *optional*, defaults to `False`): Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace) which it will tokenize. normalize (`bool`, *optional*, defaults to `None`): Whether or not to apply punctuation and casing normalization to the text inputs. Typically, VITS is trained on lower-cased and un-punctuated text. Hence, normalization is used to ensure that the input text consists only of lower-case characters. kwargs (`dict[str, Any]`, *optional*): Keyword arguments to use for the tokenization. Returns: `tuple[str, dict[str, Any]]`: The prepared text and the unused kwargs. aCText to the tokenizer contains non-Roman characters. To apply the `uroman` pre-processing step automatically, ensure the `uroman` Romanizer is installed with: `pip install uroman` Note `uroman` requires python version >= 3.10Otherwise, apply the Romanizer manually as per the instructions: https://github.com/isi-nlp/uroman phonemizerzen-usespeakT)r&backendstrippreserve_punctuation with_stressz\s+ rKc"<VSP9#r=)r0)charr5s&r8VitsTokenizer.prepare_for_tokenization..sTT\\=Qr)r(rSrYrr*rloggerwarningurUromanromanize_stringr)rrbr subjoinrLfilterre)r5rXr\r(r6rPuromansf&&&, rprepare_for_tokenization&VitsTokenizer.prepare_for_tokenizations 4"+!6IDNN &&t,D--d3 #M 2 2t~~~&((y & 6 6} E >>> d\N 3&00  %) MFF63 >M $$ GGD0QS`)a$bciikM$$rc6<V^8dQhRS[RS[S[,/#)rrXr)r]rL)rrs"rrrs  c d3i rc \V4pVP'd7VP^4.\V4^,^,,pW#R&TpV#)z]Tokenize a string by inserting the `` token at the boundary between adjacent characters.Nr)rLr'_convert_id_to_tokenr>)r5rXtokens intersperseds&& r _tokenizeVitsTokenizer._tokenizesKd >>> 55a89S[1_q=PQL!' !F rc6<V^8dQhRS[S[,RS[/#)rr}r)rLr])rrs"rrrstCySrczVP'd\V4^8d VR,pRPV4#)r{rzrK)r'r>rs)r5r}s&&rconvert_tokens_to_string&VitsTokenizer.convert_tokens_to_strings, >>>c&kAoD\Fwwvrc `WP9dVPV,#VP#)z0Converts a token (str) in an id using the vocab.)r0 unk_token_id)r5tokens&&r_convert_token_to_id"VitsTokenizer._convert_token_to_ids' LL <<& &   rc 8VPPV4#)z=Converts an index (integer) in a token (str) using the vocab.)r2get)r5indexs&&rr|"VitsTokenizer._convert_id_to_tokens||&&rcX<V^8dQhRS[RS[R,RS[S[,R,/#)rsave_directoryfilename_prefixNr)r]r_)rrs"rrrs3  c C$J Z_`cZdgkZk rc \PPV4'g\P RV R24R#\PP Y'd VR,MR\ R,,4p\VRRR7;_uu_4pVP\P!VP^R R R 7R ,4RRR4V3# +'giT3#;i) zVocabulary path (z) should be a directoryN-rKrwr!r"TF)indent sort_keys ensure_ascii ) ospathisdirrmerrorrsVOCAB_FILES_NAMESr-writer.dumpsr0)r5rrrfs&&& rsave_vocabularyVitsTokenizer.save_vocabularysww}}^,, LL,^,<zNTTTF)FNr=)__name__ __module__ __qualname____firstlineno____doc__rvocab_files_namesmodel_input_namesr4propertyr?rHrSrYrvrrrr|r__static_attributes____classdictcell__ __classcell__)r:rs@@rrr.s**$&67! ! F!! * >%>%@   ! '   rr)rr.rr typingrtokenization_pythonrutilsrrrrrbruro get_loggerrrmrrr__all__rrrrss# 6]]   H %!<0F'FR  r