+ jRt^RIt^RIt^RIt^RIHt^RIHt^RIH t ^RI H t ^RI H t^RIHt^RIHtHt^RIHt^R IHt^R IHt^R IHtHt^R IHtH t H!t!H"t"^R I#H$t$^RI%H&t&^RI'H(t(^RI)H*t*^RI+H,t,^RI-H.t.H/t/H0t0H1t1H2t2H3t3H4t4^RI5H6t6H7t7H8t8H9t9]9Pt!];4tRt?Rt@RtARtB].R, t.R]R] R]!R]"/tCR]=R ]@/tD]7!].4!R!R"]144tE]EtFR#)#z Tokenization classes for fast tokenizers (provided by HuggingFace's tokenizers library). For slow (python) tokenizers see tokenization_utils.py N) defaultdict)Iterable)copyfile)Any)is_offline_mode) AddedToken processors)Encoding) Tokenizer)Decoder)BPEUnigram) BpeTrainerUnigramTrainerWordLevelTrainerWordPieceTrainer) cached_file) SpmConverter)convert_gguf_tokenizer)is_tekken_vocab_filename)load_gguf_checkpoint)INIT_TOKENIZER_DOCSTRING BatchEncodingPreTokenizedInputPreTrainedTokenizerBase TextInputTruncationStrategygenerate_merges)PaddingStrategyadd_end_docstringshf_apiloggingztokenizer.jsonzspecial_tokens_map.jsonztokenizer_config.jsonztokenizer.modelzadded_tokens.jsonztiktoken.modelu tokenizer_object ([`tokenizers.Tokenizer`]): A [`tokenizers.Tokenizer`] object from 🤗 tokenizers to instantiate from. See [Using tokenizers from 🤗 tokenizers](../fast_tokenizers) for more information. tokenizer_file ([`str`]): A path to a local JSON file representing a previously serialized [`tokenizers.Tokenizer`] object from 🤗 tokenizers. r r WordLevel WordPiecetokenizer_file vocab_filecaa]tRt^WtoRt]tRtRt] RHRl4t ] V3RlRl4t ] V3RlRl4t V3R lt ]V3R lR l4t]V3R lR l4tRIV3RlRlltRJV3RlV3RllltRt]R4t]R4t]P,R4t]P,R4tRt]V3RlRl4tV3RlRlt]V3RlRl4t]V3RlRl4t]V3R lR!l4t]t]tV3R"lR#ltV3R$lR%lt V3R&lR'lt!]V3R(lR)l4t"]V3R*lR+l4t#RKV3R-lR.llt$V3R/lR0lt%V3R1lR2lt&RHV3R3lR4llt'RHV3R5lR6llt(RHV3R7lR8llt)RLV3R9lR:llt*V3R;lR<lt+RR,],PZ].P^R^RRRRRRRRRRR,R3V3R=lR>llt0V3R?lR@lt1RMV3RAlRBllt2RNV3RClRDllt3RORElt4] RPRFl4t5RGt6Vt7V;t8#)QTokenizersBackenda5 Base class for all fast tokenizers (wrapping HuggingFace tokenizers library). Inherits from [`~tokenization_utils_base.PreTrainedTokenizerBase`]. Handles all the shared methods for tokenization and special tokens, as well as methods for downloading/caching/loading pretrained tokenizers, as well as adding tokens to the vocabulary. This class also contains the added tokens in a unified way on top of all tokenizers so we don't have to handle the specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...). NFc na\V4pVPRR4pVed\PP V4'd?V\ JgRVP 9g V'd\P!V4VR&V#VEe?\PP V4'Ed\VRR7;_uu_4p\P!V4pRRR4XPR/4PR4pVR&9dc\V4p\VR,4p /V R &VR 8Xd.V R &WR&.VR &\P!\P!V44p M\P!V4p V PVR&V P VR&V P"VR&V P"eV P"VR&V P eV P VR&VPR4p V 'dV PRR4R8Xd V R,p M\%V \&4'gV .p V FCp V PR4R8XgKRV 9gK$^RIp V P+V R,4VR&M VPR/4PR R4pVP,f2\%V\&4'd\'\/\0V44pEM>VP,P2R 8Xd]\%V\&4'dFV'd>\%V^,\&\034'dVUu.uFp\1V4NK ppMVP,P2R8Xd"\5V4UUu/uF wppVVbK pppMVP,P2R 8XgVP,P2R8XdV\%V\&4'd@\5V4UUu/uF(wpp\%V\&4'd V^,MTVbK* pppWR &\7VRR4pR VPR/49dyV'dqVP2R 8Xd`VR,R ,pVUu.uF>p\%V\84'd\1VP;R44M \1V4NK@ ppVVR &V#VPR4pVPR4pVPR 4pVPR 4p\%V\84'dZ\PP V4'd5\=V4'd$^RIH pV!V4pVPC4VR&V#\%V\84'd\PP V4'diVPER4'dR\PPGV4\H8wdVPKVV4pV#VPSVV4pV#VfJ\%V\84'd4\PP V4'dVVR &VR ,pVfJ\%V\84'd4\PP V4'dVVR &VR ,pVfVP,eVP,P2R 8Xdq\%V\4'd[R#V3R$llo.R'Op\U4pVF+pVV9gK VPWS!VV,.44K- \YVVR%7pVVR &V# +'giEL;iuupiuuppiuuppiuupi \Ld(p\NPQR T R!T R"24Rp?ELRp?ii;i)(z Build a `tokenizers.Tokenizer` backend from the available serialization files (tokenizer.json, sentencepiece models, tekken.json, vocab/merges). r$N__init__tokenizer_objectutf-8encodingmodeltyper vocabr merges added_tokenspost_processortokenizer_paddingtokenizer_truncation_json_truncation _json_padding normalizerSequence normalizers Precompiledprecompiled_charsmap_spm_precompiled_charsmapr"r# r% merges_file)MistralConverter.modelz+Could not extract SentencePiece model from z$ using sentencepiece library due to z%. Falling back to TikToken extractor.c\V^8dQhR\\,R\\,/#)valuesreturn)rrliststr)formats"k/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/tokenization_utils_tokenizers.py __annotate__@TokenizersBackend.convert_to_native_format..__annotate__s" ! !Xc] !tCy !c<.pVFXpVfK \V\\34'dVPS!V44K>VP \ V44KZ V#N) isinstancerFtupleextendappendrG)rD collectedval_iter_special_tokenss& rIrUHTokenizersBackend.convert_to_native_format.._iter_special_tokenssY') !C{ !#e}55!(()=c)BC!((S2 "! rL) skip_tokens)Nr ) pad_token unk_token bos_token eos_token sep_token cls_token mask_tokenadditional_special_tokensextra_special_tokens)-dictpopospathisfiler'__dict__ TokenizerFast from_fileopenjsonloadgetfrom_strdumpsr3padding truncationrOrFbase64 b64decoder.maprP__name__ enumerategetattrrGsplitrintegrations.mistral.tokenizerr@ convertedendswithbasenameTIKTOKEN_LEGACY_NAME_convert_from_sentencepiece Exceptionloggerwarning_convert_from_tiktokensetupdater)clstrust_remote_codekwargs local_kwargsfast_tokenizer_filetokenizer_handletokenizer_json model_typeminimal_tokenizer_json minimal_model tok_from_filenormalizer_configr8rqr0itemitokenr1merger%r?r@ converterespecial_tokens_keysrWkeyrUs&&, @rIconvert_to_native_format*TokenizersBackend.convert_to_native_formatisF| *../?F +233))Zs||-KO`/CY]gCg%DJDTDT&'=>E %@A#4#&&w377FEyy eT** UE!23E##y0eT**uE!HtUZm9\9\5:;UTU4[UE;##{22;E2BC2Bha2BC##u, 0B0Bk0QeT**_hin_op_oS[STV[E4)@)@U1XeQN_oEp$) ! gt4J>--gr:: zObObfkOk'0:kqrkqbgZs5K5K% C 01QVW\Q]]kqr)/ X& !%%l3 "&&}5   )!!(+ j# & &277>>*+E+EJbcmJnJn H(4I/8/B/B/DL+ ,  j# & &277>>*+E+E*J]J]^fJgJgww +/CC#&#B#B:|#\L''55j,OL  =Z C88RWW^^J=W=W$.L ! )E >jc::rww~~k?Z?Z%0L "!(+F >cii3 8J8Je8SXbchjnXoXo ! ! # %(EK*,&&&';\#=N!VZ[^`c[cVd!rLc  ^RIHpVPR4;'g/P4UUu/uF9wrE\ V4\ V\ 4'd VR,M \V4bK; pppVPR4;'g.p\ V\ 4'd\VP44p\VPV44pV'd\VP44M^p W8de\W^,4U u.uFp VPV RV R24NK p p \V 4p VFp W9gK V PV 4K T pV!WR7pVP!4VR&V#uuppiuup i) z8Path to a vocab file and a list of extra special tokens.)TikTokenConverteradded_tokens_decodercontentr`z<|reserved_token_z|>)r%r`r*)convert_slow_tokenizerrrlitemsintrOrarGrFkeyslenload_tiktoken_bpemaxrangerrRry)rr%rrtoken_idrrr`base_vocab_sizemax_added_token_idindex new_extrasset_new_extras special_tokenrs&&& rIr(TokenizersBackend._convert_from_tiktokens > %1$4$45K$L$R$RPR#Y#Y#[0 #[ Mz%/F/F5+CPUJ V#[ 0 +7*:*:;Q*R*X*XVX *D 1 1#'(<(A(A(C#D /AA*MNBVS!5!:!:!<=[\  0#?4JKKE%((2CE7"0MNK !_N!5  6%%m4"6$. %g +4+>+>+@ '(90 s ?FF cX<V^8dQhRS[RS[S[S[3,RS[S[S[3,/#rr)rHrs"rIrJr5s899S9SRUX9[_`ceh`h[i9rLc^RIHpV!V4pVP!VP3/VBp^RIHpVP VP 4pVe%\VR4'dVP!R/VBp\TR4'dTP!R/TBpR T9EdT\JgR TP9EdTPR R4pTPR R4p TP R 4;'g/p TeT 'dTP!4U U u/uFwrYbK p p p T P!4F]wr\#T 4p \%T4pT P T 4pT'gK6Y8wgK>Y9gKFTPT4Y&YT &K_ \&P(!TP*TT R7pTeTTR &TP*P,pTP.^8d'TP1RTP2;'gR4TP4^8d'TP1RTP6;'gR4TP8^8d'TP1RTP:;'gR4T# \d2p\PRTP RT R24Rp?ELGRp?ii;iuup p i))SentencePieceExtractor)SLOW_TO_FAST_CONVERTERSNconvert_from_spmz,Could not reorder vocab using converter for z due to z/. Falling back to raw SentencePiece extraction.convert_from_spm_modelr*r)r0r1r)protor0r1rZr[rYz)rrextractr.rrlrthasattrrr~rrrr'rfrbrrrGrbuild_tokenizer_from_spm_protor trainer_specbos_id setdefault bos_pieceeos_id eos_pieceunk_id unk_piece)rr%rr extractorrconverter_classrr0r1rrr id_to_token new_token current_tokenr* proto_specs&&& rIr}-TokenizersBackend._convert_from_sentencepiece4sB+:6  ((ClC   G599#,,GO*wHZ/[/[.??O,O 30 1 155E EL \ 1s>O7OS]eheqeqSq $$Wd3E!%%h5F$0#3#34J#K#Q#Qr  %9FKkkmTm?5xm T+?+E+E+G'H"8}H #II$/OOH$=M$})C H^+099]+C(09H- ,H ,JJoo    +3C /0'__99 $$) ++K9M9M9V9VQVW$$) ++K9M9M9W9WQWX$$) ++K9M9M9X9XQXYW  NN>s||nHUVTWXGH   "UsA J K K&J==Kc <VPRR4pVPRR4pVPRR4VPRR4pVPRR4pVPRR4pVPR/4pVPRR 4p VPR 4p VPR 4p VPR 4p Rp Ve\P!V4p EMVe>\P P V4'd\P!V4p EMVe^R I H pH pH p\VPRR4V3/VBpV!V4wppVR,V9dV!V4wpppM-\V4pVR,R,pVR,pVR,p\VV4wp pVP!V4\#V4^8dVP!V4EMVP$fV eV eX\'V \(4'dT M%\+V 4UUUu/uF wpwppVVbK upppp\\-VV RRR74p M\'V \(4'd\\-V .RRR74p M\'V \.4'dQV 'dI\'V ^,\0\.34'd&\\3WPR^4R74p MVP$f \5R4hVf7Vf3VP$f%VP7RR4VP7RR4V eWnVP$f \5R4hVPRR4;'g!VP$P8;'gTpVeVP$P:!R9/VBVP7R VR ,4VP7R!VR",4VP7R#VR#,4VP7R$VR%,4MVP$P=4VPR&R4;'g!VP$P>;'gTpVeVP$P@!R9/VBVP7R'VR',4VP7R(VR),4VP7R*VR",4VP7R VR+,4VP7R,VR,,4R-V9dR.VR-&R/V9;'gR0V9pVPR/R 4Vn!VPR0R 4Vn"VPR1R4;p'dVVP$n#T;'gVP$PFRJVn$\JS*V`!R9/VBV eWn'Wn(VPRVP$n*VPVU u0uFp \Y\[V 44kK p!p \]VP_4R2R37U"U u.uF#wp"p \Y\[V 44V!9gK!V NK% p#p"p \/VP`Pc44V#U u.uFp \eV 4NK up ,p$VPfPi4F5p%V%fK \eV%4V$9gKV%V#9gK$V#PkV%4K7 VPlF/p \eV 4V$9gKV V#9gKV#PkV 4K1 \#V#4^8d.p&VPfPi4U'u.uFp'V''gK \eV'4NK p(p'V#Fwp \'V \d4'd\oV RR47p M?\'V \n4'd*V Pp'g\eV 4V(9dRV n8V&PkV 4Ky V&'dVPsV&4VP$Pu4p)V)R58d\yVP$R6R4enVPRR4VPz!VP$VP|PRR43R7VP|R8VPR8R4/VBVnVPH;'gVP$PFRJVn$VPH'dVP4R#R#uupppiuup iuup p"iuup iuup'i \vd^p)ELi;i):r6Nr7r=r* gguf_filer$radd_prefix_spaceFr%r0r1)GGUF_TOKENIZER_ARCHSget_gguf_tokenizerread_gguf_metadata name_or_pathzgeneral.architectureconfigr tokenizertokenizer_configT)r0r1fuse_unkdropoutr)r0ra9Couldn't instantiate the backend tokenizer from one of: (1) a `tokenizers` library serialization file, (2) a slow tokenizer instance to convert or (3) an equivalent slow tokenizer class to instantiate and convert. You need to have sentencepiece or tiktoken installed to convert a slow tokenizer to a fast one.rZrr[rz3The backend tokenizer is not correctly initialized.r5 max_lengthtruncation_side directionstridetruncation_strategystrategyr4rXpad_token_type_id pad_type_id padding_sidelengthpad_to_multiple_ofbackend tokenizers add_bos_token add_eos_tokenr3cV^,#r)xs&rI,TokenizersBackend.__init__..s STUVSWrLr)speciali pre_tokenizer init_kwargsfix_mistral_regexr)@rbrlcopydeepcopyrcrdrergrhintegrations.ggufrrrrrrrr _tokenizerrOrarur rFrPr ValueErrorrrpenable_truncation no_truncationroenable_padding_add_bos_token_add_eos_tokenr3_should_update_post_processorsuperr)r%rsplit_special_tokensencode_special_tokensrhashreprsortedradded_tokens_encoderrrG_special_tokens_maprDrR_extra_special_tokensrr add_tokensget_vocab_sizeNotImplementedErrorrv_patch_mistral_regexrupdate_post_processor)+selfargsrr6r7r*rrrrr%r0r1fast_tokenizerrrr gguf_pathmetadata_ architecturetokenizer_dictr gguf_paramadditional_kwargsrw vocab_dict _truncation_paddingexplicit_bos_eos_in_kwargsr3radded_tokens_decoder_hashr tokens_to_addencoderspecial_token_valuetokenstall_named_tokens vocab_size __class__s+&*, rIr)TokenizersBackend.__init__ps"::&8$? ?D9   .5!::&8$?JJ{D1 $jj)94@%zz*@"E!::&8%@ZZ -  7#H%  '!]]+; n.>1)< )(3LA !+K!8#-.@#A 0F|Uc0d -N- MM* +$%) /0 __ $):!&0&=&=UZcdiZjCkZjYQPVQRTUAqDZjCk !.sF]ako/p!qE4((!.srTXbf/g!hE4((Uz%(UTXM7Z7Z!.wU::V^`aKb/c!d __ $r   &+;+CH_   k5 1   k6 2  %,O ?? "RS Sjj!7>pp$//B\B\pp`p  " OO - - < <   lK ,E F   / J   h H(= >   3[5L M OO ) ) +::148ddDOO ?   2H=Q4R S F " ,F9 %4%>%[%[/U[B["$jj%@$jj%@#ZZ(8$? ?> ?-;DOO *-G-q-q4??KiKimqKq* "6"  !(O 0040I0I-DHD]D]$^D]5T$u+%6D]!$^!'';'A'A'C X X uDK (AA E X  t005578Ta;bTa5CJTa;bb$(#;#;#B#B#D "*&'w6;NVc;c$$%89 $E//E5z(U--G$$U+0 }  !F040H0H0O0O0QW0Q1UVA0Q W&eS))&ud;Ez22 ===SZ;K-K(,  e$'' 779J  74??OT#R#^ JJ{D )"77  $$^T:!,,#)**-@$"G   DO  . . X X$//2P2PTX2X *  - - -  & & ( .oDl~%_ 5 53K?Bfgh h ) # J " rww}}^<<#'#=#=n#M +DAJ!',z:0z:1z $B:1)singlepairspecial_tokens) rZ bos_token_idrr[ eos_token_idrrRrTemplateProcessingrr3)rbosr]eosr^rZr[r\s& rIr'TokenizersBackend.update_post_processor}sFnn(( ;4---!&D nn(( ;4---!&D %)%7%7%7S5[R@[_[m[m[mcCiRVFVsuDvw0B0B0B39t+K5gkgygygyRUX[R[^bRb@BQCD      ! !3"5 6      ! !3"5 6)3)F)F^* &rLc\VRR4#)rFrvr.s&rIrTokenizersBackend.add_eos_tokent-u55rLc\VRR4#)rFrdr.s&rIrTokenizersBackend.add_bos_tokenrfrLcT\PVRV4VP4R#)rNobject __setattr__rrvalues&&rIrre!4!159 ""$rLcT\PVRV4VP4R#)rNrjrms&&rIrrhrorLc  .pVPP4FgpVfK \V\4'dVP V4K2\V\ 4'gKJVP \VRRR74Ki VP Fap\V\4'dVP V4K,\V\ 4'gKDVP \VRRR74Kc V'dVPVRR7\VRR4'gVPPfVP4R#R#)a3 Post-initialization hook that runs after the tokenizer is fully set up. This is called by from_pretrained() after loading the tokenizer, which allows us to add any special tokens that may have been passed as AddedToken objects. Child classes should call super()._post_init() if they override this method. NTF)r normalized)r\r) r rDrOrrRrGr r rvrr3r)rr  token_valuers& rI _post_initTokenizersBackend._post_inits 33::ArLc:VPPRR7#)Try)r get_vocabr.s&rIrTokenizersBackend.get_vocabs((4(@@rLc6<V^8dQhRS[S[S[3,/#r*r~)rHrs"rIrJrs  tCH~ rLc"VP4#rN)rr.s&rIr0TokenizersBackend.vocabs~~rLc6<V^8dQhRS[S[S[3,/#r*r~)rHrs"rIrJrsnnd38nnrLc \VPP4RR7UUu/uFwrVPVbK upp#uuppi)z Returns the sorted mapping from string to index. The added tokens encoder is cached for performance optimisation in `self._added_tokens_encoder` for the slow tokenizers. cV^,#rrrs&rIr8TokenizersBackend.added_tokens_encoder.. dhijdkrLrrrrrrvks& rIr&TokenizersBackend.added_tokens_encoders? *00I0I0O0O0QWk)lm)l 1 )lmmmAc6<V^8dQhRS[S[S[3,/#r*)rarr)rHrs"rIrJrs::d3 ?&;:rLc 6VPP4#)z Returns the added tokens in the vocabulary as a dictionary of index to AddedToken. Returns: `dict[str, int]`: The added tokens. )rget_added_tokens_decoderr.s&rIr&TokenizersBackend.added_tokens_decoders7799rLc6<V^8dQhRS[S[S[3,/#r*r~)rHrs"rIrJrsnnc3hnrLc \VPP4RR7UUu/uFwrVPVbK upp#uuppi)z Returns the added tokens in the vocabulary as a dictionary of token to index. Returns: `dict[str, int]`: The added tokens. cV^,#rrrs&rIr3TokenizersBackend.get_added_vocab..rrLrrrs& rIget_added_vocab!TokenizersBackend.get_added_vocabs?*00I0I0O0O0QWk)lm)l 1 )lmmmrc <V^8dQhRS[/#r*r+)rHrs"rIrJrs$rLc R#)z> Returns True, to avoid expensive `assert tokenizer` gotchas. Trr.s&rI__bool__TokenizersBackend.__bool__srLc <V^8dQhRS[/#r*rw)rHrs"rIrJrsFFFrLc :VPPRR7#)z4 Size of the full vocabulary with the added tokens. Tryr{r.s&rI__len__TokenizersBackend.__len__s---EErLc <V^8dQhRS[/#r*)rg)rHrs"rIrJrs=rLc VP#)zS `tokenizers.implementations.BaseTokenizer`: The Rust tokenizer used as a backend. )rr.s&rIbackend_tokenizer#TokenizersBackend.backend_tokenizers rLc <V^8dQhRS[/#r*) DecoderFast)rHrs"rIrJr s'''rLc .VPP#)zE `tokenizers.decoders.Decoder`: The Rust decoder for this tokenizer. )rdecoderr.s&rIrTokenizersBackend.decoder s &&&rLTc<V^8dQhRS[RS[R,RS[R,RS[RS[RS[RS[R S[R S[S[S[S[3,S[S[,3,/ #) rCr-return_token_type_idsNreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverboserE) EncodingFastr,rPrarGrrF)rHrs"rIrJrs-(-(-( $d{-( $d{ -( $( -( %) -(!%-(-(-( tCH~tL11 2-(rLc  VfRVP9pVfRVP9pV'd$VPeV.VP,p MV.p \\4p V EFp V R,P V P 4V'd#V R,P V P 4V'd#V R,P V P4V'd#V R,P V P4V'd#V R,P V P4V'gKV R,P \V P 44EK W3#)ar Convert the encoding representation (from low-level HuggingFace tokenizer output) to a python Dict and a list of encodings, take care of building a batch from overflowing tokens. Overflowing tokens are converted to additional examples (like batches) so the output values of the dict are lists (overflows) of lists (tokens). Output shape: (overflows, sequence length) token_type_idsattention_mask input_idsspecial_tokens_maskoffset_mappingr) model_input_names overflowingrrFrRidstype_idsrroffsetsr) rr-rrrrrrr encodings encoding_dictrs &&&&&&&&& rI_convert_encoding#TokenizersBackend._convert_encodings( ! ($48N8N$N ! ($48N8N$N ! $)=)=)I! X%9%99I! I#D) A + & - -aee 4$./66qzzB$./66q7G7GH)34;;A This encodes a dummy input and checks the number of added tokens, and is therefore not efficient. Do not put this inside your training loop. Args: pair (`bool`, *optional*, defaults to `False`): Whether the number of added tokens should be computed in the case of a sequence pair or a single sequence. Returns: `int`: Number of special tokens added to sequences. )rnum_special_tokens_to_add)rr[s&&rIr+TokenizersBackend.num_special_tokens_to_addOs&88>>rLcl<V^8dQhRS[S[S[,,RS[RS[S[S[,,/#)rCrskip_special_tokensrErrFr,rG)rHrs"rIrJrds7tCyt`cfjknfo`orLc F\V\4'dVPPV4#.pV'd\ VP 4M \ 4pVF@p\V4pWT9dKVP VPPV44KB V#)a Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and added tokens. Args: ids (`int` or `list[int]`): The token id (or token ids) to convert to tokens. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. Returns: `str` or `list[str]`: The decoded token(s). )rOrrrrall_special_idsrR)rrrr# ids_to_skiprs&&& rIconvert_ids_to_tokens'TokenizersBackend.convert_ids_to_tokensds c3  ??..s3 33Fc$../CE EJE# MM$//55e< =   rLc P<V^8dQhRS[RS[R,RS[RS[S[,/#)rCtextr[NrrE)rGr,rF)rHrs"rIrJr~s:vvSvd vtvjnorjsvrLc NVP!RRVRVRV/VBP4#)r text_pairrr) _encode_plusr#)rrr[rrs&&&&,rItokenizeTokenizersBackend.tokenize~s0  ldldlOaleklssuurLc Z<V^8dQhRS[RS[RS[RS[RS[R,RS[R,/#)rCpadding_strategyrrrrNr)rrrrG)rHrs"rIrJrsWI9I9)I90I9 I9  I9  $J I9DjI9rLc  VPPpVPPpV\P8Xd VeVPP 4MhRVRVRVP RVP/p VfRp M!V U u/uFqVPV R4bK p p W8wdVPP!R /V BV\P8Xd#VeVPP4R#R#V\P8XdTMRp RT RVeTM VPRVPRVP R VP"R V/p W8wdVPP$!R /V BR#R#uup i) a Define the truncation and the padding strategies for fast tokenizers (provided by HuggingFace tokenizers library) and restore the tokenizer settings afterwards. The provided tokenizer has no padding / truncation strategy before the managed section. If your tokenizer set a padding / truncation strategy before, then it will be reset to no padding / truncation when exiting the managed section. Args: padding_strategy ([`~utils.PaddingStrategy`]): The kind of padding that will be applied to the input truncation_strategy ([`~tokenization_utils_base.TruncationStrategy`]): The kind of truncation that will be applied to the input max_length (`int`): The maximum size of a sequence. stride (`int`): The stride to use when handling overflow. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). padding_side (`str`, *optional*): The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. Nrrrrrpad_idrXrrr)rrprorDO_NOT_TRUNCATErrnrrlrr DO_NOT_PAD no_padding MAX_LENGTHr pad_token_idrXrr) rrrrrrrrrtargetcurrentrrs &&&&&&& rIset_truncation_and_padding,TokenizersBackend.set_truncation_and_paddingshBoo00 ??** "4"D"D D&--/j&/55T11 F"@FG1kooa66G 11;F; 99 9#**,$$47Q7Q#QZW[F&\-E\4K\K\$++T^^t55$&8 F!..88"%Hs E;c(<V^8dQhRS[S[,S[S[,,S[S[,,RS[S[,S[S[,,S[S[,,R,RS[RS[RS[RS[R,RS[R S[R S[R,R S[R,R S[R,R S[R,RS[R,RS[RS[RS[RS[RS[RS[R,RS[/#)rCrrNrrrrris_split_into_wordsrrreturn_tensorsrrrrrrrrrE) rrrFr,rrrrGr)rHrs"rIrJrs?XX++d9o=EV@WWX004 ?BTJ[E\\_ccX! X * X 0 X$JXX"X $JXDjXt X $d{X $d{X$(X %)!X"!%#X$%X&'X(#Tk)X, -XrLc RpV!V4'g \R4hVeV!V4'g \R4hV'dG\V\\34;'d(T;'d\V^,\\34pM\V\\34pV'd~\V\4'd \ R4hVe>\ V4\ V48wd%\R\ V4 R\ V4 R24hVe\\W44MTpMV'dW3.MV.p\V\\34'g\ R\V4 R24hVPVVVVV V R7Vf VPpVPPV8wdVVPn VPPVVVR 7pVUu.uFpVPVV V VVVVVR 7NK pp/pV^,^,F0pVUUUu.uFwppVV,FpVNK K p pppV VV&K2 VUUUu.uFwppVFpVNK K p!pppV'dA.p"\V4F*wp#wp$pV"V#.\ V$R ,4,, p"K, V"VR &VR ,Fp%VP!V%VV4K \#VV!V R 7p&V'g}V fyV'gq\#V&P%4UU'u/uF?wpp'T\ V'4^8d'\V'^,\4'd V'^,MT'bKA up'pV&P&4p&V&#uupiuupppiuupppiuup'pi)c\V\4'dR#\V\\34'Ed \ V4^8XdR#\V^,\4'dR#\V^,\\34'd\ V^,4^8Xg%\V^,^,\4'dR#\V^,^,\\34'dJ\ V^,^,4^8H;'g&\V^,^,^,\4#R#R#R#)TF)rOrGrFrPr)r$s&rI_is_valid_text_input._is_valid_text_inputs!S!!Ae}--q6Q;!c**!tUm441Q4yA~AaDGS)A)A##AaDGdE];;"1Q47|q0OOJqtAwqz34OO$ rLztext input must be of type `str` (single example), `list[str]` (batch or single pretokenized example) or `list[list[str]]` (batch of pretokenized examples) or `list[tuple[list[str], list[str]]]` (batch of pretokenized sequence pairs).zdwhen tokenizing batches of text, `text_pair` must be a list or tuple with the same length as `text`.zbatch length of `text`: z- does not match batch length of `text_pair`: .z:batch_text_or_text_pairs has to be a list or a tuple (got ))rrrrrr)ris_pretokenized)r-rrrrrrrroverflow_to_sample_mapping) tensor_type)rrOrFrPrG TypeErrorrzipr/rrrr encode_batchrru&_eventual_warn_about_too_long_sequencerrr)(rrrrrrrrrrrrrrrrrrrrrr is_batchedbatch_text_or_text_pairsrr-tokens_and_encodingssanitized_tokensrrrrstacksanitized_encodingsrrtoksrbatched_outputrns(&&&&&&&&&&&&&&&&&&&&, rIrTokenizersBackend._encode_pluss0 ($D))W   )=i)H)HW  #D4-8hhThhjQUVWQX[_afZgFhJ#D4-8J )S))$Tc)n)D .s4yk:I'q*FOEZtC,@'A`d $?H(9':dV $2UDMBBLTRjMkLllmn  ''- 3!1% (  '#'#<#< ?? 0 04H H4HDOO 1OO00 $1/1 $& &  " "!&;&;*C+E'=+ # &  '*1--C&:N&:74DIIqQIQ&:EN$) S !.1ES0DWQdqdq0DS %)+ & )*> ? 9D!*qcC[8I4J.JJ*!@=W 9 :)+66I  7 7 :w W7''79LZhin4=V*'5&:&:&<&< Uc%j1nE!Hd9S9S%(Y^^&<(( NW "OS"s"M%M* M1 AM8 c6<V^8dQhRS[S[,RS[/#)rCr#rE)rFrG)rHrs"rIrJrfs  tCy S rLcVPPe&VPPPV4#RPV4#)Nr>)rrdecoder=)rr#s&&rIconvert_tokens_to_string*TokenizersBackend.convert_tokens_to_stringfsJ%%--9  " " * * 1 1& 9 &! rLc`<V^8dQhRS[S[S[,,RS[RS[R,RS[/#)rC token_idsrclean_up_tokenization_spacesNrEr)rHrs"rIrJrms?))c?)")'+Tk ) )rLc VPRR4\V\4'dV.p\V\4'd VR,pVPP WR7pVeTM VP pV'd\VPP4PR8XdBVP'g0\PRVPP R24V#VPV4pV#)use_source_tokenizerNr)rr z=Ignoring clean_up_tokenization_spaces=True for BPE tokenizer aE. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.)rbrOrrarrrr/rr.rtGclean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_outputr warning_oncer'clean_up_tokenization)rrrrrrs&&&&, rI_decodeTokenizersBackend._decodems  )40 i % %" I i & &!+.I%%i%Y,7 )22 % ( T++112;;uDddd##//01-- 11$7 rLc <V^8dQhRS[S[P,RS[S[R3,RS[R,RS[R,RS[S[R3,/#)rCr7 file_names.rDNr8rE)rGrcrGrPr,)rHrs"rIrJrs^bkk)#s(Od{  t  sCx rLc\V4p\PPY'd VR,MR\,4pVP P V4W%3,pV#)r:r)rGrcrdr=TOKENIZER_FILErsave)rr7rrDr8r$s&&&&& rI_save_pretrained"TokenizersBackend._save_pretraineds[^, o_s22Q__  ##N3"33 rLc \P!VPP44pVP R4pVP R4p Rp VR,R,R8Xd/VR,R&.VR,R&MVR,R,R 8XdoVR,R ,e\VR,R ,p VR,R,V ,^,p VeW9d WZ,p ^VR,R &V R ..VR,R&M?VR,R,R&9d/VR,R&M\ R VR,R, R 24hVeDRVR,9d6VR,R,V9d!WWR,R,,VR,R&\ P!\P!V44p .p VFpVP RR4pVP RR4pVR,R,R 8wd V'gKFVe!VR,V9dW^R,,VR&V P\R'/VB4K VeV PV4VR,R,R8Xd-RV9d&VR,R,eVR,R,VR&VR,R,R8Xd-RV9d&VR,R,eVR,R,VR&VR,R,R 8Xd V eWR&VR,eVR,R,R8XgzVR,R,R8XdRVR,9dx\;QJd*RVR,R,4F 'gK RM RM!RVR,R,44'd"\PP4VR&\ VR,R,,pV!R'RVRV /VBpV P#WVR7V Een\P!V P44pRV 9dV R,FpV R,V,R,pVe"VUu.uFpVP%VV4NK ppVV R,V,R&VF#pV P'V4pVeK\ R 4h VUu.uFpV P'V4NK upV R,V,R!&K R(FSpVV 9gK V V,wppVeVV9d VV,pV P'V4pVf \ R 4hVV.V V&KU V VR&\ P!\P!V44p VP(P+4p\,P.Fp\1VV4fK\1VV4pVeVV9d VV,pVP2P%VR4p\5V\4'd?\VVP6VP8VP:VP<RR"7VV&KVVV&K VP>'dVP>P+4M.pVeVPV4\AV4^8dVVR#&WR$&VPB!R'/VB#uupiuupi \DdIpR%\GT49d3TP R$R4TPB!R'/TBpT TnTuRp?#hRp?ii;i))u Trains a tokenizer on a new corpus with the same defaults (in terms of special tokens or tokenization pipeline) as the current one. Args: text_iterator (generator of `list[str]`): The training corpus. Should be a generator of batches of texts, for instance a list of lists of texts if you have everything in memory. vocab_size (`int`): The size of the vocabulary you want for your tokenizer. length (`int`, *optional*): The total number of sequences in the iterator. This is used to provide meaningful progress tracking new_special_tokens (list of `str` or `AddedToken`, *optional*): A list of new special tokens to add to the tokenizer you are training. special_tokens_map (`dict[str, str]`, *optional*): If you want to rename some of the special tokens this tokenizer uses, pass along a mapping old special token name to new special token name in this argument. kwargs (`dict[str, Any]`, *optional*): Additional keyword arguments passed along to the trainer from the 🤗 Tokenizers library. Returns: [`PreTrainedTokenizerFast`]: A new tokenizer of the same type as the original one, trained on `text_iterator`. r2r3Nr.r/r r0r1r rgz;This method does not support this type of tokenizer (found z-) only BPE, Unigram, WordLevel and WordPiece.rYridrcontinuing_subword_prefixend_of_word_suffixr ByteLevelr9 pretokenizersc3:"TFpVR,R8HxK R#5i)r/r'Nr).0 pretokenizers& rI .s!(X !(K7(XsTFinitial_alphabetr&r\)rtrainerr#zQAttempted to set a token in the post processor that does not exist in the mappingr) single_wordlstriprstriprrrr`r*z7multiple values for keyword argument 'tokenizer_object')r"r#r)rrS)$rjloadsrto_strrbrrgrmrnrRrrQanypre_tokenizers_fastr'alphabetMODEL_TO_TRAINER_MAPPINGtrain_from_iteratorrlrrrrSPECIAL_TOKENS_ATTRIBUTESrvr rOr0r1r2rrr`rr'rrG)r text_iteratorr&rnew_special_tokensspecial_tokens_maprrr2r3rYrrr\ added_tokenrr trainer_classr/trained_tokenizer_jsonrr#rrrspecial_token_fullr`r new_tokenizers&&&&&&, rItrain_new_from_iterator)TokenizersBackend.train_new_from_iteratorsDDOO$:$:$<=%)).9 '++,<= ' "6 *e 3/1N7 #G ,02N7 #H - G $V , 9g&x0<'0:*73G4D3Ew'0 G $V ,0J J/1N7 #G ,Mn]dNeflNmMno>>   *~g66w' 48JJ3EU\F]^iFj3kN7 #K 0!**4::n+EF 'K!ooi6Gd+Ag&v.);G!-+i2HL^2^); 1)*:;;C+,<=cB8LF)5TZ![TZ5"4"8"8"FTZ![FLN#34S9(C!'#,#8#8#?#+", s#"(ouCuntejIDYDYZ_D`ntCuN#34S9%@<"0  N2-m>+F+ +{"\Dvj HCPQFR -t4 $ 8 8 +4 ($$ s0Z Z$ Z)) [<4<[70[<6[77[<c aa^RIo^RIHp ^RIHp ^RIHp HoV !^R7RVV3Rll4pV'g\4'dRpVEeV'gV'EgpV!V4'EdaV !VR VVVR R VR 7pR pVe\VR R 7;_uu_4p\P!V4pRRR4XPR4pVPR4pV'd<V PV4V PR48dV'dVe VR9dV#M/V'd(V PV4V PR48dV#RpV'gV'EgpV!V4'EdaV'dRV9d\VRVR,4V f<\VRR 4'g)\VRR 4\ P#RV R24V#V RJg\VRR 4'd\VRR4^RIpVP&P)VP+R4RR7pVP,p\/VVP&P04'dVVP,^&V#\/VVP&P24'dVP&P5R R R7pVP&P1VV.4VnV# +'giEL);i)a6 Patches mistral related tokenizers with incorrect regex if detected 1) Local file with an associated config saved next to it >> Model type one of the mistral models (on older versions) 2) Remote models on the hub from official mistral models >> Tags including `base_model:.*mistralai` N) lru_cache)version)rr )maxsizec0V^8dQhR\R\/#)rCmodel_idrE)rGr,)rHs"rIrJ.__annotate__s  c d rLc<S!4PV4pTPe4SPRRP TP44'dR#R# \dR#i;i)Fzbase_model:.*mistralairT) model_infor~tagssearchr=)rJr.r res& rIis_base_mistral?TokenizersBackend._patch_mistral_regex..is_base_mistrals_ ++H5zz%995rwwuzz7JKK   sA A+*A+Tz config.jsonF) cache_dirrlocal_files_only%_raise_exceptions_for_missing_entries'_raise_exceptions_for_connection_errors _commit_hashr+r,transformers_versionrz5.0.0rz$The tokenizer you are loading from 'a' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.z[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\p{Lo}\p{M}]+|[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]+[\p{Ll}\p{Lm}\p{Lo}\p{M}]*|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n/]*|\s*[\r\n]+|\s+(?!\S)|\s+isolated)patternbehavior)r use_regex)rImistral3voxtral ministralpixtral)rP functoolsrF packagingrGtransformers.utils.hubrr rrirjrkrlparsesetattrrvrrrpre_tokenizersSplitRegexrrOr9 Metaspacer')rrpretrained_model_name_or_pathrrSrTrWis_localrrrrFrGrrQ _config_filemistral_config_detectedf_configrXtransformers_model_typersplit_pretokenizercurrent_pretokenizerr rPs&&&&&&&&&&, @@rIr&TokenizersBackend._patch_mistral_regexns* '%> 3      00H ( 4 XX/:W*X*X&-#!16;8=) L', #',99Q"iilG:'.{{3I'J$*1++l*C' (GMM:N,ORYR_R_`gRh,h 3?3  )()gmm?\>]^eeH?'$.')EXZ_2`2`I':DA%)3)B)B)H)H * 0 0s!", *I*& ,5+B+B(!"6 8Q8Q8Z8Z[[5G //2"&&:Jr3r.r classmethodrrr}r)propertyr/r4rArVrrrsetterrtr&rr0rr_added_tokens_encoder_added_tokens_decoderrrrrrrrrrrrrrrrrrrrrr!rCr__static_attributes____classdictcell__ __classcell__)r'rs@@rIr'r'Ws * EJddL!!F99vh)T   ! !B B H 46666%%%%)@GG AA  nn::10nn FF  '' -(-(^ 7766 ??*4vvI9I9\gk#',;,F,F2D2T2T!%$))-#'&*-1-1*/+0',#,0)XXt  ))V"CJBBBrLr')Grwrrjrc collectionsrcollections.abcrshutilrtypingrtokenizers.pre_tokenizersrfr6huggingface_hubrrrrr rr rgtokenizers.decodersr rtokenizers.modelsr r tokenizers.trainersrrrrrcrrrintegrations.ggmlrintegrations.mistral.constantsrmodeling_gguf_pytorch_utilsrtokenization_utils_baserrrrrrrutilsrrr r! get_loggerrtrrSPECIAL_TOKENS_MAP_FILETOKENIZER_CONFIG_FILETIKTOKEN_VOCAB_FILEADDED_TOKENS_FILEr|r8r>r'PreTrainedTokenizerFastrrLrIrs  #$7+-/16*^^.05D=HG   H %"3/'(( : ~!! &~|EXY,-Y/Y.Yz,,rL