+ "ݳj)\^RIt^RIt^RIt^RIHt^RIHtHtHtH t ^RI H t H t H t ^RIHt^RIHt^RIHtHt^R IHtHtHtHtHtHt]P<!]4t ]PB!]PDR 7]#]$]%]&]'](3t)]e ])]3,t)R]*R 3R]*R 3R]+R 3R],R 3.t-!RR4t.R# ]d^RIHtL ]dRtLi;ii;i)N) import_module)AnyListOptionalUnion) AutoTokenizerPreTrainedModelPreTrainedTokenizerBase)get_class_from_dynamic_module)StrictDataclassError)#StrictDataclassFieldValidationError)DEFAULT_PAD_TOKENSMODEL_PAD_TOKEN_MAP) auto_config config_pathcustom_tokenizer_class_refhas_custom_tokenizer_codetokenizer_class_name!tokenizer_special_token_overrides)levelzeDeprecated in 0\.9\.0: BPE\.__init__ will not create from files anymore, try `BPE\.from_file` insteadz?`rope_config_validation` is deprecated and has been removed\..*zinvalid escape sequence '\\p'cDaa]tRt^YtoRtRt]RV3RlRll4t]V3RlRl4t ]R4t ]R4t ]V3R lR l4t RV3R lR llt V3R lRlt]RRl4tV3RlRltRtV3RlRltV3RlRltV3RltRtV3RltRtRtVtV;t#) TokenicercR#N)selfs&T/home/luca/screen-checking/.venv/lib/python3.14/site-packages/tokenicer/tokenicer.py__init__Tokenicer.__init__[s Fc ~<V^8dQhRS[S[S[3,RS[RS[S[S[S[S[3,,,RS[/#)pretrained_model_name_or_pathstrict pad_tokens model_config)rstrr boolrrintr)format __classdict__s"r __annotate__Tokenicer.__annotate__asP11',S2I-I'J11T%S/23 1  1r c 8Vf \R4hVPRR4pVPV4p\P!4;_uu_4VFwr\P !RWR7K \ V\4'dTp \V 4p Me\ V\4'd8VP!V3/VBp \ V \4'dTp M#\R4h\R\V4 R24hVf \W4pVPV4p RRR4X f\PR 4\X 4p \V P W 3/4pV!4pWnWnVP'W#R 7V# +'giLt;i) Nz(>'?3BVXZ [ ! # . FBG' &s C F  F c <V^8dQhRS[/#)r"r#r')r*r+s"rr,r-s<`<`s<`r c \P4\V4pVPRR4\P !V3/VB# \ Edj\T4p\T4pTPT4\T4pTeG\PTT4pTe-\PRTT4TP !T3/TBu#\T4pTeu\PRTT4\Yp4pTPRR4TP !T3RR/TBu# \ d\PRTT4Mi;iTP#RR4'g\%T4'gh\T4pRTR&\PRT4\P !T3/TBu#i;i) fix_mistral_regexTNzGTokenicer: Retrying tokenizer load via `%s.from_pretrained()` for `%s`.zMTokenicer: Retrying tokenizer load via dynamic tokenizer class `%s` for `%s`.r/zXTokenicer: Failed custom tokenizer load via `%s` for `%s`, trying local tokenizer files.FzaTokenicer: Retrying tokenizer load with `trust_remote_code=True` for local custom tokenizer `%s`.)r&_install_tokenizer_compatibility_shimsdict setdefaultrfrom_pretrained_TOKENIZER_LOAD_EXCEPTIONSrupdater_resolve_tokenizer_classr?r@rr pop Exceptionr6r)r#rE load_kwargs overrides retry_kwargstokenizer_cls_namerIcustom_class_refs&, rr<Tokenicer._load_tokenizers88:6l 2D94 ` 001N^R]^ ^)1 `9:WXI ,L    *!56S!T !- ) B B1&! !,NNa*5 )889VgZfgg9:WX +NNg(5 %BBR$rM $$%8$?(889Vjnr~ NNr(52E::B[\yBzBz ,L04L, - NNs- !001N_R^_ _c1 `sDA BF?F?"AD-*F?-"EF?EF?-AF?>F?c#a"\V\4'g.#Vo\FMwrp\;QJdV3RlV4F 'gK RM RM!V3RlV44'gKHW#3xKO R#5i)c3,<"TF qS9xK R#5irr).0part model_paths& r -Tokenicer._warning_filters..s>+$:%+sTFN)r;r' _KNOWN_LOAD_WARNING_SUPPRESSIONSany)r# model_partswarning_categorypatternres& @rr7Tokenicer._warning_filterssX7==I2 6V 2K7s>+>sss>+>>>&//7WsAA7 A7, A7c\P'dR#R\n^RIHp^RIHp\ VR4'g WnR#R# \dR#i;i)NT)bytes_to_unicode)tokenization_gpt2ro)r_compatibility_shims_installed#transformers.convert_slow_tokenizerrotransformers.models.gpt2rphasattrrZ)rorps rrR0Tokenicer._install_tokenizer_compatibility_shimssQ  3 3 3 37 0  L B,.@AA5E2B   s$A AAc&<V^8dQhRS[RS[/#)r"r#r^rO)r*r+s"rr,r-s..'*..r c^RIpVR8Xd\VRR4#\W!R4pVeV#RV9d3VPR^4wrE\V4p\WeR4pVeV#\ V\ 4'd&\PPV4'gR#\P!V4Uu.uFApVPR4'gKVPR4'g VR9gK<VRRNKC ppVFp\V RV 2V4pVu# R# \dRpLi;iuupi \dKCi;i) rNPreTrainedTokenizerPreTrainedTokenizerFast.z.py tokenization_> tiktoken.py) transformersgetattrrsplitrrZr;r'osrGisdirlistdirendswith startswithr ) r#r^r~rI module_name class_namemodulef local_moduless && rrX"Tokenicer._resolve_tokenizer_classsm  !6 6<)BDI I $G  $!  $ $&8&?&?Q&G #K %&{3 'D A ($$7==RWW]]SpEqEqZZ => >zz%  &'ll?&C&CqOG[ AcrF>  )K  ="m1%7$891! %$ )3 % $  %   s<D!9D4D4.D47 D4D9! D10D19 EEc<V^8dQhRS[S[S[S[3,,RS[S[S[S[S[3,,,RS[/#)r" model_or_pathr%r$)rrr'r rr)r()r*r+s"rr,r- sTXpXpc?&: ;<XpT%S/23Xp Xpr cVe\V\4'd!\WPP4pM<\V\ 4'd\ VRR4pM\R\V4 R24hVf \R4hMVPeVPpM\ VPRR4pVe<\ VPRR4f!VPV4VPn R#V'g{\ VPRR4ebVPPVPn VPPVPn \P!R4R#\R 4hVP#V4pW@n VP%V4\ VRR4p\'VR 4;'d/\'VR4;'dWTP(VP39pVf Ve V'dqVP+WBR 7pV'gHVfDVPPe,VPPp\P!R 4Vf \R 4hWPPn VPV4VPn \ VRR4fWTn \P-RV RVPP R24R#)Nconfigz\Tokenicer: Unsupported `model_or_path` type: Expected `str` or `PreTrainedModel`, actual = `r3zETokenicer: Can not retrieve config from the provided `model_or_path`. pad_token_id pad_token eos_token_idzSTokenicer: No model config found; falling back to tokenizer.eos_token as pad_token.r4 bos_token_id)r&r%zTokenicer: Auto model config unable to fix `pad_token`, Use tokenizer.eos_token as pad_tokenpad_token = eos_token, There may be problems with the model during training or inference.It is recommended that you manually pass a `pad_tokens` to `load()`zTokenicer: Model tokenizer requires fixing but we are unable to auto-fix `pad_token`. Please consult model docs and pass `pad_tokens` to `load()` or set `strict`= False.z#Tokenicer: Auto fixed pad_token_id=z (token='z').)r;r'rrBr/r rr5r=r&_token_literal_for_idrrr eos_tokenr?r@r>auto_fix_model_configrtr_auto_map_pad_tokeninfo)rrr%r$r&rhas_invalid_config_pads&&&& rrCTokenicer.auto_fix_pad_token s  $---*=..:Z:Z[ M?;;&}hE  rswyFtGsHHJKLL# !hii$  ,#00  't~~~tL +t~~{DAI 483M3Ml3[0'$...$"O"[26..2M2MDNN//3~~/G/GDNN,NNm @ 66|D ( ""<0|^TB L. 1 W W n5 W W!:!:L>#>#>LNN^ # @'3#$(#=#=l#K < 6 >)5 % 9,yQUQ_Q_QiQiPjjmnor c&<V^8dQhRS[RS[/#)r"token_idreturn)r)r')r*r+s"rr,r-zs11c1c1r cxRpVFDp\VPV R2R4p\VPVR4pWA8XgK<VfKBVu# \VPR/4;'g/pVP4Fp\V\4'dJVFAp\V\ 4'gKVPP V4V8XgK=Vuu# Kb\V\ 4'gKzVPP V4V8XgKVu# VPPV.4#)r_idNspecial_tokens_map)r unk_tokenr bos_token cls_token sep_token mask_token)rrBvaluesr;listr'convert_tokens_to_idsdecode)rrspecial_token_attrsattrattr_id attr_valueritems&& rrTokenicer._token_literal_for_idzs  (DdnnclDAG t)r&rmodel_config_idrtext_model_config attr_namenested_model_configrHs&& rr>$Tokenicer._resolve_text_model_configs   <5D\*  "  !!,0A4H O $ $ )$3$5!!,1B1V ;;>CCc0<V^8dQhRS[S[,/#r"rrr))r*r+s"rr,r-sx}r c(RpVeVPV4pVf0\VPRR4eVPP#\VRR4pVfb\P !VR4eI\P !V4pVPVP .4pVeWeP8XdTpVfVP\4pVf9\VRR4p\V\4'dV'd V^,pV#TpV#)Nr model_typer) _candidate_id_from_tokenizerrrBrrr6tokenrrr;r)rr&r%rr token_tuplerrs&&& rrTokenicer._auto_map_pad_tokens    !<>.. .\<>  $7$;$;J$M$Y-11*=K88+:K:K9LMH#4H4H(H'   <<=OPL  "<FL,--,+A  , r cH\VRR4fP\VPRR4e7VPPVnVPPVnVP 4p\VRR4f%VeVP V4VnW!nR#R#R#)rNr)rrBrr_tokenizer_eos_token_idrrr)rr&tokenizer_eos_token_ids&& rrTokenicer.auto_fix_model_configs < 6 >74>>[ikoCpC|%)^^%=%=L "(,(C(CL %!%!=!=!? < 6 >CYCe%)%?%?@V%WL "(> %Df >r c\<V^8dQhRS[S[S[S[3,,RS[S[,/#)r" token_listr)rrr'r)r)r*r+s"rr,r-s/  tE#s(O7L QYZ]Q^ r cVPP4pVFp\V\4'dVVP V4pVfVPP V4p\V\ 4'dV^8dVu#KlKn\V\ 4'gK^Tu;8:d!\VP48gKVu#K R#r)rB get_vocabr;r'r6rr)len)rrvocabrrs&& rr&Tokenicer._candidate_id_from_tokenizers((*D$$$ 99T?##~~CCDIHh,,Q#O2?,D#&&1+Js4>>7J+J ,Kr c0<V^8dQhRS[S[,/#rr)r*r+s"rr,r-s77#7r c\VPRR4p\V\4'dV'd V^,#R#VeV#\VPRR4#)rNeod_id)rrBr;r)rrs& rr!Tokenicer._tokenizer_eos_token_idsQt~~~tD lD ) )&2<? < <  # t~~x66r c<VR9d\SV`V4#\SV`R4PV4# \d\ST`T4u#i;i)rB>rB)super__getattribute__r)rname __class__s&&rrTokenicer.__getattribute__sa   7+D1 1 27+K8II$O O 27+D1 1 2s7AAc.\VPV4#r)rrB)rrs&&r __getattr__Tokenicer.__getattr__ st~~t,,r c<VR9d\SV`W4#\SV` R4pVe\ W14'd \ W1V4#\SV`W4# \dRpL?i;i)rBN>rBr&)r __setattr__rrrtsetattr)rrvaluerBrs&&& rrTokenicer.__setattr__ sw 0 07&t3 3 0=I  WY%=%=9E2 2w"4// I sA A('A(c (VP!V3/VB#r)rB)rdatarEs&&,r__call__Tokenicer.__call__s~~d-f--r )r&)FNN)NNFr)rA __module__ __qualname____firstlineno__rrq classmethodrL staticmethodr<r7rRrXrCrr>rrrrrrrr__static_attributes____classdictcell__ __classcell__)rr+s@@rrrYs &+"111f<`<`|00"..`XpXpt11BB@?  77 2- 0..r r)zopt-125)zQwen3-Omni-30B-A3B-Instruct)zLongCat-Flash-Chat)zOpenCoder-8B-Instruct)/loggingrr8 importlibrtypingrrrrr~rr r !transformers.dynamic_module_utilsr huggingface_hub.errorsr rZr constrrutilrrrrrr getLoggerrAr? basicConfigINFOrOSErrorRecursionErrorrr5KeyErrorrVDeprecationWarning FutureWarning SyntaxWarningrhrrr rrs" #-- L$;;   8 $',,'   #!;?S>U!U p )p J #)!$ 0C.C.s$$ g $#$ $s)B11C9C C C CC