+
j R t ^ RIHtHtHtHtHtHt ^ RIH t ^RI
H t ^RI H
t
]
P ! ]4 tRR/t ! R R] 4 tR.tR # )
z Tokenization classes for XGLM.)Regex Tokenizerdecoders normalizerspre_tokenizers
processors)Unigram)TokenizersBackend)loggingtokenizer_fileztokenizer.jsonc V a a ] tR t^ t oRt]tRR.t]t RV3R lV 3R lllt
Rt Vt V ;t
# )
XGLMTokenizera
Construct a XGLM tokenizer (backed by HuggingFace's tokenizers library). Based on BPE.
This tokenizer inherits from [`TokenizersBackend`] which contains most of the main methods. Users should
refer to this superclass for more information regarding those methods.
Args:
tokenizer_file (`str`, *optional*):
Path to a tokenizers JSON file containing the serialization of a tokenizer.
bos_token (`str`, *optional*, defaults to `""`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
sep_token (`str`, *optional*, defaults to `""`):
The separator token, which is used when building a sequence from multiple sequences.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token which is used when doing sequence classification.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding.
vocab (`str`, `dict` or `list`, *optional*):
Custom vocabulary dictionary. If not provided, a minimal vocabulary is created.
merges (`list[tuple[str, str]]`, *optional*):
Custom merge rules for BPE. If not provided, merges are generated from the vocabulary.
add_prefix_space (`bool`, *optional*, defaults to `True`):
Whether to add a prefix space before encoding.
input_idsattention_maskc < V ^8 d QhRS[ S[S[S[ S[3, , , R, RS[ RS[ RS[ RS[ RS[ RS[ R S[ R, R
S[/ # ) vocabN bos_token eos_token sep_token cls_token unk_token pad_token_spm_precompiled_charsmapadd_prefix_space)strlisttuplefloatbool)format
__classdict__s "k/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/xglm/tokenization_xglm.py __annotate__XGLMTokenizer.__annotate__> s ?
?
T%U
+,,t3?
?
?
?
?
?
?
$':?
?
c
< ^V n \ V P 4 U u. uF p RV R2NK p p V
P R. 4 ;' g . V
R& T
R;;, V U
u. uF qV
R, 9 g K V
NK up
,
uu& Wn Ve Wn M7\ V4 R3\ V4 R3\ V4 R3\ V4 R3.V n \
\ V P ^RR7 4 V n \ P ! \ R4 R 4 .pVe' VP ^ \ P ! V4 4 \ P ! V4 V P n V ' d R
MR p\ P" ! R VR
7 V P n \& P" ! R VR
7 V P n \* SV `X ! RRVRVRVRVRVRVRV /V
B \. P0 ! V P2 R2V P2 RV P2 R V P2 R2V P4 V P6 3V P2 V P8 3.R7 V P n R# u up i u up
i ) z additional_special_tokensNg F)r unk_id
byte_fallbackz {2,} alwaysneveru ▁) replacementprepend_schemer r r r r r r z $Az $A z $B)singlepairspecial_tokens ) num_madeup_wordsrangegetr _vocabr r r
_tokenizerr Replacer insert PrecompiledSequence
normalizerr Metaspace
pre_tokenizerr decodersuper__init__r TemplateProcessingr r bos_token_id eos_token_idpost_processor)selfr r r r r r r r r kwargsi madeup_wordsword normalizers_r0 __class__s &&&&&&&&&&, r" rC XGLMTokenizer.__init__> sc !" 49$:O:O4PQ4Pq+aS*4P Q.4jj9TVX.Y._._]_*+*+ )0
)T@[9\-\DD\0
+ !1 K Y %Y %Y %Y % DK $G$++aW\$]^ $++E(OSAB $ 0 ;#:#:;T#U V%0%9%9,%G"%57(6(@(@Ucq(r%"*"4"4We"f
.
*4)F)Fnn%S)NN#4'7q 8HL !2!23 !2!23 *
&U R0
s I& I+
2I+
)r9 r8 r r5 ) N