+
jE ^ RI HtHtHtHtHt ^ RIHt ^RIH t ^RI
H t ] P ! ]
4 tRRRR/t ! R R
] 4 tR
.tR # ) ) Tokenizerdecoders normalizerspre_tokenizers
processors)BPE)TokenizersBackend)logging
vocab_filez
vocab.json merges_filez
merges.txtc V a a ] tR t^ t oRt]tRR.t]t RV3R lV 3R lllt
Rt Vt V ;t
# )HerbertTokenizera~
Construct a BPE tokenizer for HerBERT (backed by HuggingFace's tokenizers library).
Peculiarities:
- uses BERT's pre-tokenizer: BertPreTokenizer splits tokens on spaces, and also on punctuation. Each occurrence of
a punctuation character will be treated separately.
This tokenizer inherits from [`TokenizersBackend`] which contains most of the methods. Users should refer to the
superclass for more information regarding methods.
Args:
vocab_file (`str`):
Path to the vocabulary file.
merges_file (`str`):
Path to the merges file.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token.
pad_token (`str`, *optional*, defaults to `""`):
The padding token.
mask_token (`str`, *optional*, defaults to `""`):
The mask token.
sep_token (`str`, *optional*, defaults to `""`):
The separator token.
vocab (`str`, `dict` or `list`, *optional*):
Custom vocabulary dictionary.
merges (`str` or `list[str]`, *optional*):
Custom merges list.
input_idsattention_maskc < V ^8 d QhRS[ S[S[ S[3, , R, RS[ S[S[ , , R, RS[ RS[ RS[ RS[ RS[ R S[ R, R
S[ R, / # ) vocabNmerges cls_token unk_token pad_token
mask_token sep_tokenr r )strdictintlist)format
__classdict__s "q/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/herbert/tokenization_herbert.py __annotate__ HerbertTokenizer.__annotate__A s +
+
T#s(^#d*+
d3i$&+
+
+
+
+
+
$J+
4Z+
c
f < Ve TM \ V4 ^ /V n T;' g . V n \ \ V P V P R \ V4 RR7 4 V n \ P ! RRRRR7 V P
n \ P ! 4 V P
n \ P ! RR7 V P
n \ S V `@ ! R RVRVR VR
VR V/V
B \" P$ ! V P& ^3V P( ^ 34 V P
n R # )
Nz)r r dropoutr end_of_word_suffixFT) lowercase
strip_accents
clean_texthandle_chinese_chars)suffixr r r r r )r _vocab_mergesr r
_tokenizerr BertNormalizer
normalizerr BertPreTokenizer
pre_tokenizerr
BPEDecoderdecodersuper__init__r BertProcessingr r post_processor) selfr r r r r r r r r kwargs __class__s &&&&&&&&&&,r r7 HerbertTokenizer.__init__A s % 0es9~q6I || # kk|| i.#)
&1%?%?5TX\&
" )7(G(G(I%"*"5"5V"D
"
*4)B)B
^^Q
^^Q *
&r# )r. r/ r- ) NNzzzzzNN)__name__
__module__ __qualname____firstlineno____doc__VOCAB_FILES_NAMESvocab_files_namesmodel_input_namesr modelr7 __static_attributes____classdictcell__
__classcell__)r< r s @@r r r s3 @ *$&67 E+
+
+
r# r N)
tokenizersr r r r r tokenizers.modelsr tokenization_utils_tokenizersr utilsr
get_loggerr> loggerrC r __all__r, r# r rQ sR T S ! >
H %!< M P
( P
f
r#