+
j r ^ RI t ^ RIt^ RIHtHtHtHtHt ^ RIH t ^RI
H t RRRR/t ! R R
] 4 t
R
.tR# ) N) Tokenizerdecoders normalizerspre_tokenizers
processors)Unigram)TokenizersBackend
vocab_filez spiece.modeltokenizer_fileztokenizer.jsonc r a a ] tR t^!t oRt]tRR.t]t R
V 3R llt
R t R t R V3R lV 3R lllt
R tVtV ;t# )
LasrTokenizeraP
Construct a LASR tokenizer (backed by HuggingFace's *tokenizers* library). Based on
[Unigram](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=unigram#models).
This tokenizer inherits from [`TokenizersBackend`] which contains most of the main methods. Users should
refer to this superclass for more information regarding those methods.
Args:
vocab_file (`str`, *optional*):
[SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that
contains the vocabulary necessary to instantiate a tokenizer.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
extra_ids (`int`, *optional*, defaults to 100):
Add a number of extra ids added to the vocabulary for use as sentinels. These tokens are accessible as
"" where "{%d}" is a number between 0 and extra_ids-1. These tokens can be retrieved by
calling get_sentinel_tokens method and token ids can be by calling get_sentinel_token_ids method
additional_special_tokens (`list[str]`, *optional*):
Additional special tokens used by the tokenizer.
vocab (`str`, `dict` or `list`, *optional*):
Custom vocabulary dict. If not provided, a minimal vocabulary is created using the special tokens.
input_idsattention_maskc
< WPn Ve V U
u. uF p
R\ V
4 9 g K V
NK p p
\ V 4 ^8 d( T\ V4 U u. uF p RV R2NK up ,
pMJV^ 8 d# V\ V 4 8w d \ RV RV R24 hM \ V4 U u. uF p RV R2NK p p T pVe Wpn Mh\ V4 R3\ V4 R3\ V4 R3R.V n \ V^,
RR4 F$ p V P
P
RV R2R34 K& \ \ V P
^RR 7 4 V n Ve& \ P ! V4 V P n \ P ! \ P ! 4 \ P ! RR
R R 7 .4 V P n \$ P ! RR
R R 7 V P n \( S
V `T ! RR
VRVRVRVRV/V B \, P. ! RR.. RORV P0 3.R7 V P n R # u up
i u up i u up i )Nz
zBoth extra_ids (z!) and additional_special_tokens (zm) are provided to LasrTokenizer. In this case the additional_special_tokens must include the extra_ids tokensg ▁F)unk_id
byte_fallbackalwaysT) replacementprepend_schemesplit eos_token unk_token pad_token extra_idsadditional_special_tokens$A)singlepairspecial_tokens)r g )r r z$Br )
_extra_idsstrlenrange
ValueError
_vocab_scoresappendr r
_tokenizerr Precompiled
normalizerr SequenceWhitespaceSplit Metaspace
pre_tokenizerr decodersuper__init__r TemplateProcessing eos_token_idpost_processor)selfr r r _spm_precompiled_charsmapr r vocabr
kwargsx extra_tokensi __class__s &&&&&&&&&, k/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/lasr/tokenization_lasr.pyr5 LasrTokenizer.__init__J s| $ % 0'@['@!LTWXYTZDZAA'@L[< 1$)yIY-ZIYA
1#Q.?IY-ZZ)Q 9L0A#A &yk1RSlRm n 8=Y7GH7G!j1 -7GLH(4 % !& Y %Y %Y % "D 9q="b1""))Zs!+-**+ *
&k \-Z Is I I
I I
c T \ \ \ R V P 4 4 4 # )zQGet the list of sentinel tokens (extra_id tokens) from additional_special_tokens.c F \ \ P ! R V 4 4 RJ# )zN)boolresearch)r= s &rA 3LasrTokenizer.get_sentinel_tokens.. s bii0A1&E!Fd!R )listsetfilterr )r9 s &rA get_sentinel_tokens!LasrTokenizer.get_sentinel_tokens s& RTXTrTrs t
rJ c h V P 4 Uu. uF qP V4 NK up# u upi )z&Get the token IDs for sentinel tokens.)rN convert_tokens_to_ids)r9 tokens & rA get_sentinel_token_ids$LasrTokenizer.get_sentinel_token_ids s. ?C?W?W?YZ?Ye**51?YZZZs /c
f < V ^8 d QhRS[ S[S[ , , RS[RS[R, RS[RS[/# ) token_idsskip_special_tokens clean_up_tokenization_spacesN group_tokensreturn)intrK rE r& )format
__classdict__s "rA __annotate__LasrTokenizer.__annotate__ sI
c?
"
'+Tk
rJ c " < \ V\ 4 ' d V.pV' d, \ P ! V4 Uu. uF qf^ , NK
ppV Uu. uF qwV P 8w g K VNK pp\
SV ` ! RRVRVRV/VB # u upi u upi )r rW rX rY r$ )
isinstancer\ itertoolsgroupby pad_token_idr4 _decode) r9 rW rX rY rZ r< token_grouprR r@ s &&&&&, rA rf LasrTokenizer._decode s i % %" I ;D;L;LY;WX;WKQ ;WIX )2P ud>O>O5OUU Pw
3
*F
Y Qs BB )B )r% r, r* )r zzNd NNN)FNT)__name__
__module__ __qualname____firstlineno____doc__VOCAB_FILES_NAMESvocab_files_namesmodel_input_namesr modelr5 rN rS rf __static_attributes____classdictcell__
__classcell__)r@ r^ s @@rA r
r
! sE "H *$&67 EK
Z
[
rJ r
)rc rF
tokenizersr r r r r tokenizers.modelsr tokenization_utils_tokenizersr ro r
__all__r$ rJ rA rz sF * S S % > ">3CEUV U
% U
p
rJ