+
j ' n R t ^ RIt^RIHtHt ^RIHt ]P ! ]4 t ! R R]4 t
R.t R# )z"Tokenization class for model ByT5.N)
AddedTokenPreTrainedTokenizer)loggingc a a ] tR t^t oRtRR.tRV3R lV 3R lllt]R 4 tR t RV3R lV 3R lllt
V3R
lR lt RV3R lR
llt RV3R lR llt
V3R lR ltR tR tR tRV3R lR lltRtVtV ;t# )
ByT5Tokenizera7
Construct a ByT5 tokenizer. ByT5 simply uses raw bytes utf-8 encoding.
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
extra_ids (`int`, *optional*, defaults to 125):
Add a number of extra ids added to the end of the vocabulary for use as sentinels. These tokens are
accessible as "" where "{%d}" is a number between 0 and extra_ids-1. Extra tokens are
indexed from the end of the vocabulary up to beginning ("" is the last token in the vocabulary
like in ByT5 preprocessing see
[here](https://github.com/google-research/text-to-text-transfer-transformer/blob/9fd7b14a769417be33bc6c850f9598764913c833/t5/data/preprocessors.py#L2117)).
additional_special_tokens (`list[str]`, *optional*):
Additional special tokens used by the tokenizer.
input_idsattention_maskc < V ^8 d QhRR/# ) returnN )format
__classdict__s "k/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/byt5/tokenization_byt5.py __annotate__ByT5Tokenizer.__annotate__< s %
%
%
c
~ < V^ 8 d$ Vf \ V4 Uu. uF pRV R2NK ppMRV^ 8 dL VeH \ V4 ^ 8 d8 \ \ \ R V4 4 4 pW8w d \ RV RV R24 h\ V\ 4 ' d \ VRRR 7 MTp\ V\ 4 ' d \ VRRR 7 MTp\ V\ 4 ' d \ VRRR 7 MTp^ V^V^V/V n \ V P 4 V n RV n
\ S V `0 ! RR
VR VR VR
^ RV/VB R# u upi ) Nz
c 0 \ R \ V 4 9 4 # )extra_id)boolstr)xs &r (ByT5Tokenizer.__init__..J s Ds1v9M4Nr zBoth extra_ids (z!) and additional_special_tokens (zm) are provided to ByT5Tokenizer. In this case the additional_special_tokens must include the extra_ids tokensT)lstriprstrip eos_token unk_token pad_token extra_idsadditional_special_tokens r )
rangelensetfilter
ValueError
isinstancer r _added_tokens_decoderoffset_utf_vocab_sizesuper__init__)
selfr r r! r" r# kwargsi extra_tokens __class__s
&&&&&&, r r/ ByT5Tokenizer.__init__< sw q= 6 >DI)DT(UDTq:aS):DT %(U %
] 8 DMfIgjkIk s6*NPi#j klL( &yk1RSlRm n( ( HRR[]`GaGaJydCgp GQR[]`GaGaJydCgp GQR[]`GaGaJydCgp &'Ay!Y%O"$445 #
'@
' )Vs D:c V P # N)r- )r0 s &r
vocab_sizeByT5Tokenizer.vocab_sizec s ###r c \ V P V P , 4 Uu/ uF qP V4 VbK ppVP V P
4 V# u upi r7 )r% r8 r, convert_ids_to_tokensupdateadded_tokens_encoder)r0 r2 vocabs & r get_vocabByT5Tokenizer.get_vocabg sX ;@ SWS^S^A^;_`;_a++A.1;_`
T../ as A"c p < V ^8 d QhRS[ S[, RS[ S[, R, RS[RS[ S[, /# )r
token_ids_0 token_ids_1Nalready_has_special_tokensr )listintr )r
r s "r r r l sD O O 9O379t3COhlO
cOr c < V' d \ SV ` WRR7 # Vf ^ .\ V4 , ^., # ^ .\ V4 , ^., ^ .\ V4 , , ^., # )ad
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`list[int]`):
List of IDs.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not the token list is already formatted with special tokens for the model.
Returns:
`list[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
T)rB rC rD )r. get_special_tokens_maskr& )r0 rB rC rD r4 s &&&&r rH %ByT5Tokenizer.get_special_tokens_maskl sw $ &72 ']a 3
C#k**qc1 1c+&&1#-!s;7G1GHA3NNr c F < V ^8 d QhRS[ S[, RS[ S[, /# )r
token_idsr rE rF )r
r s "r r r s# 3 3c 3tCy 3r c \ V4 ^ 8 d? VR, V P 8X d' \ P ! RV P R24 V# WP ., # )z.Do not add eos again if user already added it.zThis sequence already has zQ. In future versions this behavior may lead to duplicated eos tokens being added.)r& eos_token_idwarningswarnr )r0 rK s &&r _add_eos_if_not_present%ByT5Tokenizer._add_eos_if_not_present s[ y>A )B-43D3D"D MM,T^^,< =+ +
1 1 22 2r c j < V ^8 d QhRS[ S[, RS[ S[, R, RS[ S[, /# r
rB rC Nr rL )r
r s "r r r s; @ @ 9@379t3C@
c@r c V P .pVf \ W, 4 ^ ., # \ W, V, V, 4 ^ ., # )ay
Create a mask from the two sequences passed to be used in a sequence-pair classification task. ByT5 does not
make use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`list[int]`):
List of IDs.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`list[int]`: List of zeros.
)rO r& )r0 rB rC eoss &&& r $create_token_type_ids_from_sequences2ByT5Tokenizer.create_token_type_ids_from_sequences sL ! {()QC/ /;${2S89QC??r c j < V ^8 d QhRS[ S[, RS[ S[, R, RS[ S[, /# rU rL )r
r s "r r r s6 - - 9-379t3C-
c-r c b V P V4 pVf V# V P V4 pW, # )a"
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
adding special tokens. A sequence has the following format:
- single sequence: `X `
- pair of sequences: `A B `
Args:
token_ids_0 (`list[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`list[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
)rR )r0 rB rC s &&&r build_inputs_with_special_tokens.ByT5Tokenizer.build_inputs_with_special_tokens s9 & 22;? 66{CK , ,r c 6 < V ^8 d QhRS[ RS[S[ , /# )r
textr )r rE )r
r s "r r r s c d3i r c b VP R4 Uu. uF p\ V4 NK ppV# u upi )zPTake as input a string and return a list of strings (tokens) for words/sub-wordsutf-8)encodechr)r0 r_ r2 tokenss && r _tokenizeByT5Tokenizer._tokenize s/ "&++g"67"6Q#a&"67
8s ,c f \ V4 ^8w d RpV# \ V4 V P , pV# )z0Converts a token (str) in an id using the vocab.N)r& ordr, )r0 tokentoken_ids && r _convert_token_to_id"ByT5Tokenizer._convert_token_to_id s4 u:?H 5zDKK/Hr c < \ WP ,
4 pV# )z=Converts an index (integer) in a token (str) using the vocab.)rc r, )r0 indexri s && r _convert_id_to_token"ByT5Tokenizer._convert_id_to_token s EKK'( r c & RpV Fu pW0P 9 d$ V P V, P R4 pM7W0P 9 d VP R4 pM\ \ V4 .4 pW$,
pKw VP RRR7 pV# )z:Converts a sequence of tokens (string) in a single string.r ra ignore)errors)r+ rb _added_tokens_encoderbytesrh decode)r0 rd bstringri
tok_stringstrings && r convert_tokens_to_string&ByT5Tokenizer.convert_tokens_to_string s E222 ! 7 7 > E Eg N
444 "\\' 2
"CJ< 0
!G 9
r c J < V ^8 d QhRS[ RS[ R, RS[S[ , /# )r
save_directoryfilename_prefixNr )r tuple)r
r s "r r r s- c C$J Z_`cZd r c R# )Nr r )r0 r} r~ s &&&r save_vocabulary ByT5Tokenizer.save_vocabulary s r )r+ r- r, )zzz} N)NFr7 )__name__
__module__ __qualname____firstlineno____doc__model_input_namesr/ propertyr8 r? rH rR rX r\ re rk ro rz r __static_attributes____classdictcell__
__classcell__)r4 r s @@r r r s @ %&67%
%
N $ $
O O8 3 3@ @,- -4
r r ) r rP tokenization_pythonr r utilsr
get_loggerr loggerr __all__r r r r s? ) B
H %N' Nb
r