+
j3 R t ^ RIt^ RIt^ RIHt ^RIHt ^RIHt ]P ! ]
4 t RRRR/t R t
! R
R ]4 tR .tR# ) z Tokenization classes for PhoBERTN)copyfile)PreTrainedTokenizer)logging
vocab_filez vocab.txt merges_filez bpe.codesc \ 4 pV ^ , pV R, F pVP W#34 TpK \ V4 pV# )zy
Return set of symbol pairs in a word.
Word is represented as tuple of symbols (symbols being variable-length strings).
: NN)setadd)wordpairs prev_charchars & q/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/phobert/tokenization_phobert.py get_pairsr ! sH
EEQIR
9#$
JE L c a a ] tR t^1t oRt]tRV 3R lltRV3R lR lltRV3R lV 3R lllt RV3R lR llt
] R 4 t R
t
R tR tR
tR tR tRV3R lR lltR tRtVtV ;t# )PhobertTokenizera
Construct a PhoBERT tokenizer. Based on Byte-Pair-Encoding.
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
Path to the vocabulary file.
merges_file (`str`):
Path to the merges file.
bos_token (`str`, *optional*, defaults to `""`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
When building a sequence using special tokens, this is not the token that is used for the beginning of
sequence. The token used is the `cls_token`.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
sep_token (`str`, *optional*, defaults to `""`):
The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
sequence classification or for a text and a question for question answering. It is also used as the last
token of a sequence built with special tokens.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token which is used when doing sequence classification (classification of the whole sequence
instead of per-token classification). It is the first token of the sequence when built with special tokens.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
mask_token (`str`, *optional*, defaults to `""`):
The token used for masking values. This is the token used when training this model with masked language
modeling. This is the token which the model will try to predict.
c
* < Wn W n / V n ^ V P \ V4 & ^V P \ V4 & ^V P \ V4 & ^V P \ V4 & V P V4 V P P 4 U U u/ uF w rWbK up p V n \ VRR7 ;_uu_ 4 p
V
P 4 P R4 RR pRRR4 X Uu. uF p\ VP 4 RR 4 NK! pp\ \ V\ \ V4 4 4 4 V n / V n \" SV `H ! R
RVRVRVRVR VR
VR V /V
B R# u up p i + ' g i L; iu upi ) utf-8encoding
N bos_token eos_token unk_token sep_token cls_token pad_token
mask_token )r r encoderstr
add_from_fileitemsdecoderopenreadsplittupledictziprangelen bpe_rankscachesuper__init__)selfr r r r r r r r r kwargskv
merges_handlemergesmerge __class__s &&&&&&&&&&, r r3 PhobertTokenizer.__init__d sz %& '( S^$'( S^$'( S^$'( S^$ :&)- ););)=>)=)=>
+
0
0M"'')//5cr:F 19?@%
cr*+@ c&%F *< =>
"
?
0
0@s #
E7#E==%F= F
c j < V ^8 d QhRS[ S[, RS[ S[, R, RS[ S[, /# token_ids_0 token_ids_1Nreturnlistint)format
__classdict__s "r __annotate__ PhobertTokenizer.__annotate__ s; A A 9A379t3CA
cAr c Vf' V P .V, V P ., # V P .pV P .pW1, V, V, V, V, # )a6
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
adding special tokens. A PhoBERT sequence has the following format:
- single sequence: ` X `
- pair of sequences: ` A B `
Args:
token_ids_0 (`list[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`list[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
) cls_token_id sep_token_id)r4 r@ rA clsseps &&& r build_inputs_with_special_tokens1PhobertTokenizer.build_inputs_with_special_tokens se ( %%&48I8I7JJ J ! ! 3&,{:S@@r c p < V ^8 d QhRS[ S[, RS[ S[, R, RS[RS[ S[, /# )r? r@ rA Nalready_has_special_tokensrB )rD rE bool)rF rG s "r rH rI sD X X 9X379t3CXhlX
cXr c < V' d \ SV ` WRR7 # Vf$ ^.^ .\ V4 , , ^., # ^.^ .\ V4 , , ^^., ^ .\ V4 , , ^., # )ad
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`list[int]`):
List of IDs.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not the token list is already formatted with special tokens for the model.
Returns:
`list[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
T)r@ rA rR )r2 get_special_tokens_maskr/ )r4 r@ rA rR r; s &&&&r rU (PhobertTokenizer.get_special_tokens_mask s & &72 ']a 3
31#K 001QC7 7sqcC ,,-A61#K@P:PQUVTWWWr c j < V ^8 d QhRS[ S[, RS[ S[, R, RS[ S[, /# r> rC )rF rG s "r rH rI s; L L 9L379t3CL
cLr c V P .pV P .pVf! \ WA, V, 4 ^ ., # \ WA, V, V, V, V, 4 ^ ., # )a|
Create a mask from the two sequences passed to be used in a sequence-pair classification task. PhoBERT does not
make use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`list[int]`):
List of IDs.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`list[int]`: List of zeros.
)rL rK r/ )r4 r@ rA rN rM s &&& r $create_token_type_ids_from_sequences5PhobertTokenizer.create_token_type_ids_from_sequences si " ! ! s(3./1#5 53$s*S0;>DEKKr c , \ V P 4 # N)r/ r# r4 s &r
vocab_sizePhobertTokenizer.vocab_size s 4<< r c B \ V P 3/ V P B # r\ )r, r# added_tokens_encoderr] s &r get_vocabPhobertTokenizer.get_vocab s DLL>D$=$=>>r c a VS P 9 d S P V, # \ V4 p\ \ VR R 4 VR, R, ., 4 p\ V4 pV' g V# \ VV 3R lR7 pVS P
9 d MVw rV. p^ pV\
V4 8 d VP WX4 p VP W(V 4 T pY(, T8X dN T\
T4 ^,
8 d7 Y(^, , T8X d# TP YV, 4 T^,
pK TP Y(, 4 T^,
pK \ V4 pTp\
V4 ^8X d M\ V4 pEK RP V4 pVR R pVS P V& V# \ d TP Y(R 4 Kt i ; i)Nzc N < SP P V \ R 4 4 # )inf)r0 getfloat)pairr4 s &r &PhobertTokenizer.bpe.. s 1C1CD%PU,1Wr )key@@ r! ) r1 r+ rD r minr0 r/ indexextend
ValueErrorappendjoin)
r4 tokenr r bigramfirstsecondnew_wordijs
f& r bpePhobertTokenizer.bpe s DJJ ::e$ $U|T$s)_R6(9'::;$L$WXFT^^+ "MEHAc$i-
5,A
OOD1I.A7e#CIM(9dq5kV>S OOEN3FA OODG,FA XHD4yA~!$zz$ CRy
5 - " OODH-s 0F% % GGc . p\ P ! RV4 pV F; pVP \ V P V4 P R4 4 4 K= V# )zTokenize a string.z\S+\n? )refindallrq rD r| r* )r4 text split_tokenswordsru s && r _tokenizePhobertTokenizer._tokenize sM
9d+E TXXe_%:%:3%? @ A r c ~ V P P WP P V P 4 4 # )z0Converts a token (str) in an id using the vocab.)r# rg r )r4 ru s &&r _convert_token_to_id%PhobertTokenizer._convert_token_to_id s* || ||'7'7'GHHr c L V P P WP 4 # )z=Converts an index (integer) in a token (str) using the vocab.)r' rg r )r4 rp s &&r _convert_id_to_token%PhobertTokenizer._convert_id_to_token s || ~~66r c d RP V4 P RR4 P 4 pV# )z:Converts a sequence of tokens (string) in a single string.r rm )rt replacestrip)r4 tokens
out_strings && r convert_tokens_to_string)PhobertTokenizer.convert_tokens_to_string# s, XXf%--eR8>>@
r c J < V ^8 d QhRS[ RS[ R, RS[S[ , /# )r? save_directoryfilename_prefixNrB )r$ r+ )rF rG s "r rH rI ( s- . .c .C$J .Z_`cZd .r c \ P P V4 ' g \ P R V R24 R# \ P P Y' d
VR, MR\ R, , 4 p\ P P Y' d
VR, MR\ R, , 4 p\ P P V P 4 \ P P V4 8w dG \ P P V P 4 ' d \ V P V4 Mw\ P P V P 4 ' gI \ VR4 ;_uu_ 4 pV P P 4 pVP V4 RRR4 \ P P V P 4 \ P P V4 8w d \ V P V4 W43# + ' g i Lt; i)zVocabulary path (z) should be a directoryN-r r r wb)ospathisdirloggererrorrt VOCAB_FILES_NAMESabspathr isfiler r( sp_modelserialized_model_protowriter )r4 r r out_vocab_fileout_merge_fileficontent_spiece_models &&& r save_vocabulary PhobertTokenizer.save_vocabulary( su ww}}^,, LL,^,< 'r! )
isinstancer$ r( r% FileNotFoundError UnicodeError Exception readlinesr rfindrr r/ r# ) r4 ffdfnfelineslineTmplineidxr s && r r% PhobertTokenizer.add_from_fileE s a
c!S733r&&r* 4
G==?D**S/Cby !XYY:D!$T\\!2DLL
43
%
c "A!D` abb
cs: C B? C ? C
C C C> C C>-C>)r0 r1 r' r# r r )