+ jX_ jRt^RIt^RIt^RIt^RIt^RIt^RIHt^RIH t ] P!] 4t RRRR/t Rt!R R ]4tR tR t]R ]RRRRRR3 t]P&!RRP)]4,]P*]P,,]P.,4t]P&!R4t]P&!]]P*]P,,]P.,4t]P&!R4tRRltR Rlt!RR4tRtRt R!Rlt!R .t"R#)"z!Tokenization classes for BERTweetN)PreTrainedTokenizer)logging vocab_file vocab.txt merges_file bpe.codesc\4pV^,pVR,FpVPW#34TpK \V4pV#)zy Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). :NN)setadd)wordpairs prev_charchars& s/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/bertweet/tokenization_bertweet.py get_pairsr$sH EEQIR 9#$  JE Lcaa]tRt^4toRt]tRV3Rllt]R4t Rt Rt Rt Rt RtR tR tR tRV3R lR lltRtRtVtV;t#)BertweetTokenizera8 Constructs a BERTweet tokenizer, using Byte-Pair-Encoding. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Path to the merges file. normalization (`bool`, *optional*, defaults to `False`): Whether or not to apply a normalization preprocess. bos_token (`str`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. When building a sequence using special tokens, this is not the token that is used for the beginning of sequence. The token used is the `cls_token`. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the `sep_token`. sep_token (`str`, *optional*, defaults to `""`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. cls_token (`str`, *optional*, defaults to `""`): The classifier token which is used when doing sequence classification (classification of the whole sequence instead of per-token classification). It is the first token of the sequence when built with special tokens. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. mask_token (`str`, *optional*, defaults to `""`): The token used for masking values. This is the token used when training this model with masked language modeling. This is the token which the model will try to predict. c  <^RIHp WnWnW n/Vn^VP\V4&^VP\V 4&^VP\V4&^VP\V4&VPV4VPP4U Uu/uFwrWbK upp Vn \VRR7;_uu_4pVP4PR4RRpRRR4XUu.uFp\!VP4RR4NK! pp\#\%V\'\)V4444Vn/VnW0n\14VnRRR R /Vn\6SV`p!RR VR VR VRVRVRVRV RV RRRRRR/ V BR# \d!\P R4RTnELi;iuupp i +'giL;iuupi))demojizezsemoji is not installed, thus not converting emoticons or emojis into text. Install emoji: pip3 install emoji==0.6.0Nutf-8encoding u’'u…z... normalization bos_token eos_token sep_token cls_token unk_token pad_token mask_tokentoken_type_ids_pattern all_zeros%token_type_ids_include_special_tokensTspecial_tokens_patterncls_double_sep)emojir demojizer ImportErrorloggerwarningrrencoderstr add_from_fileitemsdecoderopenreadsplittupledictziprangelen bpe_rankscacherTweetTokenizertweetPreprocessorspecial_punctssuper__init__)selfrrrrrr r!r"r#r$kwargsrkv merges_handlemergesmerge __class__s&&&&&&&&&&&, rrDBertweetTokenizer.__init__is " &%N%& '( S^$'( S^$'( S^$'( S^$ :&)-););)=>)=)=> + 0 0M"'')//5cr:F19?@% cr*+@c&%F *<=> *!/!1$c5%8  '            " $/ 37 $4  = " NN( "DN  "$? 0 0@s) F,0 G#G  %G3,'GG G0 c,\VP4#N)r=r1rEs&r vocab_sizeBertweetTokenizer.vocab_sizes4<<  rcB\VP3/VPB#rO)r:r1added_tokens_encoderrPs&r get_vocabBertweetTokenizer.get_vocabsDLL>D$=$=>>rcaVSP9dSPV,#\V4p\\VRR4VR,R,.,4p\V4pV'gV#\ VV3RlR7pVSP 9dMVwrV.p^pV\ V48dVPWX4p VPW(V 4T pY(,T8XdNT\ T4^, 8d7Y(^,,T8Xd#TPYV,4T^, pKTPY(,4T^, pK\V4pTp\ V4^8XdM\V4pEK RPV4pVRRpVSPV&V# \dTPY(R4Kti;i)NzcN<SPPV\R44#)inf)r>getfloat)pairrEs&r'BertweetTokenizer.bpe..s1C1CD%PU,1Wrkey@@ r*) r?r9listrminr>r=indexextend ValueErrorappendjoin) rEtokenr r bigramfirstsecondnew_wordijs f& rbpeBertweetTokenizer.bpes DJJ ::e$ $U|T$s)_R6(9'::;$L$WXFT^^+"MEHAc$i- 5,A OOD1I.A7e#CIM(9dq5kV>SOOEN3FAOODG,FAXHD4yA~!$zz$CRy  5 -"OODH-s0F%%GGc VP'dVPV4p.p\P!RV4pVF;pVP \ VP V4PR444K= V#)zTokenize a string.z\S+\n? )rnormalizeTweetrefindallrfrcrqr8)rEtext split_tokenswordsrjs&& r _tokenizeBertweetTokenizer._tokenizesi    &&t,D  9d+E   TXXe_%:%:3%? @ Arc 4VPF%pVPW PV,4pK' VPPV4pRP VUu.uFq@P V4NK up4pVPRR4PRR4PRR4PRR4PR R 4pVPR R 4PR R4PRR4PRR4PRR4PRR4pVPRR4PRR4PRR4PRR4pRP VP 44#uupi)z Normalize a raw Tweet rtzcannot zcan not zn't z n't zn 't zca n'tzcan'tzai n'tzain'tz'm z 'm z're z 're z's z 's z'll z 'll z'd z 'd z've z 've z p . m .z p.m.z p . m z p.m z a . m .z a.m.z a . m z a.m )rBreplacerAtokenizerinormalizeTokenr8)rEtweetpuncttokensrj normTweets&& rru BertweetTokenizer.normalizeTweets^((EMM%)<)>@ rcN<V^8dQhRS[RS[R,RS[S[R3,/#)save_directoryfilename_prefixNreturn.)r2r9)format __classdict__s"r __annotate__BertweetTokenizer.__annotate__/s3((c(C$J(Z_`ceh`hZi(rc  z\PPV4'g\P RV R24R#\ VR/4pV'dV R2MRp\PP WVPRR4,4p\VRR R 7;_uu_4p\VPP4R R 7F%wrxV^8gKVPV R V R24K' RRR4\PP WVPRR4,4p \V RR R 7;_uu_4p V PR\VPP4RR 744RRR4WY3# +'giL;i +'giYY3#;i)z6 Save the vocabulary and merges files to a directory. zVocabulary path (z) should be a directoryvocab_files_names-rrrwrrcV^,#r r+kvs&rr]3BertweetTokenizer.save_vocabulary..>s rRSurr_rtrNrrc3V"TFwrRPV4R,xK! R#5i)rtrN)ri).0 bpe_tokens token_indexs& r 4BertweetTokenizer.save_vocabulary..Fs+/c+J$t++/cs')cV^,#rr+rs&rr]rHs ]_`a]brr+)ospathisdirr/errorgetattrrirZr6sortedr1r4write writelinesr>) rErrrprefixrfrjtoken_id merge_filewriters &&& rsave_vocabulary!BertweetTokenizer.save_vocabulary/smww}}^,, LL,^,<DT#Uq=GGugQxj34$V5WW\\.;L;P;PQ^`k;l2lm *cG 4 4   /5dnn6J6J6LRb/c 5 ''5 45 4 ''s)0FF =F( F% ( F: c \V\4'd4\VRRR7;_uu_4pVPV4RRR4R#VP4pVF]pVP4pVPR4pVR 8Xd \R4hVRVp\VP4VPV&K_ R# +'giR#;i \dpThRp?i\ d\ RT R24hi;i) zY Loads a pre-existing dictionary from a text file and adds its symbols to this instance. rrrNzIncorrect encoding detected in z, please rebuild the datasetrtz5Incorrect dictionary format, expected ' 'r*) isinstancer2r6r3FileNotFoundError UnicodeError Exception readlinesrrfindrgr=r1) rErfdfnfelineslineTmplineidxr s && rr3BertweetTokenizer.add_from_fileMs a   c!S733r&&r*4  G==?D**S/Cby !XYY:D!$T\\!2DLL  43  %   c"A!D` abb cs:CB?C? C CC C>C  C>-C>) r>r?r5r-r1rrrBrAr)FrrzzzrO)__name__ __module__ __qualname____firstlineno____doc__VOCAB_FILES_NAMESrrDpropertyrQrUrqr{rurrrrrr3__static_attributes____classdictcell__ __classcell__)rLrs@@rrr4sl0d*> @!!?*X  +D&I7((<33rrac (?: [<>]? [:;=8] # eyes [\-o\*\']? # optional nose [\)\]\(\[dDpP/\:\}\{@\|\\] # mouth | [\)\]\(\[dDpP/\:\}\{@\|\\] # mouth [\-o\*\']? # optional nose [:;=8] # eyes [<>]? | <3 # heart )u # Capture 1: entire matched URL (?: https?: # URL protocol and colon (?: /{1,3} # 1-3 slashes | # or [a-z0-9%] # Single letter or digit or '%' # (Trying not to match e.g. "URI::Escape") ) | # or # looks like domain name followed by a slash: [a-z0-9.\-]+[.] (?:[a-z]{2,13}) / ) (?: # One or more: [^\s()<>{}\[\]]+ # Run of non-space, non-()<>{}[] | # or \([^\s()]*?\([^\s()]+\)[^\s()]*?\) # balanced parens, one level deep: (...(...)...) | \([^\s]+?\) # balanced parens, non-recursive: (...) )+ (?: # End with: \([^\s()]*?\([^\s()]+\)[^\s()]*?\) # balanced parens, one level deep: (...(...)...) | \([^\s]+?\) # balanced parens, non-recursive: (...) | # or [^\s`!()\[\]{};:'".,<>?«»“”‘’] # not a space or one of these punct chars ) | # OR, the following to match naked domains: (?: (?\s]+>z [\-]+>|<[\-]+z (?:@[\w_]+)z(?:\#+[\w_]+[\w\'_\-]*[\w_]+)z#[\w.+-]+@[\w-]+\.(?:[\w-]\.?)+[\w-]a (?:[^\W\d_](?:[^\W\d_]|['\-_])+[^\W\d_]) # Words with apostrophes or dashes. | (?:[+\-]?\d+[,/.:-]\d+[+\-]?) # Numbers, including fractions, decimals. | (?:[\w_]+) # Words without apostrophes or dashes. | (?:\.(?:\s*\.){1,}) # Ellipsis dots. | (?:\S) # Everything else that isn't whitespace. z(%s)|z([^a-zA-Z0-9])\1{3,}z&(#?(x?))([^&;\s]+);c`VfRp\V\4'dVPW4#V#)Nr)rbytesdecode)rxrerrorss&&&r_str_to_unicoders.${{8,, KrcRaaVV3Rlp\PV\W44#)u Remove entities from text by converting them to their corresponding unicode character. Args: text: A unicode string or a byte string encoded in the given *encoding* (which defaults to 'utf-8'). keep (list): List of entity names which should not be replaced. This supports both numeric entities (`&#nnnn;` and `&#hhhh;`) and named entities (such as ` ` or `>`). remove_illegal (bool): If `True`, entities that can't be converted are removed. Otherwise, entities that can't be converted are kept "as is". Returns: A unicode string with the entities removed. See https://github.com/scrapy/w3lib/blob/master/w3lib/html.py Examples: ```python >>> from nltk.tokenize.casual import _replace_html_entities >>> _replace_html_entities(b"Price: £100") 'Price: \xa3100' >>> print(_replace_html_entities(b"Price: £100")) Price: £100 ```c><VP^4pVP^4'd`VP^4'd\V^4pM \V^ 4p^Tu;8:d^8:dMM]\V34PR4#MAVS9dVP^4#\ P PPV4pVe \V4#S'dR#VP^4# \dRpL:i;i \\3dLAi;i)cp1252Nr) groupintrrrghtmlentitiesname2codepointrZchr OverflowError)match entity_bodynumberkeepremove_illegals& r_convert_entity/_replace_html_entities.._convert_entity<skk!n ;;q>> ;;q>> b1F b1F 6)T) &+228<<* d"{{1~%5599+F   6{"$r7Q7   .  s/C5*C5-C5 D5 DDDD)ENT_REsubr)rxrrrrs&ff& r_replace_html_entitiesrs :8: ::ot'F GGrc4a]tRtRtoRtRRltRtRtVtR#)r@i_a Examples: ```python >>> # Tokenizer for tweets. >>> from nltk.tokenize import TweetTokenizer >>> tknzr = TweetTokenizer() >>> s0 = "This is a cooool #dummysmiley: :-) :-P <3 and some arrows < > -> <--" >>> tknzr.tokenize(s0) ['This', 'is', 'a', 'cooool', '#dummysmiley', ':', ':-)', ':-P', '<3', 'and', 'some', 'arrows', '<', '>', '->', '<--'] >>> # Examples using *strip_handles* and *reduce_len parameters*: >>> tknzr = TweetTokenizer(strip_handles=True, reduce_len=True) >>> s1 = "@remy: This is waaaaayyyy too much for you!!!!!!" >>> tknzr.tokenize(s1) [':', 'This', 'is', 'waaayyy', 'too', 'much', 'for', 'you', '!', '!', '!'] ```c*WnW nW0nR#rO preserve_case reduce_len strip_handles)rErrrs&&&&rrDTweetTokenizer.__init__ss*$*rc \V4pVP'd \V4pVP'd \ V4p\ P RV4p\PV4pVP'g=VUu.uF0p\PV4'dTMVP4NK2 ppV#uupi)z Args: text: str Returns: list(str) A tokenized list of strings; concatenating this list returns the original string if `preserve_case=False` \1\1\1) rrremove_handlesrreduce_lengtheningHANG_RErWORD_RErwr EMOTICON_REsearchr)rErx safe_textrzxs&& rrTweetTokenizer.tokenizexs&d+    !$'D ???%d+DKK 40  *!!!HMN1+,,Q//QQWWY>EN Os C'CrNTFF) rrrrrrDrrr)rs@rr@r@_s &+ rr@cR\P!R4pVPRV4#)zY Replace repeated character sequences of length 3 or greater with sequences of length 3. z (.)\1{2,}rregexcompilerrxpatterns& rrrs#mmL)G ;;y$ ''rcR\P!R4pVPRV4#)z, Remove Twitter username handles from text. zv(?r!sN(  6   H %+;  n3+n3x L $)\  (. A+ ` -- chhw&779PSXS`S`9` a --/ 0mmIu}}uww'>'NO  . /:H@00p("   r