+
jxq R t ^ RIt^ RIt^ RIHt ^ RIt^RIHt ^RI H
t
H t ^RI H
t
HtHtHt ^RIHtHt ^RIHtHtHtHt ^RIHt ]P6 ! ] 4 t ]! 4 ' d ^ RI t R t R
t R t!R t"R
t#Rt$Rt%Rt&Rt' ! R R]RR7 t(R t)R t*R R lt+R R lt,R R lt-R R lt.R R! lt/R" R# lt0R$ R% lt1]! R)R&7 ] ! R' R(]4 4 4 t2R(.t3R# )*z$
Image/Text processor class for GIT
N)Union) BatchFeature)
ImageInputmake_flat_list_of_images)MultiModalDataProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInput)auto_docstringis_torch_availableloggingrequires_backends)requireszzzzz<0x00>z<0x01>z<0x02>z<0x03>z<0x04>c & ] tR t^8tRRRRR//tRtR# )FuyuProcessorKwargs text_kwargsadd_special_tokensTpaddingF N)__name__
__module__ __qualname____firstlineno__ _defaults__static_attributes__r i/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/fuyu/processing_fuyu.pyr r 8 s $ u
Ir r F)totalc t \ P ! W, 4 P \ P 4 # Nnproundastypeint32)original_coordsscale_hs &&r original_to_transformed_h_coordsr) B #
88O- . 5 5bhh ??r c t \ P ! W, 4 P \ P 4 # r! r" )r' scale_ws &&r original_to_transformed_w_coordsr- G r* r c ^ V ^8 d QhR\ R\ R\ R\ \ , /# ) xy scale_factorreturnfloatlistint)formats "r __annotate__r9 K s/ % u QUVYQZ r c \ \ P ! V ^, .4 V4 ^ , p\ \ P ! V^, .4 V4 ^ , pW4.# r/ r- r# arrayr) )r0 r1 r2 x_scaledy_scaleds &&& r scale_point_to_transformed_imager@ K sM /!a%0A<PQRSH/!a%0A<PQRSH r c
v V ^8 d QhR\ R\ R\ R\ R\ R\ \ , /# )r/ topleftbottomrightr2 r3 r4 )r8 s "r r9 r9 Q sH B B BB%*B38BHMB #YBr c \ \ P ! V ^, .4 V4 ^ , p\ \ P ! V^, .4 V4 ^ , p\ \ P ! V^, .4 V4 ^ , p\ \ P ! V^, .4 V4 ^ , pWVWx.# r; r< ) rB rC rD rE r2
top_scaled left_scaled
bottom_scaled right_scaleds &&&&& r scale_bbox_to_transformed_imagerK Q s 2"((C!G92E|TUVWJ2288TAXJ3G VWXYK4RXXvzl5K\Z[\]M3BHHeai[4I<XYZ[L ] AAr c 0 V ^8 d QhR\ R\ /# r/ promptr3 )str)r8 s "r r9 r9 [ s r c V P \ \ 4 p V P \ \ 4 p V P \
\ 4 p V P \ \ 4 p V # r! ) replaceTEXT_REPR_POINT_OPENTOKEN_POINT_OPEN_STRINGTEXT_REPR_POINT_CLOSETOKEN_POINT_CLOSE_STRINGTEXT_REPR_BBOX_OPENTOKEN_BBOX_OPEN_STRINGTEXT_REPR_BBOX_CLOSETOKEN_BBOX_CLOSE_STRING)rN s &r $_replace_string_repr_with_token_tagsrZ [ sO
^^ 02I
JF
^^ 13K
LF
^^ /1G
HF
^^ 02I
JF Mr c 0 V ^8 d QhR\ R\ /# rM )rO r6 )r8 s "r r9 r9 c s r c . p\ P ! R\ R\ R\ R\
R2 4 pVP
V 4 p\ V4 Fk w rE\ V4 ^ 8X g V\ \ \ \
39 d K2 TP YT^8 ;' d W4^,
, \ \ 39 34 Km V# )zQ
Given a string prompt, converts the prompt into a list of TextTokenConversions.
(|))
recompilerW rY rS rU split enumeratelenappend)rN prompt_text_list
regex_pattern prompt_splitielems & r +_segment_prompt_into_text_token_conversionsrk c s
JJ
" #1%<$=Q?V>WWXYqXrrstM !&&v.L \* t9>T " # # $ &
q5ee\a% 05KMd4e e f
+ r c < V ^8 d QhR\ R\ R\ /# )r/ textr2 r3 )rO r5 )r8 s "r r9 r9 | s& G G G5 GS Gr c N V P R4 p\ V4 ^8X d \ p\ pM \ p\
pV Uu. uF p\
VP 4 4 NK pp\ V4 ^8X d \ V^ , V^, VR7 pMT\ V4 ^8X d. \ V^ , V^, V^, V^, VR7 pM\ R\ V4 24 hRP V Uu. uF p\ V4 NK up4 pW8, V, # u upi u upi )z
Given a bounding box of the fashion 1, 2, 3, 4 | 1, 2 This function is responsible for
converting 1, 2, 3, 4 into tokens of 1 2 3 4 without any commas.
,)r0 r1 r2 )rB rC rD rE r2 zInvalid number of ints: )
rb rd rS rU rW rY r5 stripr@ rK
ValueErrorjoinrO ) rm r2 num_int_strstoken_space_open_stringtoken_space_close_stringnumnum_intsnum_ints_translatedtokenss && r transform_within_tagsr{ | s
::c?L
<A "9 #; "8 #: /;;lsciik"lH;
8} >!PXYZP[jv w X ! = !A;1+%
3CM?CDD
WW*=>*=3c#h*=>
?F " +.F FF < ?s "D 3D"c v V ^8 d QhR\ \ , R\ R, R\ \ , /# )r/ rm
scale_factors torch.Tensorr3 r6 rO )r8 s "r r9 r9 s1 S $~:N SWX[S\ r c j . p\ W4 F w r4\ V4 p\ V4 p. pV F^ pV^, ' d5 \ V^ , VP 4 4 pVP V4 KF VP V^ , 4 K` VP RP
V4 4 K V# )a
This function transforms the prompts in the following fashion:
- and to their respective token mappings
- extract the coordinates from the tag
- transform the coordinates into the transformed image space
- return the prompts with the transformed coordinates and new tags
Bounding boxes and points MUST be in the following format: y1, x1, y2, x2 x, y The spaces
and punctuation added above are NOT optional.
rp )ziprZ rk r{ itemre rs ) rm r} processed_textrN r2 rf transformed_promptsrj prompt_within_tagss && r transform_coordinatesr s N #D 8 5f=FvN)+$DAww%:47LDUDUDW%X"#**+=>#**473
% bgg&9 :; !9 r c V ^8 d QhR\ \ \ \ , , R3, RRR\ \ R, , R\ R\ R, /# )r/ num_real_text_tokensr~ input_stream image_tokens
batch_sizer3 )r r6 r7 )r8 s "r r9 r9 s[ T#Y ?@ tN+,
. r c
\ P ! V^ , ^ , VR, .^ R7 pV^ , ^ , P ^ , V ^ , ^ , , p\ P ! VRV .^ R7 .pV# )a Takes an input_stream tensor of shape B x S x ?. For each subsequence, adds any required
padding to account for images and then unpacks the subsequences to create a single sequence per item in the batch.
Returns a list of tensors, one for each item in the batch.dimN) r )torchcatshape)r r r r subsequence_streamnum_real_tokens
all_bi_streams &&&& r construct_full_unpacked_streamr sw LOA$6 T8J#KQRS"1oa(..q14H4KA4NNOYY 23CO D E1MNM r )backendsc & a a ] tR t^t o]t]RR l4 tV 3R lt] V3R lR l4 t
] RV3R lR ll4 t RV3R lR llt
RV3R lV 3R
llltV3R lR ltV3R
lR lt] V3R lR l4 tRR ltRR ltRR lt] R 4 tRtVtV ;t# )
FuyuProcessorc ^RI Hp VP ! V3/ VB pRVP 9 d VP P R4 V# )z
Override for BC. Fuyu uses TokenizersBackend and requires token_type_ids to be removed from model_input_names
because Fuyu uses mm_token_type_ids instead for multimodal token identification. `
)TokenizersBackendtoken_type_ids) tokenization_utils_tokenizersr from_pretrainedmodel_input_namesremove)clssub_processor_type pretrained_model_name_or_path subfolderkwargsr tokenizers &&&&, r _load_tokenizer_from_pretrained-FuyuProcessor._load_tokenizer_from_pretrained sH G%556S^W]^ y : : : ' ' . ./? @r c < ^
V n VP 4 p^ Vn RV n RV n VR, V n VR, V n \ SV `! WR7 R# )
z |SPEAKER|z |NEWLINE|)image_processorr N) max_tokens_to_generate get_vocab pad_token_id image_tokenimage_newline_tokenimage_token_idimage_newline_idsuper__init__)selfr r r vocab __class__s &&&, r r FuyuProcessor.__init__ s[ &(###%!" &#. #K 0 %k 2
Nr c 0 < V ^8 d QhRS[ S[, /# r/ r3 )r6 r7 )r8
__classdict__s "r r9 FuyuProcessor.__annotate__ s < <c `return_attention_mask=False` is not supported for this model.NFr r ptreturn_tensors
images_kwargsr rm r return_mm_token_type_ids return_text_replacement_offsetsimage_scale_factorsz(?<=))maxsplit input_idsr attention_maskrC ) padding_side
max_lengthimage)
modalitiestext_replacement_offsetsmm_token_type_ids) tensor_typeskip_tensor_conversionr ). NN) r
_merge_kwargsr r init_kwargs
setdefaultrr prepare_inputs_layoutvalidate_inputshasattr_process_imagesgetpopr r get_text_with_replacementsr` rb rd re r r squeezepad_check_special_mm_tokenscreate_mm_token_type_idsitemsunused_input_namesr r )r r rm r
merged_kwargs_processed_imagesimages_replacements text_inputsr r r r samplescaler batch_input_idsbatch_attention_mask split_sample
prompt_inputsplaceholder_inputsdatakvs &&&, r __call__FuyuProcessor.__call__ s $ * **
"&.."<"<
]+667NPTUU ]^ ^ 2AFM- ()= >6;M- ( 3 =AM- ()9 :?CM/ *+; < 55YVY$YRXYq @F@@@ '$0A"B"B484H4H4rS`apSq4r 1 &}599:JDQ '4]'C'G'GHbdi'j $.;M.J.N.N15/ + (8';';Trs"151P1P'2. ' "nnT R]=5Q R 9;B!5 "F#%88K!#LL$(NN<3C$d}UbGc$dM<(A- .2^^LO-l}]jOk-l*'.. !II!3K!@!I=YdKe fln %gaj
-33 !II!34D!Eg!NP]^nPo p$& &gaj (..}[/I/Q/QRS/TU,33MBR4S4[4[\]4^_+ #0 #nn 0 0 /3CEYZ!' # 1 #--dWI-V.:R 67'373P3PQ\]hQi3j /0 3+2!12!% R A$:Q:Q1Q1 RDnUpUpqqq tn Ss :O< PPc < V ^8 d QhRS[ R, RS[S[, S[S[, , S[S[, , /# )r/ r Nrm )r r r
r6 )r8 r s "r r9 r ` s> 1 1T! 1 ++d9o=EV@WW 1r c ( Ve Ve . p\ V\ 4 ' d V.MTp\ V\ \ 34 ' g V.MTp\ W!4 FQ w rV\ V\ 4 ' d V. 8X g V P
V, pVP
V\ , 4 KS MF-f5F}dD00r c < V ^8 d QhRS[ S[S[ , , R, RS[S[, S[S[, , S[S[, , RS[S[, /# )r/ r Nrm r )r r6 r r
r r )r8 r s "r r9 r s] L LT* --4 L ++d9o=EV@WW L )* Lr c Z < \ SV ` ! RR VRV/VB Vf Vf \ R4 hR# R# )r rm Nz+You must provide either `text` or `images`.r )r r rr )r r rm r r s &&&,r r FuyuProcessor.validate_inputs s; CvCDCFC > "* ? K ,6sFUFFUF7JKE 0 0 ? ?eXb ? cM & &}'<')r re r r rs )r r' r r image_tokens_with_newlinesr s &&&, r r !FuyuProcessor.replace_image_token sg %'"q& / 1DEA & - -d.>.> AT.TW[WoWo.o p F ww12U::r c 0 < V ^8 d QhRS[ S[, /# r r )r8 r s "r r9 r s i iDI ir c
. RO# )r
)r
r image_sizesr r r s &r r FuyuProcessor.unused_input_names s hhr c n / pVEe VP R4 ;' g V P P pVR, VR, re. p^.\ V4 , pV EF3 p WY^ , , p
Wi^, , p \ W4 p \ \ V ^ , V , 4 V ^ , 4 p
\ \ V ^, V , 4 V ^, 4 pV P P
\ P ! ^^^WV4 \ P ! ^^^4 \ P ! V
..4 \ P ! V..4 ^ ^ RR7 pVP VR, ^ , ^ , P R , 4 EK6 VP RVRV/4 \ R
/ VB # ) a{
Computes the number of placeholder tokens needed for multimodal inputs with the given sizes.
Args:
image_sizes (`list[list[int]]`, *optional*):
The input sizes formatted as (height, width) per each image.
Returns:
`MultiModalData`: A `MultiModalData` object holding number of tokens per each of the provided
input modalities, along with other useful data.
sizer r T) image_input
image_presentimage_unpadded_himage_unpadded_wimage_placeholder_idr variable_sizedimage_input_idsnum_image_tokensnum_image_patchesr r )r r r1 rd r r7 preprocess_with_tokenizer_infor zerosonestensorre r updater )r r. r vision_datar1
padded_height padded_widthr9 r:
image_sizeheight_scale_factorwidth_scale_factoroptimal_scale_factorr4 r5 model_image_inputs &&, r _get_num_multimodal_tokens(FuyuProcessor._get_num_multimodal_tokens s "::f%BB)=)=)B)BD*.x.$w-< ! !"c+&6 6 )
&3m&C#%1qM%A"'*+>'S$#&s:a=;O+O'PR\]^R_#` #&s:a=;O+O'PR\]^R_#` %)$8$8$W$W % Aq!] Q"'**Q1"5%*\\4D3E2F%G%*\\4D3E2F%G)*%&#' %X %! !''(9:K(LQ(OPQ(R(X(XY[(\]' *(
24DFY[l m n , ,,r c a aa R V 3R llo V 3R loVV V 3R lpVV V 3R lpVfM S P P R, S P P R, 33\ V4 , pM#VP ^, ^8w d \ R4 h\ V4 \ V4 8w d \ R4 h. p\ W4 F& w rgV! Wg4 pV! Wg4 pVP
V4 K( V# )
a.
Transforms raw coordinates detected by [`FuyuForCausalLM`] to the original images' coordinate space.
Coordinates will be returned in "box" format, with the following pattern:
`top, left, bottom, right`
Point coordinates are not supported yet.
Args:
outputs ([`GenerateOutput`]):
Raw outputs from `generate`.
target_sizes (`torch.Tensor`, *optional*):
Tensor of shape (batch_size, 2) where each entry is the (height, width) of the corresponding image in
the batch. If set, found coordinates in the output sequence are rescaled to the target sizes. If left
to None, coordinates will not be rescaled.
Returns:
`GenerateOutput`: Same output type returned by `generate`, with output token ids replaced with
boxed and possible rescaled coordinates.
c < V w r#Vf< SP P R, pSP P R, pMVw rEW58: d W$8: d R# \ WB, WS, 4 # )Nr r g ?)r r1 r )
original_size target_sizer r
max_height max_widthr s && r scale_factor_to_fitGFuyuProcessor.post_process_box_coordinates..scale_factor_to_fit se )MF" ! 1 1 6 6x @
0 0 5 5g > (3%
!f&:z*I,=> >r c t < SP P V4 pSP P V4 pW8H P R R7 ^ , pW8H P R R7 ^ , p\ P ! V4 ' d/ \ P ! V4 ' d V^ , V^ , 3# R# )Tas_tupleNN)r convert_tokens_to_idsnonzeror any)rz start_token end_tokenstart_idend_idstarting_positionsending_positionsr s &&& r find_delimiters_pairHFuyuProcessor.post_process_box_coordinates..find_delimiters_pair s ~~;;KHH^^99)DF"("4!=!=t!=!LQ!O & 0 9 94 9 H K yy+,,;K1L1L*1-/?/BCC r c
x < S
! V \ \ 4 ;pR8w Ed Vw r4WC^, 8w d K- SP P W^, V 4 pS! V4 pV Uu. uF% p^\ \ V4 V, 4 , NK' upw rrR\ V RV RV
RV \ 2
p SP P V 4 R, p SP P V 4 p \ P ! V 4 P V 4 p \ P ! V R V WV^, R .^ 4 p EK3 V # u upi N z, r rU )rW rY r convert_ids_to_tokensr7 r5 rV rX tokenizerV r r> tor )rz rL pairstartendcoordsr crB rC rD rE replacementr_ rP r s && r tokens_to_boxesCFuyuProcessor.post_process_box_coordinates..tokens_to_boxes s7 /8NPghh4 n "
!)# ==fQYQT>UV ,M:OU+Vv!AE!Hu4D0E,E,Ev+V(6 !""5!6se2dV2fXRPUwWkVl m "nn 5 5k B2 F "nn B B; O #ll; 7 : :6 B F6ENKa AR#SUVWM ,Ws +D7c j < S ! V \ \ 4 ;pR8w Ed Vw r4WC^, 8w d K- S
P P W^, V 4 pS ! V4 pV Uu. uF% p^\ \ V4 V, 4 , NK' upw rR\ V RV \ 2p
S
P P V
4 R, p
S
P P V
4 p
\ P ! V
4 P V 4 p
\ P ! V R V WV^, R .^ 4 p EK, V # u upi rb )rS rU r rd r7 r5 rR rT re rV r r> rf r )rz rL rg rh ri rj r rk r0 r1 rl r_ rP r s && r tokens_to_pointsDFuyuProcessor.post_process_box_coordinates..tokens_to_points* s' /8OQijj4 p "
!)# ==fQYQT>UV ,M:;AB6aCa5 0 1116B !""6!7s"QC@U?V W "nn 5 5k B2 F "nn B B; O #ll; 7 : :6 B F6ENKa AR#SUVWM Cs +D0r r zTEach element of target_sizes must contain the size (h, w) of each image of the batchzCMake sure that you pass in as many target sizes as output sequencesr! )r r1 rd r rr r re )
r outputs target_sizesrm rp resultsseqr1 r_ rP s
f&& @@r post_process_box_coordinates*FuyuProcessor.post_process_box_coordinates s * ? 2 2 ! 1 1 6 6x @$BVBVB[B[\cBd egjmnujvvL
"a
' st t w<3|, , bc c W3IC!#,C"3-C NN3 4
r c V P P \ 4 pV Uu. uF' qUWT8H P RR7 ^ , ^, R NK) pp\ R V 4 4 p\
P ! \ V4 V3V P P 4 p\ V4 F* w r\
P ! V4 WR\ V4 13& K, V P ! V3RV/VB # u upi )a
Post-processes the output of `FuyuForConditionalGeneration` to only return the text output.
Args:
generated_outputs (`torch.Tensor` or `np.ndarray`):
The output of the model. The output is expected to be a tensor of shape `(batch_size, sequence_length)`
containing the token ids of the generated sequences.
skip_special_tokens (`bool`, *optional*, defaults to `True`):
Whether or not to remove special tokens in the output. Argument passed to the tokenizer's `batch_decode` method.
**kwargs:
Additional arguments to be passed to the tokenizer's `batch_decode method`.
Returns:
`list[str]`: The decoded text output.
TrS Nc 3 8 " T F p\ V4 x K R # 5ir! )rd ).0ru s & r @FuyuProcessor.post_process_image_text_to_text..i s D*C3c#hh*Cs skip_special_tokens) r rV r rW maxr fullrd r rc r> batch_decode)
r generated_outputsr} r beginning_of_answerru unpadded_output_sequencesmax_lenpadded_output_sequencesri s
&&&, r post_process_image_text_to_text-FuyuProcessor.post_process_image_text_to_textS s #nn B BC] ^ Zk%
YjRU+44d4CAFJL MYj " %
D*CDD"'**c2K.Lg-VX\XfXfXsXs"t 9:FA5:\\#5F #zSzM 2 ; !8lNalekll%
s -C)c T V P P p\ VR ., 4 # )r )r r r6 )r tokenizer_input_namess & r r FuyuProcessor.model_input_namesq s& $ @ @ )_,==>>r )r r r r r )rp rU r! )T)r r r r r valid_processor_kwargs classmethodr r propertyr r r r r r r r rH rv r r r __classdictcell__
__classcell__)r r s @@r r r s 1
O < < or or orb 1 1@ L L!4 !4F; ; i i)-VkZ m< ? ? ?r r )vision)4__doc__r r` typingr numpyr# image_processing_utilsr image_utilsr r processing_utilsr r r r tokenization_utils_baser
r utilsr r
r r utils.import_utilsr
get_loggerr r r rV rX rR rT rW rY rS rU r r r) r- r@ rK rZ rk r{ r r r __all__r r r r s 2 ? D S S *
H % " ! " " # % *% @
@ B 2 GD6"
; h?N h? h?V
r