+
j+t R t ^ RIt^ RIt^ RIt^RIHt ^RIHt ^RIH t H
t
H t H t H
t
^RIHt ^RIHtHt ^RIHt ]]]]3, , ]]]]]]3, , , ]]]]]3, , , , ]]]]]]3, , , , t]]R, ]]R, ]]R, ]]R, , , , , , , , t ! R R ] R
R 7 t ! R R
] R
R 7 t ! R R]
R
R 7 t ] ! R R] 4 4 t R R lt R R lt R t!R t"R t#RR lt$R.t%R# )z Processor class for KOSMOS-2.N) BatchFeature)
ImageInput) ImagesKwargsProcessingKwargsProcessorMixin
TextKwargsUnpack)
AddedToken)
BatchEncoding TextInput)auto_docstringc 0 a ] tR t^'t o RtV 3R ltRtV tR# )Kosmos2ImagesKwargsa
bboxes (`Union[list[tuple[int]], list[tuple[float]], list[list[tuple[int]]], list[list[tuple[float]]]]`, *optional*):
The bounding bboxes associated to `texts`.
num_image_tokens (`int`, *optional* defaults to 64):
The number of (consecutive) places that are used to mark the placeholders to store image information.
This should be the same as `latent_query_num` in the instance of `Kosmos2Config` you are using.
first_image_token_id (`int`, *optional*):
The token id that will be used for the first place of the subsequence that is reserved to store image
information. If unset, will default to `self.tokenizer.unk_token_id + 1`.
c Z < V ^8 d Qh/ S[ R, ;R& S[;R& S[R, ;R& # ) Nbboxesnum_image_tokensfirst_image_token_id)
NestedListint)format
__classdict__s "o/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/kosmos2/processing_kosmos2.py __annotate__ Kosmos2ImagesKwargs.__annotate__' s5
*$ N__name__
__module__ __qualname____firstlineno____doc____annotate_func____static_attributes____classdictcell__r s @r r r ' s r r F)totalc 0 a ] tR t^8t o RtV 3R ltRtV tR# )Kosmos2TextKwargsz
add_eos_token (`bool`, defaults to `False`):
Whether or not to include `EOS` token id in the encoding when `add_special_tokens=True`.
c & < V ^8 d Qh/ S[ ;R& # )r
add_eos_token)bool)r r s "r r Kosmos2TextKwargs.__annotate__8 s
r r Nr r& s @r r) r) 8 s r r) c ` a ] tR t^At o RRRRRR^ RRRRR RR
RR RR R/ R
R^@//tV 3R ltRtV tR# )Kosmos2ProcessorKwargs text_kwargsadd_special_tokensTpaddingFstridereturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mappingreturn_token_type_idsverboser+
images_kwargsr c 2 < V ^8 d Qh/ S[ ;R& S[;R& # )r r0 r9 )r) r )r r s "r r #Kosmos2ProcessorKwargs.__annotate__A s "" && r r N)r r r r! _defaultsr# r$ r% r& s @r r/ r/ A sY $ u a ' (% $e #U t U
I r r/ c a a ] tR t^Vt o]tRV 3R llt]RV3R lR ll4 tR t R t
RV3R lR llt RR lt RR lt
]R
4 tV3R lR ltV3R
lR ltRtVtV ;t# )Kosmos2Processorc < RVn RV n RV n RV n RV n RV n RV n RV n R V n R
V n R V n
R V n V P V P V P V P V P
V P V P V P V P V P V P . V n W0n
\ V P 4 Uu. uF! pR
\ V4 P! ^4 R2NK# pp. pV P V, F! pVP# \% VRRRR7 4 K# VP' V4 \( S V `U W4 R# u upi )zv
num_patch_index_tokens (`int`, *optional*, defaults to 1024):
The number of tokens that represent patch indices.
Fzzzzzzz z z T)lstriprstrip
normalizedN)r7 eod_token boi_token eoi_token eoc_token eol_token bop_token eop_token boo_token eoo_token dom_token grd_token
tag_tokensnum_patch_index_tokensrangestrzfillappendr
add_tokenssuper__init__)
selfimage_processor tokenizerrR kwargsxpatch_index_tokens
tokens_to_addtoken __class__s
&&&&* r rY Kosmos2Processor.__init__Z sJ
+0 '!"##"#$#$9&
NN NN NN NN NN NN NN NN NN NN NN
'=#JOPTPkPkJl mJlQ
c!fll1o->a @Jl m
__'999E E$uY^!_ ` : ]+
4 ns 3'E8c p < V ^8 d QhRS[ R, RS[S[S[, , RS[S[, RS[/# )r imagesNtextr] return)r r listr r/ r )r r s "r r Kosmos2Processor.__annotate__ sM J JT!J $y/)J /0 J
Jr c
&
Vf Vf \ R4 hV P ! \ 3RV P P /VB pVR, P RR 4 pVR, P R^@4 pVR, P RR 4 pVR, P RR 4 pVR, R
, p VR, R , p
VR, P
R R 4 p \ 4 p Ve, V P ! V3/ VR, B p
V P V
4 VEe V P W!WVR
7 pV ' dx V' gp \ V\ 4 ' d V P P V 2pM>\ V\ 4 ' d) V Uu. uF qP P V 2NK ppVR, R
, ;' d TVR, R
& Vf T
MR VR, R & Vf T MR VR, R & V P ! RRV/VR, B pV P V4 WR, R
& WR, R & WR, R & VEe! VEe Vf V P P ^, pT p\! V4 ^, p\ \# WwV, 4 4 p^ .^.V, , ^ ., p. p. pV R, p\ V\ 4 ' d V.pV R, .V R& V F pVR V V, VVV, R , pVP% V4 \& P& ! V4 pV' d ^ .V, pV^ .\) V4 \) V4 ,
, ,
pVP% V4 K \ V\ 4 ' Edf \+ \- XP. 4 UUu. uF w ppV\) V4 3NK uppR R7 pV^ , w p p VR, w pp VR, R
, ;' d TVR, R
& R VR, R & V P ! RRVV, ./VR, B p\) VP. ^ , 4 p V V 8w Ed V P P0 R8X d V Uu. uF8 pVV P P2 .V \) V4 ,
, , NK: ppV Uu. uF$ pV^ .V \) V4 ,
, , NK& ppV R, Uu. uF$ pV^ .V \) V4 ,
, , NK& upV R& MV P P0 R8X d V Uu. uF8 pV P P2 .V \) V4 ,
, V, NK: ppV Uu. uF$ p^ .V \) V4 ,
, V, NK& ppV R, Uu. uF$ p^ .V \) V4 ,
, V, NK& upV R& \ V\ 4 ' d* V f& V^ , pV R, ^ , V R& V^ , pV P \5 RVRV R, RV/V R7 4 V # u upi u uppi u upi u upi u upi u upi u upi u upi )Nz*You have to specify either images or text.tokenizer_init_kwargsr9 r r r r0 r+ Fr1 r2 return_tensors)r rf input_idsattention_maskc V R, # r )r^ s &r +Kosmos2Processor.__call__.. s defhdir )keyrightleftimage_embeds_position_mask)data tensor_typer rr )
ValueError
_merge_kwargsr/ r\ init_kwargspop
setdefaultr r[ updatepreprocess_examples
isinstancerT bos_tokenrh unk_token_idr rS rV copylensorted enumeraterm padding_side pad_token_idr
) rZ re rf r]
output_kwargsr r r r+ r1 r2 rl encodingimage_encodings
text_encodingwith_bos start_indeximage_token_ids base_image_embeds_position_maskrm rx
all_input_idstext_idsmaskidxr^
sorted_length_min_len_not_paddedmax_len_paddeds &&&, r __call__Kosmos2Processor.__call__ s >dl IJ J ** "
"&.."<"<
/33HdC( 9==>PRTU ,_ = A ABXZ^ _ %m488 %P
*= 9:N O
.y9&}5@@AQSWX > ! 1 1& [M/D d++FJKd~~ 7 7 8
BH gUZM- ( 3OU~^cgM- ()9 : NN U U
m8T UM OOM *=Om$%9:29m$Y/9Gm$%56 2#+'+~~'B'BQ'F$ *H h-!+K #5)=Vf?f#g hO/0cQC:J4J.JaS.P + I)+ & $[ 1M$$$!.
.67G.H-I)* )#L[1OCh{]mOmOoFpp *yy!@A 3:Ds8}s4y 899*11$7 * $%% &1:=;R;R1ST1SvsAc3q6]1STZi!
)6a(8%%&r*Q!-01EFXX= m,-AB BF
m,-=> $ `T#YK `=Q^C_ `
!$]%<% ? ? ?Ic 3 6 JRRbIc6IcAA~A'> ? ? ?Ic6!12 44>lu$vlughdnn&A&A%BnWZ[\W]F]%^ab%b%blu $vIc6IcAQC>CF#: ;a ? ?Ic 3 6 JRRbIc6IcAQC>CF#: ;a ? ?Ic6!12
$$$ )? %aL -56F-G-J)*-G-J*
OO #Y((3C*D46P
!/
K Lj U %w66 %w66s0
"Y*8Y/
>Y5*Y:*Y?>Z*Z *Zc $ Vf R# \ V\ 4 ' g \ R4 hV F pVf K \ V\ 4 ' g V.pV F p\ V\ 4 ' d \ V4 ^8X d= \
;QJ d R V 4 F ' d K RM RM
! R V 4 4 ' d Ke \ V4 ^8X d= \
;QJ d R V 4 F ' d K RM RM
! R V 4 4 ' d K \ R4 h K R# )a
Check `bboxes` for a single text example. It could be
- `None`: no bounding box associated to a text.
- A list with each element being the bounding boxes associated to one ` ... ` pair found
in a text. This could be:
- `None`: no bounding box associated to a ` ... ` pair.
- A tuple of 2 integers: A single bounding box specified by patch indices.
- A tuple of 4 float point number: A single bounding box specified by (normalized) coordinates.
- A list containing the above 2 tuple types: Multiple bounding boxes for a
` ... ` pair.
Nz@`bboxes` (for a single text example) should be `None` or a list.c 3 B " T F p\ V\ 4 x K R # 5iN)r r .0r^ s & r AKosmos2Processor._check_bboxes_for_single_text..1 s .S7az!S/A/A7 FTc 3 B " T F p\ V\ 4 x K R # 5ir )r floatr s & r r r 2 s 1XPW1*Q2F2FPWr a' Each element in `bboxes` (for a single text example) should be either `None`, a tuple containing 2 integers or 4 float point numbers, or a list containing such tuples. Also make sure the arguments `texts` and `bboxes` passed to `preprocess_text` are both in batches or both for a single example.)r rh r{ tupler all)rZ r bboxelements && r _check_bboxes_for_single_text.Kosmos2Processor._check_bboxes_for_single_text s > FD)) _` ` D| d++ v !'511\Q&33.S7.S333.S7.S+S+SG )cc1XPW1Xccc1XPW1X.X.X$@ r c ^ VP 4 pVe V RV 2pV P W4 pV# )N )strip_insert_patch_index_tokens)rZ rf imager img_info_tokenss &&&&&r _preprocess_single_example+Kosmos2Processor._preprocess_single_example; s; zz| %&av.D ..t< r c < V ^8 d QhRS[ S[S[ , , RS[R, RS[RS[R, RS[S[S[, , /# )r textsre Nr r rg )r rh r BboxInputr rT )r r s "r r ri E s^ @ @4 ?*@ T!@ @
* @
tCy
@r c V P .V, pRP V P .V, V P ., 4 pRp\ V\ 4 ' d RpV.pVf R.\ V4 , pM\ V\ 4 ' g V.p\ V4 \ V4 8w d% \ R\ V4 R\ V4 R24 hV' g V P V4 V.pMSVe= \ V\ 4 ' g \ R4 hV F pV P V4 K MR.\ V4 , p\ V4 \ V4 8w d% \ R \ V4 R\ V4 R24 h\ WV4 U U
U u. uF w rp V P WW4 NK p p
p p V' g
V ^ , p V # u up p
p i )
a Add image and bounding box information to `texts` as image and patch index tokens.
Args:
texts (`Union[TextInput, list[TextInput]]`): The texts to be processed.
images (`ImageInput`, *optional*): The images associated to `texts`.
bboxes (`Union[list[tuple[int]], list[tuple[float]], list[list[tuple[int]]], list[list[tuple[float]]]]`, *optional*):
The bounding bboxes associated to `texts`.
num_image_tokens (`int`, *optional*, defaults to 64):
The number of image tokens (used as latent queries). This should corresponds to the `latent_query_num`
attribute in `Kosmos2Config`.
Returns:
`Union[TextInput, list[TextInput]]`: The processed texts with image and patch index tokens.
r TFNzGThe number of examples in `texts` and `images` should be the same. Got v.s. instead.zS`bboxes` should be `None` or a list (as a batch) when `texts` is passed as a batch.zGThe number of examples in `texts` and `bboxes` should be the same. Got ) rG joinrH r rT r rh r{ r zipr )
rZ r re r r
img_tokensr batchedr^ rf r r results
&&&&& r r $Kosmos2Processor.preprocess_examplesE s , nn%(88
((DNN#3j#@DNNCS#ST eS ! !GGE >Vc%j(FFD)) XF u:V $ YZ]^cZdYeeklopvlwkx yB C
. .v 6 XF
fd++ !vww2215 Vc%j(F v;#e* $ YZ]^cZdYeeklopvlwkx yB C
&)%?
%?!T
+ +D O%?
AYF
s Gc p VP V P 4 R, pV' d \ V4 # V# rp )splitrH +clean_text_and_extract_entities_with_bboxes)rZ rf cleanup_and_extractcaptions &&& r post_process_generation(Kosmos2Processor.post_process_generation s, **T^^,R0 >wG Gr c z V P ! V3RV/VB pV Uu. uF qPP VRR7 NK up# u upi )ah
Post-process the output of the model to decode the text.
Args:
generated_outputs (`torch.Tensor` or `np.ndarray`):
The output of the model `generate` function. The output is expected to be a tensor of shape `(batch_size, sequence_length)`
or `(sequence_length,)`.
skip_special_tokens (`bool`, *optional*, defaults to `True`):
Whether or not to remove special tokens in the output. Argument passed to the tokenizer's `batch_decode` method.
**kwargs:
Additional arguments to be passed to the tokenizer's `batch_decode method`.
Returns:
`list[str]`: The decoded text.
skip_special_tokensF)r ) batch_decoder )rZ generated_outputsr r] generated_textsrf s &&&, r post_process_image_text_to_text0Kosmos2Processor.post_process_image_text_to_text sJ ++,=qSfqjpqZijZiRV,,Tu,MZijjjs 8c z V P P pV P P pW, R ., # )rx )r\ model_input_namesr[ )rZ tokenizer_input_namesimage_processor_input_namess & r r "Kosmos2Processor.model_input_names s6 $ @ @ &*&:&:&L&L#$BFbEcccr c | < V ^8 d QhRS[ RS[S[S[, , S[S[S[, , , RS[ /# )r rf r rg )rT rh r r r )r r s "r r ri s= + +s +Ds.+?)stringzuThe number of elements in `bboxes` should be the same as the number of ` ... ` pairs in `text`. Got r r c 3 ( " T F qR Jx K
R # 5ir r )r boxs & r r >Kosmos2Processor._insert_patch_index_tokens.. s 7$3$$s FTzTThe multiple bounding boxes for a single phrase should not contain any `None` value.r z z )
r rh refinditerr{ r spanrV r r r #_convert_bbox_to_patch_index_tokensr )rZ rf r matched_phrasescurr_posbuffermatchedr r endpatch_index_stringsr
patch_index_1
patch_index_2 position_strs &&& r r +Kosmos2Processor._insert_patch_index_tokens s >S[A -K r{{+B4 PQ 3v; . H IL M\ I] H^ ^d eh io ep dq qz {
9MG \\^FA MM$, -H|$&& v"$ 37$73337$777 j /3/W/WX[/\,
#**m_Am_+MN &'1,=BBCVWL MMIl^:> ?/ :2 c$i MM$y/ *wwv r c < V ^8 d QhRS[ S[S[3, S[ S[S[S[S[3, , RS[ S[S[3, /# )r r rg )r r r rT )r r s "r r ri s@ #s(OeE5%,F&GG sCx r c \ V4 ^8X d Vw r#M6\ \ P ! V P 4 4 p\ W4 w r#R\
V4 P ^4 R2pR\
V4 P ^4 R2pWV3# )r rA rB )r r mathsqrtrR coordinate_to_patch_indexrT rU )rZ r idx_1idx_2num_patches_per_sidetoken_1token_2s && r r 4Kosmos2Processor._convert_bbox_to_patch_index_tokens s t9> LE5 $'tyy1L1L'M#N 4TPLE!#e*"2"21"5!6a8!#e*"2"21"5!6a8 r )
rG rM rK rO rI rF rH rJ rN rL rP rR rQ )i )NN)NN@ )T)r r r r! r/ valid_processor_kwargsrY r r r r r r r propertyr r r r$ r%
__classcell__)rb r s @@r r> r> V s 3/5b J J JX!F@ @D k& d d
+ +Z r r> c V ^8 d QhR\ \ \ \ \ 3, R\ R\ \ \ 3, /# )r r r rg )r r r )r s "r r r s= E%u*D$E ]` ejknpskset r c x V w r#rEWB8 d WS8 g \ R4 h\ P ! W!, 4 p\ P ! W1, 4 p\ P ! WA, ^,
4 p\ P ! WQ, ^,
4 p Wq, V, p
W, V, p W3# )a Convert a bounding box to a pair of patch indices.
Args:
bbox (`tuple[float, float, float, float]`):
The 4 coordinates of the bounding box, with the format being (x1, y1, x2, y2) specifying the upper-left and
lower-right corners of the box. It should have x2 > x1 and y2 > y1.
num_patches_per_side (`int`): the number of patches along each side.
Returns:
`tuple[int, int]`: A pair of patch indices representing the upper-left patch and lower-right patch.
zTThe coordinates in `bbox` should be `(x1, y1, x2, y2)` with `x2 > x1` and `y2 > y1`.)r{ r floorceil) r r x1y1x2y2ul_xul_ylr_xlr_yul_idxlr_idxs && r r r s RR Gopp ::b/ 0D ::b/ 0D 99R.2 3D 99R.2 3D
(4
/F
(4
/F >r c < V ^8 d QhR\ R\ R\ /# )r r r r )r )r s "r r r s! ( (c (3 (c (r c RV, pW, pW, pW, pW, pW8X d0 WC, pWS, p Wc, V, p
Ws, V, p MWF8X g WW8X d0 WC, pWS, p Wc, V, p
Ws, V, p MXWC, V^, , pWS, V^, , p Wc, V^, , p
Ws, V^, , p WW3# )ak
Given a grid of length `num_patches_per_side` and the indices of the upper-left and lower-right corners of a
bounding box, returns the normalized coordinates of the bounding box, in the form (x1, y1, x2, y2).
Args:
ul_idx (`int`): the index of the grid cell that corresponds to the upper-left corner of the bounding box.
lr_idx (`int`): the index of the grid cell that corresponds to the lower-right corner of the bounding box.
num_patches_per_side (`int`): the number of patches along each side.
Returns:
`tuple[float]`: the normalized coordinates of the bounding box, in the form (x1, y1, x2, y2).
g ?r ) r r r cell_sizer r r r r r r r s &&& r patch_index_to_coordinater
s **I (D )D (D )D
)
)
)
)
A
-
A
-
A
-
A
-
2>r c
Rp\ P ! W4 p. pV EF pVP ^4 pVP 4 w rgpV' g3 RpVP ^ 4 ^ , VP ^ 4 ^ , 3pVP R4 p . p
V F p \ P
! RV 4 p \ P
! RV R, 4 p
V ' g KB V
' g KL V' dF V
P
\ V P ^4 4 \ V
P ^4 4 34 K V
P
\ V P ^4 4 \ V
P ^4 4 34 K V' d VP
WuV
34 EKu V
F. pRV^ , RV^, R2pVP
WV.34 K0 EK V# ) a Extract entities contained in `text`. The bounding bboxes is given in the form of patch indices.
This function is only intended to be used within `clean_text_and_extract_entities_with_bboxes` where further
processing happens, including converting to normalized coordinates and whitespace character cleaning up.
Examples:
```python
>>> text = " An image of a snowman warming himself by a fire."
>>> entities = extract_entities_with_patch_indices(text)
>>> entities
[(' a snowman', (31, 41), [(44, 863)]), (' a fire', (130, 137), [(5, 911)])]
```z(?:(([^<]+)))?Nr@ z:rq NNrA z>r N)r r sub)r rf entity_namestartr adjusted_start adjusted_endadjusted_entitys && r adjust_entity_positionsr$ k s[ & K %T&5\:;Nrvvgr4:67L"\$BCO r c V P 4 p\ V 4 \ V P 4 4 ,
p. pV F w pw rgp\ V4 \ VP 4 4 ,
p \ V4 \ VP 4 4 ,
p
Wc,
V , pWs,
V
,
pVP 4 pVP WVV3V34 K W$3# )z9Remove the spaces around the text and the entities in it.)r r rC rD rV ) rf entitiesnew_textleading_spaces new_entitiesr r r r entity_name_leading_spacesentity_name_trailing_spacess && r _cleanup_spacesr, u s zz|HYT[[]!33NL-5) \e6%(%5K>> text = " An image of a snowman warming himself by a fire."
>>> clean_text, entities = clean_text_and_extract_entities_with_bboxes(text)
>>> clean_text
'An image of a snowman warming himself by a fire.'
>>> entities
[('a snowman', (12, 21), [(0.390625, 0.046875, 0.984375, 0.828125)]), ('a fire', (41, 47), [(0.171875, 0.015625, 0.484375, 0.890625)])]
```r r : r N)r r r r$ r
rV r, ) rf r processed_textr r& itemr r r# r bboxes_in_coordss && r r r s VVGR.N"Ed"K H+ cDG1&?jpqjpbf 5d1gtAwH\ ]jpq*==> , > 44 rs $B) )&r" r r r image_processing_utilsr image_utilsr processing_utilsr r r r r tokenization_pythonr tokenization_utils_baser
r utilsr rh r r r r r r r) r/ r> r r
r r$ r, r __all__r r r r: sC $ 2 % b b - ? # sCx
5ue+ ,-.
4c3h !" 4eUE)* +,-
%$,edlT%$,eVZlI[:[5\&\!]]
^
%,e %"
% -U * K ~ K K \ >(Z7't"*5:
r