+ j^RIt^RIt^RIHt^RIHt^RIt^RIt^RI H uH t ^RIH t H t ^RIHt^RIHt^RIHtHtHt^RIHt^R IHtHt^R IHt^R IHtH t H!t!^R I"H#t#^R I$H%t%^RI&H't']!4'd^RI(H)t)]!!4'd ^RI*H+t+^RI,H-t-!RR] P\4t/!RR] P\4t0!RR] P\4t1!RR] P\4t2RERRllt3!RR] P\4t4!RR] P\4t5!R R!] P\4t6!R"R#]4t7!R$R%] P\4t8] !R&R'7]!R(R)]444t9RFR*R+llt:R,R-lt;R.R/lt<!R0R1] P\4t=R2R3lt>R4R5lt?!R6R7] P\4t@] !R8R9]44tA!R:R;] P4tC!R<R=] P\4tD!R>R?] P\4tE!R@RA] P\4tF] !RBR'7!RCRD]A44tGR9RD.tHR#)GN)Callable) dataclass)Tensornn)initialization)ACT2FN) ModelOutputis_scipy_availablerequires_backends)GradientCheckpointingLayer)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringis_accelerate_available)merge_with_config_defaults)capture_outputs VideomtConfig)linear_sum_assignment) PartialState)reducecHaa]tRt^3toRtV3RltV3RlRltRtVtV;t #)VideomtPatchEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. c <\SV`4VPVPr2VPVP rT\ V\PP4'dTMW"3p\ V\PP4'dTMW33pV^,V^,,V^,V^,,,pW nW0nW@nW`n \P!WEW3R7Vn R#) kernel_sizestrideN)super__init__ image_size patch_size num_channels hidden_size isinstance collectionsabcIterable num_patchesrConv2d projection)selfconfigr#r$r%r&r+ __class__s&& m/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/videomt/modeling_videomt.pyr"VideomtPatchEmbeddings.__init__:s !'!2!2F4E4EJ$*$7$79K9Kk#-j+//:R:R#S#SZZdYq #-j+//:R:R#S#SZZdYq !!} 1 5*Q-:VW=:XY $$(&))L:icN<V^8dQhRS[PRS[P/#) pixel_valuesreturntorchr)format __classdict__s"r1 __annotate__#VideomtPatchEmbeddings.__annotate__Is#  ELL U\\ r3cDVP^,pW P8wd\RVP RV R24hVPVPP P R7pVP V4P^4P^^4pV#)rzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .dtype) shaper% ValueErrortor-weightrAflatten transpose)r.r6r% embeddingss&& r1forwardVideomtPatchEmbeddings.forwardIs#))!, ,, ,!../yaI  $T__-C-C-I-IJ __\2::1=GG1M r3)r#r%r+r$r- __name__ __module__ __qualname____firstlineno____doc__r"rI__static_attributes____classdictcell__ __classcell__r0r;s@@r1rr3s j   r3rcXaa]tRt^VtoRtV3RlV3RlltRV3RlRlltRtVtV;t #)VideomtEmbeddingszE Construct the CLS token, mask token, position and patch embeddings. c$<V^8dQhRS[RR/#r5r/r7Nr)r:r;s"r1r<VideomtEmbeddings.__annotate__[sNN}NNr3cX<\SV`4WnVPVn\P !\ P!^^VP44Vn \P !\ P!^VPVP44Vn \V4VnVPPp\P !VP"4Vn^VP,Vn\P(!W!P4Vn\P,!\ P.!V4P1R4RR7Vn\P !\ P!^^VP44VnR#)rF) persistentNr)r!r"r/r$r Parameterr9randnr& cls_tokenzerosnum_register_tokensregister_tokensrpatch_embeddingsr+Dropouthidden_dropout_probdropoutnum_prefix_tokens Embeddingposition_embeddingsBufferarangeexpand position_ids mask_token)r.r/r+r0s&& r1r"VideomtEmbeddings.__init__[s   ++ekk!Q8J8J&KL!||EKK6;U;UW]WiWi,jk 6v >++77 zz&"<"<= !"V%?%?!?#%<< =O=O#P IIell;&?&F&Fw&O\ab,,u{{1a9K9K'LMr3cv<V^8dQhRS[PRS[PR,RS[P/#)r5r6bool_masked_posNr7r8)r:r;s"r1r<rYks7ELL5<->ejj%Q[[\^_DT??JGJ^^**:r2> ..55j"bI":":4;L;L"MM YY ZHaP \\*- r3) r`r/rgrorhrdr$rjrnrcNrKrTs@@r1rVrVVs%NN r3rVcPaa]tRt^toV3RlV3RlltV3RlRltRtVtV;t#) VideomtMLPc<V^8dQhRR/#r5r7N)r:r;s"r1r<VideomtMLP.__annotate__s G G$ Gr3c<\SV`4VP;r#\VPVP,4p\ P !W$RR7Vn\VP\4'd\VP,Vn MVPVn \ P !WCRR7Vn R#)TbiasN)r!r"r&int mlp_ratiorLinearfc1r' hidden_actstrr activationfc2r.r/ in_features out_featureshidden_featuresr0s&& r1r"VideomtMLP.__init__s %+%7%77 f0063C3CCD99[E f'' - -$V%6%67DO$//DO99_Fr3cN<V^8dQhRS[PRS[P/#r5 hidden_stater7r8)r:r;s"r1r<r#ELLU\\r3clVPV4pVPV4pVPV4pV#r)rrrr.rs&&r1rIVideomtMLP.forwards2xx - |4 xx - r3)rrr rLrMrNrOr"rIrQrRrSrTs@@r1rrs  G Gr3rcPaa]tRt^toV3RlV3RlltV3RlRltRtVtV;t#)VideomtGatedMLPc<V^8dQhRR/#rr)r:r;s"r1r<VideomtGatedMLP.__annotate__OO$Or3cj<\SV`4VP;r#\VPVP,4p\V^,^, 4^,^,^,p\ P !V^V,RR7Vn\ P !WCRR7VnR#r5TrN r!r"r&rrrr weights_in weights_outrs&& r1r"VideomtGatedMLP.__init__ %+%7%77 f0063C3CCD2Q67!;AAE))K_1D4P99_Nr3cN<V^8dQhRS[PRS[P/#rr8)r:r;s"r1r<r#((ELL(U\\(r3cVPV4pVP^RR7wr#\PP V4V,pVP V4#r5rvr]rchunkr functionalsilurr.rx1x2hiddens&& r1rIVideomtGatedMLP.forwardQ|4 ##A2#.##B'",''r3rrrrTs@@r1rr OO(((r3rcV^8dQhR\PR\PR\PR\PR\PR,R\R\/#) r5modulequerykeyvalueattention_maskNscalingrg)rModuler9rfloat)r:s"r1r<r<sg%% II% <<% % << % LL4' %  %%r3c \P!WPRR44V,pVe W,p\PP VR\P R7PVP4p\PPWVPR7p\P!W4p V P^^4P4p W3#)r)rwrA)ptrainingr]) r9matmulrGrrsoftmaxfloat32rDrArgr contiguous) rrrrrrrgkwargs attn_weights attn_outputs &&&&&&&, r1eager_attention_forwardrs<<}}R'<=GL!#4 ==((2U]](SVVW\WbWbcL==((6??([L,,|3K''1-88:K  $$r3cLaa]tRt^toRtV3RltRV3RlRlltRtVtV;t #)VideomtAttentionz=Multi-headed attention from 'Attention Is All You Need' paperc<<\SV`4WnVPVnVP VnVPVP ,VnVPVP ,VP8wd'\RVP RVP R24hVPR,Vn VPVn RVn \P!VPVP4Vn\P!VPVP4Vn\P!VPVP4Vn\P!VPVP4VnR#)z;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).FNg)r!r"r/r& embed_dimnum_attention_heads num_headshead_dimrCscaleattention_dropoutrg is_causalrrk_projv_projq_projout_projr.r/r0s&&r1r"VideomtAttention.__init__s  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( // ii? ii? ii?  $..$..A r3c <V^8dQhRS[PRS[PR,RS[S[PS[PR,3,/#r5 hidden_statesrNr7)r9rtuple)r:r;s"r1r<VideomtAttention.__annotate__sM!)!)||!) t+!) u||U\\D00 1 !)r3c VPRRp.VORNVPN5pVPV4PV4P ^^4pVP V4PV4P ^^4pVP V4PV4P ^^4p\P!VPP\4p T !TTTTTVPVPVP'gRM VPR7wrV P !.VORN5!P#4p VP%V 4p W3#)z#Input shape: Batch x Time x ChannelN)rrrgr])rBrrviewrGrrr get_interfacer/_attn_implementationrrrrrgryrr) r.rrr input_shape hidden_shapequerieskeysvaluesattention_interfacerrs &&&, r1rIVideomtAttention.forwards8$))#2. 88b8$--8 ++m,11,?II!QO{{=)..|<FFq!L]+00>HHAN(?(M(M KK , ,.E) %8     nnJJ#}}}C$,, % ! "));;;;FFH mmK0 ((r3) r/rgrrrrrrrrrrrKrTs@@r1rrsGB(!)!)!)r3rcPaa]tRt^toV3RlV3RlltV3RlRltRtVtV;t#)VideomtSwiGLUFFNc<V^8dQhRR/#rr)r:r;s"r1r<VideomtSwiGLUFFN.__annotate__rr3cj<\SV`4VP;r#\VPVP,4p\V^,^, 4^,^,^,p\ P !V^V,RR7Vn\ P !WCRR7VnR#rrrs&& r1r"VideomtSwiGLUFFN.__init__rr3cN<V^8dQhRS[PRS[P/#rr8)r:r;s"r1r<rrr3cVPV4pVP^RR7wr#\PP V4V,pVP V4#rrrs&& r1rIVideomtSwiGLUFFN.forwardrr3rrrTs@@r1rrrr3rcjaa]tRtRtoRtR V3RlV3RllltV3RlRltV3RlRltR tVt V;t #) VideomtDropPathi zStochastic depth (DropPath) per sample, for residual blocks. Identity when ``drop_prob`` is 0 or outside training. See `Deep Networks with Stochastic Depth `_. c$<V^8dQhRS[RR/#)r5 drop_probr7N)r)r:r;s"r1r<VideomtDropPath.__annotate__s##%#$#r3c0<\SV`4WnR#r)r!r"r)r.rr0s&&r1r"VideomtDropPath.__init__s "r3cN<V^8dQhRS[PRS[P/#r5rr7r8)r:r;s"r1r<rs#<>S   & $$Q')DM4F4F4J,KK 50C0CML`L`a  M$=>   +m;;r3c <V^8dQhRS[/#)r5r7)r)r:r;s"r1r<r s%%C%r3c RVP 2#)zp=rr.s&r1 extra_reprVideomtDropPath.extra_repr sDNN#$$r3rr) rLrMrNrOrPr"rIrrQrRrSrTs@@r1rr s- ##<<%%%r3rcXaa]tRtRtoRtV3RlV3RlltRV3RlRlltRtVtV;t #) VideomtLayeri$zCThis corresponds to the Block class in the original implementation.c$<V^8dQhRS[RR/#rXr)r:r;s"r1r<VideomtLayer.__annotate__'s66}66r3c.<\SV`4\P!VPVP R7Vn\V4Vn\V4Vn VPR8d\VP4M\P!4Vn\P!VPVP R7VnVP 'd\#V4VnM\'V4Vn\V4VnR#)epsrN)r!r"r LayerNormr&layer_norm_epsnorm1r attentionVideomtLayerScale layer_scale1drop_path_raterIdentity drop_pathnorm2use_swiglu_ffnrmlpr layer_scale2rs&&r1r"VideomtLayer.__init__'s \\&"4"4&:O:OP )&1-f5CICXCX[^C^)>)>?dfdododq\\&"4"4&:O:OP  '/DH!&)DH-f5r3cv<V^8dQhRS[PRS[PR,RS[P/#rr8)r:r;s"r1r<r7s:|| t+  r3c6VPV4pVPW24wrEVPV4pVPV4V,pVP V4pVP V4pVP V4pVPV4V,pV#r)r r!r#r&r'r)r*)r.rrhidden_states_normself_attention_output_ layer_outputs&&& r1rIVideomtLayer.forward7s "ZZ 6#'>>2D#U  $ 1 12G H'<= M zz-0 xx - ((6 ~~l3mC r3)r!r&r#r*r)r r'rrKrTs@@r1rr$s!M66 r3rcPaa]tRtRtoV3RlV3RlltV3RlRltRtVtV;t#)r"iNc<V^8dQhRR/#rr)r:r;s"r1r<VideomtLayerScale.__annotate__Os^^$^r3c<\SV`4\P!VP\ P !VP4,4VnR#r) r!r"rr^layerscale_valuer9onesr&lambda1rs&&r1r"VideomtLayerScale.__init__Os8 ||F$;$;ejjI[I[>\$\] r3cN<V^8dQhRS[PRS[P/#rr8)r:r;s"r1r<r5Ss#++ELL+U\\+r3c&WP,#rr9rs&&r1rIVideomtLayerScale.forwardSsll**r3r=rrTs@@r1r"r"Ns ^^+++r3r"a Class for outputs of [`VideomtForUniversalSegmentationOutput`]. This output can be directly passed to [`~VideomtVideoProcessor.post_process_semantic_segmentation`] or [`~VideomtVideoProcessor.post_process_instance_segmentation`] or [`~VideomtVideoProcessor.post_process_panoptic_segmentation`] to compute final segmentation maps. Please, see [`~VideomtVideoProcessor`] for details regarding usage. ) custom_introcHa]tRtRtoRtRtRtRtRtRt Rt V3Rlt Rt Vt R#)%VideomtForUniversalSegmentationOutputiWa< loss (`torch.Tensor`, *optional*): The computed loss, returned when labels are present. class_queries_logits (`torch.FloatTensor`): A tensor of shape `(batch_size, num_queries, num_labels + 1)` representing the proposed classes for each query. Note the `+ 1` is needed because we incorporate the null class. masks_queries_logits (`torch.FloatTensor`): A tensor of shape `(batch_size, num_queries, height, width)` representing the proposed masks for each query. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Last hidden states (final feature map) of the last layer. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states all layers of the model. attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `tuple(torch.FloatTensor)` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Self and Cross Attentions weights from transformer decoder. NcN<V^8dQh/S[PR,;R&S[PR,;R&S[PR,;R&S[PR,;R&S[S[P,R,;R&S[S[P,R,;R&#)r5Nlossclass_queries_logitsmasks_queries_logitslast_hidden_stater attentions)r9 FloatTensorr)r:r;s"r1r<2VideomtForUniversalSegmentationOutput.__annotate__Ws>   d ")?@ ++d29AB ++d29CD((4/6EF**+d29GHe''(4/6Ir3r)rLrMrNrOrPrCrDrErFrrG__annotate_func__rQrR)r;s@r1rArAWs5&&*D59592659M26JIr3rAcxV^8dQhR\PR\PR\P/#)r5input_featurespoint_coordinatesr7r8)r:s"r1r<r<s0LL5:\\ \\r3c VP4^8XdRpVP^4p\PPP !VRV,R, 3/VBpV'dVP ^4pV#)a A wrapper around `torch.nn.functional.grid_sample` to support 3D point_coordinates tensors. Args: input_features (`torch.Tensor` of shape (batch_size, channels, height, width)): A tensor that contains features map on a height * width grid point_coordinates (`torch.Tensor` of shape (batch_size, num_points, 2) or (batch_size, grid_height, grid_width,: 2)): A tensor that contains [0, 1] * [0, 1] normalized point coordinates add_dim (`bool`): boolean value to keep track of added dimension Returns: point_features (`torch.Tensor` of shape (batch_size, channels, num_points) or (batch_size, channels, height_grid, width_grid): A tensor that contains features for points in `point_coordinates`. Tg@?)rwr{r9rr grid_samplesqueeze)rLrMadd_dimrpoint_featuress&&&, r1 sample_pointrTst(!#-77:XX((44^SK\E\_bEbmflmN'//2 r3c<V^8dQhR\R\R\/#r5inputslabelsr7)r)r:s"r1r<r<s!6r3c8VP4P^4p^\P!WP4,pVP R4R,VP R4R,,p^V^,V^,, , pV#)a A pair wise version of the dice loss, see `dice_loss` for usage. Args: inputs (`torch.Tensor`): A tensor representing a mask labels (`torch.Tensor`): A tensor with the same shape as inputs. Stores the binary classification labels for each element in inputs (0 for the negative class and 1 for the positive class). Returns: `torch.Tensor`: The computed loss between each pairs. r]NNNN)Nr[)sigmoidrFr9rTsum)rWrX numerator denominatorrCs&& r1pair_wise_dice_lossrasr^^  % %a (FELL22I**R.)FJJrN7,CCK A +/2 2D Kr3cxV^8dQhR\PR\PR\P/#rVr8)r:s"r1r<r<s0u||X]XdXdr3cVP^,p\P!RR7pV!V\P!V44pV!V\P !V44p\P !WB, VP4p\P !WR, ^V, P4pWg,pV#)a A pair wise version of the cross entropy loss, see `sigmoid_cross_entropy_loss` for usage. Args: inputs (`torch.Tensor`): A tensor representing a mask. labels (`torch.Tensor`): A tensor with the same shape as inputs. Stores the binary classification labels for each element in inputs (0 for the negative class and 1 for the positive class). Returns: loss (`torch.Tensor`): The computed loss between each pairs. none reduction)rBrBCEWithLogitsLossr9 ones_like zeros_likerr]) rWrXheight_and_width criterioncross_entropy_loss_poscross_entropy_loss_negloss_posloss_negrCs && r1$pair_wise_sigmoid_cross_entropy_lossrps||A$$v6I&vuv/FG&vu/?/?/GH||2EvxxPH||2EF ~~VH  D Kr3caa]tRtRtoRtRV3RlV3Rlllt]P!4V3RlRl4tRt Vt V;t #) VideomtHungarianMatcheriaaThis class computes an assignment between the labels and the predictions of the network. For efficiency reasons, the labels don't include the no_object. Because of this, in general, there are more predictions than labels. In this case, we do a 1-to-1 matching of the best predictions, while the others are un-matched (and thus treated as non-objects). c2<V^8dQhRS[RS[RS[RS[/#)r5 cost_class cost_mask cost_dice num_points)rr)r:r;s"r1r<$VideomtHungarianMatcher.__annotate__s-###27#JO#cf#r3c <\SV`4V^8XdV^8XdV^8Xd \R4hW@nWnW nW0nR#)aCreates the matcher Params: cost_class (`float`, *optional*, defaults to 1.0): Relative weight of the classification error in the matching cost. cost_mask (`float`, *optional*, defaults to 1.0): This is the relative weight of the focal loss of the binary mask in the matching cost. cost_dice (`float`, *optional*, defaults to 1.0): This is the relative weight of the dice loss of the binary mask in the matching cost. num_points (`int`, *optional*, defaults to 12544): No. of points to sample on which the mask loss will be calculated. The same set of K points are uniformly sampled for all prediction and ground truth masks to construct the cost matrix for bipartite matching. zAll costs can't be 0N)r!r"rCrwrtrurv)r.rtrurvrwr0s&&&&&r1r" VideomtHungarianMatcher.__init__sC"  ?yA~)q.34 4$$""r3c <V^8dQhRS[PRS[PRS[PRS[PRS[S[S[,,/#)r5rErD mask_labels class_labelsr7)r9rlistr)r:r;s"r1r<rxs]DD#llD$llD\\ D ll D eFm  Dr3c  .pVP^,p\V4EFpW',PR4pW,p VRWG,3,)p W7,PV 4p V R,p V R,p \P !^VP ^V PR7p V PV P^,^^4p \WRR7P^4p V PV P^,^^4p\WRR7P^4p \W4p\W4pVPV,VPV ,,VPV,,p\P !V\P"!R44p\P$!V\P"!R 44p\P&!V^4p\)VP+44pVP-V4EK VUUu.uFRwpp\P.!V\P0R7\P.!V\P0R73NKT pppV#uuppi) a Params: masks_queries_logits (`torch.Tensor`): A tensor of dim `batch_size, num_queries, num_labels` with the classification logits. class_queries_logits (`torch.Tensor`): A tensor of dim `batch_size, num_queries, height, width` with the predicted masks. class_labels (`torch.Tensor`): A tensor of dim `num_target_boxes` (where num_target_boxes is the number of ground-truth objects in the target) containing the class labels. mask_labels (`torch.Tensor`): A tensor of dim `num_target_boxes, height, width` containing the target masks. Returns: matched_indices (`list[tuple[Tensor]]`): A list of size batch_size, containing tuples of (index_i, index_j) where: - index_i is the indices of the selected predictions (in order) - index_j is the indices of the corresponding selected labels (in order) For each batch element, it holds: len(index_i) = len(index_j) = min(num_queries, num_target_boxes). r[ruF align_cornersg _Br@r]rZg _)rBrangerrDr9r rwrurepeatrTrQrprarurtrvminimumtensormaximum nan_to_numrcpuappend as_tensorint64)r.rErDr|r}indicesr~i pred_probs pred_maskrt target_maskrMtarget_coordinatespred_coordinatesrurv cost_matrixassigned_indicesjmatched_indicess&&&&& r1rIVideomtHungarianMatcher.forwards8*,*//2 z"A-088ejjY_` 36wP]3^ /F< +FLLO+->v||A->@Q&,,q/@QQ R X XY_ `AFL*6<<?*,=fll1o,== >4_,,&PsEc |<V^8dQhRS[RS[S[,RS[S[P,RS[S[S[3,/#)r5rDr}rr7)rr~rnparrayrr)r:r;s"r1r<rsF  $* :>v, QVWYW_W_Q`  c6k  r3c  TpVPwrVp\P!VPR7pVP V4p \ P !\W#4U UU u.uFwp wr{W,NK up pp 4p \ P!WV3VP\ PVPR7p WV &VP^^4pV!W4pRV/pV#uup pp i)a-Compute the losses related to the labels using cross entropy. Args: class_queries_logits (`torch.Tensor`): A tensor of shape `batch_size, num_queries, num_labels` class_labels (`list[torch.Tensor]`): List of class labels of shape `(labels)`. indices (`tuple[np.array])`: The indices computed by the Hungarian matcher. Returns: `dict[str, Tensor]`: A dict of `torch.Tensor` containing the following key: - **loss_cross_entropy** -- The loss computed using cross entropy on the predicted and ground truth labels. )rE) fill_valuerAruloss_cross_entropy) rBrCrossEntropyLossr$_get_predictions_permutation_indicesr9r}rfullrrrurG)r.rDr}r pred_logitsr~ num_queriesr0rkidxtargetrtarget_classes_otarget_classespred_logits_transposedloss_celossess&&&& r1 loss_labelsVideomtLoss.loss_labelss"+ %0%6%6" ''t/@/@A 77@ 99-0-G H-G>66AVYY-G H   %$//]h]o]o /s!,!6!6q!!<2C&0  Is#C# c <V^8dQhRS[PRS[S[P,RS[S[P ,RS[RS[S[S[P3,/#)r5rEr|rrr7) r9rr~rrrrrr)r:r;s"r1r<rs`<<#ll<%,,'<rxx <  < c5<< <r3c  BaSPV4pSPV4pW,pSPV4wrW,pVR,pVR,p\P!4;_uu_4SP VV3RlSP SPSP4p \WRR7P^4p RRR4\VX RR7P^4p R\V X V4R\WV4/p ??V # +'giLL;i)a$Compute the losses related to the masks using sigmoid_cross_entropy_loss and dice loss. Args: masks_queries_logits (`torch.Tensor`): A tensor of shape `(batch_size, num_queries, height, width)`. mask_labels (`torch.Tensor`): List of mask labels of shape `(labels, height, width)`. indices (`tuple[np.array])`: The indices computed by the Hungarian matcher. num_masks (`int)`: The number of masks, used for normalization. Returns: losses (`dict[str, Tensor]`): A dict of `torch.Tensor` containing two keys: - **loss_mask** -- The loss computed using sigmoid cross entropy loss on the predicted and ground truth. masks. - **loss_dice** -- The loss computed using dice loss on the predicted on the predicted and ground truth, masks. Nc&<SPV4#r)calculate_uncertainty)logitsr.s&r1(VideomtLoss.loss_masks..st99&Ar3Fr loss_mask loss_dicerZ) r _get_targets_permutation_indicesrr9rsample_points_using_uncertaintyrwrrrTrQrr)r.rEr|rrsrc_idxtgt_idx pred_masks target_masksr0rM point_labels point_logitsrsf&&&& r1 loss_masksVideomtLoss.loss_maskss4;;GD77@)2 ::;G #,  ( #G, ]]__ $ D DA%%,, ! ( W\]eefghL$J0AQVW__`ab  3L,PYZ ?"JJyaPVP]P]^#FUS 2<@"#:#GH&=jj,W59MSTUVWX"U\\)5::V\VcVc%dd W~-222q9#((2,/JOOPYqrs q % "EJJyQW]WdWd$ef! ! r3c<V^8dQhRS[PRS[PRS[S[P,RS[S[P,RS[S[S[P3,R,RS[S[S[P3,/#)r5rErDr|r}auxiliary_predictionsNr7)r9rr~rr)r:r;s"r1r<r^s55#ll5$ll5%,,' 5 5<<( 5 $C$56= 5 c5<< 5r3c VPWW44pVPWD^,PR7p/VPWWg4CVP W$V4CpVes\ V4FcwrV R,pV R,pVP WW44p V P4U U u/uF wrV RV 2V bK p p p VPV 4Ke V#uup p i)a This performs the loss computation. Args: masks_queries_logits (`torch.Tensor`): A tensor of shape `(batch_size, num_queries, height, width)`. class_queries_logits (`torch.Tensor`): A tensor of shape `(batch_size, num_queries, num_labels)`. mask_labels (`torch.Tensor`): List of mask labels of shape `(labels, height, width)`. class_labels (`list[torch.Tensor]`): List of class labels of shape `(labels)`. auxiliary_predictions (`dict[str, torch.Tensor]`, *optional*): if `use_auxiliary_loss` was set to `true` in [`VideomtConfig`], then it contains the logits from the inner layers of the VideomtMaskedAttentionDecoder. Returns: losses (`dict[str, Tensor]`): A dict of `torch.Tensor` containing three keys: - **loss_cross_entropy** -- The loss computed using cross entropy on the predicted and ground truth labels. - **loss_mask** -- The loss computed using sigmoid cross_entropy loss on the predicted and ground truth masks. - **loss_dice** -- The loss computed using dice loss on the predicted on the predicted and ground truth masks. if `use_auxiliary_loss` was set to `true` in [`VideomtConfig`], the dictionary contains additional losses for each auxiliary predictions. rrErDr0) r get_num_masksrurrrrIitemsupdate)r.rErDr|r}rrrrr aux_outputs loss_dictrrs&&&&&& r1rIVideomtLoss.forward^sH,,3;e&&|O,VideomtLoss.get_num_masks..sALG Lsr )min) r^r9rrrr _shared_stater num_processesclamp)r.r}rur world_sizes&&& r1rVideomtLoss.get_num_maskssvALAA OOIU[[P  " $ $))R/"9- )^99 KK 6A> r3)rrrrrrwrrr)rLrMrNrOr"rrrrrrrrrIrrQrRrSrTs@@r1rrmss! ! F--"  D<<|2 - """5!5!n55n   r3rcaa]tRtRtoRtRtRtRtRtR.t Rt R] R ] /t ]P!4V3R lV3R ll4tV3R ltR tVtV;t#)VideomtPreTrainedModelizz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. videomtpixel_values_videosFrTrrGc8<V^8dQhRS[PRR/#)r5rr7N)rr)r:r;s"r1r<#VideomtPreTrainedModel.__annotate__s..BII.$.r3c<\SV`V4VPPp\ V\ P \ P\ P34'd\P!VP\P!^4R7VPe\P PP!VP4wr4V^8d^\P!V4, M^p\P"!VPV)V4EM]\ V\ P$4'd\P&!VPR^R7VP(eO\+VPRR4'g2\P,!VPVP(,4EM\ V\.4'dJ\1VR4'd6\P2!VP4VPP64EM]\ V\84'd\P:!VP<RVR7\P,!VP>4\P@!VPB\PD!VPBPFR,4PIR 44M\ V\J4'dY\PL!VPN^,4pVPPVR&\P@!VPRV4M6\ V\T4'd!\PV!VPX4\ V\84'd,\ PP-VPZ4R#R#) rt)aNr)rstd_is_hf_initializedFr9r]r\).r! _init_weightsr/initializer_ranger'rrr,ConvTranspose2dinitkaiming_uniform_rEmathsqrtrr9_calculate_fan_in_and_fan_outuniform_rinormal_ padding_idxgetattrzeros_r"hasattr constant_r9r7rV trunc_normal_r`rcrrnrlrBrmrr8rrrVideomtForUniversalSegmentationones_attn_mask_probsro)r.rr6fan_inr0boundrr0s&& r1r8$VideomtPreTrainedModel._init_weightssT f%kk++ fryy"))R5G5GH I I  ! !&--499Q< @{{&!HHMMGG V 17!DIIf-- fkkE659  - - LLSa 8!!-gfmmMach6i6i FMM&*<*<=>  1 2 2vy))v~~t{{/K/KL  1 2 2   v//cs C KK.. / JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h  , , ::f&7&7!&;TrrTs@@r1rYrYsKr3rYcPaa]tRtRtoV3RlV3RlltV3RlRltRtVtV;t#)VideomtScaleLayeric <V^8dQhRS[/#rOr)r:r;s"r1r<VideomtScaleLayer.__annotate__s;;};r3c <\SV`4VPp\P!W"^^R7Vn\ VP,Vn\P!VV^^VRR7Vn \V4Vn R#)r5rF)rpaddinggroupsrN) r!r"r&rr:conv1rrrr,conv2rY layernorm2dr.r/r&r0s&& r1r"VideomtScaleLayer.__init__ss (( '' aXYZ  !2!23YY     .k:r3cN<V^8dQhRS[PRS[P/#rr8)r:r;s"r1r<ris#U\\ellr3cVPV4pVPV4pVPV4pVPV4pV#r)rmrrnror.rs&&r1rIVideomtScaleLayer.forwardsB =1  6  =1 ((7 r3)rrmrnrorrTs@@r1rgrgs;; r3rgcPaa]tRtRtoV3RlV3RlltV3RlRltRtVtV;t#)VideomtScaleBlockic <V^8dQhRS[/#rOr)r:r;s"r1r<VideomtScaleBlock.__annotate__s``}`r3c<\SV`4VPVn\P !\ VP4Uu.uFp\V4NK up4VnR#uupir) r!r"num_upscale_blocks num_blocksr ModuleListrrgblockr.r/r0r0s&& r1r"VideomtScaleBlock.__init__sN  33]]uT__G]#^G]!$5f$=G]#^_ #^sA*cN<V^8dQhRS[PRS[P/#rr8)r:r;s"r1r<rys#U\\ellr3c<VPF pV!V4pK V#r)r~)r.rr~s&& r1rIVideomtScaleBlock.forwards ZZE!-0M r3)r~r|rrTs@@r1rwrws `` r3rwcPaa]tRtRtoV3RlV3RlltV3RlRltRtVtV;t#)VideomtMaskHeadic <V^8dQhRS[/#rOr)r:r;s"r1r<VideomtMaskHead.__annotate__ s44}4r3c<\SV`4VPp\P!W"4Vn\P!W"4Vn\P!W"4Vn\VP,Vn R#r) r!r"r&rrrrfc3rrrrps&& r1r"VideomtMaskHead.__init__ s[ (( 99[699[699[6 !2!23r3cN<V^8dQhRS[PRS[P/#rr8)r:r;s"r1r<rs#U\\ellr3cVPVPV44pVPVPV44pVPV4pV#rrrrrrts&&r1rIVideomtMaskHead.forwardsD(?@ (?@ / r3rrrTs@@r1rrs44r3rzY The Videomt Model with head on top for instance/semantic/panoptic segmentation. caa]tRtRtoRtV3RlV3RlltV3RlRltV3RlRlt]] ] RV3R lR ll444t R t V3R lR lt RtVtV;t#)rHir1c <V^8dQhRS[/#rOr)r:r;s"r1r<,VideomtForUniversalSegmentation.__annotate__!s}r3cx<\SV`V4WnVPVn\ V4Vn\ P!VPVPR7Vn \ P!VPVP4Vn \ P!\VP4Uu.uFp\!V4NK up4Vn\%V4Vn\)V4Vn\ P,!VPVP.^,4VnVP2VP4,VP2VP4,3VnRVP8RVP:RVP</Vn\AWP>R7Vn!\ PD!\FPH!VPJ44Vn&\ P,!VPVP4Vn'VPQ4R#uupi)rrrr)r/rN))r!r"r/num_hidden_layersrVrHrrr&r layernormrirrr}rrlayersrw upscale_blockr mask_headrrclass_predictorr#r$ grid_sizerrrrrrkrkr9r8r|rJ query_updater post_initrs&& r1r"(VideomtForUniversalSegmentation.__init__!s   !'!9!9+F3f&8&8f>S>ST\\&"4"4f6H6HI mm5IaIaCb$cCba\&%9Cb$cd .v6(0!yy););V=N=NQR=RS ++v/@/@@&BSBSW]WhWhBhi &"5"5 ++ ++.  %F@P@PQ!yyF4E4E)FGYYv'9'96;M;MN '%dsH7cj<V^8dQhRS[RS[RS[RS[RS[S[S[3,RS[S[S[3,/#)r5rErDr|r}rr7)rrr)r:r;s"r1r<r>sY$%    $CK0  c6k r3cVPVVVVVR7pVPP4F.wrxVP4FwrWy9gK W,p K K0 V#))rErDr|r}r)rkrr) r.rErDr|r}rrrrEloss_keyrCs &&&&&& r1 get_loss_dict-VideomtForUniversalSegmentation.get_loss_dict>sj(,~~!5!5#%"7 (6(  ++113KC"+//"3?ND#44 r3c<<V^8dQhRS[S[S[3,RS[/#)r5rr7)rrr)r:r;s"r1r<rVs$''$sF{"3''r3c4\VP44#r)r^r)r.rs&&r1get_loss(VideomtForUniversalSegmentation.get_lossVs9##%&&r3c <V^8dQhRS[PR,RS[S[P,R,RS[S[P,R,RS[S[P,R,RS[S[,RS[/#)r5r1Nr|r} patch_offsetsrr7)r9rr~rrrA)r:r;s"r1r<r\sO O "\\D0O %,,'$.O 5<<(4/ O ELL)D0 O +, O  /O r3c RV9d \R4hVf \R4hVP^8wd \R4hVfVe \R4hVPwrgrp VPWg,WV 4p VP V 4p VP VP P, p VPRV F pV!V 4p K V PWgV P^,V P^,4p .p.p.pRp\V4EFrpV RV3,pVfIVPPR ,PVR R 4PVP4pMeVP!V4PVP4VPPR ,PVP4,p\"P$!VV3^R7pVPV RF pV!V4pK VP'V4pVP)V4wppVP+V4VP+V4VP+V4VRRVP P,1R3,pEKu \/R\"P$!V^R7\"P$!V^R7\"P$!V^R7R 7#) a pixel_values_videos (`torch.Tensor`, *optional*): Video inputs of shape `(batch_size, num_frames, num_channels, height, width)`. mask_labels (`list[torch.Tensor]`, *optional*): Not supported for 5D video inputs. class_labels (`list[torch.LongTensor]`, *optional*): Not supported for 5D video inputs. patch_offsets (`list[torch.Tensor]`, *optional*): Unused for video inputs and only kept for modular compatibility. r6zAUse `pixel_values_videos` with `VideomtForUniversalSegmentation`.Nz'You have to specify pixel_values_videoszyVideomtForUniversalSegmentation only supports 5D video inputs of shape (batch_size, num_frames, channels, height, width).zTraining with 5D video inputs is not supported in `VideomtForUniversalSegmentation`. Flatten frames and use `EomtForUniversalSegmentation` instead.r[rv)rCrErDrF)Nr[r[r])rCrxrBryrHrr/r|rrrrrErmrDrurr9r}rpredictrrrA)r.r1r|r}rrr~rr%rrflat_pixel_valuesrquery_start_idx layer_moduleall_masks_queries_logitsall_class_queries_logitsall_last_hidden_statespropagated_query frame_idxframe_hidden_states query_tokenssequence_outputrErDs&&&&&, r1rI'VideomtForUniversalSegmentation.forwardYs* V #`a a  &FG G  # #q (E   "l&>Q  ?R>W>W; e/77 8OQ]glm(9: 004;;3I3II KK(89L(7M:&**:=CVCVWXCY[h[n[nop[qr #% #% !#z*I"/9 "= '#zz00<CCJPRTVWZZ[n[u[uv #112BCFFGZGaGabeieoeoevevf"(//0 1 #())\;N,OUV"W  $ O,< = &23F&G#!>#nn-@AO9=o9V 6 "6 $ + +,@ A $ + +,@ A " ) )/ :216O 8O8O6OQR3RS )+,5!&+C!K!&+C!K#ii(>AF   r3c.VPP#r)rHrdrs&r1get_input_embeddings4VideomtForUniversalSegmentation.get_input_embeddingss///r3c4<V^8dQhRS[P/#)r5rr8)r:r;s"r1r<rs))ell)r3cVRRVPP1R3,pVPV4pVRVPPVPP,R1R3,pVP ^^4pVP !VP^,R.VPO5!pVPV4pVPV4p\P!RW$4pWS3#)r[Nzbqc, bchw -> bqhwr]) r/rrrHrhrGryrBrrrr9einsum)r.rr class_logits prefix_tokens mask_logitss&& r1r'VideomtForUniversalSegmentation.predictsa!:4;;#:#:!:A=> ++L9 q$++"9"9DOO<]<]"]"_abbc %//15 %--m.A.A!.DbZ4>>Z ~~l3 **=9 ll#6 T ((r3)rJrr/rkrHrrrrrrrrr)NNNN)rLrMrNrOrRr"rrrrrrIrrrQrRrSrTs@@r1rHrHsk ,O:0'' O O  O b0)))r3rHr)F)Icollections.abcr(r=r dataclassesrnumpyrr9torch.nn.functionalrrrbrrr; activationsr file_utilsr r r modeling_layersr modeling_utilsr rprocessing_utilsrutilsrrr utils.genericrutils.output_capturingrconfiguration_videomtrscipy.optimizer accelerateraccelerate.utilsrrrrVrrrrrrrr"rArTrarprrrrrr/rrYrgrwrrH__all__rr3r1rs* $! &!LL9F&PP7504'' RYY F, ,^&(bii("%.8)ryy8)v(ryy("%bii%0'-'T+ +  7K7  7:@,8gbiigT<(u"))up /._/./.d 2  bii" `)&<`)  `)F $%F Gr3