+ jl^RIHt^RIHt^RIt^RIHt^RIHt^RI H t ^RI H t H t ^RIHt^R IHtHt^R IHt^R IHt^R IHt^R IHtHtHtHt^RIHtH t ^RI!H"t"H#t#^RI$H%t%^RI&H't'H(t(H)t)H*t*^RI+H,t,^RI-H.t.H/t/^RI0H1t1^RI2H3t3^RI4H5t5H6t6]!R4!RR]Pn44t8!RR]Pn4t9!RR]Pn4t:!RR]Pn4t;!R R!]Pn4t!R%R&]Pn4t?!R'R(]Pn4t@R)tA]!R*4RKR+l4tBR,R-ltCRLR.R/lltD]!]B4!R0R1]Pn44tE!R2R3]4tF](!R4R5]#44tG](!R6R7]#44tH!R8R9]Pn4tI](!R:R;]G44tJ](!R<R=]G]44tK](!R>R?7]!R@RA]444tL](!RBR?7]!RCRD]444tM](!RER?7!RFRG]H44tN](!RHR?7!RIRJ]H]44tO.RMOtPR#)N)Callable) dataclassN)nn)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hubuse_kernelized_func)create_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastBaseModelOutputWithPoolingCausalLMOutputWithPast ModelOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupletorch_compilable_check)deprecate_kwarg)maybe_autocastmerge_with_config_defaults)capture_outputs) AutoModel) AriaConfigAriaTextConfigRMSNormcZaa]tRt^4toRV3RlV3RllltV3RlRltRtRtVtV;t #)AriaTextRMSNormc$<V^8dQhRS[RR/#)epsreturnN)float)format __classdict__s"g/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/aria/modeling_aria.py __annotate__AriaTextRMSNorm.__annotate__6s$$$$$c <\SV`4\P!\P !V44VnW nR#)z. AriaTextRMSNorm is equivalent to T5LayerNorm N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizer) __class__s&&&r.r4AriaTextRMSNorm.__init__6s/ ll5::k#:; #r1cN<V^8dQhRS[PRS[P/#r( hidden_statesr*r6Tensor)r,r-s"r.r/r0>s#;;U\\;ell;r1c:VPpVP\P4pVP ^4P RRR7pV\P !W0P,4,pVPVPV4,#)r(T)keepdim) dtypetor6float32powmeanrsqrtr9r8)r:r@ input_dtypevariances&& r.forwardAriaTextRMSNorm.forward>sw#)) %((7  $$Q',,R,>% H?T?T4T(UU {{]--k:::r1c^\VPP4 RVP 2#)z, eps=)tupler8shaper9r:s&r. extra_reprAriaTextRMSNorm.extra_reprEs* ))*+6$2G2G1HIIr1)r9r8)gư>) __name__ __module__ __qualname____firstlineno__r4rNrT__static_attributes____classdictcell__ __classcell__r<r-s@@r.r&r&4s%$$;;JJr1r&c<aa]tRt^ItoRtV3RltRtRtVtV;t #)AriaProjectorMLPz Feed-Forward Network module for the Aria Projector. Args: in_features (`int`): Input embedding dimension. hidden_features (`int`): Hidden dimension of the feed-forward network. output_dim (`int`): Output dimension. c<\SV`4\P!WRR7Vn\P!W#RR7Vn\ R,VnR#)Fbiasgelu_newN)r3r4rLinear linear_in linear_outract)r: in_featureshidden_features output_dimr<s&&&&r.r4AriaProjectorMLP.__init__Vs> ;eL))OeL*%r1chVPVPV44pVPV4pV#Nrgrerf)r:r@s&&r.rNAriaProjectorMLP.forward\s- !>?  6 r1rn rVrWrXrY__doc__r4rNrZr[r\r]s@@r.r_r_Is & r1r_cPaa]tRt^btoRtRV3RlV3RllltRRltRtVtV;t #)AriaCrossAttentionzb Aria Cross-Attention module. Args: config (`AriaConfig`): The configuration to use. c&<V^8dQhRS[RS[/#)r(config dropout_rate)r"r+)r,r-s"r.r/AriaCrossAttention.__annotate__ks77z77r1cH<\SV`4VPPpVPPpW@n\ P!W3RR7Vn\ P!W3RR7Vn \ P!W3RR7Vn \ P!W4RR7Vn \ P!W34Vn \ P!V4Vn\ P !V4Vn\ P !V4VnR#)FraT) batch_firstN)r3r4 vision_configr;num_attention_heads num_headsrrdq_projk_projv_projMultiheadAttentionmultihead_attnlinearDropoutdropout LayerNorm layer_norm layer_norm_kv)r:rurvr;r|r<s&&& r.r4AriaCrossAttention.__init__ks **66 ((<< "ii uE ii uE ii uE !33KX\]ii 9 zz,/ ,,{3\\+6r1c VPVPV44pVPV4pVPV4pVP V4pVP WEWcR7wrxVP VPV44pV#)ai Forward pass of the AriaCrossAttention module. Args: key_value_states (`torch.Tensor`): Input tensor for key and value. hidden_states (`torch.Tensor`): Input tensor for query. attn_mask (`torch.Tensor`, *optional*, defaults to None): Attention mask. Returns: torch.Tensor: Output tensor after cross-attention.  attn_mask)r}rrr~rrrr) r:key_value_statesr@rquerykeyvalue attn_output_s &&&& r.rNAriaCrossAttention.forward|s  DOOM:;--.>?kk*+ ,-,,U,T ll4;;{#;< r1) rr~rrrrr|r}r)rrmrpr]s@@r.rsrsbs77"r1rscXaa]tRt^toRtV3RlV3RlltRV3RlRlltRtVtV;t #) AriaProjectorz Aria Projector module. This module projects vision features into the language model's embedding space, enabling interaction between vision and language components. Args: config (`AriaConfig`): Configuration object for the model. c <V^8dQhRS[/#r(rur")r,r-s"r.r/AriaProjector.__annotate__sfffr1c<\SV`4VPVnVPP VnVPPVnVPP Vn VPP Vn VPP Vn \P!\P !VP"VP 44Vn\'V4Vn\P*!VP 4Vn\/VP VPVP4VnR#rm)r3r4projector_patch_to_query_dictpatch_to_query_dictrzr;rhr{r|kv_dim text_configrirjrr5r6zeros'max_value_projector_patch_to_query_dictrrs cross_attnrrr_ feed_forwardr:rur<s&&r.r4AriaProjector.__init__s #)#G#G !//;;--AA**66 %11== ,,88\\%++f.\.\^b^n^n"op ,V4,,t'7'78,T-=-=t?S?SUYUdUder1c\<V^8dQhRS[PRS[PR,/#)r(rrNrA)r,r-s"r.r/rs) PTATr1c PVP^,VP^,rCW@P9d+\RV RVPP4 R24hVPV,pVPRVP ^4P V^^4pVeNVPVP^4pVP ^4PRVP^4R4pVPWVR7pVPVPV44pV#)aH Forward pass of the Projector module. Args: key_value_states (`torch.Tensor`): Input tensor of shape (batch_size, num_patches, kv_dim). attn_mask (`torch.Tensor`, *optional*, default is None): Attention mask. Returns: `torch.Tensor`: Output tensor of shape (batch_size, query_number, output_dim). zNumber of patches z: not found in patch_to_query_dict amongst possible values .NrrE)rRrKeyErrorkeysr unsqueezerepeatrepeat_interleaver|expandsizerrr) r:rr batch_size num_patches query_numqueries attention_outouts &&& r.rNAriaProjector.forwards0#3"8"8";=M=S=STU=VK 66 6$[M1klpmEmEmJmJmLlMMNO ,,[9 **Zi(2215<? r1) rrrirhrrr|rjrrrmrpr]s@@r.rrs%ff(r1rcHaa]tRt^toRtV3RlV3RlltRtRtVtV;t #)AriaSharedExpertsMLPa Shared Expert MLP for shared experts. Unlike routed experts, shared experts process all tokens without routing. This class reconfigures the intermediate size in comparison to the LlamaMLP. Args: config (`AriaTextConfig`): Configuration object for the Aria language model. c <V^8dQhRS[/#rr#)r,r-s"r.r/!AriaSharedExpertsMLP.__annotate__s00~0r1c6<\SV`4WnVPVnVPVP ,Vn\ P!VPVPVPR7Vn \ P!VPVPVPR7Vn \ P!VPVPVPR7Vn \VP,VnR#)raN)r3r4rur;intermediate_sizemoe_num_shared_expertsrrdmlp_bias gate_projup_proj down_projr hidden_actact_fnrs&&r.r4AriaSharedExpertsMLP.__init__s  !--!'!9!9FpWh,p Wh^,,p W V p \P!WV,4p WW%K@ V#)a  Compute the matrix multiplication (GEMM) for each expert sequentially. This approach is computationally inefficient, especially when dealing with a large number of experts. Args: token_states (torch.Tensor): Input tensor of shape (num_tokens, in_features). expert_weights (torch.Tensor): Weight tensor of shape (num_experts, in_features, out_features). tokens_per_expert (torch.Tensor): Number of tokens assigned to each expert. Returns: torch.Tensor: Output tensor of shape (num_tokens, out_features). rFdevicedimrE) rRr6rrFrcumsumlongcatrangematmul) token_statesexpert_weightstokens_per_expert num_tokens out_featuresoutputcumsum_num_tokens zero_tensor expert_numstartendtokensrs &&& r.sequential_experts_gemmrs##A&J!''+L [[9K9KT`TgTg hF %6A>++auzz:K:R:RSK ;"BCN0034 !-Q/C(ll6*#=>u 5 Mr1c<aa]tRtRtoRtV3RltRtRtVtV;t #)AriaGroupedExpertsGemmia Grouped GEMM (General Matrix Multiplication) module for efficient expert computation. This module utilizes the grouped_gemm library (https://github.com/fanshiqing/grouped_gemm) for optimized performance. If the grouped_gemm library is not installed, it gracefully falls back to a sequential GEMM implementation, which may be slower but ensures functionality. Args: in_features (`int`): Number of input features. out_features (`int`): Number of output features. groups (`int`): Number of expert groups. c<\SV`4WnW nW0n\ P !\P!W1V44Vn R#rm) r3r4rhrgroupsrr5r6emptyr8)r:rhrrr<s&&&&r.r4AriaGroupedExpertsGemm.__init__$s: &( ll5;;vL#QR r1c L\VVPVP44#)a- Perform grouped matrix multiplication. Args: input (`torch.Tensor`): Input tensor of shape (num_tokens, in_features). tokens_per_expert (`torch.Tensor`): Number of tokens assigned to each expert. Returns: torch.Tensor: Output tensor of shape (num_tokens, out_features). )rr8cpu)r:inputrs&&&r.rNAriaGroupedExpertsGemm.forward+s''  KK  ! ! #  r1)rrhrr8rpr]s@@r.rrs S  r1rcVaa]tRtRtoV3RlV3RlltRtV3RlRltRtVtV;t #) AriaExpertsi?c$<V^8dQhRS[RR/#)r(rur*Nr)r,r-s"r.r/AriaExperts.__annotate__@spp~p$pr1c<\SV`4Wn\VPVP ^,VP 4Vn\VP VPVP 4VnR#)r(N) r3r4rurr;rmoe_num_expertsfc1fc2rs&&r.r4AriaExperts.__init__@s_  )&*<*V>VYZ>Z\b\r\rs)&*B*BFDVDVX^XnXnor1c\P!WPP^R7wr#\P P VRR7pW43#))krrrE)r6topkrumoe_topkr functionalsoftmax)r: router_logits top_logits top_indicesscoress&& r.route_tokens_to_experts#AriaExperts.route_tokens_to_expertsFsB"'**]kk>R>RXY"Z &&zr&:""r1c4<V^8dQhRS[P/#r(r*rA)r,r-s"r.r/rKsu||r1clVPV4wr4VPp\P!VP 4P \P 4VPP^VPP^, R7P V4pTpVPR4p\P!V4p VP^WPP,4p VPW4p \P!V ^RR7wr\P P#V 4V ,p VP%W4p\P&!VP(^,VPP,VP+^43VPVP,R7pVP/^W4VPRVPPVP+^44pWP1R4,P3^R7pV#)r)binsminmaxrrrE)rrFr6histcflattenrGrHrurviewargsort index_selectrrchunkrrsilurrrRrr index_copy_rsum)r:r@r top_k_index top_k_weightsoriginal_dtyperindicesflatten_indicessorted_indicespermuted_tokens fc1_output projectiongate expert_outputunpermuted_tokensrs&&& r.rNAriaExperts.forwardKs%)%A%A-%P" $**!KK    ! $ $U]] 3,, ++a/  "^   !,,r*7'44Q++J^J^8^_XXoA  ;;z1"= ]]'' 3d: ? !KK   #dkk&:&: :M