+ j84^RIt^RIHt^RIHt^RIHt^RIHtH t H t H t H t ^RI Ht^RIHt]P"!]4tRt!RR ]4t!R R ] 4tR tRR lt]!RR]44t!RR] 4t!RR] 4t!RR] 4t.ROtR#)N)logging)no_inherit_decorator)LlamaAttentionLlamaForCausalLMLlamaForSequenceClassificationLlamaForTokenClassificationLlamaRotaryEmbedding)Phi3MLP) GlmConfigzTHUDM/glm-4-9bc]tRt^%tRtR#)GlmMLPN__name__ __module__ __qualname____firstlineno____static_attributes__rd/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/glm/modular_glm.pyr r %rr c6a]tRt^)toRV3RlRlltRtVtR#)GlmRotaryEmbeddingNcP<V^8dQhRS[RS[S[PS[3,/#)configreturn)r tupletorchTensorfloat)format __classdict__s"r __annotate__GlmRotaryEmbedding.__annotate__*s.55 5UZ[`[g[gin[nUo5rc VPR,pVPPRR4p\VRR4;'gVPVP,p\ WT,4pRpRV\ P!^V^\ PR7V, ,, pVPV4V3#)a~ Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetapartial_rotary_factorg?head_dimN)dtype) rope_parametersgetgetattr hidden_sizenum_attention_headsintraranger!to) rdevicekwargsbaser(r)dimattention_factorinv_freqs &&, rcompute_default_rope_parameters2GlmRotaryEmbedding.compute_default_rope_parameters*s%%l3 & 6 6 : :;RTW X6:t4hh8J8JfNhNh8h(23$5<<3#MPS#STU{{6"$444rrN)rrrrr9r__classdictcell__)r#s@rrr)s555rrczVR,pVR,p\P!V)V3RR7PR4#)z*Rotates half the hidden dims of the input.r6).:rNr).:Nr)rstackflatten)xx1x2s& r rotate_halfrG?s6 9B 9B ;;Ryb ) 1 1" 55rcVPV4pVPV4pVRRVPR,^,13,P^RR7pVRRVPR,^,13,P^RR7pVPR,pVRRV13,VRVR13,rvVRRV13,VRVR13,rWb,\V4V,,p W,\V4V,,p \P !W.RR7p \P !W.RR7p W3#)aIApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. .Nr>r@) unsqueezeshaperepeat_interleaverGrcat) qkcossin unsqueeze_dim rotary_dimq_rotq_passk_rotk_passq_embedk_embeds &&&&& rapply_rotary_pos_embrYFs6$ -- &C -- &C c'SYYr]a''' ( : :1" : EC c'SYYr]a''' ( : :1" : EC2Jc;J;&'3 +;)<6c;J;&'3 +;)<6{{51C78G{{51C78Gii)r2Gii)r2G  rcBaa]tRt^ntoRV3RlV3RllltRtVtV;t#) GlmAttentionc4<V^8dQhRS[RS[R,/#)rr layer_idxN)r r0)r"r#s"rr$GlmAttention.__annotate__ps#llylS4Zlrc<\SV`W4\P!VPVP ,VP RR7VnR#)F)biasN)super__init__nnLinearr/r)r.o_proj)selfrr] __class__s&&&rrbGlmAttention.__init__ps: +ii : :T]] JFL^L^ejk r)rer;)rrrrrbrr< __classcell__)rgr#s@@rr[r[nslllrr[c]tRt^utRtR#)GlmForCausalLMrNrrrrrkrkurrrkc]tRt^ytRtR#)GlmForSequenceClassificationrNrrrrrmrmyrrrmc]tRt^}tRtR#)GlmForTokenClassificationrNrrrrroro}rrro)r?)GlmPreTrainedModelGlmModelrkrmro)rtorch.nnrcutilsr utils.genericrllama.modeling_llamarrrrr phi3.modeling_phi3r configuration_glmr get_loggerrlogger_CHECKPOINT_FOR_DOCr rrGrYr[rkrmro__all__rrrr|s  1)(   H %& W 5-5,6%Pl>ll  %  #A  ;  r