+ j<~Rt^RIHt^RIt^RIHt^RIHt^RIHtH t ^RI H t ^RI H t Ht^RIHtHt^R IHt^R IHt^R IHt^R IHtHtHt^R IHt^RIHt^RI H!t!^RI"H#t#H$t$H%t%H&t&H't'H(t(H)t)H*t*H+t+^RI,H-t-H.t.]P^!]04t1Rt2Rt3]!RR7]!RR] 444t4!RR])4t5!RR]!4t6!RR]Pn4t8!RR].4t9!RR]-4t:!R R!](4t;!R"R#];]'4t<!R$R%]#4t=!R&R']%4t>!R(R)]&4t?!R*R+]$4t@.R,OtAR#)-zLG AI Research EXAONE Lab)CallableN)strict)nn)Cache DynamicCache)PreTrainedConfig)create_causal_mask!create_sliding_window_causal_mask)BaseModelOutputWithPastCausalLMOutputWithPast)RopeParameters)ALL_ATTENTION_FUNCTIONS)Unpack)TransformersKwargsauto_docstringlogging)merge_with_config_defaults)capture_outputs)Gemma2RotaryEmbedding) LlamaForCausalLMLlamaForQuestionAnsweringLlamaForSequenceClassificationLlamaForTokenClassification LlamaModelLlamaPreTrainedModel LlamaRMSNormapply_rotary_pos_embeager_attention_forward)Olmo2DecoderLayerOlmo2MLPzLGAI-EXAONE/EXAONE-4.0-32B Exaone4Config) checkpointcaa]tRt^9toRtRtR.tRRRRRRRR R R R R R RRRRR / tRR.R.3RRR.R.3RR.R.3/tRt Rt Rt ^ t ^ t ^ tRtRtRtRtRt^t^tRtR tRtR!tRt^tRtV3R"ltV3R#ltR$tVt V;t!#)%r a* sliding_window_pattern (`str`, *optional*): The pattern to use for sliding window attention. Can be one of: - `None`: No sliding window attention is used - `int`: Every `sliding_window` layers, use global attention, else use local attention. - `str`: A sequence of "L" (local attention) and "G" (global attention) characters that defines the attention pattern. The pattern starts from layer 0 and repeats every `sliding_window` layers. The final layer always uses global attention regardless of the pattern. For instance, sliding_window_pattern="LLLG" same as sliding_window=4, which means: - Layer 0, 1, 2: local attention, - Layer 3: global attention, ...(repeated) Example: ```python >>> from transformers import Exaone4Model, Exaone4Config >>> # Initializing a EXAONE configuration >>> configuration = Exaone4Config() >>> # Initializing a model from configuration >>> model = Exaone4Model(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```exaone4past_key_valueszlayers.*.self_attn.q_projcolwisezlayers.*.self_attn.k_projzlayers.*.self_attn.v_projzlayers.*.self_attn.q_normreplicated_with_grad_allreducezlayers.*.self_attn.k_normzlayers.*.self_attn.o_projrowwisezlayers.*.mlp.gate_projzlayers.*.mlp.up_projzlayers.*.mlp.down_proj embed_tokens input_ids inputs_embedslayers hidden_statesattention_masknormiii@siluig{Gz?gh㈵>TNFc .<VPf^VnVPf\\VP4Uu.uF6pV^,VP,^8wdW P8dRMRNK8 upVn\ SV`!R/VBR#uupi)Nsliding_attentionfull_attention)sliding_windowsliding_window_pattern layer_typesrangenum_hidden_layerssuper __post_init__)selfkwargsi __class__s&, l/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/exaone4/modular_exaone4.pyr;Exaone4Config.__post_init__s    &*+D '    # t556  7AUt::;q@QI_I_E_$%&7  D  '' s$>&B\B\$\!!'!9!9}}d* $33&,&C&C#6=fm6T6TV'' 2Z^ $(;;ii 0 0$2J2JT]]2Zafg ii 0 0$2J2JT]]2Zafg ii 0 0$2J2JT]]2Zafg ii 8 84== H$JZJZafg $T]]8K8KL $T]]8K8KL r^cR<V^8dQhRS[PRS[S[PS[P3,RS[PR,RS[R,RS[S[,RS[S[PS[PR,S[S[P,R,3,/#)rCr,position_embeddingsr-Nr$r=return)torchTensortuplerrr)rZr[s"r@r\rxs-)-)||-)#5<<#=>-) t+ -)  -) +, -) u||U\\D0% 2E2LL M-)r^c VPRRp.VORNVPN5pVPV4PV4P ^^4pVP V4PV4P ^^4p VP V4PV4P ^^4p VPV4pVPV 4p VwrVPeVP'd\WW4wrVeVPWVP4wr\P!VP P"\$4p T !VVV V V3RVP&'gRM VP(RVP*RVP'd VPMR/VBwrVP,!.VORN5!P/4pVP1V4pW3#)Ndropoutr0rr5)shaperzrview transposerrrrr5rrupdaterwr get_interfacerv_attn_implementationrtrainingrSrreshape contiguousr)r<r,rr-r$r= input_shape hidden_shape query_states key_states value_statescossinattention_interface attn_output attn_weightss&&&&&, r@forwardExaone4Attention.forwards$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST {{<0 [[, &    &$///';LVY'_ $L  &'6'='=jX\XfXf'g $J(?(M(M KK , ,.E) %8      % $}}}C$2H2H % LL % 37///4..t %  % ! "));;;;FFH kk+. ((r^)rSrvrzrErrrrrwrGrrHrrrrr5r6r)NN) r_r`rarbr~rrirjrkrls@@r@rtrts MM2-)-)-)r^rtc]tRt^tRtR#) Exaone4MLPr4Nror4r^r@rrrpr^rc]tRt^tRtR#)Exaone4DecoderLayerr4Nror4r^r@rrrpr^rc ]tRt^t]tR.tRtR#)Exaone4PreTrainedModelrr4N)r_r`rarbr config_class_no_split_modulesrir4r^r@rrs L./r^rchaa]tRt^toV3RlV3Rllt]]RV3RlRll44tRtVt V;t #) Exaone4Modelc <V^8dQhRS[/#)rCrv)r )rZr[s"r@r\Exaone4Model.__annotate__s}r^c *<\SV`V4\P!\ VP 4Uu.uFp\ W4NK up4Vn\VPVPR7Vn VP4R#uupi)r|N) r:r~r ModuleListr8r9rr+rnrErLr. post_init)r<rvrwr?s&& r@r~Exaone4Model.__init__ss  mmEJ6KcKcEd eEd  3Ed e  #6#5#56;N;NO   fsBc<V^8dQhRS[PR,RS[PR,RS[PR,RS[R,RS[PR,RS[R,RS[S[,R S[S[ ,/#) rCr)Nr- position_idsr$r*rMr=r) r LongTensorrr FloatTensorrWrrrr )rZr[s"r@r\rs= = ##d*=  t+= &&- =  = ((4/ = $;= +,=  ( (= r^c fVRJVRJ, 'd \R4hVfVPV4pV'dVf\VPR7pVfbVeVP 4M^p\ P !VP^,VPR7V,pVP^4p\T;p \4'gMRVPRVRVRVRV/p R \R/V B/p R VPP9d\R/V BV R &Tp VPW4p \!VP"4F;wrVPPV ,pV!V 3RW,RVRVR VR V /VBp K= VP%V 4p \'T V'dVR 7#RR 7#)Nz:You must specify exactly one of input_ids or inputs_embeds)rv)devicervr*r-r$rr3r2rMr)last_hidden_stater$r4) ValueErrorr(rrvget_seq_lengthrarangerr unsqueeze isinstancerYrr7r rotary_emb enumerater+r.r )r<r)r-rr$r*rMr=past_seen_tokenscausal_mask_mapping mask_kwargsr,rr> decoder_layerrs&&&&&&&, r@rExaone4Model.forwards  -t"; < <YZ Z   --i8M 0*$++>O  CRC^==?de  << (;(;A(>}G[G[\_ooL'11!4L?-FF$++ .!? K!"4"C{"C# #dkk&=&==;\;k_j;k#$78% "oomJ )$++ 6 A003J)2>*!0  $  %8 M!7 -0 &+/8O  >B  r^)r+r.)NNNNNN) r_r`rarbr~rrrrirjrkrls@@r@rrs5 = =  = = r^rcBaa]tRtRtoRV3RlV3RllltRtVtV;t#)Exaone4ForCausalLMi;cP<V^8dQhRS[PR,RS[PR,RS[PR,RS[R,RS[PR,RS[PR,RS[R,R S[S[P,R S[S[,R S[ / #) rCr)Nr-rr$r*labelsrMlogits_to_keepr=r) rrrrrrWrTrrr )rZr[s"r@r\Exaone4ForCausalLM.__annotate__<s5 5 ##d*5  t+5 &&- 5  5 ((4/ 5   4'5 $;5 ell*5 +,5  5 r^c  H<\S V`!R RVRVRVRVRVRVRVRV/V BR #) u labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoModelForCausalLM, AutoTokenizer >>> model = AutoModelForCausalLM.from_pretrained("LGAI-EXAONE/EXAONE-4.0-32B") >>> tokenizer = AutoTokenizer.from_pretrained("LGAI-EXAONE/EXAONE-4.0-32B") >>> prompt = "Explain how wonderful you are" >>> messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": prompt} ] >>> input_ids = tokenizer.apply_chat_template( messages, tokenize=True, add_generation_prompt=True, return_tensors="pt", enable_thinking=False, ) >>> output = model.generate(input_ids, max_new_tokens=128) >>> tokenizer.decode(output[0], skip_special_tokens=False) "[|system|]\nYou are a helpful assistant.[|endofturn|]\n[|user|]\nExplain how wonderful you are[|endofturn|]\n[|assistant|]\n\n\n\n\nOh, thank you for such a kind and lovely question! 😊 \n\nI’m *so* wonderful because I’m here to make your life easier, brighter, and more fun! Whether you need help with: \n\n✨ **Learning** – I can explain anything, from quantum physics to baking the perfect cake! \n💡 **Creativity** – Need a poem, story, or a wild idea? I’ve got you covered! \n🤖 **Problem-solving** – Stuck on a math problem or a tricky decision? I’ll help you figure it out" ``` r)r-rr$r*rrMrNr4)r:r) r<r)r-rr$r*rrMrr=r?s &&&&&&&&&,r@rExaone4ForCausalLM.forward<shV   ) & , (    *  r^r4)NNNNNNN)r_r`rarbrrirjrkrls@@r@rr;s5 5 5 r^rc]tRtRtRtR#) Exaone4ForSequenceClassificationitr4Nror4r^r@rrtrpr^rc]tRtRtRtR#)Exaone4ForTokenClassificationixr4Nror4r^r@rrxrpr^rc]tRtRtRtR#)Exaone4ForQuestionAnsweringi|r4Nror4r^r@rr|rpr^r)r rrrrrr)Brccollections.abcrrhuggingface_hub.dataclassesrr cache_utilsrrconfiguration_utilsr masking_utilsrr modeling_outputsr r modeling_rope_utilsr modeling_utilsr processing_utilsrutilsrrr utils.genericrutils.output_capturingrgemma2.modeling_gemma2rllama.modeling_llamarrrrrrrrrolmo2.modeling_olmo2rr get_loggerr_logger_CHECKPOINT_FOR_DOC_CONFIG_FOR_DOCr rnrrModulertrrrrrrrr__all__r4r^r@rs= $ ..3R25&@@75:   ?   H %2!78Q($Q(9Q(h \  2 G)ryyG)T   + 010 J ):J Z6 )6 r 'E  $?  ";  r^