+ j}Rt^RIt^RIt^RIHuHt^RIHt^RIHt^RI H t ^RI H t ^RIHtHt^RIHt^RIHtHt^R IHt^R IHtHt^R IHt]P:!]4t!R R ]P@4t!!RR]P@4t"!RR]P@4t#!RR]P@4t$!RR]P@4t%!RR]P@4t&!RR]P@4t'!RR]P@4t(!RR]P@4t)!RR]P@4t*!R R!]P@4t+]!R"R#]44t,]!R$R%],44t-]!R&R'7!R(R)],]44t..R*Ot/R#)+zPyTorch CPMAntN)nn)CrossEntropyLoss)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)BaseModelOutputWithPastCausalLMOutputWithPast)PreTrainedModel)auto_docstringlogging CpmAntConfigcTaa]tRt^$toRtV3RlV3RlltV3RlRltRtVtV;t #)CpmAntLayerNormzv We use Root Mean Square (RMS) Layer Normalization, please see https://huggingface.co/papers/1910.07467 for details." c <V^8dQhRS[/#configr)format __classdict__s"k/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/models/cpmant/modeling_cpmant.py __annotate__CpmAntLayerNorm.__annotate__)sDD|Dc<\SV`4VPVnVPVn\ P !\P!VP44Vn R#N) super__init__eps hidden_sizedim_normr Parametertorchemptyweightselfr __class__s&&rrCpmAntLayerNorm.__init__)sE ::** ll5;;v/A/A#BC rc4<V^8dQhRS[P/#r hidden_statesr$Tensor)rrs"rrr0  U\\ rc VPR4VP8wd \R4hVPpVP \ P 4P^4PRRR7pV\ P!W0P,4,P V4VP,pV#)N Args: hidden_states (`torch.Tensor` of shape `(batch, seq_len, dim_in)`) z'hidden_states.size(-1) != self.dim_normT)dimkeepdim) sizer"AssertionErrordtypetor$float32powmeanrsqrtr r&)r(r- old_dtypevariances&& rforwardCpmAntLayerNorm.forward0s   b !T]] 2 !JK K!''  ##EMM266q9>>2t>T&X5H)IIMMiX[_[f[ff r)r"r r&) __name__ __module__ __qualname____firstlineno____doc__rr@__static_attributes____classdictcell__ __classcell__r)rs@@rrr$s%DD   rrcXaa]tRt^=toRV3RlV3RllltRV3RlRlltRtVtV;t#)CpmAntAttentionc <V^8dQhRS[/#rr)rrs"rrCpmAntAttention.__annotate__>s  | rc<\SV`4VPVnVPVnVP VnW n\P!VPVP VP ,RR7Vn \P!VPVP VP ,RR7Vn \P!VPVP VP ,RR7Vn \P!VP VP ,VPRR7Vn \PPRR7VnVP"e2\PP%VP"R7VnR#RVnR#)Fbiasr3N)pr5)rrr! dim_modelnum_attention_heads num_headsdim_head layer_idxrLinear project_q project_k project_v attention_outr$Softmaxsoftmax dropout_pDropoutdropoutr(rrXr)s&&&rrCpmAntAttention.__init__>s ++33 "4>>4>>DMM3QX]^4>>4>>DMM3QX]^4>>4>>DMM3QX]^YYt~~ 'Et~~\abxx''B'/    ' 88++f.>.>+?DLDLrc<V^8dQhRS[PRS[PRS[PRS[PRS[R,RS[R,RS[R,/#) rhidden_q hidden_kvattention_mask position_biasoutput_attentionsNpast_key_values use_cache)r$r/ BoolTensorboolr)rrs"rrrNRsxM#M#,,M#<<M#(( M# || M#  $; M#M#$;M#rc VP^4p VP^4p VP^4p VPV4p VPV4p VPV4pV P WVP VP 4P^^^^4p V P WVP VP 4P^^^^4p VP WVP VP 4P^^^^4pVe0VPWVP4wrV PR4p \P!WPRR44\P!VP 4, pW,p\P!VVP V ^W4\P !R48H\P"!\%R4VP&VP(R74pVP+V4p\P!VVP V ^W4\P !R48H\P"!^VP&VP(R74pV'dTpMRpVP,eVP-V4p\P!W4pVP WP WP 4P^^^^4pVP/4P WVP VP ,4pVP1V4pVV3#)a Args: hidden_q (`torch.Tensor`): Input of transformer block(self-attention block). It can be the raw embedding of a batch of sequences. hidden_kv (`torch.Tensor` of shape `(batch, len_k, dim_model)`)): Tensor *key_value* and *query* of shape `(batch, len_k, dim_model)` attention_mask (`torch.Tensor` of shape `(batch, len_seq, len_seq)`): Avoid invalid areas to participate in the calculation of self-attention. position_bias (`torch.Tensor` of shape `(batch, len_seq, len_seq)`): Provide positional information to self-attention block. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. past_key_values (`Cache`, *optional*): Cached past key and value projection states. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). NFz-inf)devicer8r5)r6rZr[r\viewrVrWpermuteupdaterXr$matmul transposemathsqrt masked_filltensor scalar_tensorfloatrpr8r_rb contiguousr])r(rfrgrhrirjrkrlkwargs batch_sizelen_qlen_kquerykeyvaluescore attn_weightss&&&&&&&&, rr@CpmAntAttention.forwardRs:]]1%  a q!x(nnY'y) :dnndmmLTTUVXY[\^_`hhz$..$--HPPQRTUWXZ[\ :dnndmmLTTUVXY[\^_`  &(//DNNKJCHHRLE UMM"b$9:TYYt}}=UU%!!     Au < U@S S   f ell%++ V   U#!!     Au < U@S S   %,,ekk J   LL << #LL'E U* :~~ummLTTUVXY[\^_`  "'' 4>>DMM;YZ""5)l""r) r]rWrTrbrXrVr[rZr\r_r)FNN rBrCrDrErr@rGrHrIrJs@@rrLrL=s!  (M#M#M#rrLcXaa]tRt^toRV3RlV3RllltRV3RlRlltRtVtV;t#)CpmAntSelfAttentionBlockc <V^8dQhRS[/#rr)rrs"rr%CpmAntSelfAttentionBlock.__annotate__  | rc<\SV`4\V4Vn\ WR7VnVP 'd1\PPVP 4Vn R#RVn R#)rXN) rrrlayernorm_before_attentionrLself_attentionr`r$rrarbrcs&&&rr!CpmAntSelfAttentionBlock.__init__sW *9&*A'-fJ     88++F,<,<=DLDLrc <V^8dQhRS[PRS[PRS[PR,RS[R,RS[R,RS[R,/#rr-rhriNrjrkrlr$r/rnr)rrs"rrrsd)+)+||)+ )+||d* )+  $; )+  )+$;)+rc VPV4pVPVVVVVVV4wrVPeVPV4pW,pW3#)av Args: hidden_states (`torch.Tensor` of shape `(batch, len_seq, dim_model)`): Input of transformer block(self-attention block). It can be the raw embedding of a batch of sequences. attention_mask (`torch.Tensor` of shape `(batch, len_seq, len_seq)`): Avoid invalid areas to participate in the calculation of self-attention. position_bias (`torch.Tensor` of shape `(batch, len_seq, len_seq)`): Provide positional information to self-attention block. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. past_key_values (`Cache`, *optional*): Cached past key and value projection states. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). )rrrb) r(r-rhrirjrkrlr~outputsrs &&&&&&&, rr@ CpmAntSelfAttentionBlock.forwardsg411-@ $ 3 3       !  << #ll7+G%/ **r)rbrrrNFNNrrJs@@rrrs  )+)+)+rrcPaa]tRt^toV3RlV3RlltV3RlRltRtVtV;t#)CpmAntDenseGatedACTc <V^8dQhRS[/#rr)rrs"rr CpmAntDenseGatedACT.__annotate__s##|#rc2<\SV`4\P!VPVP RR7Vn\P!VPVP RR7Vn\PP4Vn R#FrPN) rrrrYr!dim_ffw_0w_1r$GELUactr's&&rrCpmAntDenseGatedACT.__init__s[ 99V//UK99V//UK88==?rc4<V^8dQhRS[P/#r,r.)rrs"rrrr0rc xVPVPV44pVPV4pW!,pV#)zTransform an input tensor from one feature space to another via a nonlinear operation Args: hidden_states (`torch.Tensor` of shape `(batch, seq_len, dim_in)`) rrr)r(r- gate_scores&& rr@CpmAntDenseGatedACT.forwards7 XXdhh}56 / "2 rrrrJs@@rrrs##   rrcPaa]tRt^toV3RlV3RlltV3RlRltRtVtV;t#)CpmAntFeedForwardc <V^8dQhRS[/#rr)rrs"rrCpmAntFeedForward.__annotate__sNN|Nrc0<\SV`4\V4VnVPe0\ P PVP4VnMRVn\ P!VPVPRR7Vn R#)NFrP) rrrw_inr`r$rrarbrYrr!w_outr's&&rrCpmAntFeedForward.__init__sg '/    ' 88++F,<,<=DLDLYYv}}f.@.@uM rc4<V^8dQhRS[P/#r,r.)rrs"rrrs  U\\ rc VPV4pVPeVPV4pVPV4pV#)r2)rrbrr(r-s&&rr@CpmAntFeedForward.forwards>  -0 << # LL7M =1 r)rbrrrrJs@@rrrs NN   rrcPaa]tRtRtoV3RlV3RlltV3RlRltRtVtV;t#)CpmAntFFNBlockic <V^8dQhRS[/#rr)rrs"rrCpmAntFFNBlock.__annotate__rrc<\SV`4\V4Vn\ V4VnVP 'd1\PPVP 4Vn R#RVn R#r) rrrlayernorm_before_ffnrffnr`r$rrarbr's&&rrCpmAntFFNBlock.__init__sV $3F$;!$V,     88++F,<,<=DLDLrc4<V^8dQhRS[P/#r,r.)rrs"rrrs||rc VPV4pVPV4pVPeVPV4pW,pV#)z Args: hidden_states (`torch.Tensor` of shape `(batch, len_seq, dim_model)`): Hidden states before feed forward layer. )rrrb)r(r- ln_outputsrs&& rr@CpmAntFFNBlock.forwardsH..}= ((:& << #ll7+G%/ r)rbrrrrJs@@rrrs  rrcXaa]tRtRtoRV3RlV3RllltRV3RlRlltRtVtV;t#) CpmAntTransformerBlocki!c <V^8dQhRS[/#rr)rrs"rr#CpmAntTransformerBlock.__annotate__"s**|*rcf<\SV`4\WR7Vn\ V4VnR#r)rrrself_attrrrcs&&&rrCpmAntTransformerBlock.__init__"s& 0M !&)rc <V^8dQhRS[PRS[PRS[PR,RS[R,RS[R,RS[R,/#rr)rrs"rrr'sd$+$+||$+ $+||d* $+  $; $+  $+$;$+rc \VPVVVVVVR7wrVPV4pW3#)a Args: hidden_states (`torch.Tensor`): Input to the layer of shape `(batch, seq_len, dim_model)` attention_mask (`torch.Tensor`): Avoid invalid areas to participate in the calculation of shape `(batch, seq_len, seq_len)` position_bias (`torch.Tensor`): Provides position information to attention mechanism of shape `(num_heads, seq_len, seq_len)` output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. past_key_values (`Cache`, *optional*): Cached past key and value projection states use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). )rhrirjrkrl)rr) r(r-rhrirjrkrlr~rs &&&&&&&, rr@CpmAntTransformerBlock.forward'sE4'+mm )'/+ '4' # / **r)rrrrrrJs@@rrr!s** $+$+$+rrcTaa]tRtRtoV3RlV3RlltRV3RlRlltRtVtV;t#) CpmAntEncoderiNc <V^8dQhRS[/#rr)rrs"rrCpmAntEncoder.__annotate__Os88|8rc <\SV`4VPVn\P !\ VP4Uu.uFp\WR7NK up4Vn\V4Vn R#uupir) rrnum_hidden_layers num_layersr ModuleListrangerlayersroutput_layernorm)r(rir)s&& rrCpmAntEncoder.__init__Osc  22mmZ_`d`o`oZp$qZpUV%;F%PZp$qr / 7%rsA;c<V^8dQhRS[PRS[PRS[PRS[R,RS[R,RS[R,RS[R,/#) rr-rhrirjNoutput_hidden_statesrkrlr)rrs"rrrVsv4@4@||4@ 4@|| 4@  $; 4@ #Tk 4@4@$;4@rc &V'dRMRp V'dRMRp \VP4F;wrV'd W3, p V !VVVVVVR7p V wrV'gK2W3, p K= VPV4pV'd W3, p WV 3#)a Args: hidden_states (`torch.Tensor`): Input to the layer of shape `(batch, seq_len, dim_model)` attention_mask (`torch.Tensor`): Avoid invalid areas to participate in the calculation of shape `(batch, seq_len, seq_len)` position_bias (`torch.Tensor`): Provides position information to attention mechanism of shape `(num_heads, seq_len, seq_len)` output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. past_key_values (`Cache`, *optional*): Cached past key and value projection states use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). N)rjrkrl) enumeraterr)r(r-rhrirjrrkrlr~all_hidden_statesall_self_attnsrlayer layer_outputsrs&&&&&&&&, rr@CpmAntEncoder.forwardVs:#7BD0d!$++.HA#!%55!!"3 /# M+8 'M  /1/--m<  !1 1 ??r)rrr)NNNNrrJs@@rrrNs!884@4@4@rrcDaa]tRtRtoV3RltV3RlRltRtVtV;t#)CpmAntIntermediateic"<\SV`4\P!VPVP 4Vn\VP\4'd\VP,Vn R#VPVn R#r) rrrrYr!intermediate_sizedense isinstance hidden_actstrrintermediate_act_fnr's&&rrCpmAntIntermediate.__init__s` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $rcN<V^8dQhRS[PRS[P/#)rr-returnr.)rrs"rrCpmAntIntermediate.__annotate__s#U\\ellrcJVPV4pVPV4pV#rrrrs&&rr@CpmAntIntermediate.forwards& =1 00? rrrrJs@@rrrs9rrc`aa]tRtRtoV3RlV3RlltV3RlRltRtR RltRtVt V;t #) CpmAntSegmentPositionEmbeddingic <V^8dQhRS[/#rr)rrs"rr+CpmAntSegmentPositionEmbedding.__annotate__s   |  rcx<\SV`4VPVnVPVnVP VnVPVn \P!\P!VPVP,VP,VP44VnR#r)rrrUrVposition_bias_num_buckets num_bucketsposition_bias_max_distance max_distance segment_types num_segmentsrr#r$r%relative_attention_biasr's&&rr'CpmAntSegmentPositionEmbedding.__init__s 33!;;"=="00')|| KK$$v';';;f>^>^^** ( $rc<V^8dQhRS[PRS[PRS[PRS[P/#)rkey_pos query_pos key_segment query_segmentr.)rrs"rrrsC222<<2\\ 2 || 2rc b\P!4;_uu_4VP^4pVP^4pVP^4pVP^4VP^48wd1\RVP^4 RVP^4 R24hWcP^48wgWtP^48wd"\RV RVP^4 R24hWtP^48wd"\RV RVP^4 R24hVP VR V4pVP WWR 4pVP VR V4pVP WWR 4pVP WC4pWP ,pVP\P!V\PVPR7R ,\P!V\PVPR7R ,, VP VPR7p \P!W48HV R ,V4pRRR4\P!XVP4p V P!^^^^4P#4p V # +'giLU;i) z>key_pos.size(0) should be equal to query_pos.size(0), but got z and !z7keylen should be equal to key_segment.size(1), but got z;querylen should be equal to query_segment.size(1), but got r8rpN)rrr5NNNNr N)Nr r )r$no_gradr6r7rr!_segment_relative_position_bucketr_position_bucketarangeint32rprwhereF embeddingrrsr}) r(rrrrbatchkeylenquerylenrelative_position_bucketabsolute_position_bucketembedss &&&&& rr@&CpmAntSegmentPositionEmbedding.forwardsl]]__LLOE\\!_F ~~a(H||A).."33$TU\UaUabcUdTeejktkykyz{k|j}}~))!,,>rc^pV^,pV^8P\P4V,p\P!V4pV^,pW8pV\P!VP 4V, 4\ P!W5, 4, W%, ,P\P4,p\P!V\P!Wr^, 44pV\P!WaP\P4V4, pV#)r) r9r$rabslogr|rwmin full_liker)r(relative_positionrrrelative_buckets max_exactis_smallrelative_position_if_larges&&&& rr/CpmAntSegmentPositionEmbedding._position_buckets -155ekkB[P!II&781$ $0%. II'--/); <hh|/0 1& ( "U[[/ &" &+YY & OO6a H& " EKK2F2Fu{{2SUoppr)rrrVrr) ) rBrCrDrErr@rrrGrHrIrJs@@rrrs)   22h?  rrcDaa]tRtRtoV3RltV3RlRltRtVtV;t#) CpmAntOutputic0<\SV`4\P!VPVP 4Vn\P!VP VPR7Vn\P!VP4Vn R#))r N) rrrrYrr!r LayerNormlayer_norm_epsrahidden_dropout_probrbr's&&rrCpmAntOutput.__init__s` YYv779K9KL f&8&8f>S>STzz&"<"<= rch<V^8dQhRS[PRS[PRS[P/#)rr- input_tensorrr.)rrs"rrCpmAntOutput.__annotate__s1U\\RWR^R^rcxVPV4pVPV4pVPW,4pV#r)rrbr.)r(r-r3s&&&rr@CpmAntOutput.forwards5 =1  ]3 }'CD r)r.rrbrrJs@@rr,r,s> rr,cjaa]tRtRtoRt]P !4V3Rl4tV3RltRt Vt V;t #)CpmAntPreTrainedModelicpmantc 4<\SV`V4\V\4'd#\P !VP 4R#\V\4'd:\P!VPRVPPR7R#R#)zInitialize the weightsg)r<stdN) r _init_weightsrrinitones_r&rnormal_rrinit_std)r(moduler)s&&rr<#CpmAntPreTrainedModel._init_weights sc f% fo . . JJv}} %  > ? ? LL77ct{{G[G[ \@rc&<V^8dQh/S[;R&#rr)rrs"rr"CpmAntPreTrainedModel.__annotate__s rr) rBrCrDrEbase_model_prefixr$r r<__annotate_func__rGrHrIrJs@@rr8r8s.! ]]_]] rr8cpaa]tRtRtoV3RlV3RlltRtRtRt]R V3RlRll4t R t Vt V;t #) CpmAntModelic <V^8dQhRS[/#rr)rrs"rrCpmAntModel.__annotate__s  | rc<\SV`V4\V4Vn\P !VP VP4Vn\P !VPVPVP,,VP4Vn \V4VnVPVn VPVn VP4R#r)rrrencoderr Embeddingrr!segment_embedding vocab_size prompt_types prompt_lengthinput_embeddingrri post_initr's&&rrCpmAntModel.__init__s  $V, !#f.B.BFDVDV!W!||    3 3f6J6J J JFL^L^ &CpmAntModel.forward..s^a^s )last_hidden_staterkr- attentionsr5r)rrjrrnrlr8r$rr9rprsumrbrrQrOrar6zerosfullrget_seq_lengthr}rRrNrkrirLror )r(rdrjrrkrlrnr~r8rpsegmentrgr seq_lengthrfpositionre past_lengthr-segment_statesrhrirall_attentionsnew_attentions attentionnew_hidden_states hidden_states&&&&&&&, rr@CpmAntModel.forward>s(2C1N-TXT_T_TqTq$8$D $++JjJj &1%O:I:Uo446[\ ((* ,,Y7 //8 ! +ArsAI6N%6 55iwW**8wP ';<(:;%aKL!&;< %aq&89 ;?<<      < 8 (. ! )!T-?-?-A1*DEM)!#!/I"yAt7I7I7KTM_M_Ma1a'b&ddN"0!/ ,$&!$5L%,q$:L:L:NPQ7Q*R)TT%%6$5!5)?>> from transformers import CPMAntTokenizer, CpmAntForCausalLM >>> texts = "今天天气不错，" >>> model = CpmAntForCausalLM.from_pretrained("openbmb/cpm-ant-10b") >>> tokenizer = CPMAntTokenizer.from_pretrained("openbmb/cpm-ant-10b") >>> input_ids = tokenizer(texts, return_tensors="pt") >>> outputs = model.generate(**input_ids) >>> output_texts = tokenizer.batch_decode(outputs) >>> print(output_texts) ['今天天气不错，阳光明媚，我和妈妈一起去超市买东西。\n在超市里，我看到了一个很好玩的玩具，它的名字叫“机器人”。它有一个圆圆的脑袋，两只圆圆的眼睛，还有一个圆圆的'] ``` Nr :NN)losslogitsrkr-rwr5)rrnr9rvrrslicerrrrr6r rkr-rw)r(rdrkrlrjrrrnrhrr~ model_outputr- slice_indicesrr loss_funcoutputs&&&&&&&&&&, rr@CpmAntForCausalLM.forwardsR&1%SV8W8W~ot4]k mA}a,?@A  (*IV[[V[[_=v{{2ODYb!11F)-)9TGf$ Ev E%(88&44#..   rc.VPP#rr9rRrWs&rrX&CpmAntForCausalLM.get_input_embeddingss{{***rc&WPnR#rr)r(r[s&&rr\&CpmAntForCausalLM.set_input_embeddings s &0 #r)r9rr) NNNNNNNNr)rBrCrDrE_tied_weights_keysrrr r@rXr\rGrHrIrJs@@rrrsIeF F F P+11rr)rrHr8)0rFrwr$torch.nn.functionalr functionalrtorch.nnrrr= activationsr cache_utilsrr generationrmodeling_outputsr r modeling_utilsr utilsr r configuration_cpmantr get_loggerrBloggerModulerrLrrrrrrrrr,r8rHr__all__rrrrss %&!.)O-,.   H %bii2b#biib#J3+ryy3+l"))( 4RYY6*+RYY*+Z<@BII<@@  Y RYYY z 299  ]O ] ]N 'N N b ]1-]1  ]1@ Hr