+ ܳjqc T^RIt^RIt^RIt^RIt^RIt^RIHt^RIHt^RIH t H t H t ^RI H t Ht^RIHtHt^RIHtHtHtHtHtHtHtHtHtHtHt^RIHtH t H!t!H"t"^RI#H$t$H%t%H&t&^R IH't'H(t(H)t)H*t*H+t+H,t,H-t-^R I.H/t/H0t0^R I1H2t2^R I3H4t4^R I5H6t6^RIHtH t /RRbRRbRRbRRbRRbRRbRRbRRbRRbRRbRRbRRbRRbRRbRRbRRbR Rb/R!R"bR#RbR$R"bR%RbR&RbR'RbR(RbR)RbR*RbR+R"bR,RbR-RbR.RbR/RbR0RbR1RbR2RbCR3RR$R"R4R"R5R6/Ct7!R7R8]Pp4t9R#)9N)tqdm)ListUnionDict)Doc Annotated)snapshot_downloadsave_torch_state_dict) WQLinear_GEMM WQLinear_GEMV WQLinear_IPEXWQLinear_MarlinWQLinear_ExllamaWQLinear_ExllamaV2WQLinear_GEMVFastmarlin_post_initexllama_post_initexllamav2_post_initipex_post_init)get_named_linearsset_op_by_nameexclude_layers_to_not_quantize try_import)get_best_deviceipex_availabletriton_available) AutoConfigPreTrainedModelPretrainedConfig AutoProcessorBaseImageProcessorProcessorMixinPreTrainedTokenizer)init_empty_weightsload_checkpoint_and_dispatch) AwqConfig)ScaledActivation) AwqQuantizer)rrmptAutoModelForCausalLMllamaopt RefinedWebRefinedWebModelexaonefalconbloomgptj gpt_bigcodemistralmixtralgpt_neoxaquilaYiqwenbaichuanllavaAutoModelForVision2Seqqwen2qwen2_vlqwen3 qwen3_moegemmagemma2stablelm starcoder2 llava_nextphi3phi3_vcohere deepseek_v2 deepseek_v3minicpmminicpm3 internlm2 qwen2_5_vl qwen2_5_omniAutoModelForTextToWaveformcaa]tRt^atoV3RlV3RlltV3RlRltRtRt]P!4R/RR R R R R R^R R] 3 V3RlRll4t ]P!4R4t ] R4tR V3RlRllt]]P"R R RRR R 3V3RlRll4t]RR]P"R R R R R R RRRR3 V3RlRll4tR!RltR"Rlt] R4tRtVtV;t#)#BaseAWQForCausalLMc <V^8dQhRS[S[S[!R43,RS[S[S[!R43,RS[S[S[!R43,RS[S[S[!R43,R S[S[S[!R 43,R S[S[S[!R 43,/#) modelz"The pretrained or quantized model. model_typez%The model type, found in config.json. is_quantizedz,Indicates if the current model is quantized.configzThe config of the model. quant_configz%The quantization config of the model. processorz.An optional processor, e.g. for vision models.)rrrstrboolrr&r!)format __classdict__s"P/home/luca/screen-checking/.venv/lib/python3.14/site-packages/awq/models/base.py __annotate__BaseAWQForCausalLM.__annotate__bs33#.R*SST3c3'N#OOP3 #DE E 3*C0J,KKL3 sBC C 3 $T U U 3c z<\SV`4WnW nW0nRVnW@nWPnW`nR#)z&The base model for all AutoAWQ models.N) super__init__rUrVrW search_resultrXrYrZ)selfrUrVrWrXrYrZ __class__s&&&&&&&r_reBaseAWQForCausalLM.__init__bs8 &+ )".!(. '3)2rbcB<V^8dQhRS[S[S[!R43,/#)rTdevicez!The device to move your model to.)rr[r)r]r^s"r_r`ra{s$%%3,O(P#PQ%rbc 8VPPV4#)z4A utility function for moving the model to a device.)rUto)rgrks&&r_rmBaseAWQForCausalLM.to{szz}}V$$rbc &VP!V/VB#)z1A forward function that mimics the torch forward.rUrgargskwargss&*,r_forwardBaseAWQForCausalLM.forwardszz4*6**rbc \P!4;_uu_4VPP!V/VBuuRRR4# +'giR#;i)z9A generate function that mimics the HF generate function.N)torchinference_moderUgeneraterqs&*,r_ryBaseAWQForCausalLM.generates7  ! ! # #::&&77$ # # #s A A NpilevaltraintextTFicH<V^8dQhRS[S[S[!R43,RS[S[S[!R43,RS[S[S[S[S[,3,S[!R43,RS[S[S[!R43,R S[S[S[!R 43,R S[S[S[!R 43,R S[S[S[!R43,RS[S[S[!R43,RS[S[S[!R43,RS[S[S[!R43,RS[S[S[!R43,RS[S[S[!R43,RS[S[ S[!R43,/ #)rT tokenizerz&The tokenizer to use for quantization.rYz(The quantization config you want to use. calib_datazaThe calibration dataset. Either a string pointing to Huggingface or a list of preloaded examples.splitzThe split of calib_data. text_columnzThe text column of calib_data. duo_scalingz*Whether to scale using both w/x or just x.export_compatiblezcThis argument avoids real quantization by only applying the scales without quantizing down to FP16. apply_clipzrWhether to apply clipping to the model during quantization. Some models may perform better with this set to False.n_parallel_calib_samplesa"The number of parallel samples to run through the model. A high number of parallel samples can result in OOM during quantization if max_calib_samples is high enough. If None, runs through all samples at the same time. You can set this to a low number for more memory efficient quantization.max_calib_samplesz7The maximum number of samples to run through the model.max_calib_seq_lenzgThe maximum sequence length of the calibration dataset. Discard samples greater than max_calib_seq_len.max_chunk_memoryzThe loss computation and per-channel mean is optimized into chunked computations. Adjust this parameter to increase or decrease memory usage for these computations. Default is 1GB (1024 * 1024 * 1024). quantizer_clszZIf you want to customize the quantization class, you can use AwqQuantizer as a base class.) rr#rrrr[rr\intr()r]r^s"r_r`rasn!n! %M!N N n!  #@A A n! #tCy. ! s   n!c"<==>n!sC(H$IIJn!  #BC C !n!&%  u   'n!2  E   3n!>#,  [  # ?n!P% NO O Qn!V%  y   Wn!b$  8   cn!v!  l m n wn!rbc \P!V4Vn\VR4'dVPVPnV !VVP VVPP VPPVPPVPPVVVV3 RVPPRVRVRV RV RV RV /VBVn VPP4RVn R #) a The main quantization function that you can use to quantize your model. Example: ```python from awq import AutoAWQForCausalLM from transformers import AutoTokenizer model_path = "..." model = AutoAWQForCausalLM.from_pretrained(model_path) tokenizer = AutoTokenizer.from_pretrained(model_path) quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" } model.quantize(tokenizer, quant_config) ``` modules_to_not_convertrrrrrrTN) r& from_dictrYhasattrrrUw_bit q_group_size zero_pointversion quantizerquantizerW)rgrrYrrrrrrrrrrrrss&&&&&&&&&&&&&&,r_rBaseAWQForCausalLM.quantizesf(1':':<'H 41 2 27;7R7RD   4&  JJ     # #    * *    ( (    % %     $(#4#4#K#K 0 " &> 0! "0# $.% &' * ! rbc :VPP4R#)a A utility function for the following scenario. Note that save_quantized will overwrite existing weights if you use the same quant_path. Example: ```python model.quantize( tokenizer, quant_config=quant_config, export_compatible=True ) model.save_quantized(...) # produces GGUF/other compat weights model.pack(...) # makes the model CUDA compat model.save_quantized(...) # produces CUDA compat weights ``` N)rpack)rgs&r_rBaseAWQForCausalLM.packs& rbcR#Nrps&r_ fuse_layersBaseAWQForCausalLM.fuse_layerss rbc <V^8dQhRS[S[S[!R43,RS[S[S[!R43,RS[S[S[!R43,/#)rTsave_dirz$The directory to save your model to. safetensorsz8Whether to save the model as safetensors or torch files. shard_sizez>The shard size for sharding large models into multiple chunks.)rr[rr\)r]r^s"r_r`ras_- - C%K!LLM-  #PQ Q -  UV V - rbc aVR ,R8XdVRR MTp!V3RlR\P4oVPP4VPP nRVPPnVPPVS!4P4R7VPeVPPV4V R2V R2.pVF@p\PPV4'gK*\P!V4KB \!VPP4VVVRVPP"R 7R#) /Nc<<aa]tRtRtoVV3RltRtRtVtV;t#)6BaseAWQForCausalLM.save_quantized..EmptyModuleic$<\SV`4R#r)rdre)rg EmptyModulerhs&r_re?BaseAWQForCausalLM.save_quantized..EmptyModule.__init__ sk413rbcV#rr)rgxs&&r_rt>BaseAWQForCausalLM.save_quantized..EmptyModule.forward#srbr) __name__ __module__ __qualname____firstlineno__rert__static_attributes____classdictcell__ __classcell__)rhr^rs@@r_rrs 4  rbrT) state_dictz/model.safetensorsz/pytorch_model.bin)rsave_directorymax_shard_sizesafe_serializationforce_contiguousshared_tensors_to_discard)nnModulerYto_transformers_dictrUrXquantization_configgeneration_config do_samplesave_pretrainedrrZospathexistsremover _tied_weights_keys)rgrrr default_pathsrrs&&&& @r_save_quantized!BaseAWQForCausalLM.save_quantizeds%-RLC$78CR=X ")) 150A0A0V0V0X -15 $$. ""8 8P8P8R"S >> % NN * *8 4j* +j* + "Dww~~d## $" zz,,.#%*!&*jj&C&C  rbautoc<V^8dQhRS[S[S[!R43,RS[S[S[!R43,RS[S[PS[!R43,RS[S[S[!R43,R S[S[S[!R 43,R S[S[S[S[3,S[!R 43,R S[S[S[!R43,RS[S[S[!R43,RS[S[S[!R43,RS[S[S[!R43,/ #)rT model_path,A Huggingface path or local path to a model.rV(The model type, loaded from config.json. torch_dtypeLThe dtype to load the model as. May not work with other values than float16.trust_remote_codeXUseful for Huggingface repositories that have not been integrated into transformers yet.r>Whether to download/load safetensors instead of torch weights. device_mapQA device map that will be passed onto the model loading method from transformers.download_kwargs!Used for configure download modellow_cpu_mem_usagez5Use low_cpu_mem_usage when loading from transformers. use_cachez&Use use_cache argument in transformersmodel_init_kwargszEAdditional kwargs that are passed to the model during initialization.)rr[rrwdtyper\rr)r]r^s"r_r`raBsbU U c3'U#VVWU c3'Q#RRSU  KK ^   U %  j   U  #VW W !U & #t)  c   'U 2#  3 4 6 3U :%  G H I ;U B  8 9 : CU J'  W   KU rbc VPVVRVVVR7wrp \V P,p\\V4pRpVR8XgVR8Xd\ P !V 4pV PR4fWR&V PR4fVR8Xg VR8XgWR&VP !V 3RVR VR VR V/V BpVP4V!VVR V V VR 7#)zBA method for initialization of pretrained models, usually in FP16.)rrNr<rPrrrruse_safetensorsrFrWrXrYrZ) _load_configTRANSFORMERS_AUTO_MAPPING_DICTrVgetattr transformersr from_pretrainedgeteval)rgrrVrrrrrrrrmodel_weights_pathrXrYtarget_cls_name target_clsrZrUs&&&&&&&&&&, r_r"BaseAWQForCausalLM.from_pretrainedAs8^483D3D    /+ 4E4 0L99J9JK\?;  6 6/Mi:i%556HII  !4 5 =5F1 2   - 5Sk@krAEara-6k ***  / $ (  "        %   rbrbalancedc$<V^8dQhRS[S[S[!R43,RS[S[S[!R43,RS[S[S[!R43,RS[S[S[!R43,R S[S[P S[!R 43,R S[S[S[!R 43,R S[S[S[!R43,RS[S[S[!R43,RS[S[S[!R43,RS[S[S[!R43,RS[S[S[!R43,RS[S[S[S[3,S[!R43,RS[S[S[S[S[3,S[S[S[3,3,S[!R43,RS[S[S[!R43,RS[S[S[!R43,RS[S[S[!R 43,/#)!rTrrrVrmodel_filenamez=Load a specific model's filename by specifying this argument. max_seq_lenzsThe maximum sequence cached sequence length of the model. Larger values may increase loading time and memory usage.rrrrrrrzIWhether to use fused/optimized combination of layers for increased speed. use_exllamaz0Whether to map the weights to ExLlamaV1 kernels.use_exllama_v2z0Whether to map the weights to ExLlamaV2 kernels.use_ipexzBWhether to map the weights to ipex kernels for CPU and XPU device.rr max_memoryuA dictionary device identifier to maximum memory which will be passed onto the model loading method from transformers. For example：{0: "4GB",1: "10GB"offload_folderz#The folder ot offload the model to.rr config_kwargszFAdditional kwargs that are passed to the config during initialization.) rr[rrrwrr\rr)r]r^s"r_r`rasM` ` c3'U#VVW` c3'Q#RRS` " TU U `   F   `  KK ^   ` &%  j   '` 2 #VW W 3` 8  [   9` D #HI I E` J" #HI I K` P #Z[ [ Q` V #t)  c   W` b sCx%S/1 2 k   c` n"  5 6 8 o` v#  3 4 6 w` ~#  X   ` rbc VP!VVVVV3RVRV/VBwppp\VP,p\\V4p\ 4;_uu_4VP VVVR7pRRR4\4pVR8XgVR8Xd\'gRp V 'd\'g \R4hVPVXVVPV V V R 7VP4\VVV V VP.VVR 7\!R 4wppV'd;VR9d#Vf\"P$!R V,4MVP'V4V 'd \)V4pMoVPR 8Xd \+V4pMRV 'd \-V4pM>V 'd7\/TT;'gR\1\2P4!R^44R7pVP74V!VVRVVRR7# +'giEL;i)zBA method for initialization of a quantized model, usually in INT4.rr)rXrrNcpuzxpu:0TzlPlease install intel_extension_for_pytorch with `pip install intel_extension_for_pytorch` for 'ipex' kernel!)rrr) checkpointrrno_split_module_classesrrawq_extz?Skipping fusing modules because AWQ extension is not installed.marlinAWQ_BATCH_SIZE) max_input_lenmax_batch_sizer)mpszcuda:0)rrrVrrr$ from_configrrr ImportError_load_quantized_modulesr tie_weightsr% layer_typerwarningswarnrrrrrrrgetenvr)rgrrVrrrrrrrrrrrrrrrrXrYrrrU best_devicermsgs&&&&&&&&&&&&&&&&, r_from_quantized!BaseAWQForCausalLM.from_quantizedsR483D3D      4 $ 4 , 4  4 0FL99J9JK\?;  ! !**'"3+E"&' % K7$:CSCSH NNO  $$     #) %   % )!!%)__$5) "),  //GO _beef  ' "5)E  ! !X -$U+E %e,E ')11T"299-=q#ABE    %   E" ! !s G;; H c V\PPV4'g.R OpV'dVP.R O4MVP R4Vf/pRV9daVP R4p \ V \4'dVP V 4M'\ V \4'dVPV 4\V3RV/VBpVR8wdVRV 2,p MTp \P!V4p Vf\VR4'dm\P!V3RV/VBp \WPR4V n\V R4'd&\WPR4V P"nM'VfRMTp\P!V3RV/VBp W\nWV 3#) *msgpack*z*.safetensors*ignore_patternsrrmax_seq_len_keyrr text_config)rz*h5*z optimizer.ptz*.onnx*)z*.pt*z*.bin*z consolidated*)rrisdirextendappendpop isinstancer[listr r&rrrrrrr) rgrrrrrrrrdownload_kwargs_ignore_patternsrrYrXs &&&&&&&, r_rBaseAWQForCausalLM._load_config<sww}}Z((NO&&'KL&&'78&"$ O32A2E2EFW2X/=sCC#**+JK ?FF#**+JK*,;?NJ R !+.1A.B!B !+ !00<   741B#C#C//.?CPF")1E1Et!LF v}--1800$2"".#."5$;K//.?CPF"- !<77rbcpVR8Xd V'gV'g V'dQR4hVPV4p\\\V44RR7EFZpWx,p \ V 4p \ WP 4p VPW 4V P4FwrV'd\p MUVR8Xd\p MGV'd\p M8V'd\p M)VR8Xd\p MVR8Xd\p M VR8Xd\p X P!WP"VP$R4pVP'\)V P+44P,4\/WV4K V'g\0P2P54\6P8!4EK] R#) gemvz*Exllama kernels only support GEMM version.zReplacing layers...)descrgemm gemv_fastTN)get_model_layersrrangelenrrr_scale_activationsitemsr rrrr r r from_linearrrrmnext parametersrkrrwcuda empty_cachegccollect)rgrUrYrrrrlayersilayer named_linearsnamemoduleq_linear_moduleq_linears&&&&&&& r_r*BaseAWQForCausalLM._load_quantized_moduleszs` v ;.H 8 7 8 &&u-eCK(/DEEAIE.e4M;BBM  # #D 0!. 3 3 5 &3O(&5O &6O#&8O&&3O&&3O +&7O+66.. 0I0I4 D!1!1!34;;<uH5+!6. &&( JJLQFrbcnVPV4pVR,'d\VR,\4'gv\VP 44p\ P !VR,VPVPR7p\VR,V4p\WR,V4R#R#R#) is_scalable scale_layer scale_shape)rrk scale_nameN) get_act_for_scalingrr'r%r&rwonesrrkr)rgr- scale_dictparam scale_like scaled_acts&& r_r"%BaseAWQForCausalLM._scale_activationss--e4 m $ $j79IJJU--/0#ZZ}-U[[ .j.GT u&> KK %rb)rXrWrUrVrZrYrrfi@)T5GB)TTiN)F)rrrrrermrtryrwno_gradr(rr staticmethodrr classmethodfloat16rr rrr"rrr)rhr^s@@r_rRrRas332%%+8  ]]_   AHMS        }n!n!n!` ]]_(  - - ^ MM      IU U U n   MM          }` ` ` D<8|3jLLLrbrR):rr)rrwrtorch.nnrrtypingrrrtyping_extensionsrrhuggingface_hubr r awq.modules.linearr r r rrrrrrrrawq.utils.modulerrrrawq.utils.utilsrrrrrrr r!r"r#accelerate.big_modelingr$r%awq.models._configr&awq.modules.actr'awq.quantize.quantizerr(rrrRrrbr_rPsi  $$,D     NM ),/> '" !'" #'" !'"( '" - '"  $ '" $'" #'" "'")'"%'"%'"&'" $'" '"  "!'""&#'"$ %%'"& #''"(()'"* #+'",'-'". #/'"0 $1'"2&3'"4(5'"6*7'"8 "9'": $;'"< $='">)?'"@)A'"B%C'"D&E'"F'(*0M'"T] L] Lrb