+ jCRt^RIt^RIHt^RIHtHt^RIHt^RIH t ]P!] 4t Rt !RR]4t]Pt]P"!]4P$t]P"!]4P(tR R ltRR R llt!R R]P04tRRltRt!RR]4t!RR]4tR#)z"CompressedTensors integration fileN)nn) ConversionOps _IdentityOp)logging)is_torch_greater_or_equalc\VPP44pVFUp\;QJd&RVP4F 'gK RM RM!RVP44'gKSVu# VFUp\;QJd&RVP4F 'gK RM RM!RVP44'gKSVu# V^,#)aResolve which config group quantizes the MoE experts. Mixed configs quantize different layers with different schemes (e.g. Kimi: FP8 attention + INT4 experts), so the experts' scheme cannot be assumed global: the experts' group is the one whose `re:` targets mention the expert modules. c3\"TF"qPR4;'dRV9xK$ R#5i)re:expertsN startswith.0targets& m/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/integrations/compressed_tensors.py %get_experts_scheme..#s)\mF  '??I,??ms,,TFc3L"TFqPR4'*xK R#5i)r Nr r s& rrr(sH-$$U+++-s"$)list config_groupsvaluesanytargets)quantization_configgroupsgroups& rget_experts_schemers %33::< =F 3\emm\333\emm\ \ \L  3H%--H333H%--H H HL !9c`a]tRt^-toRtR RltR V3RlRllt]V3RlRl4tRt Vt R#) DecompressExpertsa Dequantize MoE layers when they are in new layout, because they aren't `nn.Module` anymore! Takes packed weights and scales from the loaded state dict, creates a dummy Module to take advantage of higher-lvl API `decompress_module` and dequantizes all weights. Requires MoE conversion to be defined on conversion mapping, so that decompressed weights are stacked/merged for all experts. NcWnW nR#N hf_quantizerscheme)selfr#r$s&&&r__init__DecompressExperts.__init__8s( rc <V^8dQhRS[S[S[P3,RS[S[,RS[S[,RS[R,RS[S[S[P3,/#) input_dictsource_patternstarget_patternsfull_layer_nameNreturn)dictstrtorchTensorr)format __classdict__s"r __annotate__DecompressExperts.__annotate__>slggell*+gcgc g t g c5<< grc  ^RIHp^RIHpVPP P pVP;'g \V4p V Pp V PR8XEdV P^8XEdu/p VP4EF[wrRV 9dKV PRR4pW9dWV &K-\V \4'dT MV .pW,p\V\4'dTMV.pRp\!\#VV44FwpwppVP%\&P(4VP%\&P(4,P%\&P*4pVfE\&P,!\/V4.VP0O5VP2VP4R7pVV,P7V4?K VfEKWVW&EK^ V #V P8;'gV!\:P<V 4pVP?V4p!R R \:P@4p^ V PP,p/p VP4EFwrR V 9dKT pWPR R4,pVPCV PR R 44pRp\!\#VV44EF"wpwppVfRMVV,pVeVPE4'dTpMA\&PF!VP0^,VP0^,V,.4pV!VVV4pV Vn$VPKV4Vfc\&P,!\/V4.VPLP0O5VPLP2VPLP4R7pVV,P7VPL4?EK% ??VfEKVW&EK V #) )BaseCompressor)infer_module_formatfloat weight_scalez.weightz .weight_scaleN)dtypedevicec2aa]tRt^ltoV3RltRtVtV;t#).DecompressExperts.convert..DummyModulec<\SV`4\P!VRR7Vn\P!VRR7Vn\P!VRR7VnR#)F) requires_gradN)superr&r Parameter weight_packedr< weight_shape)r%weightscaleshape __class__s&&&&rr&7DecompressExperts.convert..DummyModule.__init__msG "%'\\&%N"$&LLe$L!$&LLe$L!r)rEr<rF)__name__ __module__ __qualname____firstlineno__r&__static_attributes____classdictcell__ __classcell__rJr4s@@r DummyModuler@ls M MrrTrErF)'compressed_tensors.compressorsr9%compressed_tensors.compressors.formatr:r# compressorrr$rweightstypenum_bitsitemsreplace isinstancer enumerateziptor1float32bfloat16emptylenrIr=r>copy_r3rLinearget_value_from_registryModulegetnumeltensorquantization_schemedecompress_modulerG)r%r*r+r,r-kwargsr9r:ct_quantization_configrl weights_args processed_outkeyvalue scale_key quantizedscalesoutputiquantrHdenser3rWrT pack_factorshapes stored_shaperImodules&&&&&, rconvertDecompressExperts.convert>sg BM!%!2!2!=!=!Q!Q"kkWW-?@V-W*22    'L,A,AQ,FM(..0 !S(KK ?C .).#&%/t%<%E5 (.~t6!9 + 0B0B0D0D(E!LL%++a.%++a.;:V)WXE$UE59-@*,,V4>"[[Y>&--*=*=>$mm11%}}33F q  ./&G26!&, "Q-Trc<V^8dQhRR/#)r)r.r)r3r4s"rr5r6sOrc\4#r!)rr%s&r reverse_opDecompressExperts.reverse_ops }rr"r!) rLrMrNrO__doc__r&rpropertyrrPrQr4s@rrr-s0 ggRrrc8V^8dQhR\P/#)r)xr1r2)r3s"rr5r5sELLrcVP\P4pVP4P RR7p\P !V^8V\P !V44pV\, p\P!WPR4, \\R7P\4pWT3#)a"Quantize a 2D tensor to FP8 per-row using pure PyTorch (torch.compile compatible). Args: x: Input tensor of shape (num_rows, hidden_dim). Returns: Tuple of (x_fp8, scales) where x_fp8 is the quantized tensor and scales is a 1D tensor of shape (num_rows,) with per-row scales. )dim)minmax) r`r1raabsamaxwhere ones_like_FP8_MAXclamp unsqueeze_FP8_MIN _FP8_DTYPE)rx_floatrow_maxsafe_maxrvx_fp8s& rquantize_fp8_per_rowrsdd5==!Gkkm  R (G{{7Q;1IJH  F KK"2"22"66H( S V VWa bE =rcV^8dQhR\PR\PR\PR\PR\PR\PR,R\P/#) r)inputrGscale_ascale_b out_dtypebiasNr.)r1r2r=)r3s"rr5r5sx m m << m LL m\\ m\\ m {{ m ,,  m \\ mrc ,\RRR7'dj\PPPp\PPP VVVVP VVP VVR7#\P!WW#WER7#)aRow-wise scaled FP8 matmul dispatcher that uses the public `torch.nn.functional.scaled_mm` (torch >= 2.10) if available, else falls back to `torch._scaled_mm`. Args: input (`torch.Tensor`): FP8 input of shape (M, K), scaled per-row. weight (`torch.Tensor`): FP8 weight of shape (K, N), scaled per-column. scale_a (`torch.Tensor`): Input scales of shape (M, 1), float32. scale_b (`torch.Tensor`): Weight scales of shape (1, N), float32. Returns: `torch.Tensor`: Output tensor of shape (M, N) in `out_dtype`. z2.10T) accept_dev)r output_dtyperrrr)rr1r functional ScalingType scaled_mmRowWise _scaled_mm)rrGrrrrrs&&&&&& r_scaled_mm_rowwisers(!D99hh))55 xx"",,          "-     E7W` llrcXaa]tRt^toRtRV3RlV3RllltV3RlRltRtVtV;t #)CompressedTensorsFP8LinearaLinear layer for compressed-tensors FP8 models. Stores weights in FP8 format and uses a row-wise scaled matmul (`_scaled_mm_rowwise`) for FP8 matmul. Activations are always dynamically quantized per-row via ``quantize_fp8_per_row``, which works for both ``dynamic`` and ``static`` checkpoints (the checkpoint's static input scale, if any, is simply not needed). The weight scale (per-channel or per-tensor) is stored as ``weight_scale``. c,<V^8dQhRS[RS[RS[/#)r) in_features out_featureshas_bias)intbool)r3r4s"rr5'CompressedTensorsFP8Linear.__annotate__s" 2 2C 2s 2d 2rc<\SV`W4W0n\P!\ P !W!\R74Vn\P!\ P!^V3\ PR74Vn VP'd<\P!\ P !VP44Vn R#VPRR4R#))r=rN)rCr&rrrDr1rcrrGzerosrar<rrregister_parameter)r%rrrrJs&&&&rr&#CompressedTensorsFP8Linear.__init__s 3 ll5;;|PZ#[\ LLa5Femm)\] === U[[1B1B%CDDI  # #FD 1rcN<V^8dQhRS[PRS[P/#)r)rr.r)r3r4s"rr5rs#,,U\\,ell,rc X.VPRRORN5pVPRVPR,4p\V4wrE\VVPP 4VP R4VPVPVPR7pVPV4#)Nrr) rIreshaperrrGtrr<r=r)r%r output_shaper x_quantizedx_scalerws&& rforward"CompressedTensorsFP8Linear.forwards/Sb).2. MM"ekk"o .3A6 #  KKMMO%%b)%%kk  ~~l++r)rrrGr<)F) rLrMrNrOrr&rrPrQrRrSs@@rrrs# 2 2,,,rrc^RIHp^RIHpRpVeV!WVR7UUu0uFwrxVkK ppp.p VP 4FwrVe W9dKV!W4'gK!\ V \ P4'gKC\V PV PV PRJR7p VPW4V PV 4K V 'g\PR4V #uuppi)uReplace nn.Linear modules with CompressedTensorsFP8Linear for compressed-tensors FP8 loading. Quantization may only cover part of the model: `targets` / `ignore` are the compressed-tensors patterns (class names or `re:` regexes) from the config — only matching modules are replaced, everything else keeps its dense weights. `modules_to_not_convert` holds additional transformers-side exclusions (e.g. `_keep_in_fp32_modules`). )match_named_modules)should_convert_moduleN)ignore)rrrzYou are loading your model using compressed-tensors FP8 but no linear modules were found. Please double check your model architecture.)compressed_tensors.utils.matchrquantizers.quantizers_utilsr named_modulesr]rrfrrrr set_submoduleappendloggerwarning) modelrrmodules_to_not_convertrrmatchedname_replaced_names module_namer~ new_modules &&&& r*replace_with_compressed_tensors_fp8_linearrsCDG':5RX'YZ'YGD4'YZN$224  ;#= $[II  fbii ( (4"..#00D0J    8  ! !+ . 5"  ;  3[s C8cVPVPR^4^,4p\V\4'dV#R#)zReturn the `CompressedTensorsFP8Linear` owning `full_layer_name` (a `weight_scale` key), or None if the key belongs to another kind of module (e.g. a non-FP8 group of a mixed config)..N) get_submodulersplitr]r)rr-r~s&& r_get_fp8_linearr=s> !7!7Q!?!B CF(BCC6MMrcJa]tRtRtoRtRRlt]V3RlRl4tRtVt R#) ConvertFP8LinearScaleiDa#Reshape the checkpoint ``weight_scale`` into the row-wise kernel layout at load time. compressed-tensors checkpoints store the dequantization scale as ``(out_features, 1)`` (channel strategy) or a single element (tensor strategy). The row-wise scaled matmul in :class:`CompressedTensorsFP8Linear` needs a contiguous float32 ``(1, out_features)`` scale; converting once at load time avoids a transpose + expand + copy on every forward call. Scales of modules that were not replaced with :class:`CompressedTensorsFP8Linear` pass through untouched. Nc \\VP444wrV\V\4'd V^,MTp\ W#4pVfWV/#VP \P4P^R4pVP^,VP8wd+VP^VP4P4pWV/#)r8r)nextiterr[r]rrr`r1rarrIrexpand contiguous)r%r*rr-rnrrrHr~s&&&&, rrConvertFP8LinearScale.convertOs$z//123 &ud33a 8 >< '//26 ;;q>V00 0LLF$7$78CCEE|rc <V^8dQhRS[/#r)r.r)r3r4s"rr5"ConvertFP8LinearScale.__annotate__]s&&M&rc\4#r!)RevertFP8LinearScalers&rr ConvertFP8LinearScale.reverse_op\s #%%rrNN rLrMrNrOrrrrrPrQrs@rrrDs# &&rrcJa]tRtRtoRtRRlt]V3RlRl4tRtVt R#) riaaPSave-time inverse of :class:`ConvertFP8LinearScale`: restore the checkpoint layout, ``(out_features, 1)`` for the channel strategy and ``(1, 1)`` for the tensor strategy. The strategy is read from the config group targeting the module, so the saved checkpoint stays loadable by other compressed-tensors consumers (llm-compressor, vLLM).Nc ^RIHp\\VP 444wrg\ V\ 4'd V^,MTp\W#4pVfWg/#VPR^4^,p VPPPp Rp V PP4FYp V PfKT!YV PV P;'gR4'gKCV PP p M V R8XdWgR,P#4/#WgP%4P'4/#)r8)is_matchNrrkr):NNN:NN)rrrrr[r]rrrconfigrrrrXrrstrategyclonerr) r%r*rr-rnrrrrHr~r ct_configrrs &&&&, rrRevertFP8LinearScale.convertgs ;$z//123 &ud33a 8 >< %,,S!4Q7 LL44HH ,,335E}}(Xk5==ZcZjZjZpZpnp-q-q ==116 x u++-. .WWY))+,,rc <V^8dQhRS[/#rr)r3r4s"rr5!RevertFP8LinearScale.__annotate__}s''M'rc\4#r!)rrs&rrRevertFP8LinearScale.reverse_op|s $&&rrrrrs@rrras$T -*''rrr!)NNN)rr1rcore_model_loadingrrutilsrutils.import_utilsr get_loggerrLrrr float8_e4m3fnrfinforrrrrrrfrrrrrrrrrs% ;:   H %$| |~ ;;z " & & ;;z " & && mF&,&,R*ZN&M&:'='r