+ "ݳjRt^RIt^RIt^RIt^RIt^RIt^RIHt^RIH t H t ^RI H t ^RI HtHtHtHtHtHt^RI Ht^RIt^RIHt^RIHuHuHt^RIt^RIHt^RIHt^RI H!t!^R I"H#t#H$t$H%t%H&t&H't'^R I(H)t)^R I*H+t+^R I,H-t-H.t.H/t/H0t0H1t1H2t2H3t3H4t4H5t5H6t6H7t7H8t8H9t9H:t:H;t;HH?t?H@t@^RIAHBtBHCtCHDtDHEtE^RIFHGtGHHtHHItI^RIJHKtKHLtLHMtMHNtNHOtO^RIPHQtQHRtR^RISHTtT^RIUHVtVHWtWHXtX]P!]Z4t[.RcOt\RdRRllt]Rt^ReRlt_RfRlt`RtaRRRR/Rltb]^R3RRlltc] !R R!]44tdR"RR#R/R$R%llte]@!]d4R&R'R"RR#R/R(R)ll4tf] !R*R+]44tgR,th]@!]g4R&R'/R-R.ll4ti] !R/R0]44tj]@!]j4R&R'/R1R2ll4tk] !R3R4]44tlR5tm]@!]l4R&R'/R6R7ll4tnR8R9lto] !R:R;]44tpR<tq]@!]p4R&R'/R=R>ll4tr] !R?R@]44ts]@!]s4R&R'/RARBll4tt] !RCRD]44tuREtv]@!]u4R&R'/RFRGll4tw] !RHRI]44txRJtyRKRLltz]@!]x4R&R'/RMRNll4t{RORPlt|] !RQRR]44t}R"RR#R/RSRTllt~]@!]}4R&R'R"RR#R/RURVll4t] !RWRX]44t]tRYRZltR[R\ltR]R^ltR_R`ltRaRbltR#)gap Quantization APIs Generally these APIs can be applied directly to any model with Linear modules to obtain quantized linear ops. The intended usage involves applying torch.compile to the model afterwards both because primitives were designed based on the fusions that come along with it and because that is how we access the intended quantized and mixed GEMM kernels N) OrderedDict) dataclassfield)partial)AnyCallableListOptionalTupleUnion) AOBaseConfig) e4m3_dtype) Float8Linear)Float8MMConfigFP8Granularity_check_hardware_support!_granularity_is_a_1_128_w_128_128_normalize_granularity)(Float8StaticActivationFloat8WeightConfig)KernelPreference)Float8PackingFormat&Float8Sparse2x4_1DData1DMetadataTensor&Float8Sparse2x4_2DData2DMetadataTensor Float8TensorInt4ChooseQParamsAlgorithmInt4PackingFormatInt4PlainInt32TensorInt4PreshuffledTensor Int4TensorInt4TilePackedTo4dTensor Int8TensorIntxChooseQParamsAlgorithmIntxOpaqueTensorIntxPackingFormatIntxUnpackedToInt8TensorQuantizeTensorToFloat8KwargsQuantizeTensorToInt8Kwargs)_QUANTIZE_CONFIG_HANDLER register_quantize_module_handler)_fp8_mm_compat_linear_extra_repr_module_extra_repr_quantization_type)is_MI300is_MI350is_sm_at_least_89) GranularityPerAxisPerGroupPerRow PerTensor)Int4WeightOnlyQuantizerInt8DynActInt4WeightQuantizer) intx_quantization_aware_training)_DTYPE_TO_QVALUE_BOUNDS MappingTypequantize_affinecXV^8dQhR\\\R3,,RR/#) extra_args.returnN)r r r)formats"_/home/luca/screen-checking/.venv/lib/python3.14/site-packages/torchao/quantization/quant_api.py __annotate__rAzs-++ sCx) + +c <V!WRR4'd#VeVPVR7V!V.VO5!pV#\VP44pVF3wrx\VVVV V R2VV4p WJgK!V fK'\ WV 4K5 VeVPVR7V#)a Recursively replaces each child module in `model` with the result of `replacement_fn(child)` if `filter_fn(child)` returns `True`. Args: model (torch.nn.Module): The model containing modules to be replaced. replacement_fn (Callable[[torch.nn.Module], torch.nn.Module]): The function to replace matching modules. filter_fn (Callable[[torch.nn.Module], bool]): The filter function to determine which modules to replace. cur_fqn (str, optional): The current fully qualified name of the module being processed. Defaults to "". device (device, optional): Device to move the model to before applying `filter_fn`. Defaults to None. extra_args (Tuple[Any, ...], optional): optional extra args to pass to `replacement_fn`. Returns: None Ndevice.)tolistnamed_children)_replace_with_custom_fn_if_matches_filtersetattr) modelreplacement_fn filter_fncur_fqnrEr=named_children_listnamechild new_childs &&&&&& r@rKrKzs. %%   HHFH #u2z2 "5#7#7#9:.KDA)D6# I%)*?Y//   HHFH # rBc:^RIHp\V\PP 4;'dk\ VR4;'dW\VPV4'*;'d4\V\PPP4'*#))_AffineFakeQuantizedTensorweight) 5torchao.quantization.qat.affine_fake_quantized_tensorrW isinstancetorchnnLinearhasattrrXmoduleslinearNonDynamicallyQuantizableLinear)modargsrWs&* r@ _is_linearrds 3( S S C " S S3::'AB B S S3 1 1 Q QR R rBFc laaaaaSPRR4oSPRR4oVVVVV3RlpV#)a Returns a function which inserts the given subclass into all linear modules in the model. The inserted module will have its weight set to the result of `cls(mod.weight, **kwargs)`. If parametrization is enabled then this will be done using torch.nn.utils.parametrize instead of directly setting the attribute on the module. Args: cls (torch.Tensor): The class to insert as a child module. kwargs (Any): Any additional arguments for the constructor. constructorsubclass_constructormethod from_floatc<S'd\PPSP!VP3/SBRR7VnVPP 4wr\ P!VR\SS4!V!4V#\PP\SS4!VP3/SBRR7VnV#)F requires_gradrX) r[r\ ParameterrirX__tensor_flatten__ parametrizeregister_parametrizationgetattr)lin_rcclsrfenable_parametrizationrikwargss& r@insert_subclass/_get_subclass_inserter..insert_subclasss !++szz4V4E,CJjj335GA  0 0XwsK8$?  ++Z(>v>#,CJ  rB)pop)rtrurvrwrfrisffl @@r@_get_subclass_inserterrzs8**],BCKHl3J" rBca!RR\PP4oV3RlpVfRp\WVR7R#)za Changes all conv2d 1x1 modules to equivalent linear modules so that they can then be quantized. c8aa]tRt^toV3RltRtRtVtV;t#)2swap_conv2d_1x1_to_linear..PermuteSandwichc0<\SV`4WnR#N)super__init__rb)selfrb __class__s&&r@r;swap_conv2d_1x1_to_linear..PermuteSandwich.__init__s G  HrBczVPV^,P^^^^44P^^^^4#)rV)rbpermute)rrcs&*r@forward:swap_conv2d_1x1_to_linear..PermuteSandwich.forwards588DGOOAq!Q78@@Q1M MrB)rb) __name__ __module__ __qualname____firstlineno__rr__static_attributes____classdictcell__ __classcell__)r __classdict__s@@r@PermuteSandwichr}s  N NrBrc\<VPR8XgQh\PPVPVP VP RJR7p\PPVPPRR44VnVP VnS!V4#)N)biasrrrG) kernel_sizer[r\r] in_channels out_channelsrrmrXsqueeze)convrrrs& r@replace_conv2d_1x15swap_conv2d_1x1_to_linear..replace_conv2d_1x1s6)))hhoo   d//tyyD7H XX'' (;(;B(CD 99s##rBNcx\V\PP4;'dVPR8H#)rr)rZr[r\Conv2dr)rbrcs&*r@+swap_conv2d_1x1_to_linear..s1z ( ((((oo'((rB)rO)r[r\ModulerK)rMrOrrs&& @r@swap_conv2d_1x1_to_linearrs< N%((//N$( . YrBcRVPP^, RVPP^, R\VP4 2#)znum_embeddings=z, embedding_dim=z , weight=)rXshaper,rs&r@_embedding_extra_reprrsb T[[..q122B4;;CTCTUVCWBXXabtuyvAvAcBbC DDrBallow_requires_gradpropagate_biasc  aaaaVVVV3RlpV#)zHelper function to apply the constructor that quantizes the weight Tensor (with additional kwargs) to the weight of linear module c"<S;'dVPPpSR8XdVPSR&\PP S!VP3/SBVR7Vn\ P!\V4Vn V#)Trrk) rXrlrr[r\rmtypes MethodTyper* extra_repr)rrrlrrfrvrs& r@rw6_get_linear_subclass_inserter..insert_subclasssy+HH 0H0H T ! XXF6NXX''  -f -](  ))*?H  U[['' ( H rBc.\PPR4\V\4'Ed Ve \ R4h\ VP44pVP4FwrV\WQ4'g9\WeV4'g'RVP9gK;\V4'gKN\WeV4pVeVPVR7pWvJgKwVR8wgKVPR4R ,pVP!V4P!R4p WI,p \#WV4K R#\V\$4'd6Vf\MTp\&\)V4,p \+VV VVV3R7R#\-R 4h) anConvert the weight of linear modules in the model with `config`, model is modified inplace Args: model (torch.nn.Module): input model config (AOBaseConfig): a workflow configuration object. filter_fn (Optional[Callable[[torch.nn.Module, str], bool]]): function that takes a nn.Module instance and fully qualified name of the module, returns True if we want to run `config` on the weight of the module device (device, optional): Device to move module to before applying `filter_fn`. This can be set to `"cuda"` to speed up quantization. The final model will be on the specified `device`. Defaults to None (do not change device). Example:: import torch import torch.nn as nn from torchao import quantize_ # quantize with some predefined `config` method that corresponds to # optimized execution paths or kernels (e.g. int4 tinygemm kernel) # also customizable with arguments # currently options are # Int8DynamicActivationInt8WeightConfig (optimized with int8 mm op and torch.compile) # Int4WeightOnlyConfig (optimized with int4 tinygemm kernel and torch.compile) # Int8WeightOnlyConfig (optimized with int8 mm op and torch.compile from torchao.quantization.quant_api import Int4WeightOnlyConfig m = nn.Sequential(nn.Linear(32, 1024), nn.Linear(1024, 32)) quantize_(m, Int4WeightOnlyConfig(group_size=32)) ztorchao.quantization.quantize_NzuCustom filter_fn and FqnToConfig were both specified. Only filter_fn=None is supported when FqnToConfig is specified._defaultrDrF)rEr=zPassing a generic Callable to `quantize_` is no longer recommended and will be deprecated at a later release. Please see https://github.com/pytorch/ao/issues/1690 for instructions on how to pass in workflow configuration instead.rG)r[_C_log_api_usage_oncerZ FqnToConfig ValueErrordict named_modulesitemsfqn_matches_fqn_config _module_param_matches_fqn_config fqn_to_configrd_fqn_to_config_handlerrHsplit removesuffixrLr r'typerKAssertionError) rMrrOrEr module_fqnmodule replacement child_name parent_fqn parent_modulehandlers &&&& r@ quantize_rsbF HH  !AB&+&&  H U0023 "/"5"5"7 J&z::3FOO&"6"66:f;M;M4VP %"-..."?K,r1A!+!1!1#!6r!:J!+!8!8!D!Q!QRU!VJ$1$=MM{C#8 FL ) )"+"3J *4<81   y   x  rBca]tRtRtoRt]P t]!^ 4t ] Pt Rt ] Pt]P"t]P(t^tRtV3RltRtVtR#)%Int8DynamicActivationIntxWeightConfigi^a^ Configuration for dynamically quantizing activations to torch.int8 and weights to torch.intx, with 1 <= x <= 8. More specifically, activations are dynamically quantized to 8-bits at a per-token granularity with scales/zeros. Weights are quantized with scales/zeros in a groupwise or channelwise manner using the number of bits specified by weight_dtype. args: `weight_dtype`: The dtype to use for weight quantization. Must be torch.intx, where 1 <= x <= 8. `weight_granularity`: The granularity to use for weight quantization. Must be PerGroup or PerAxis(axis=0). `weight_mapping_type`: The type of mapping to use for the weight quantization. Must be one of MappingType.ASYMMETRIC or MappingType.SYMMETRIC. MappingType.SYMMETRIC requires ZeroPointDomain.NONE `weight_scale_dtype`: The dtype to use for the weight scale. `act_mapping_type`: The type of mapping to use for the activation quantization. Must be one of MappingType.ASYMMETRIC or MappingType.SYMMETRIC. `intx_packing_format`: The format to use for the packed weight tensor (version 2 only). - unpacked_to_int8: this format is the default and is intended for export applications like ExecuTorch. - opaque_torchao_auto: this format is optimized for CPU performance. `intx_choose_qparams_algorithm`: The algorithm to use for choosing the quantization parameters. `version`: version of the config to use, only subset of above args are valid based on version, see note for more details. Example: .. literalinclude:: ../../examples/inference/int8_dynamic_activation_intx_weight.py :language: python Nc >\PPR4VP\ ^^ 4Uu.uFp\ \RV 24NK up9gQRVP 24h\ VP\\34'gQRVP 24h\ VP\4'd:VPP^8XgQRVPP 24hVP\P\P\P39gQRVP 24hVP \P\P39gQRVP 24hR#uupi)z:torchao.quantization.Int8DynamicActivationIntxWeightConfigint $TT J4K\K\J] ^ T$11GX3FGG FtG^G^F_ ` G d--w 7 7**//14 *4+B+B+G+G*HI 4''  " "  ! !  1 1,   MNRNfNfMg h   $$  " "  ! !)   aaeavav`w x   #%UsFc<V^8dQh/S[P;R&S[;R&S[;R&S[S[P,;R&S[;R&S[;R&S[;R&S[;R&#) r<rrrweight_scale_dtyperintx_packing_formatintx_choose_qparams_algorithmversionr[dtyper0r9r r#r!r)r?rs"r@rA2Int8DynamicActivationIntxWeightConfig.__annotate__^s:++*;<$2=>%GNOrBr)rrrr__doc__r[int8rr2rr9rrrrrr#UNPACKED_TO_INT8rr!AFFINErrr__annotate_func__rrrs@r@rr^sk6!& L&.rl'2'<'<04$/$:$:->-O-O"))"G SrBr custom_scalecustom_zero_pointcV^8dQhR\\P,R\\P,/#r<rrr r[Tensor)r?s"r@rArAs6G G 5<<( G   - G rBc tVPpVPpVPpVPpVPp VP p VP p VP4^8XgQRVP4 24h\V\4'dVPp M]\V\4'd:VP^8XgQRVP 24hVPR ,p M\RV 24h^V 3p VP^8XgQhV \ P"8XgQh\$P&\$P(\$P*\$P,.pV \$P.8XgW9g QRV 24hVe?VP0\2P48Xd VP7\2P84p\:P<!VV VVRV VVR7pVeWP08wd \?WV4TpW9d\@PB!VVV R7pRpVV3#) r<zFInt8DynamicActivationIntxWeightConfig only works for 2-d Tensor, got: %axis must be 0 with PerAxis, but got z4weight_granularity must be PerGroup or PerAxis, got Unsupported packing format: Nint8_asym_per_token) mapping_typeactivation_quantizationrrr)rrrG)"rrrrrrrdimrZr2 group_sizer1rrrrr9rr#OPAQUE_ATEN_KLEIDIAIOPAQUE_TORCHAO_AUTOOPAQUE_TORCHAO_KLEIDIAIOPAQUE_TORCHAO_LOWBITrrr[int32rHrr$from_hp+_adjust_scale_dtype_in_intx_unpacked_tensorr"!from_intx_unpacked_to_int8_tensor)rXrrrrrrrrrrrr block_sizeopaque_formats new_weightnew_biass&&&$$ r@4_int8_dynamic_activation_intx_weight_quantize_tensorr sF&&L22 4422.. 44$*$H$H! ::<1  PQWQ[Q[Q]P^_ $h//'22 & 0 0!&&!+ 34F4K4K3L M +\\"% BCUBV W  ZJ >>Q   {55 55 5..--11// N 0AAA  0< &&9%:;< 1$):)@)@EKK)O-00<)11( 5&C!+ J%*< *L3  2 H,%GG X;N  x rBparameter_namerXc V^8dQhR\PPR\R\R\ \P ,R\ \P ,R\PP/#r<rrr rrr>)r[r\rrrr r)r?s"r@rArAsf HHOO 1  5<<(    -  XX__rBc *\\W4VPVVVR7wrV\VV\P P VRR74VfRVn\P!\\VPVR7V4Vn V#)rrFrkNoriginal_extra_reprr ) r rqrrLr[r\rmrrrr+r)rrr rrr r s&&$$$ r@._int8_dynamic_activation_intx_weight_transformrsP' !+ J  :U;  ((  & 1 1)  F MrBcva]tRtRtoRt^tRt]Pt ] Pt ^t ^tRtV3RltRtVtR#)Int4WeightOnlyConfigia Configuration for int4 weight only quantization, only groupwise quantization is supported. Args: `group_size`: parameter for quantization, controls the granularity of quantization, smaller size is more fine grained, choices are [256, 128, 64, 32] `int4_packing_format`: the packing format for int4 tensor `int4_choose_qparams_algorithm`: variants of choose qparams algorithm to use for int4, currently support TINYGEMM ("tinygemm") and HQQ ("hqq") `set_inductor_config`: if True, adjusts `torchinductor` settings to recommended values. `int4_tile_packed_ntile`: ntile size for TILED_PACKED_TO_4D format, default is 8 for CUDA platform, 16 for ROCm platform Example: .. literalinclude:: ../../examples/inference/int4_weight_only.py :language: python TctVPR9gQR4h\PPR4R#)z-int4_tile_packed_ntile must be either 8 or 16z)torchao.quantization.Int4WeightOnlyConfigN)r)int4_tile_packed_ntiler[rrrs&r@r"Int4WeightOnlyConfig.__post_init__-s5**g5 ; 5 $$%PQrBcb<V^8dQh/S[;R&S[;R&S[;R&S[;R&S[;R&S[;R&#)r<rset_inductor_configint4_packing_formatint4_choose_qparams_algorithmrr)rrrr)r?rs"r@rA!Int4WeightOnlyConfig.__annotate__sU()*$+,+D-.$>/4 #567rBrN)rrrrrrrrPLAINrrTINYGEMMrrrrrrrrs@r@rrsK$J $->-D-D"++"#$GR;rBrcVPpVPpVPpVPpVPR,V,^8wd(\ P RVP RV 24V#\\VP^, 4Uu.uFp^NK upV.,4pVP^8XgQh\V4pV\P8Xd!V\P8Xg QRV R24hV\P 8Xd*\"P$!VV\&P(R7pV#V\P*8Xd\,P$!VV4pV#V\P.8Xd\0P$!VV4pV#V\P8Xd\2P$!VVVVR7pV#\5RV 24huupi) rzZSkipping quantizing weight with int4 weight only quantization because the shape of weight z# is not compatible with group_size zBInt4ChooseQParamsAlgorithm.HQQ is not supported by packing format zF, it's only supported by Int4PackingFormat.TILE_PACKED_TO_4D currentlyactivation_dtype)r ntile_sizez!Unsupported int4 packing format: rG)rrrrrloggerinfotuplerndimrrIrHQQrTILE_PACKED_TO_4D PRESHUFFLEDrrr[bfloat16r r PLAIN_INT32rrr) rXrrrrrrsrr s && r@!_int4_weight_only_quantize_tensorr/4s ""J$*$H$H! 44#:: ||B*$) hioiuiuhvwZ[eZf g  5q#9:#9a#9:j\IJJ >>Q  j!J$(B(F(FF"&7&I&II PQdPefS T I /;;;*22  "^^   1 7 7 7''     1 = = =)11     1 C C C-55  *G-  <=P)r[r\rrr)r?s"r@rArAps@ HHOO    XX__ rBc VP'd)\PPP 4\ W4'gQRV R2RV R2,4h\ \W4V4p\VV\PPVRR74\P!\\VP VR7V4VnV#)z8applying int4 weight only quant requires module to have attribute but  does not have oneFrkr)rtorchao quantizationutils"recommended_inductor_config_setterr^r/rqrLr[r\rmrrrr+rrrr r s&&$ r@_int4_weight_only_transformr;os!!!""EEG 6 * * B>BRR\] &+ , - *3'J  :U; ((  & 1 1)  F MrBc4a]tRtRtoRtRtV3RltRtVtR#)'Float8DynamicActivationInt4WeightConfigiaConfiguration for apply float8 dynamic per row quantization and int4 per group weight quantization to linear (only group_size 128 is supported right now since underlying kernel used only supports 128 and above and no benefits of making it bigger) Args: `int4_packing_format`: how the weight is packed, supported values are "preshuffled" and "plain" Example: .. literalinclude:: ../../examples/inference/float8_dynamic_activation_int4_weight.py :language: python preshuffledc&<V^8dQh/S[;R&#)r<r)r)r?rs"r@rA4Float8DynamicActivationInt4WeightConfig.__annotate__s +:!rBrN) rrrrrrrrrrs@r@r=r=s .;!rBr=cV^8dQhR\PPR\R\R\PP/#r1)r[r\rr=r)r?s"r@rArAs@11 HHOO1 31 1  XX__ 1rBc \W4'gQRV R2RV R2,4hVPpVR 9g QRV 24h\W4p^p\\ VP ^, 4Uu.uFp^NK upV.,4pVR8Xd3\ P!VPV\PR7pM1\P!VPV\PR7p\VV\PPVRR74\P !\#\$VP&VR 7V4VnV#uupi) zMapplying float8 dynamic activation int4 weight quant requires module to have r3r4r5r>zIonly preshuffled and plain int4_packing_format supported right now, got: r#Frkr)r>plain)r^rrqrIrr)rrrXr[ float8_e4m3fnrrLr\rmrrrr+r) rrr rrXrrsrr s &&$ r@0_float8_dynamic_activation_int4_weight_transformrEsj 6 * * WXfWggqr &+ , - *!44 #  TTgShi   V ,FJ% a"89"8Qq"89ZLHIJm+*22 MM "00  '' MM "00   :U; ((  & 1 1)  F M;:s5 EcPa]tRtRtoRtRt]!4tRt^t Rt V3Rlt Rt Vt R#)Int8WeightOnlyConfigia Configuration for applying int8 weight-only symmetric per-channel quantization to linear layers. Args: group_size (version 1) - Controls the granularity of quantization. If None, applies per-channel quantization. Otherwise, applies per-group quantization with the specified group size. granularity (version 2) - Quantization granularity. PerRow() for per-channel quantization, PerTensor() for per-tensor quantization, PerGroup(group_size) for per-group quantization. set_inductor_config: bool = True - If True, adjusts `torchinductor` settings to recommended values for better performance with this quantization scheme. Example: .. literalinclude:: ../../examples/inference/int8_weight_only.py :language: python NTcZ\PPR4VP^8Xd \ R4hVP e#QRVP RVP R24h\ VP\\\34'gQRVP 24hR#)z)torchao.quantization.Int8WeightOnlyConfigzversion 1 of Int8WeightOnlyConfig has been removed, please use version 2, see https://github.com/pytorch/ao/issues/2752 for more detailsNz1Only support version 2 with group_size=None, got z. Use granularity=PerGroup(z ) instead.z>Q G"6v~~6F GG ##F8J8JKJ rBcdV^8dQhR\PPR\R\/#r<rrr )r[r\rrGr)r?s"r@rArAs. HHOO  rBc VP'd)\PPP 4\ W4'gQR4h\ \W4V4p\VV\PPVRR74\P!\\VP VR7V4VnV#)zRapplying int8 weight only quant requires module to have {parameter_name} attributeFrkrzqapplying int8 weight only quant requires module to have {parameter_name} attribute but {module} does not have one)rr6r7r8r9r^rPrqrLr[r\rmrrrr+rrrr quantized_tensors&&$ r@_int8_weight_only_transformrVs!!!""EEG 6 * * , *9'  +5A ((  & 1 1)  F MrBc4V^8dQhR\R\RR/#)r<act_granularityrr>N)r0)r?s"r@rArA(s&    #   rBcB\\3p\W4'g\R\ V4 R24h\V\4'd+VP R8wd\RVP R24h\W4'g\R\ V4 R24hR#)z)Unsupported activation granularity type: z*. Only PerTensor and PerRow are supported.zMOnly PerRow(dim=-1) is supported for activation quantization, got PerRow(dim=zS). Per-feature activation quantization is not supported due to slicing limitations.z%Unsupported weight granularity type: NrG)r4r3rZrrr)rXr supporteds&& r@_validate_granularity_int8r[(sF#I o 1 17_8M7NO7 8  /6**/B/Bb/H-1123_ `  ( 4 43D9K4L3MN7 8   5rBcla]tRtRtoRt]P tRt] !4t Rt ^t Rt RtV3RltRtVtR#) %Int8DynamicActivationInt8WeightConfigiBa Configuration for applying int8 dynamic per-token activation and int8 per-channel weight quantization to linear layers. Args: granularity: Optional[Union[Granularity, List[Granularity]]] = PerRow() The granularity for quantization. Can be either a single granularity (applied to both activations and weights) or a list of two granularities (first for activations, second for weights). If None, defaults to PerRow for both. Only PerTensor and PerRow are supported. act_mapping_type: Optional[MappingType] = MappingType.SYMMETRIC - Mapping type for activation quantization. SYMMETRIC and ASYMMETRIC are supported. set_inductor_config: bool = True - If True, adjusts `torchinductor` settings to recommended values for better performance with this quantization scheme. version (int): the version of the config reduce_range (Optional[bool] = False): If True, use reduced activation and weight quantization ranges to avoid overflow on CPU without VNNI. Users can call should_reduce_range() to help determine. Example: .. literalinclude:: ../../examples/inference/int8_dynamic_activation_int8_weight.py :language: python FTcB\PPR4VP^8Xd \ R4h\ P !VP4wr\W4VP\P\P39gQR4hR#)z:torchao.quantization.Int8DynamicActivationInt8WeightConfigzversion 1 of Int8DynamicActivationInt8WeightConfig has been removed, please use version 2, see https://github.com/pytorch/ao/issues/2752 for more detailszInt8DynamicActivationInt8WeightConfig requires `act_mapping_type` in (MappingType.SYMMETRIC, MappingType.ASYMMETRIC). Please set it to MappingType.SYMMETRIC or MappingType.ASYMMETRIC.N) r[rrrrr rrIr[rr9rrrrXrs& r@r3Int8DynamicActivationInt8WeightConfig.__post_init__bs $$ H  <<1 Q /9.O.O   / + #?G$$  ! !  " ")   &  rBc<V^8dQh/S[S[,;R&S[;R&S[S[S[S[S[,3,,;R&S[;R&S[;R&S[S[,;R&#)r<rweight_only_decoderIrr reduce_range)r r9rr r0rIr)r?rs"r@rA2Int8DynamicActivationInt8WeightConfig.__annotate__Bsp2{+C34$56% T+-> >?@K78$9:;<4.(=rBrN)rrrrrr9rrrbr3rIrrrcrrrrrs@r@r]r]BsB./:.C.C$CI8K $G#(L ArBr]c VP^8XgQRVP 24h\P!VP4wr#\P!VV\ VVP VPR7VPR7pV#)r<rNrIrrc)rIact_quant_kwargsrc)rr rrIrr&rrc)rXrrXrquantized_weights&& r@4_int8_dynamic_activation_int8_weight_quantize_tensorri{s >>Q G"6v~~6F GG *4*K*K+'O"))&3'00,, ((  rBcV^8dQhR\PPR\R\PP/#)r<rrr>)r[r\rr])r?s"r@rArAs6 HHOO 1  XX__ rBc VP'd)\PPP 4\ W4'gQRV R2RV R2,4h\ \W4V4p\VV\PPVRR74\P!\\VP VR7V4VnV#)zKapplying int8 dynamic activation int8 weight quant requires module to have r3r4r5Frkr)rr6r7r8r9r^rirqrLr[r\rmrrrr+rr:s&&$ r@._int8_dynamic_activation_int8_weight_transformrls!!!""EEG 6 * * UVdUeeop &+ , - *F'J  :U; ((  & 1 1)  F MrBca]tRtRtoRtRtRt]!4t] Pt Rt ^t RtRtV3RlRltV3R ltR tVtR#) $Int8StaticActivationInt8WeightConfigiac Configuration for applying int8 static quantization to both activation and weight Args: act_quant_scale (torch.Tensor): The scale tensor for activation quantization. act_quant_zero_point (torch.Tensor): The zero_point tensor for activation quantization (asymmetric only). granularity (Optional[Union[Granularity, List[Granularity]]] = PerRow()): The granularity for quantization. Can be either a single granularity (applied to both activations and weights) or a list of two granularities (first for activations, second for weights). If None, defaults to PerRow for both. Only PerTensor and PerRow are supported. act_mapping_type (MappingType): The mapping type for activation quantization. SYMMETRIC and ASYMMETRIC are supported. set_inductor_config (bool): if True, adjusts `torchinductor` settings to recommended values. version (int): the version of the config reduce_range (Optional[bool] = False): If True, use reduced activation and weight quantization ranges to avoid overflow on CPU without VNNI. Users can call should_reduce_range() to help determine. NTFc \PPR4\P!VP 4wr\ W4VP\P\P39gQR4hR#)z9torchao.quantization.Int8StaticActivationInt8WeightConfigzInt8StaticActivationInt8WeightConfig requires `act_mapping_type` in (MappingType.SYMMETRIC, MappingType.ASYMMETRIC). Please set it to MappingType.SYMMETRIC or MappingType.ASYMMETRIC.N) r[rrr rrIr[rr9rrr_s& r@r2Int8StaticActivationInt8WeightConfig.__post_init__sx $$ G /9.O.O   / + #?G$$  ! !  " ")   &  rBc <V^8dQhRS[/#)r<r>)r&)r?rs"r@rA1Int8StaticActivationInt8WeightConfig.__annotate__s  &@ rBc \P!VP4wr\VVPVP R7#)zGet the activation quantization kwargs for static quantization. Returns: QuantizeTensorToInt8Kwargs with the configured granularity and mapping type. rf)r rrIr&rrcr_s& r@get_act_quant_kwargs9Int8StaticActivationInt8WeightConfig.get_act_quant_kwargssD /9.O.O   / +*'..**  rBc <V^8dQh/S[S[P,;R&S[S[P,;R&S[S[S[S[S[,3,,;R&S[S[,;R&S[;R&S[;R&S[S[,;R&#)r<act_quant_scaleact_quant_zero_pointrIrrrrc) r r[rr r0rIr9rr)r?rs"r@rArrs&ell+2'(#5<<07)*% T+-> >?@K+,{+C-.$/0124.(3rBr)rrrrrrwrxr3rIr9rrrrrcrrtrrrrs@r@rnrnsQ"/3O37CI8K.9.C.C $G#(L &  ]rBrncXV^8dQhR\PPR\/#)r<rr)r[r\rrn)r?s"r@rArAs$.. HHOO. 0.rBc \P!VP4wr4\W4'g QRV R24hVP'd)\ P PP4RpVPeVPP4p\P!\W4V\VVPVPR7VP P4VVPR7p\#VV\$P&P)VRR74\*P,!\/\0VP2VR7V4VnV#)z#Expected module to have attribute `z` but not foundNrf)rIrgrwrxrcFrkr)r rrIr^rr6r7r8r9rxdetachrrqr&rrcrwrLr[r\rmrrrr+r)rrr activation_granularityrrxrUs&&$ r@-_int8_static_activation_int8_weight_transformr}sN2<1R1R2. 6 * * -n-=_M *!!!""EEG "".%::AAC!))'&3.00,, ..5571((   +5A ((  & 1 1)  F MrBcFa]tRtRtoRt]tRt^tRt Rt V3Rlt Rt Vt R#)Float8WeightOnlyConfigi!a Configuration for applying float8 weight-only symmetric quantization to linear layers. Args: weight_dtype (torch.dtype): The target data type for weight quantization. Default is torch.float8_e4m3fn. set_inductor_config (bool): if True, adjusts `torchinductor` settings to recommended values. version (int): the version of the config, version 1 is deprecated, version 2 is using Float8Tensor (default) granularity (Granularity): quantization granularity. Supported: PerTensor, PerRow (default), PerGroup. Note: The actual matmul will be computed in original precision of the weight tensor. Example: .. literalinclude:: ../../examples/inference/float8_weight_only.py :language: python TNcVPf\4Vn\VP\\\34'gQR\ VP4 24h\ PPR4R#)Nz8granularity must be PerTensor, PerRow, or PerGroup, got z+torchao.quantization.Float8WeightOnlyConfig) rIr3rZr4r2rr[rrrs&r@r$Float8WeightOnlyConfig.__post_init__:sj    #%xD $**Y,IJJ FtDL\L\G]F^ _ J $$%RSrBc^<V^8dQh/S[P;R&S[;R&S[;R&S[;R&#)r<rrrrI)r[rrrr0)r?rs"r@rA#Float8WeightOnlyConfig.__annotate__!s=(++*)*$+,-.#/rBrO)rrrrrr rrrrIrrrrrs@r@rr!s.$!+L $G#KT3rBrcVP^8XgQRVP 24hVPp\P!WVPR7pV#)r<rN) float8_dtyperI)rrrrrI)rXrrr s&& r@ _float8_weight_only_quant_tensorrCsT >>Q G"6v~~6F GG &&L%%v7I7IJ rBcV^8dQhR\PPR\R\R\PP/#r1)r[r\rrr)r?s"r@rArAMs@"" HHOO" "" "  XX__ "rBc VP'd)\PPP 4\ W4'gQR4h\ V\4'd \V4p\\W4V4p\VV\PPVRR74\P !\#\$VP&VR7V4VnV#)zTapplying float8 weight only quant requires module to have {parameter_name} attributeFrkrzsapplying float8 weight only quant requires module to have {parameter_name} attribute but {module} does not have one)rr6r7r8r9r^rZr_unwrap_float8_linearrrqrLr[r\rmrrrr+rrTs&&$ r@_float8_weight_only_transformrLs!!!""EEG 6 * * , * &,''&v.7'  +5A ((  & 1 1)  F MrBca]tRtRtoRt]t]tRt] Pt Rt Rt Rt]P tRt^t^tRtV3RltRtVtR#))Float8DynamicActivationFloat8WeightConfigira~ Configuration for applying float8 dynamic symmetric quantization to both activations and weights of linear layers. Args: activation_dtype (torch.dtype): The target data type for activation quantization. Default is torch.float8_e4m3fn. weight_dtype (torch.dtype): The target data type for weight quantization. Default is torch.float8_e4m3fn. granularity (Optional[Union[FP8Granularity, List[FP8Granularity]]]): The granularity for quantization. Can be either a single granularity (applied to both activations and weights) or a tuple of two granularities (one for activations, one for weights). If None, defaults to PerTensor for both. Currently both quantizations need to be the same type. And only PerTensor and PerRow are supported. mm_config (Float8MMConfig): Configuration for the matrix multiplication. Default uses fast accumulation. activation_value_lb (Optional[float]): the lower bound for activation value for calculating scale activation_value_ub (Optional[float]): the upper bound for activation value for calculating scale kernel_preference (KernelPreference): kernel preference for ops like matmul, grouped matmul etc. by defalut (KernelPreference.AUTO) it will be chosen for user based on hardware or other information, this only needs to be set in weight set_inductor_config (bool): if True, adjusts `torchinductor` settings to recommended values. version (int): the version of the config, version 1 is deprecated, version 2 is using Float8Tensor (default) Example: .. literalinclude:: ../../examples/inference/float8_dynamic_activation_float8_weight.py :language: python NTc\PPR4\VP4wrW.VnRp\ VP4'dQVP \P\P39gQR4hVP^8gQR4hRp\PP4'dRpVPf\VR7Vn R#R#)z>torchao.quantization.Float8DynamicActivationFloat8WeightConfigT unimplementedFN)use_fast_accum)r[rrrrIrkernel_preferencerAUTOTORCHrxpu is_available mm_configr)rr|rdefault_use_fast_accums& r@r7Float8DynamicActivationFloat8WeightConfig.__post_init__s $$ L 6L   6 23G!% ,T-=-= > >)) %% &&.  <<1$ 5o 5$%* " 99 ! ! # #%* " >> !+;QRDN "rBc<<V^8dQh/S[P;R&S[P;R&S[S[S[S[S[,3,,;R&S[S[,;R&S[S[,;R&S[S[,;R&S[S[,;R&S[ ;R&S[ ;R &S[ ;R &S[ ;R &#) r<r$rrIpacking_formatractivation_value_lbactivation_value_ubrrralg_id) r[rr r rrrrfloatrrr)r?rs"r@rA6Float8DynamicActivationFloat8WeightConfig.__annotate__rs4kk.56++*78%^0D DEFM9:01M;<'.=>"%/?@"%/AB(?CD$EFGH OIrB)rIr)rrrrrr r$rrIrr rrrrrrrrrrrrrrrs@r@rrrsb0%/ *LIMK4G4M4MN*.I+/+/*:*?*? $GFSMrBrc VPpVPpVPpVPpVPpVP pVP pVPp \V4VwrVP4R 9du\V \4'd\V \4'gQR4hVP^,^,^8wg VP^,^,^8wdV#M\V4'gV#VP^8XgQRVP 24hV \P 8Xd=\V \"4'd'VP$\&P(8XgQR4h\+VV VVVR7p V \P 8Xd\,P.!VVV VVV R7p V #V \P08Xd:\V \"4'gQR4h\2P.!VVV V R7p V #V \P48XdE\V \4'gQR4h\6P.!VVV V VP8R 7p V #R #) zH4D/5D tensor only supports per tensor activation and weight quantizationrNzBPerRow quantization only works for bfloat16 precision input weight) hp_value_lb hp_value_ubr)rrIrrrgz8Sparse packing format only supports per-row quantization)rrIrgzOSparse 1D data 1D metadata packing format only supports per-tensor quantization)rrIrgrN)r)r$rrIrrrrrrrrZr4rr)rrr r3rr[r-r%rrSPARSE_2D_DATA_2D_METADATArSPARSE_1D_DATA_1D_METADATArr)rXrr$rrIrrrrrr|rrgrhs&& r@8_float8_dynamic_activation_float8_weight_quantize_tensorrss..&&L$$K  I 44 4400**NK(1<.zz|v0)<<  B B  V U V  <<?R 1 $ Q"(<(AM)B F # # >>Q G"6v~~6F GG ,222zF88||u~~- P -4''+ ,222'// %*/-    .II I,f55 F 5BII %*-     .II I,i88 ] 8BII %*-==    JrBc(V^8dQhR\RR/#)r<rr>N)r)r?s"r@rArAs%% 5% %rBcR\PP4'gR#\PPe0\PPP R4'gR#\ ;QJd&RVP 4F 'gK RM RM!RVP 44'gR#VP\P8XdR#VPeVPP'gR#\P!R4R#)Nz12.9c3B"TFp\V\4xK R#5ir)rZr3).0gs& r@ 4_maybe_warn_rowwise_fp8_cuda_12_9..sA.@z!V$$.@sTFaRowwise fp8 (Float8DynamicActivationFloat8WeightConfig with PerRow()) on CUDA 12.9 can crash with CUBLAS_STATUS_NOT_SUPPORTED for some large-K + small-M/N GEMM shapes, due to a cuBLASLt regression specific to CUDA 12.9 (not present in 12.6, fixed in 13.0). We can't reliably predict which shapes fail, so we warn rather than guard. If you hit this, work around it with any of: (1) upgrade to CUDA 13.0 (or downgrade to 12.6), where the regression is not present; or on the affected layers (e.g. via FqnToConfig): (2) the mslk kernel, which uses CUTLASS instead of cuBLASLt and is unaffected: Float8DynamicActivationFloat8WeightConfig(granularity=PerRow(), kernel_preference=KernelPreference.MSLK). This requires the mslk package to be installed (see https://github.com/pytorch/MSLK); or (3) disabling fast accumulation, which avoids the failing cuBLASLt path: Float8DynamicActivationFloat8WeightConfig(granularity=PerRow(), mm_config=Float8MMConfig(use_fast_accum=False)). See https://github.com/pytorch/ao/issues/4582.)r[cudarr startswithanyrIrrMSLKrrwarningswarn)rs&r@!_maybe_warn_rowwise_fp8_cuda_12_9rs :: " " $ $ }}!););)F)Fv)N)N 3Af.@.@A333Af.@.@A A A #3#8#88#F,<,<,K,K,K MM 9rBcdV^8dQhR\PPR\R\/#rR)r[r\rrr)r?s"r@rArA.s.$$ HHOO$ 5$ $rBc \V4\PP4'd8\ 4'g(\ 4'g\ 4'gQR4hVP'd)\PPP4\W4'gQRV R2RV R2,4h\V\4'd \V4p\!\#W4V4p\%VV\P&P)VRR74\*P,!\/\0VP2VR7V4VnV#)zPFloat8 dynamic activation quantization is only supported on CUDA>=8.9 and MI300+zKapplying float8 dynamic activation quant requires module to have parameter r3r4r5Frkr)rr[rrr/r-r.rr6r7r8r9r^rZrrrrqrLr\rmrrrr+rrTs&&$ r@2_float8_dynamic_activation_float8_weight_transformr-s1&f- zz   ""hjjHJJ ^ >!!!""EEG 6 * * UVdUeeop &+ , - *&,''&v.O'  +5A ((  & 1 1)  F MrBchV^8dQhR\R\PR\PRR/#)r<intx_unpacked_tensor hp_tensor scale_dtyper>N)r$r[rr)r?s"r@rArAUs82|| rBc .\V\4'gQhVPPV4Vn\VP ,wr4\ VVPVPVP\PVVR7Vn R#)aP Adjusts the scale_dtype on IntxUnpackedToInt8Tensor. Updating the scale dtype requires updating the qdata because qdata is calculated after the scale. This is used in IntxWeightOnlyConfig and Int8DynamicActivationIntxWeightConfig to make version=2 and version=1 numerically equivalent when the scale_dtype differs from the input dtype ) output_dtype quant_min quant_maxN) rZr$scalerHr8 target_dtyper:r zero_pointr[rqdata)rrrqminqmaxs&&& r@rrUs *,D E EE E!5!;!;!>!>{!K()=)J)JKJD!0''""''ZZ"rBca]tRtRtoRt]P t]!^4t ] Pt Rt ]Pt]P$t^tRtV3RltRtVtR#)IntxWeightOnlyConfigina Configuration for quantizing weights to torch.intx, with 1 <= x <= 8. Weights are quantized with scales/zeros in a groupwise or channelwise manner using the number of bits specified by weight_dtype. args: `weight_dtype`: The dtype to use for weight quantization. Must be torch.intx, where 1 <= x <= 8. `granularity`: The granularity to use for weight quantization. Must be PerGroup or PerAxis(0). `mapping_type`: The type of mapping to use for the weight quantization. Must be one of MappingType.ASYMMETRIC or MappingType.SYMMETRIC. `scale_dtype`: The dtype to use for the weight scale. `intx_packing_format`: The format to use for the packed weight tensor (version 2 only). `intx_choose_qparams_algorithm`: The algorithm to use for choosing the quantization parameters. `version`: version of the config to use, only subset of above args are valid based on version, see note for more details. Example: .. literalinclude:: ../../examples/inference/intx_weight_only.py :language: python Nc \PPR4VP\ ^^ 4Uu.uFp\ \RV 24NK up9gQRVP 24h\ VP\\34'gQRVP 24h\ VP\4'd:VPP^8XgQRVPP 24hVP\P\P\P39gQRVP 24hR#uupi)z)torchao.quantization.IntxWeightOnlyConfigrrz1granularity must be PerAxis or PerGroup, but got rzvmapping_type must be MappingType.ASYMMETRIC, MappingType.SYMMETRIC, or MappingType.SYMMETRIC_NO_CLIPPING_ERR, but got N)r[rrrrrqrZrIr1r2rrr9rrrrs& r@r"IntxWeightOnlyConfig.__post_init__sV $$%PQ  aQR $T 1WUc!I%> $TT J4K\K\J] ^ T$**Wh,?@@ ?@P@P?Q R @ d&& 0 0##((A- 78H8H8M8M7NO -   " "  ! !  1 1%   EFJFWFWEX Y   %UsEc<V^8dQh/S[P;R&S[;R&S[;R&S[S[P,;R&S[;R&S[;R&S[;R&#)r<rrIrrrrrr)r?rs"r@rA!IntxWeightOnlyConfig.__annotate__nso,++*-.)/0512%++&-34+O56$>7<=rBr)rrrrrr[rrr1rIr9rrrr#rrr!rrrrrrrrs@r@rrns\(!& L&qzK + 5 5L)-K->-O-O"))"G ArBrcV^8dQhR\\P,R\\P,/#rr)r?s"r@rArAs6;O;O5<<( ;O   - ;OrBc VPpVPpVPpVPpVPpVP p VP 4^8XdRp M4VP 4^8Xd^p M\RVP 4 24h\V\4'dVPp M]\V\4'd:VP^8XgQRVP 24hVPV ,p M\RV 24hVP 4^8Xd^V 3p MVP 4^8XgQh^V ^^3p VP^8XgQhVP\P 8XdVe?VP"\$P&8Xd VP)\$P*4p\,P.!VV VVVVV R7p VeWpP"8wd \1WV4V #\RV 24h)r<z>IntxWeightOnlyConfig only works for 2-d and 4-d Tensors, got: rz-granularity must be PerGroup or PerAxis, got )rrrrrrG)rrIrrrrrrrZr2rr1rrrr#rrr[rrHrr$rr)rXrrrrrIrrrr input_dimrrr s&&$$ r@!_intx_weight_only_quantize_tensorrs&&L$$K&&L$$K 44$*$H$H! zz|q   LVZZ\N [  +x(( ++ K ) )1$ 3K4D4D3E F $\\), H VWW zz|q_ zz|q   Q* >>Q   !!%6%G%GG  (->-D-D -S 1 4 4UZZ @ -55   %%/*G   "{ll'B 7 K X78K7LMNNrBc V^8dQhR\PPR\R\R\ \P ,R\ \P ,R\PP/#r)r[r\rrrr r)r?s"r@rArAsf HHOO   5<<(    -  XX__rBc H\W4'gQRV R2RV R2,4h\\W4VVVR7p\VV\P P VRR74\P!\\VPVR7V4Vn V#)z8applying intx weight only quant requires module to have r3r4r5rFrkr) r^rrqrLr[r\rmrrrr+r)rrr rrr s&&$$$ r@_intx_weight_only_transformrs 6 * * B>BRR\] &+ , - *3'!+ J   :U; ((  & 1 1)  F MrBcda]tRtRtoRt]!]R7t]!]R7t^t Rt Rt V3Rlt Rt VtR#) riaConfiguration class for applying different quantization configs to modules or parameters based on their fully qualified names (FQNs). Args: `fqn_to_config`: typing.OrderedDict[str, Optional[AOBaseConfig]]: an ordered dictionary from a key to the config that we want to apply to the module/param (or None). A key is one of: 1. fully qualified name (fqn) of module or parameter 2. regex of fully qualified name (in python `re` module regex format), should start with prefix "re:" or 3. "_default" Config key ordered by precedence: * fully qualified parameter name, e.g. `language.layers.0.q_proj.weight` * fully qualified module name, e.g. `language.layers.0.q_proj` * regex for parameter names, must start with `re:`, e.g. `re:language\.layers\..+\.q_proj.weight`. The first regex that matches will be applied. * regex for module names, must start with `re:`, e.g. `re:language\.layers\..+\.q_proj`, whichever regex fully matches the module fqn first will be applied (order of keys for dictionary are kept consistent since we are using OrderedDict) * "_default", fallback if no match for all previous keys (Note, when using `_default`, the config is applied to all modules, to apply it to only a subset of modules, e.g. with some types, it's better to filter the modules that we don't want to quantize before hand and configure them to None, e.g. `{"re:.+norm.+": None, "_default": linear_config}`) "_default" is not supported when filter_fn is not specified. `module_fqn_to_config`: typing.OrderedDict[str, Optional[AOBaseConfig]]: To maintain BC with ModuleFqnToConfig, to be deprecated later `version`: int: Version of config to use. Note: - The order of patterns in the OrderedDict may matter as only the first matching pattern is applied - "_default" is ignored for parameter replacement. )default_factoryc^\PPR4\VP4^8dA\VP 4^8d'VPVP 8wd \ R4h\VP 4^8d,\VP4^8XdVP Vn\VP4^8d,\VP 4^8XdVPVnRVP9d\P!R4R#R#)z torchao.quantization.FqnToConfigzP`fqn_to_config` and `module_fqn_to_config` are both specified and are not equal!rzConfig Deprecation: _default is deprecated and will no longer be supported in a future release. Please see https://github.com/pytorch/ao/issues/3229 for more details.N) r[rrlenrmodule_fqn_to_configrrrrs&r@rFqnToConfig.__post_init__1s $$%GH "" #a 'D--.2""d&?&??b  t(( )A -#d6H6H2IQ2N!%!:!:D  t!! "Q &3t/H/H+IQ+N(,(:(:D % ++ + MMy  ,rBclRPR.RVPP44ORN4#) z FqnToConfig({c38"TFwrRV RV R2xK R#5i)z 'z': ,Nr)rkeyvalues& r@r&FqnToConfig.__str__..Ns(&@ #hugQ/&@sz}))joinrrrs&r@__str__FqnToConfig.__str__JsFyy &*&8&8&>&>&@      rBc<V^8dQh/S[S[S[S[,3,;R&S[S[S[S[,3,;R&S[;R&#)r<rrr)OrderedDictTyperr r r)r?rs"r@rAFqnToConfig.__annotate__sTH#3(>#>?IP*#x /E*EFQVWrB)rrN)rrrrrrrrrrrrrrrrs@r@rrsI DCH#CMJO#JG2  MrBrcV^8dQhR\\PP\.\PP3,R\ /#)r<rr>)rr[r\rr r)r?s"r@rArA[s?YY uxx 5uxxF GY YrBcf\P!V4PPRR4RJ#)zk Returns True if the handler function has a "parameter_name" kwarg in its type signature, False otherwise. r N)inspect signature parametersget)rs&r@"_handler_supports_fqn_quantizationr[s/   W % 0 0 4 45Et LTX XXrBcdV^8dQhR\PPR\R\/#r<rfqnr)r[r\rrr)r?s"r@rArAds3[[ HHOO[ [ [rBcRp.p\\VP444FFwpwrgV\V49gK\ V4^8dV RV 2MTpVP WVWx34KH \V4FwrVrxWP 9gKRpVP V,p V fVPV4KD\\V 4,p \V 4'd V !W VR7pKw\\V 4 R24h V'gHWP 9d8VP V,p V e\\V 4,p V !W 4#V#VFwrVrxVP Fp V PR4'gK\P!V R,V4'gKBRpVP V ,p V fK]\\V 4,p \V 4'd V !W VR7pK\\V 4 R24h K V'gVP F{p V PR4'gK\P!V R,V4'gKBVP V ,p V e!\\V 4,p V !W 4u#Vu# V'g?VP PR R4p V e\\V 4,p V !W 4#V#) aOThis function expects a module that either is specified in FqnToConfig or has a parameter that is specified in FqnToConfig. Args: module (torch.nn.Module): The module to be processed. fqn (str): The fully qualified name of the module containing the parameters. config (FqnToConfig): Configuration object containing regex patterns / fqn mapped to quantization configurations. Returns: torch.nn.Module: The modified module with quantized parameters. Raises: NotImplementedError: If the quantization configuration is not yet supported for parameter quantization. FrFTN)r zs does not yet support parameter quantization! Please see https://github.com/pytorch/ao/issues/3252 for more detailsre:NNr) enumeraterInamed_parametersdirrappendrryr'rrNotImplementedErrorrre fullmatchr) rrrparameter_config_foundtop_level_paramsir param parameter_fqncrpatterns &&& r@rrds&#&/V5L5L5N0O&P" "N S[ (-0X\3%q()~   # #Q$M N 'Q488H3I/5 00 0%) "$$]3Ay $$Q'247;5g>>$V~NF-7)$WX4J$ "c-A-A&A   % =.tAw7G6% %M4D/5++G!!%((R\\'"+}-U-U)-&((1=6tAw?G9'BB!(>!R1#Awi([\,4D "++G!!%((R\\'"+s-K-K((1=6tAw?G"6--!M, "  $ $Z 6 =.tAw7G6% % MrBc0V^8dQhR\R\/#)r<rr)rr)r?s"r@rArAs  rBcWP9d%VPR4'd QRV R24hR#VPFCpVPR4'gK\P!VR,V4'gKBR# R#)aMCheck if a given fqn matches the exact fqn or regex pattern specified in FqnToConfig. Args: fqn (str): The fully qualified name of the module. config (FqnToConfig): Configuration object containing regex patterns or raw FQNs for quantization. Returns: bool: True if the fqn is specified in FqnToConfig. False otherwise. rzError: Exact match but regex z specified.TrF)rrrr)rr!maybe_module_or_param_fqn_patterns&& r@rrs """>>%(( +C5 < (171E1E -0;;EBBr||1"5sHH 2F rBcPV^8dQhR\PR\R\/#r)r\rrr)r?s"r@rArAs* II  rBcVP4FDwr4V\V49gK\V4^8dV RV 2MTp\WR4'gKCR# R#)aCheck if a given module contains top-level parameters that match the exact fqn or regex pattern specified in FqnToConfig. Args: module (nn.Module): The module to be checked. fqn (str): The fully qualified name of the module. config (FqnToConfig): Configuration object containing regex patterns or raw FQNs for quantization. Returns: bool: True if the module contains top-level parameters that match the fqn or regex pattern specified in FqnTo rFTF)rrrr)rrrrRrrs&&& r@rrsS..0  3v; /23x!|se1TFOM%m<< 1 rBcDV^8dQhR\R\P/#)r<rr>)rr\r])r?s"r@rArAs  , 299 rBc\P!R4;_uu_4\P!VPVP 4pRRR4VP XnVPVnV# +'giL4;i)aE Unwrap a torchao Float8Linear by returning a nn.Linear with the same weights and bias. Torchao inference quantization techniques are generally only applicable to nn.Linear layers, so this helper is useful for unwrapping models trained with torchao float8 training, which replaces nn.Linear layers with Float8Linear layers. metaN)r[rEr\r] in_features out_featuresrXr)r new_modules& r@rrsZ f  YYv1163F3FG   JkkJO   s ,A77 B )rr5rzrr7r6ModuleFqnToConfig)rNr)Fr)rrloggingrrr collectionsr dataclassesrr functoolsrtypingrrrr r r rr[torch.nnr\torch.nn.utils.parametrizer8ror6torchao.core.configr torchao.float8.configr torchao.float8.float8_linearrtorchao.float8.inferencerrrrr(torchao.prototype.quantization.quant_apir%torchao.quantization.quantize_.commonr(torchao.quantization.quantize_.workflowsrrrrrrrrrrr r!r"r#r$r%r&%torchao.quantization.transform_moduler'r(torchao.quantization.utilsr)r*r+r, torchao.utilsr-r.r/rIr0r1r2r3r4linear_quant_modulesr5r6qatr7quant_primitivesr8r9r: getLoggerrr&__all__rKrdrzrrrrrr rrr/r;r=rErGrPrVr[r]rirlrnr}rrrrrrrrrrrrr rrrrrrrBr@r$s0  #(>>1 00,,5&     8 $ +\"D@D(->C.CM+/ H V B LB  B JG ,0 G 15 G T""GH#  ,0  15 IB  R< R  RF8Tv""67# 8@ ;l; ;$""IJ1# 1K1h % <%  % P ""67# 8@ 4 5 L5  5 p(""GH I@ ; <;  ; |""FG. .H.b T\T TB""89"# ":"J ;S ;S ;S|Q h%P""KL$# $M$N2 1 <1  1 h;O,0 ;O 15 ;O|""67#  ,0  15 8D O ,O  O f Y[|60 rB