+ "ݳj^RIt^RIt^RIt^RIt^RIt^RIHt^RIHt^RIH t ^RI H t H t H t ^RIt^RIHuHuHt^RIHt.RGOtR tR R lt]P0R R l4tRHRltRtRtRtRtRRltRt RRlt!RIRlt"Rt#RJRlt$!RR]P PJ4t&RKRlt'RRlt(R t)R!t*R"t+R#t,R$t-R%t.R&t/RLR'lt0R(t1R)t2!R*R]Pf4t4R+t5R,t6R-t7R.t8R/t9R0R1lt:R2t;R3tR7R8lt?R9R:lt@R;R<ltAR=tBR>R?ltCR@RAltDRBtERCRDltFREtG!RFR]P PJ4tHR#)MNreduce)versiongcd)AnyOptionalUnion)return_and_correct_aliasingTorchAOBaseTensor DummyModulecX\PPPV4V#)zTDecorator to register a class as a pytree constant for dynamo non-strict trace mode.)torchutils_pytreeregister_constant)clss&N/home/luca/screen-checking/.venv/lib/python3.14/site-packages/torchao/utils.pyregister_as_pytree_constantr-s KK))#. JcfV^8dQhR\R\\\P3,/#)target_device_strdevice)strr rr)formats"r __annotate__r3s(::#:uS%,,5F/G:rcH\P!V4PV8H#N)rrtype)rrs&&r _is_devicer!3s <<  $ $(9 99rcXV^8dQhR\PPR\/#)rmodulereturn)rnnModuler)rs"rrr9s"%((//crcLVP4Uu0uFqPkK upVP4Uu0uFqPkK up,p\V4^8:g QRV 24h\V4^8d\ \ V44pV#RpV#uupiuupi)zz Returns the unique device for a module, or None if no device is found. Throws an error if multiple devices are detected. zKprepare only works with cpu or single-device CUDA modules, but got devices N) parametersrbufferslennextiter)r#pdevicesrs& r_assert_and_get_unique_devicer/8s "(!2!2!45!4Axx!45 ..*9*Q*9G w<1  ") % %(L1$4T$w- F M;?F M69s BB!cx Vf/pVfG\V\PP4'gQR4h\ V4P pVR8XEd \P P4\P PRR7p\P PRR7pVP4\V4FEp\PPPR4;_uu_4V!V/VBRRR4KG VP4\P P4VPV4V, #VR8XEd\PP4\PP PRR7p\PP PRR7pVP4\V4FEp\PPPR4;_uu_4V!V/VBRRR4KG VP4\PP4VPV4V, #VR8Xd\P"P4\$P$!4p\V4FEp\PPPR4;_uu_4V!V/VBRRR4KG \$P$!4p \P"P4W, V, p V #VR 8XEd \P&P4\P&PRR7p\P&PRR7pVP4\V4FEp\PPPR4;_uu_4V!V/VBRRR4KG VP4\P&P4VPV4V, #R# +'giEK;i +'giEK;i +'giEK;i +'giK;i) z?Benchmark model runs with `args` and `kwargs` both are optionalNzFExpecting `model` to be torch.nn.Module if device_type is not providedcudaT) enable_timingz timed regionmpscpuxpu) isinstancerr%r&r/r r1 synchronizeEventrecordrangeautogradprofilerrecord_function elapsed_timer3eventr4timer5) modelnum_runsargskwargs device_type start_event end_event_ start_timeend_timeaverage_time_per_runs &&&&& rbenchmark_modelrLJso ~%11 T 14E:?? f  jj&&T&: JJ$$4$8 xA((88HHt&v&IH!   '' 2X==   iioo++$+? IIOO)))= xA((88HHt&v&IH!  '' 2X==   YY[ xA((88HHt&v&IH!99;  ( 5A##   iiooDo9 IIOO$O7 xA((88HHt&v&IH!  '' 2X== IIHHIHHIHH IHHs0 Q)* Q>> R R() Q; > R  R% ( R9 cN\PP\PPP\PPP .RR7;_uu_4pV!V/VBpRRR4XP V4X# +'giL#;i)T) activities record_shapesN)rr<profileProfilerActivityCPUCUDAexport_chrome_trace)pathfnrCrDprofresults&&*, rprofiler_runnerrYs    NN + + / / NN + + 0 0     T$V$  T" M  s 0 B B$ cLR.p\PP4'dVPR4M5\PP4'dVPR4\P P4'dVPR4V#)r4r1r5r3)rr1 is_availableappendr5r3)r.s rget_available_devicesr]skgG zz  v    ! !u yyu Nrc\PP4'gQh\PP4#r)r acceleratorr[current_acceleratorrrget_current_accelerator_devicerbs2    ) ) + ++ +    0 0 22rc\PP4'd=\PP4p\ V^, RV^, 24#R#)r.g)rr1r[get_device_capabilityfloat) capabilitys rget_compute_capabilityrhsH zz  ZZ557  1 a 1 788 rcxV^8dQhR\PR\PR\P/#)routput output_refr$)rTensor)rs"rrrs-U\\u|| rc\P!\P!W, 44\P!\P!V44, #r)rmeanabs)rjrks&&rcompute_max_diffrps; ::eii 34 5  *9 rc ^RIHuHpV!V/VBV!V/VBVPRRVRVRV/R7pVP 4pVP R,#)rNzf(*args, **kwargs)rCrDf)stmtglobalsg.A)torch.utils.benchmarkr benchmarkTimerblocked_autorangern)rrrCrDrvt0 measurements&*, r(benchmark_torch_function_in_microsecondsr{sk--tvtv  !xa8  B&&(K   c !!rc<V^8dQhR\R\R\/#)rnrCr$int)rs"rrrs!Srcr\RVR,4pW,^8XdV#W,W,, #)c2W,\W4,#rr)xys&&rfind_multiple..s#a)!3rr)r}rCks&* r find_multiplers/ 3TD[ AAuz 5AE?rcDaaaa^RIHoS'dRMRoVVVV3RlpV#)aThis decorator is used to preserve some high level operators for torch.export.export while still allow them to be decomposed for inductor path requirement: make sure `fn.__name__[1:]` is the operator name you want to register NOTE: This should be applied at the top, after all other decorators have been applied NOTE: We haven't tested the case when `fn` accepts tensor subclass instance as input, e.g. uint4 tensor subclass instance, and we'll probably need to figure out what would make sense for downstream system (like executorch) to accept as well Example: lib = torch.library.Library("my_namespace', "FRAGMENT") register_custom_op = _register_custom_op(lib) @register_custom_op def _the_op_that_needs_to_be_preserved(...) ... # after this, `_the_op_that_needs_to_be_preserved` will be preserved as # torch.ops.my_namespace.the_op_that_needs_to_be_preserved operator after # torch.export.export )register_decompositionCompositeImplicitAutogradCompositeExplicitAutogradc<a^RIHp\;QJdV3RlR4F 'gK RM RM!V3RlR44'dQRSP 24hSPpV^,R8Xd VR,pW!!S/R 7,pSP V4SP VSS4SP p\\\PV4V4pS'dS !V.4!S4V#) r) infer_schemac3@<"TFqSP9xK R#5ir)__name__).0crVs& r 9_register_custom_op..decorator..s7ABKK'sz.<>TFzEExpecting op to be defined in normal functions, not lambda or local: rHrNN) mutates_args) torch._library.infer_schemaranyrdefineimplnsgetattrrops) rVrop_nameschema lib_namespaceop dispatch_keyinductor_decomposedlibrs f r decorator&_register_custom_op..decorators<3773337777 STVT_T_S` a 7++ 1: bkG<<< 6 "l+ WUYY 6 @  "B4 ( , r)torch._inductor.decompositionr)rrrrrsff @@r_register_custom_oprs-2E  $ ( & rcaaVV3RlpV#)c.<SPSVR4pV#)Meta)r)rVrrrs& rr$_register_meta_op..decorator s XXgr6 * rra)rrrsff r_register_meta_oprs rcpaV3Rlo^pVP4Fwr4\V\PP4'd V'dK9\ P !VPRR7VPRR74FpVS!V4, pK V\WA4, pK V#)z Returns the model size in bytes. The option to ignore embeddings is useful for models with disproportionately large embeddings compared to other model parameters that get quantized/sparsified. c<\VR4'd=^pVP4^,Fp\W4pVS!V4, pK V#VP4VP 4,#)__tensor_flatten__)hasattrrrnumel element_size)tensorsize attr_name sub_tensor flat_sizes& rr*get_model_size_in_bytes..flat_sizesm 6/ 0 0D$668;; $V7  *--<K<<>F$7$7$99 9rF)recurse) named_childrenr6rr% Embedding itertoolschainr(r)get_model_size_in_bytes)rAignore_embeddings model_sizenamechildr-rs&& @rrrs :J++- 5%(("4"455:K:K__   /u1Mil*  1%K KJ . rc,a]tRtRtoRtRtRtVtR#)UnwrapTensorSubclassi.cB\V4p\VP4F[wr4p\V4p\ WRV)R4UUu/uFwrxWxbK pppW&RpVP WTRR4p VP V 4K] \V4^8XgQhV^,#uuppir)listreversed rebuild_stackr*zip__tensor_unflatten__r\) selftensorstodotpmeta inner_tensors nb_tensorabrebuilts &* rforwardUnwrapTensorSubclass.forward/sG}'/0B0B'C #BmM*I.1-yjkAR.ST.SdaQT.SMT #D--m4NG KK (D4yA~~Aw Us Bc\V4\PJgQh.p.pV.pV'dVP4pVP 4wrgVP \V4Wv34VFqp\ WX4p \V 4\PJdVP V 4K>\V \P4'gQhVP V 4Ks KW nV#r) r rrlpoprr\rr6r) rrr plain_tensorsrobjrmetadatarvals && r right_inverse"UnwrapTensorSubclass.right_inverse;sF|5<<///  x((*C&)&<&<&> #M  $s)X!E F* c-9 ,!((-%c5<<8888KK$ ++r)rN)r __module__ __qualname____firstlineno__rr__static_attributes____classdictcell__ __classdict__s@rrr.s rrcDVP4EFwr#\V\PP4'g,\V\PP 4'Ed%\ VR4'Ed\VP4\PJd\VP4\PPJd\VP\P4'd\\VP4\P4'd]\VP\4'd=\P!V4'g!\P!VR\!44\#V4EK V#)a%Unwraps (nested) tensor subclass in the model to plain tensors This is a workaround to make a model with tensor subclass to work with `torch.export.export` and `torch.aot_compile`, we hope this can be integrated into compile stack soon tracking issue: https://github.com/pytorch/ao/issues/345 weight)rr6rr%Linearrrr rrl Parameter issubclassr parametrizeis_parametrizedregister_parametrizationrunwrap_tensor_subclass)rA filter_fnrrs&& rrrQs ++- 5%((//22eUXX%7%788x((U\\"%,,6U\\"%((*<*<<5<<664 -u||<<5<<):;;//66  0 0x!5!7  u%%.& LrcDV^8dQhR\PR\/#)rdtyper$)rrbool)rs"rrrms5;;4rc\P\P\P\P0pW9#r)r float8_e4m3fnfloat8_e4m3fnuz float8_e5m2float8_e5m2fnuz)r fp8_typess& r_is_float8_typerms8     I  rc\\P!RV44p\P!RV4pV'd0\ \ VP 44wr4pV'dRpW4V.#\RV 24h)z Parse version string representing pre-release with -1 Examples: "2.5.0.dev20240708+cu121" -> [2, 5, -1], "2.5.0" -> [2, 5, 0] z (git|dev)z(\d+)\.(\d+)\.(\d+)zInvalid version string format: )rresearchmatchmaprgroups ValueError)version_string is_prereleasermajorminorpatchs& r parse_versionr wsk<@AM HH+^ :JKLLrc@\\PR4'*#) git_version)rrrrarr is_fbcoder su}}m4 44rcp\4'dR#\\P4\V48#T)r r r __version__) min_versions&rtorch_version_at_leastrs){{ ** +}[/I IIrcaa\SR4'g/SnSSP9d/SPS&\S\\34'gS.oVV3RlpV#)a\Decorator for implementing aten ops like `torch.ops.aten.linear.default` for tensor subclass, the implemented functions are called in ``__torch_dispatch__`` callback for ``torch.Tensor`` subclasses Examples:: implements = MyTensor.implements @implements(torch.ops.aten.linear.default): def _(func, types, args, kwargs): ... _ATEN_OP_TABLEc<SF8p\P!V4V3Rl4pVSPS,V&K: V#)cV!WW#4#rrarrtypesrCrD_funcs&&&&&rwrapper/_implements..decorator..wrapperQt44r) functoolswrapsr)funcrraten_opsrs& rr_implements..decoratorsIB __T "6: 5# 5+2C  s #B '  r)rrr6rtuple)rrrsff r _implementsr"s^ 3( ) ) #$$$"$3 hu . .: rcaa\SR4'g/SnSSP9d/SPS&\S\\34'gS.oVV3RlpV#)aDecorator for implementing torch functions / ops like ``torch.nn.functional.linear``, ``torch.Tensor.t`` for the tensor subclass the implemented functions are called in ``__torch_function__`` callback for ``torch.Tensor`` subclasses Examples:: implements_torch_function = MyTensor.implements_torch_function @implements_torch_function(torch.nn.functional.linear): def _(func, types, args, kwargs): ... _TORCH_FN_TABLEc<SF8p\P!V4V3Rl4pVSPS,V&K: V#)cV!WW#4#rrars&&&&&rr>_implements_torch_function..decorator..wrapperrr)rrr$)rrVrr torch_fnss& rr-_implements_torch_function..decoratorsIB __T "6: 5# 5,3C   $R (  r)rr$r6rr!)rr(rsff r_implements_torch_functionr*s^ 3) * *  #%%%#%C i$ / /K  rcaVPpVPp\PPpV!\PPP P 4R4pV!\P4R4pV!\PP.4R4pV!VPPVPPVPPVPP.4R4pRRloV!VPP4V3Rl4pV!VPP4R4pR#) caa\V4^8XgQhV^,pVP4F pVR9dK Qh RV9dVR,VP8XgQhRV9dVR,VP8XgQhVP R4oSVP 8XdV#V^,P VV3Rl4p\SW#V4#)rrlayoutrc<S!VSR7#)rra)rrrs&rr:_implements_common_tensor_ops.._..s a9Or)rr-r)r*keysrr-getr_apply_fn_to_datar )rrrCrDrr new_tensorrsf&&& @rrH(_implements_common_tensor_ops.._s4yA~~AwA55 55 f '?djj0 00 v (#t{{2 22H% T[[ K!W../OP *4zJJrc\V4^8XgQh\V^,4\V^,48H;'d(V^,PV^,P8H#)r)r*r shaperrrCrDs&&&&rrHr5sJ4yA~~DG}T!W -PP$q'--47==2PPrcDaaaS^,PVVV3Rl4#)rc*<S!V.SR,O5/SB#rrarrCrrDs&rrr0 s43ODH3O3Or)r3r8sf&ffrrHr5s Aw(()OPPrc \aaa\SSSS^,PVVV3Rl44#)rc*<S!V.SR,O5/SB#r;rar<s&rrr0sQ0Lb0LV0Lr)r r3r8sf&ffrrHr5 s.+    G % %&L M   rc<V^8dQhR\R\R\/#)rrsrcr$)r r)rs"rr3_implements_common_tensor_ops..__annotate__s#$ $ .$ 5F$ 4$ rcaa\;QJd*VV3RlSP4F 'dK RM RM!VV3RlSP44pRp\SR4'dQ\;QJd*VV3RlSP4F 'dK RM RM!VV3RlSP44p\;QJd*VV3RlSP4F 'dK RM RM!VV3RlSP44pRp\SR4'dQ\;QJd*VV3RlSP 4F 'dK RM RM!VV3RlSP 44p\ S4\ S48H;'dBSPSP8H;'d!T;'dT;'d T;'dT#)c3~<"TF2p\SV4P\SV4P8HxK4 R#5irrr7rt_namerr@s& rrH_implements_common_tensor_ops.._same_metadata..s7" 0 D& ! ' '73+?+E+E E0s:=FToptional_tensor_data_namesc3<"TFNp\SV4e.\SV4P\SV4P8HM \SV4RJxKP R#5irrDrEs& rrrG#s^/ >FtV,8D&)//733G3M3MM f-56>sAAc3V<"TFp\SV4\SV48HxK R#5irrra_namerr@s& rrrG,s+ 5 D& !WS&%9 95&)optional_tensor_attribute_namesc3V<"TFp\SV4\SV48HxK R#5irrKrLs& rrrG3s+'BFf%f)==BrN)alltensor_data_namesrrHtensor_attribute_namesrOr r7)rr@_tensor_shape_match_optional_tensor_shape_match _attr_match_optional_attr_matchsff r_same_metadata5_implements_common_tensor_ops.._same_metadatas!c" 00" ccc" 00"  (,$ 45 6 6+.3/ #== /333/ #== /, (c 55 ccc 55  $ 4: ; ;#&3'"BB'333'"BB'$ J$s) # % % cii' % %# % %- % %  % % %  rc <V^,pV^,pS!WE4'dFVP4^,pVF&p\WG4P\WW44K( R#\RV^,V^,3 24h)rNz7Not supported args for copy_ due to metadata mismatch: )rrcopy_r) rrrCrDrr@ self_tensors tensor_namerXs &&&& rrHr5As~Aw1g $ $ $224Q7L+ *001JK , Ed1gtTUwFVEW X  rcnV^,p\VR4'Ed\VR4'EdnVP!VR,/VBpVPR4pVPRR4pVPUu.uFp\ WG4P WVR7NK pp.p \VR4'dWVP FFp \ WJ4p V e$V PV P WVR74K5V PR 4KH VPU u.uFp V R8wd \ WL4MTNK p p .p\VR 4'd.VPU u.uFp V R8wd \ WL4MTNK pp VP!.VOV OV OVO5!p\WW?4#\R 4huupiuup iuup i) rrRrSrr non_blockingF)r_rHNrOzSubclasses must implement `aten._to_copy.default` or specify `tensor_data_names` and `tensor_attribute_names` for tensor class or tensor instance before using it) r_get_to_kwargsrrRrtorHr\rSrO __class__r NotImplementedError)rrrCrDrrr_rroptional_tensorstensor_data_name maybe_tensorrtensor_attributesoptional_tensor_attributests&&&& rrHr5NsAw 4, - -' *3 3 (($r(=f=FZZ)F!::ne??&*%I%I.%I 1:X0EGD,6Q%I+. "", A /t6E E! p  G ! .s:$F(F-F2N) implementsimplements_torch_functionrratenra dtype_layout!_has_compatible_shallow_copy_typerl contiguousdetachdefaultclonealiasr[_to_copy)rrjrkrlrHrXs& @r_implements_common_tensor_opsrus?J # = = 99>>D !!../K0K0uFFGQHQ LL # #  Q  Q KK   JJ   JJ   OO # #    $ L ""#  $   %%&- '- rc\VR4'd\VR4'gQh\PPW4\ VR.4F4pW P 9gK\W4'dK(\ WR4K6 \ VR.4F4pW0P 9gK\W4'dK(\ WR4K6 R#)rRrSrHNrO)rr_utils_set_obj_stater__dict__setattr)rstateoptional_tensor_data_nameoptional_tensor_attribute_names&& r _torchao_base_tensor__setstate__r~s 4, - -' &33  LL,%,T3OQS%T! $MM 9' C C  DT : &U +2 /+& * >w H H  D$ ? +rc`Vf/MTp\VR4'dIWP9d9WPV,9d"VPV,V,!WW44#\PP 4;_uu_4V!V/VBuuRRR4# +'giR#;i)zUse this util function for a common `__torch_function__` implementation that dispatches to ops/functions registered with `_implements` class MyTensor(torch.Tensor): ... __torch_function__ = classmethod(_dispatch__torch_function__) Nr$)rr$r_CDisableTorchFunctionSubclass)rrrrCrDs&&&&&r_dispatch__torch_function__rs>RvF&'' && & '', ,""3'-d4HH  . . 0 0T$V$ 1 0 0 0s B B- c \VR4'dIWP9d9WPV,9d"VPV,V,!WW44#\;QJd.RV4FNK 5M !RV44pVP4UUu/uFwrgV\ V4bK ppp\ VP RV: RV: RV: RV: 2 4huuppi)zUse this util function for a common `__torch_dispatch__` implementation that dispatches to ops/functions registered with `_implements` class MyTensor(torch.Tensor): ... __torch_dispatch__ = classmethod(_dispatch__torch_dispatch__) rc38"TFp\V4xK R#5ir)r rargs& rr._dispatch__torch_dispatch__..s04Cd3ii4szC dispatch: attempting to run unimplemented operator/function: func=z, types=z , arg_types=z, kwarg_types=)rrr!itemsr rcr) rrrrCrD arg_typesrr kwarg_typess &&&&& r_dispatch__torch_dispatch__rs %&& %% % &&s+ +!!#&t,T$GG0400400I.4lln=nFA1d3isCc@\;QJd.RV4FNK 5M !RV44pRV9dVPR4\PPP !V/VBwr4rVVf VP MTpVf VPMTpRVRVRV/pV#)zHelper function to get the device, dtype and non_blocking keyword args for `aten._to_copy.default` op Returns: {"device": device, "dtype": dtype, "non_blocking": non_blocking} c3j"TF)p\V\P4'dK%VxK+ R#5ir)r6rr-rs& rr!_get_to_kwargs..sJJsELL,Is"3 3r-rrr_)r!rrr_nn _parse_torr)rrCrDrrr_rHs&*, rr`r`s 5JJ55JJ JD6 8%*XX\\%;%;T%LV%L"F<"NT[[F-DJJUE& F Mrca]tRtRtoRt]R4t]!]4t]!] 4t ]!] 4t ]!] 4t ]!]4t]tRtRt]R4tRtRtR tR tVtR #) r iaGA util tensor subclass that provides commonly used functions new tensor subclass can inherit to get all the utility functions Attributes (defined by subclass of ``TorchAOBaseTensor``): * :attr:`tensor_data_names` (List[str]): list of names of all required tensor_data, order should match the `__init__` list of tensor subclass (optional to define to use ``TorchAOBaseTensor``, required for getting the util functions been defined (see Note section)) * :attr:`tensor_attribute_names` (List[str]): list of names of non-``Tensor`` attributes, order should match the ``__init__`` list of tensor subclass, following all the ``tensor_data_names`` arguments (optional to define to use ``TorchAOBaseTensor``, required for getting the util functions been defined (see Note section)) * :attr:`optional_tensor_data_names` (List[str]): it's not required to get the additional util functions been implemented, but this will be need if there are some optional ``Tensor`` data attributes, when defined, this will be a list of names of ``Tensor``s that can be optional * :attr:`optional_tensor_attribute_names` (List[str]): it's not required to get the additional util functions been implemented, but this will be need if there are some optional non-``Tensor`` attributes, when defined, this will be a list of names of attributes that can be optional Note: **Argument Order** Argument order in ``__init__`` and ``__new__`` of subclass of ``TorchAOBaseTensor`` should match exaclty with ``tensor_data_names`` + ``tensor_attribute_names`` + ``optional_tensor_data_names`` (if present) + ``optional_tensor_attribute_names`` (if present) Note: **How to Get Predefined Util Functions** If ``tensor_data_names`` (torch.Tensor data attribute names) and ``tensor_attribute_names`` (non-torch.Tensor attribute names) are defined, there are some additional util functions that will be added, this includes: ``__tensor_flatten__``: flattens a subclassed tensor instance, returns a ``tuple``, first element is tensor data names for valid tensor data, second element is a dict from attribute_name to non-``Tensor`` attributes ``__tensor_unflatten__``: takes a ``tensor_data_dict`` (a map from tensor name to ``Tensor``), and list of non-``Tensor`` attributes, returns a new instance of the subclassed tensor ``_apply_fn_to_data``: takes a function (``Tensor -> Tensor``), applies function to all tensor data and recreate a new subclassed Tensor with the transformed tensor data ``__repr__``: the string representation of the subclassed tensor instance ``_same_metadata``: returns whether the metadata is the same between two instances of cls ``__setstate__``: when loading a serialized tensor subclass checkpoints, it sets the new optional tensor and tensor attribute that is saved in the old checkpoint to None, to maintain BC of old checkpoints when we add new optional tensor data or attributes to the tensor subclass torch function supported (``__torch_function__``): ``torch.Tensor.contiguous`` aten ops supported (``__torch_dispatch__``): ``aten.detach.default``, ``aten.clone.default``, ``aten.alias,default``, ``aten.contiguous.default``, ``aten.copy_.default``, ``aten._to_copy.default`` (enables ``t.to``) Note: **Subclassing and Op Inheritance** Subclasses of ``TorchAOBaseTensor`` automatically inherit aten op (``__torch_dispatch__``) and torch function (``__torch_function__``) implementations from their parent classes. Each subclass gets its own independent dispatch tables, so registering a new op on a child does not affect the parent, and vice versa. A child class can override an inherited op by registering its own implementation with ``@ChildClass.implements(...)``. If no override is provided, the parent's implementation is used automatically. For multiple inheritance (e.g., ``class C(B, A)``), ops are inherited from all parents following Python's MRO (Method Resolution Order), with later bases taking priority. Example:: class Parent(TorchAOBaseTensor): tensor_data_names = ["qdata"] tensor_attribute_names = ["attr"] def __new__(cls, qdata, attr): r = torch.Tensor._make_wrapper_subclass(cls, qdata.shape) r.qdata = qdata r.attr = attr return r def __init__(self, qdata, attr): pass @Parent.implements([torch.ops.aten.cat.default]) def parent_cat(func, types, args, kwargs): # parent implementation ... # Child inherits Parent's aten.cat implementation automatically class Child(Parent): tensor_data_names = ["qdata"] tensor_attribute_names = ["attr"] # Optionally override an inherited op: @Child.implements([torch.ops.aten.cat.default]) def child_cat(func, types, args, kwargs): # child-specific implementation ... Note: **Safetensors Support** ``TorchAOBaseTensor`` subclasses can be serialized to and loaded from the `safetensors `_ format. Since safetensors only stores plain ``torch.Tensor`` objects, the serialization layer (in ``torchao.prototype.safetensors``) decomposes each tensor subclass into its constituent plain tensors (from ``tensor_data_names``/``optional_tensor_data_names``) plus JSON metadata (from ``tensor_attribute_names``/``optional_tensor_attribute_names``), and reconstructs the subclass on load. To add safetensors support for a new ``TorchAOBaseTensor`` subclass: 1. Define ``tensor_data_names``, ``tensor_attribute_names`` (and optionally ``optional_tensor_data_names``, ``optional_tensor_attribute_names``) on the subclass, which is already required for the other utility functions above. 2. Register the subclass in ``torchao/prototype/safetensors/safetensors_utils.py``: - Add the class name string to ``ALLOWED_TENSORS_SUBCLASSES``. - Add a ``"ClassName": ClassName`` entry to ``ALLOWED_CLASSES``. - If the subclass has non-Tensor attributes with custom types (dataclasses, enums, named tuples), add those types to ``ALLOWED_CLASSES`` as well. Once registered, Hugging Face Transformers users can use ``save_pretrained`` and ``push_to_hub`` with the default ``safe_serialization=True`` option. See https://docs.pytorch.org/ao/main/eager_tutorials/torchao_hf_integration.html#saving-the-model for a full end-to-end example. Examples:: class MyTensor(TorchAOBaseTensor): tensor_data_names = ["a", "b"] tensor_attribute_names = ["c", "d"] optional_tensor_data_names = ["e", "f"] optional_tensor_attribute_names = ["g", "h"] def __new__( cls, a: Tensor, b: Tensor, c: int, d: str, e: Optional[Tensor] = None, f: Optional[Tensor] = None, g: Optional[int] = None, h: Optional[int] = None, ): pass def __init__( self, a: Tensor, b: Tensor, c: int, d: str, e: Optional[Tensor] = None, f: Optional[Tensor] = None, g: Optional[int] = None, h: Optional[int] = None, ): pass c RVP9d/VnRVP9d/VnWP9d/VPV&WP9d/VPV&\VR4'd.\VR4'dVP 4\ VnVPFp\VRR4pV'd/W#9d)VPV,PW2,4\VRR4pV'gK]W$9gKeVPV,PWB,4K R#)rr$rRrSN) ryrr$rrur~ __setstate__ __bases__rupdate)rrDparentparent_aten_tableparent_torch_tables&, r__init_subclass__#TorchAOBaseTensor.__init_subclass__ts 3<< /!#C  CLL 0"$C  (( (&(C  s # )) )')C   $ 3+ , ,>V1W1W  - - /?C mmF '0@$ G  V%@""3'../@/HI!(1BD!I !!f&B##C(//0B0JK$rch\PP\\ V444R#r)rr_log_api_usage_oncerr )rrCrDs&*,r__init__TorchAOBaseTensor.__init__s $$Sd_5rc \VR4'd\VR4'dVPP4p\VR4'd6VPF%p\ W4pVfKVP V4K' VP Uu/uFqD\ W4bK pp\VR4'd.TVPUu/uFpV\ W4bK up,pW3#\R4huupiuupi)rRrSrHrOzSubclasses should implement __tensor_flatten__ or specify `tensor_data_names` and `tensor_attribute_names` for tensor class before using it) rrRcopyrHrr\rSrOrc)rrRrerfattr attr_dicts& rr$TorchAOBaseTensor.__tensor_flatten__s 4, - -' *3 3 !% 6 6 ; ; = t9::(,(G(G$#*4#BL#/)001AB)H 7;6Q6Q6Qdgd))6Q t>??% $ D D) D'$-- D) %/ /! Z  )s C7 C<c\VR4'd\VR4'dVPUu.uF qQV,NK pp/p\VR4'd,VPUu/uFpWPVR4bK ppVPUu.uF qRV,NK p p/p \VR4'd#VP Uu/uF pWRV,bK p pV!.VOV O5/VBV B#\ R4huupiuupiuupiuupi)rRrSrHNrOzSubclasses should implement __tensor_unflatten__ or specify `tensor_data_names` and `tensor_attribute_names` for tensor class before using it)rrRrHr2rSrOrc) rtensor_data_dictrg outer_size outer_striderrequired_tensorsoptional_tensor_dictrerequired_attributesoptional_attribute_dicts &&&&& rr&TorchAOBaseTensor.__tensor_unflatten__se 3+ , ,>V1W1W363H3H 3H4&&3H  $& s899-0,J,J(,J(%&:&:;KT&RR,J%( 584N4N#4ND$''4N #') #s=>>!$ C C+ CD11 C(+ !$'*  " \  7  ( # +sC6(C;DDc H\VR4'd\VR4'dVPUu.uFq!!\W44NK pp/p\VR4'd3VPF"p\W4pVe V!V4WE&KRWE&K$ VPUu.uFp\W4NK pp/p\VR4'd&VP U u/uFp V \W 4bK pp VP !.VOVO5/VBVB#\R4huupiuupiuup i)rRrSrHNrOzSubclasses should implement _apply_fn_to_data or specify `tensor_data_names` and `tensor_attribute_names` for tensor class or tensor instance before using it)rrRrrHrSrOrbrc) rrVrrrrerfrrrs && rr3#TorchAOBaseTensor._apply_fn_to_datasm 4, - -' *3 3 594J4J 4JD74&'4J  $& t9::(,(G(G$#*4#BL#/ACLAQ,>AE,> )H150K0K#0K#0K #') #t>??&*%I%I+%I wt77%I(+ >>!$'*  " l  ; # +sD"DDc \VR4'EdA\VR4'Ed.RpWP^, R\WP^,4 2, pVPR,FpVRV R\W4 2, pK VPFpVRV R\W4 2, pK \VR4'd,VPFpVRV R\W4 2, pK \VR4'd,VP FpVRV R\W4 2, pK VP P RV R 2#\R 4h) rRrS=rz, rHrO()zSubclasses must implement __repr__ or specify `tensor_data_names` and `tensor_attribute_names` for tensor class or tensor instance before using it) rrRrrSrHrOrbrrc)rrepr_strretensor_attribute_names& r__repr__TorchAOBaseTensor.__repr__s 4, - -' *3 3 H 11!45QwtE[E[\]E^7_6`a aH$($:$:2$>$> b!1 2!GD4S3TUU%?*.)D)D%./q1U0VW*E t9::(,(G(G$-.a0O/PQH)H t>??-1-Q-Q)"%:$;1WT=a>> fill_defaults([1, 2, 3], 5, [3, 4, 5]) [1, 2, 3, 4, 5] >>> fill_defaults([1, 2, 3], 5, [None, None, None]) [1, 2, 3, None, None]] z%not enough defaults to fill arguments)r* RuntimeErrorrr:r\)rCr} defaults_tailris&&& r fill_defaultsr.sf" 3} D )BCC T A 3t9a  1us='99:;! Hrc\PP4;'d\PPRJ#r)rr1r[rhiprarris_ROCMrPs, :: " " $ F F):):$)FFrc\4'd@.ROp\PP^4PpVF pW!9gK R# R#)gfx940TF)rgfx941gfx942rrr1get_device_properties gcnArchName) mxArchNamearchNamearchs ris_MI300rTs@yy3 ::33A6BBD rc\4'd3\PP^4PpRV9dR#R#)rgfx950TFrrs ris_MI350r^s1yy::33A6BB x  rc~\4'd-\PP^4PpR#R#)rTFrrs ris_Navi4rfs,yy::33A6BB  rc<V^8dQhR\R\R\/#rrrr$rr)rs"rrrns&VVVSVTVrc\PP4;'d\PPpV'd#\PP 4W38H#R#)z0Check if the CUDA version is exactly major.minorFrr1r[rre)rris_cudas&& r is_sm_versionrnsHjj%%'>>EMM,>,>GCJ5:: + + -% ?UPUUrc\PP4;'dD\PP;'d"\PP 4R8#))r rrarris_sm_at_least_89rtM ! 9 9 MM   9 9 JJ , , .& 8rc\PP4;'dD\PP;'d"\PP 4R8#)r)rrrrarris_sm_at_least_90r|rrc\PP4;'dD\PP;'d"\PP 4R8#) )rrrrarris_sm_at_least_100rsM ! : : MM   : : JJ , , .' 9rc<V^8dQhR\R\R\/#rr)rs"rrrs!66C666rc\PP4'gR#\PPpVfR#\ \ VP R4R,4wr4W43W38#)Frd:NrN)rr1r[rrrsplit)rr cuda_version cuda_major cuda_minors&& ris_cuda_version_at_leastrs_ :: " " $ $==%%L l&8&8&=b&ABJ  #~ 55rc$V^8dQhR\/#rr$r)rs"rrrs  D rc&\\PPR4'd)\PPP 4#\\P R4'd\P P 4#R#)z Safely query AMX tile support, guarding against private API absence. torch.cpu._is_amx_tile_supported / torch._C._cpu._is_amx_tile_supported are private and may be missing in certain PyTorch builds or versions. _is_amx_tile_supportedF)rrr_cpurr4rarr_cpu_is_amx_tile_supportedrsY uxx}}677xx}}3355 4 5 5yy//11 rc$V^8dQhR\/#rr)rs"rrrs   rc&\\PPR4'd)\PPP 4#\\P R4'd\P P 4#R#)z Safely query AVX512_VNNI support, guarding against private API absence. torch.cpu._is_vnni_supported / torch._C._cpu._is_vnni_supported are private and may be missing in certain PyTorch builds or versions. _is_vnni_supportedF)rrrrrr4rarr_cpu_is_vnni_supportedrsY uxx}}233xx}}//11 0 1 1yy++-- rcDV^8dQhR\PR\/#)rrr$)rrr)rs"rrrs"AA AArcNVPR8H;'d\4'*#)zy Helper to determine if int8 tensor quantization range should be reduced to avoid overflow on CPUs without VNNI support. r4)r rr/s&rshould_reduce_rangers" ;;%  @ @(>(@$@@rc.W,^, V,#rrarrs&&rceil_divrs EAI! rc<V^8dQhR\R\R\/#)rrrr$r~)rs"rrrs!rc&\W4V,#r)rrs&&rround_uprs A>A rc0V^8dQhR\R\/#)r package_namer)r)rs"rrrs00c00rct\PPV4RJpV'gR#\V4V8#)NF) importlibutil find_specr)rrpackage_existss&& ris_package_at_leastr s1^^--l;4GN  < K //rc\PPR4RJ;'g \4pV'gR#^RIpR#)mslkNFT)r r r r r)has_mslkrs r_is_mslk_availablers1~~''/t;JJy{H  rc0V^8dQhR\R\/#)rrr$)rr)rs"rrrsII#I$Ircp\4'gR#^RIp\VP4\V48#)FN)rrr r)rrs& ris_mslk_version_at_leastrs,    )) *mK.H HHrc \PPR4RJ;'dS\PPR4RJ;'d*\PPR4RJ;'g \4#) flashinferNtvm_ffipynvml)r r r r rarr_is_flashinfer_availablersu   .d: ; ; NN $ $Y /t ; ; ; NN $ $X .d :    rcFaa]tRtRtoRtRV3RlV3RllltRtVtV;t#)r iaThis is used because the TorchAO quantization functions tend to operate on modules so to apply the transform to a tensor, we can load a DummyModule with the target tensor and then apply the transformation to the module and then extract the transformed tensor. c^<V^8dQhRS[PRS[S[P,/#)rrbias)rrlr )rrs"rrDummyModule.__annotate__s(u||8ELL3Irc<<\SV`4WnW nR#r)superrrr)rrrrbs&&&rrDummyModule.__init__s   r)rrr) rrrrrrrr __classcell__)rbrs@@rr r sr)rLrYr]rhr{rrrrr rrrrrr r rrrr)raNNr)Fr)raN)Irr rrr@rimportlib.metadatarmathrtypingrr r rtorch.nn.utils.parametrizer%rrtorch.utils._python_dispatchr __all__rr!cacher/rLrYr]rbrhrpr{rrrrr&rrrr r rr"r*rur~rrr`rlr rrrrrrrrrrrrrrrr rrrr rarrr)s|  &'' 00D 2 :  "D>N 3  "4n< 588?? F8M$5J D F\ ~@(%&,([ [|  DGV 6  A0I%((//r