+ "ݳjl}aR$t=0t^RIt^RIHt^RIHtHtHtHt^RI t ^RI H t ^RI H t HtHtHt^RIHt]!R]3/4t]^k]P*!]4t] P0] P2] P4] P60t] P:R%] P<R&] P>R'] P@R(] PBR)] PDR*] PFR+/t$] PJR,] PLR-] PNR.] PPR//t)]^k])PU]$4RR lt+R t,R t-RRRRRR ]P\3R Rllt/R0RRllt0] Pb!4RR]P\3RRll4t2RR]P\Pf3RRllt4]P\Pf3RRllt5RR]P\3R] Pl/RRlllt7RR]P\Pf] Pl3RRllt8]P\Pf] Pl3RRllt9!RR] 4t:!R R!] 4t;!R"R#] 4tr?r@rAs &&&&&&&&&&&&r"choose_qparams_affine_with_min_maxrIsa0 "  "3"? FJ rcV^8dQhR\\P,R\R\\ ,R\P R\\\ \\3,,R\\\ \\3,,R\\,R\\P ,R \\P ,R \R \\,R \\P,R \\P,R\\P\P3,/#)rinputr;r1r<r(r)r=r>r?r@rAr9r:r) rrrBstrrrDrrrErrC)rs"rrrs@3@3 ELL !@3@3S @3++ @3 c5$./0 @3 c5$./0 @3 %@3%++&@3u{{+@3@3 }@3ell #@3ell #@3 5<< %&@3rc H \W4V4wrEV\PP\PP\P P39g QRV 24hV\ 9d*V\PP8Xg QRV 24hVeVf VPpVf VPpVf+\P!VP4Pp\V4VP48XgQRVP4 RV 24h\W P44wrVPV 4p\P !WRR7p \P"!WRR7p MV eV fQR4hV PV P8XgQR 4hVf V PpVf V PpVf+\P!V P4PpV 'dX\P$!V \P&!V 44p\P(!V \P&!V 44pMT pT pV\PP8Xg!V\PP8XEdIV\PP8Xd9\P(!V)V4pV\+WT, 4^, , pMbV\PP8XgQhV\+V4, pV\+V4, pVV8p\P,!VVV4pV 'g \/R 4hV e+V \0P2P8wd \/R 4h\P4!VVR 7p\P6!V\9WT,^,^, 44pEM V\P P8XgQhVV, \+WT, 4, p\P4!VVR 7pV \0P:P8XdRpMV 'd=V\P<!VV, 4, p\P4!VWE4pMLV \0P>P8XgQR 4hWT,^,^, pVVV,,pVeVPAVR7pVPAVR7V3#)aWop definition that has compatible signatures with custom op library The op does the following: 1. figure out the dimension for reduction based on block_size 2. find min_val/max_val based on the dimension for reduction 3. calculate quantization parameters based on min_val/max_val based on args like `preserve_zero` and `zero_point_domain` zUnsupported mapping type: zr?r@rAr9r:r3r4 min_val_neg max_val_posscalesminsmaxmask zero_point mid_points&&&&&&&&&&&&& rrGrGsI.4LYWI ""--22## 3 $L>2 3  y {44999 J<. Y 9   ++K  #${{  ;++ekk*..C:%))+- UYY[M | D -/D  / + ./**UF**UF"w': c :}} - k -  !--K  #&}}  ;++gmm,00Cii)9)9')BC ii)9)9')BC     --222 ;@@EE E ;0055 5))[L+>K5)>#?!#CDE;#H#H#M#MM MMy!11Dy!11D$;DKKdD1ET   )!_%8%8%=%==f  Es+__UC1F1Ja0O,PQ {55:::::{*eI4I.JJ Es+  4 4 9 9 9J&)II "[[YJ (O,A,A,F,FFNF'2Q6!; (59+<< ]])9]: 88+8 & 22rcV^8dQhR\PR\\R3,R\PR\\P,R\P R\\ \\3,,R\\ \\3,,R \\,R \P/ #) rrKr1.reri output_dtyper(r)rAr rrBrCrDrrrrEr )rs"rrr;s55 <<5c3h5 <<5& 5 ++ 5 c5j)* 5c5j)*5 05 \\5rc N\TTTTTTTVeVP4#R4#)a Args: input (torch.Tensor): original float32, float16 or bfloat16 Tensor block_size: (Tuple[int, ...]): granularity of quantization, this means the size of the tensor elements that's sharing the same qparam e.g. when size is the same as the input tensor dimension, we are using per tensor quantization scale (float): quantization parameter for affine quantization zero_point (int): quantization parameter for affine quantization output_dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor quant_min (Optional[int]): minimum quantized value for output Tensor, if not specified, it will be derived from dtype quant_max (Optional[int]): maximum quantized value for output Tensor, if not specified, it will be derived from dtype zero_point_domain (ZeroPointDomain): the domain that zero_point is in, should be either integer or float if zero_point is in integer domain, zero point is added to the quantized integer value during quantization if zero_point is in floating point domain, zero point is subtracted from the floating point (unquantized) value during quantization default is ZeroPointDomain.INT Note: How can block_size represent different granularities? let's say we have a Tensor of size: (3, 3, 10, 10), here is the table showing how block_size represents different granularities: granularity type | block_size per_tensor | (3, 3, 10, 10) per_axis (axis=0) | (1, 3, 10, 10) per_axis (axis=1) | (3, 1, 10, 10) per_group (groupsize=2) | (3, 3, 10, 2) per_group (groupsize=2) for axis = 3 | (3, 3, 2, 10) Output: quantized tensor with requested dtype N)_quantize_affinerHrKr1rerirlr(r)rAs&&&&&&&&rquantize_affinerq:sGZ   "3"?  FJ  rcV^8dQhR\PR\\,R\PR\\P,R\P R\\ \\\3,,R\\ \\\3,,R\\,R \P/ #) rrKr1rerirlr(r)rAr rrBlistrDrrrrErrL)rs"rrrss   << S   << &  ++  c5$./0 c5$./0  }  \\ rc \WEV4wrVV\9d\Pp\ VVVVVVV4P V4#)aop definition that has compatible signatures with custom op library Note: zero_point_domain is optional specifies how we quantize the floating point to quantized data: INT: quantized_val = (float_val / scale) (integer) + zero_point (integer) FLOAT: quantized_val = (float_val - (zero_point (float) - scale * mid_point)) / scale None: quantized_val = (float_val / scale) | this is primarily used for floatx quantization Where we do not want to round values to nearest integer and instead scale and cast. )r,_SUB_BYTE_UINT_BOUNDSruint8_quantize_affine_no_dtype_castrbrps&&&&&&&&rrorossU&4LYWI,,{{ )    brcNV^8dQhR\PR\\,R\PR\\P,R\ \\ 3,R\ \\ 3,R\\,R\P/#) rrKr1rerir(r)rAr)rrBrtrDrrrErL)rs"rrrs>> <<>S > <<>& > S%Z > S%Z > }> \\>rcVP\P\P\P39gQRVP 24h\ V4VP 48XgQRVP 4 RV 24h\WP44wrxVPp VPV4pTp VFp ^W&K VPV 4pVeVPV 4pV\PP8XdC\P!\P!VRV, ,4V,WE4p EMV\P P8XdFVeQR4h\P!\P!VRV, ,4WE4p MVf8VeQR4h\P!WP#4,WE4p M~V\P$P8XgQhWT,^,^, p W2V ,, p\P!\P!W, V, 4WE4p V PV 4p V #)a? The op does the following: 1. figure out the dimension for reduction based on block_size, also reshape the input to align with the shape after reduction 2. quantize the input based on the quantization parameters scale and zero_point and args like zero_point_domain 3. reshape the quantized result to origianl shape zUnsupported input dtype: rNrOg?8zero_point should be None when zero_point_domain is NONE8zero_point should be None when zero_point_domain is None)rrfloat32float16bfloat16r.rQr7rVshaperWr r\rHr]r`r_ reciprocalra)rKr1rerir(r)rAr3r4original_shapeshape_after_reductionr6quantrjr9s&&&&&&& rrxrxs*$ ;;    1 #5;;-0 1  z?eiik )  ^J<@ )+@JJL+'[[N JJ* +E/ #$  JJ, -E__%:; O//444 KKu- . ;Y  o2277 7! F ! EKKu(=> U  "! F ! E$4$4$66 M O$9$9$>$>>>>*Q.!3 y00 KKE1 2I  JJ~ &E LrrlcV^8dQhR\PR\\R3,R\PR\\P,R\P R\\ \\3,,R\\ \\3,,R \R \P R \P/ #) rrKr1.reri input_dtyper(r)rArlrrm)rs"rrrs,, <<,c3h, <<,& ,  , c5j)* ,c5j)*,',++, \\,rc V\TTTTTTTVeVPVR7 #RVR7 #)a Args: input (torch.Tensor): quantized tensor, should match the dtype `dtype` argument block_size: (List[int]): granularity of quantization, this means the size of the tensor elements that's sharing the same qparam e.g. when size is the same as the input tensor dimension, we are using per tensor quantization scale (Tensor): quantization parameter for affine quantization zero_point (Tensor): quantization parameter for affine quantization input_dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor quant_min (Optional[int]): minimum quantized value for input Tensor quant_max (Optional[int]): maximum quantized value for input Tensor output_dtype (torch.dtype): dtype for output Tensor, default is fp32 zero_point_domain (ZeroPointDomain): the domain that zero_point is in, should be either integer or float if zero_point is in integer domain, zero point is added to the quantized integer value during quantization if zero_point is in floating point domain, zero point is subtracted from the floating point (unquantized) value during quantization default is ZeroPointDomain.INT Output: dequantized Tensor, with requested dtype or fp32 N)rl)_dequantize_affinerH rKr1rerirr(r)rArls &&&&&&&&$rdequantize_affinersMD   "3"?!  FJ!  rcV^8dQhR\PR\\,R\PR\\P,R\P R\\ \\\3,,R\\ \\\3,,R\\,R \P R \P/ #) rrKr1rerirr(r)rArlrrs)rs"rrrs   << S   << &    c5$./0 c5$./0  } ++  \\ rc $V\9d)VPV8XgQRV RVP 24hV\P\P\P 39g QRV 24h\ WEV4wrV\VVVVVVVV4#)zCop definition that has compatible signatures with custom op libraryz Expected: z, got: zUnsupported output dtype: )rvrrr}r~rr,!_dequantize_affine_no_dtype_checkrs &&&&&&&&&rrrs//{{k)  WU[[M : )     3 $L>2 3  4KIVI ,    rcnV^8dQhR\PR\\,R\PR\\P,R\ \\ 3,R\ \\ 3,R\\,R\PR \P/ #) rrKr1rerir(r)rArlr) rrBrtrDrrrErLr)rs"rrr)sG9G9 <<G9S G9 <<G9& G9 S%Z G9 S%Z G9 }G9++G9 \\G9rc\V4VP48XgQRVP4 RV 24h\WP44wrVPp VP V4pTp V Fp ^W&K VP V 4pVeVP V 4pV\ PP8XdfVP\PRR7p Ve&WP\P4, p V PV4p W,p EMV\ PP8Xd&VeQR4hVPV4p W,p MVfUVeQR4h\VP4'gQRVP 24hVPV4p W,p MlV\ PP8Xg QRV 24hWT,^,^, pW, p V PV4p W,p Ve W, p V P V 4PV4#) aThis function converts AQT tensors to their high precision floating point representation The op does the following: 1. figure out the dimension for reduction based on block_size, also reshape the input to align with the shape after reduction 2. dequantize the input based on the quantization parameters scale and zero_point and args like zero_point_domain 3. reshape the quantized result to origianl shape and change dtype to the output_dtype rNrOT)copyr{r|zNdequantiztion with no zero point domain is only supported with FP8 types, got zUnexpected zero point domain: )r.rQr7rVrrWr r\rHrbrint32r_r rra)rKr1rerir(r)rArlr3r4rrr6dequantrjs&&&&&&&& rrr)s1$ z?eiik )  ^J<@ )+@JJL+'[[N JJ* +E/ #$  JJ, -E__%:; O//444((5;;T(2  ! ekk ::G**\*/ o2277 7! F !((<(/  "! F !u{{++ \]b]h]h\i j +((<(/ O$9$9$>$>> ,->,? @ >*Q.!3 #**\*  !  !G << ' * *< 88rcDa]tRtRtoV3RlRltV3RlRltRtVtR#)AffineQuantizedMinMaxObserverisc4<V^8dQhRS[P/#rrKrrB)r __classdict__s"rr*AffineQuantizedMinMaxObserver.__annotate__tsU\\rcLVP4^8XdV#VP4pVPVnVPfQR4h\ VP VP4Vn\VPVP44wr4VPV4p\P!W$RR7p\P!W$RR7p\VR4'd\VR4'gWPnW`nV#VPP VP 8Xg,QRVPP RVP 24hVP P VP 8Xg,QRVP P R VP 24h\P"!VPV4p\P$!VP V4pVPP'V4VP P'V4V# rzgranularity is NoneFrPr9r:z=Can't update existing min_val - shape mismatch, self.min_val:z != min_val:z=Can't update existing max_val - shape mismatch, self.max_val z != max_val:)numeldetachroriginal_dtype granularityr rr1r7rVrWrrXrYhasattrr9r:r$r%copy_selfrKinput_detachedr3r4r9r:s&& rforward%AffineQuantizedMinMaxObserver.forwardts ;;=A L,22+B-BB+()=)=t?O?OP.C OO^002/ +(,,-@A**^O**^OtY''wtY/G/G"L"L <<%%6 OPTP\P\PbPbOccopwp}p}o~ 6<<%%6 OPTP\P\PbPbOccopwp}p}o~ 6ii g6Gii g6G LL  w ' LL  w ' rc^<V^8dQhRS[S[PS[P3,/#rrrCrrB)rrs"rrrs&  5u||)C#D rc`\VR4'd\VR4'gQR4h\VPVPVP.VP VP VPVPVPVPVPVP4 #r9r:zhExpecting the observer has min_val and max_val, please run the observer before calling calculate_qparams rrIr9r:r;r<r(r)r=r>r?r@rArs&rcalculate_qparams/AffineQuantizedMinMaxObserver.calculate_qparamsstY''GD),D,D v D2 LL LL        NN NN HH     ! !     " "  r)r1r:r9rN)__name__ __module__ __qualname____firstlineno__rr__static_attributes____classdictcell__)rs@rrrss@  rrc aa]tRtRtoRRRRRRRR]P 3 V3RlV3RllltV3RlR ltV3R lR ltR t Vt V;t #) *AffineQuantizedMovingAverageMinMaxObserverig{Gz?NFTc<V^8dQhRS[RS[PRS[RS[S[,RS[S[,RS[S[,RS[S[P,RS[S[P,R S[R S[S[,/ # rr;r<rr(r)r=r>r?r@rA r rrrrrDrErr )rrs"rr7AffineQuantizedMovingAverageMinMaxObserver.__annotate__s$ $ !$ kk$ ! $ C= $ C=$ e_$ ekk*$ #5;;/$ $ $O4$ rc <WnW@nV'd*VP^8wd\RVP 24h\SV`VVVVVVV V V V R7 R#)z[MovingAverageMinMaxObserver doesn't support dynamic quantization for averaging constant of r;r<rr(r)r=r>r?r@rAN) is_dynamicaveraging_constantNotImplementedErrorsuper__init__)rr;r<rrr(r)r=rr>r?r@rAkwargs __class__s&&&&&&&&&&&&&,rr3AffineQuantizedMovingAverageMinMaxObserver.__init__sv"%"4 $11Q6%))-)@)@(AC  %%##-'/  rc4<V^8dQhRS[P/#rr)rrs"rrrsU\\rcVP4^8XdV#VP4pVPVnVPfQR4h\ VP VP4Vn\VPVP44wr4VPV4p\P!W$RR7p\P!W$RR7p\VR4'd\VR4'gWPnW`nV#VPP VP 8Xg,QRVPP RVP 24hVP P VP 8Xg,QRVP P R VP 24hVPVP"WPP, ,,pVP VP"W`P , ,,pVPP%V4VP P%V4V#r)rrrrrr rr1r7rVrWrrXrYrr9r:rrrs&& rr2AffineQuantizedMovingAverageMinMaxObserver.forwards ;;=A L,22+B-BB+()=)=t?O?OP.C OO^002/ +(,,-@A**^O**^OtY''wtY/G/G"L"L <<%%6 OPTP\P\PbPbOccopwp}p}o~ 6<<%%6 OPTP\P\PbPbOccopwp}p}o~ 6llT%<%<,,@V%WWGllT%<%<,,@V%WWG LL  w ' LL  w ' rc^<V^8dQhRS[S[PS[P3,/#rr)rrs"rrrs&  5u||)C#D rc`\VR4'd\VR4'gQR4h\VPVPVP.VP VP VPVPVPVPVPVP4 #rrrs&rrr?r@rArrs&&&&&&&&&&&&,rr+AffineQuantizedPlaceholderObserver.__init__s: % %%##-'/  rcr\VPVP4VnVPVnV#r)r rrr1rr)rrKs&&rr*AffineQuantizedPlaceholderObserver.forward%s+(d6F6FG#kk rc\R4h)z>calculate_qparams should not be called for PlaceholderObserver) Exceptionrs&rr4AffineQuantizedPlaceholderObserver.calculate_qparams*s L  r)r1rrrrs@@rrrs> $(#'#-126"7F7J7J  >   rrcV^8dQh/^\9d \;R&^\9dG\\\P \ 3,\\\3,3,;R&#)rrr&) __conditional_annotations__rdictrrrr rCrD)rs"rrr sP)(S(NeEKK$=>c3hOPOr)rr)r)r)r)r)r?)r)r)ir)ii)ii) NNNNNTr\NN)>rloggingabcrtypingrrrrrtorchao.quantizationr"torchao.quantization.pt2e.observerr r r r torchao.quantization.utilsr objectr getLoggerrloggerrrrrr"uint1uint2uint3uint4uint5uint6uint7rvrwint8int16rr&updater r,r7r\rIrGno_gradrqrHrorxr}rrrrrrr)rs@rrsB-- , 6 56)R ((   8 $        KK KK KK KK KK KK KK  KK JJ  KK& KK& T 45 >'/` $#)-.23B3F3F&R@3F.2-13B3F3F55|4837'6':':'?'? T(7':':'?'?>N.2-1)8)<)<,!& ,,j4837'6':':'?'? %  T(7':':'?'? % G9T2 $?2 jZ 1LZ z( )D( r