+ ܳj^RIHt^RIt^RIt^RIHt^RIHtH t ^RI t ^RI t ^RI H t ^RIHt^RIHtHt^RIHt/t!R R 4t!R R 4t] P0!R ^R7t] P4P74^8dRRltM ^RItRRltR] P<R]/RltRWRRllt RXRlt!RYRlt"RZRlt#R[Rlt$R\Rlt%R]Rlt&R^Rlt'R R!lt(R"R#lt)R$R%lt*!R&R'4t+R_R(R)llt,R`R*R+llt-RaR,lt.RRRR] P^3R-R.llt0RRRR] P^3R/R0llt1RRRRR1] P^3R2R3llt2RbR4R5llt3RbR6R7llt4RcR8R9llt5RdR:R;llt6ReR<R=llt7]!R>]8R?7] Pr3R@l4t:RfRARBllt;]!R>]8R?7RgRCRDll4t<]!R>]8R?7RgRERFll4t=R] P|3RGRHllt?RhRIRJllt@RiRKRLlltARMRNltBRjRORPlltCRkRQRRlltDRkRSRTlltERUtFRVtGR#)l)IterableN)prod)AnyOptional)Tensor) deprecated)pack_dict_to_tensorunpack_tensor_to_dict)libcJa]tRt^toRtRtRt]R4tRRlt Rt Vt R#)GlobalPageManagerNc\R4hzCall get_instance() instead RuntimeErrorselfs&X/home/luca/screen-checking/.venv/lib/python3.14/site-packages/bitsandbytes/functional.py__init__GlobalPageManager.__init__899c.VnR#N paged_tensorsrs&r initializeGlobalPageManager.initializes rcVPf1VPV4VnVPP4VP#r _instance__new__rclss&r get_instanceGlobalPageManager.get_instance"6 == KK,CM MM $ $ &}}rcVVPRRR1,Fp\W!4K R#)N)rprefetch_tensor)rto_cputs&& r prefetch_allGlobalPageManager.prefetch_all)s(##DbD))A A &*rrF) __name__ __module__ __qualname____firstlineno__r!rr classmethodr%r-__static_attributes____classdictcell__ __classdict__s@rr r s2I:  ''rr cFa]tRt^1toRtRtRt]R4tRt Rt Vt R#)CUBLAS_ContextNc\R4hrrrs&rrCUBLAS_Context.__init__4rrc/VnR#rcontextrs&rrCUBLAS_Context.initialize7s  rcVPf1VPV4VnVPP4VP#rr r#s&rr%CUBLAS_Context.get_instance:r'rcVPVP9d\PP 4p\PP V4\ P!\P!44VPVP&\PP V4VPVP,#r) indexr?torchcudacurrent_device set_devicectc_void_pr get_context)rdevice prev_devices&& rrKCUBLAS_Context.get_contextAs} <) r0r1r2r3r!rrr4r%rKr5r6r7s@rr:r:1s2I: **rr:rF)rDc8V^8dQhR\P/#arEr)formats"r __annotate__rURs''5<<'rc@\PPV4#r)rErF device_ofrRs&r_cuda_device_ofrYRszz##A&&rc8V^8dQhR\P/#rPrS)rTs"rrUrUWs((5<<(rc,\P!4#r) contextlib nullcontextrXs&rrYrYWs%%''rdtyperLcVP\V4,p\P!\P !V44p\P !V\P!\P44p\PPWRR7p\P!W`\V4R7PV4pRVnVP VnV#))shape)r^countT)itemsizerr cget_managed_ptrrIc_size_tcastPOINTERc_intnp ctypeslibas_arrayrE frombufferviewis_pagedrD page_deviceid)r^rLr` num_bytes managed_ptrc_ptr new_arrayouts$$* r get_pagedrt[se,I&&r{{9'=>K GGKBHH!5 6E %%e%9I   9e E J J5 QCCL C JrFc8V^8dQhR\P/#rQArS)rTs"rrUrUfsKKu||Krc VP'gQR4hV'dRpM VPp\P!\ V4\ P !VP4\ P!V44R#)z%Only paged tensors can be prefetched!Nr)) rmrnr cprefetchget_ptrrIrdnbytesc_int32)rwr+deviceids&& rr*r*fsQ :::>>>: ??MM'!*bkk!((3RZZ5IJrc RpVP\P8Xd-\\RV R2R4p\ P !V4pMJVP\P8Xd,\\RV R2R4p\ P!V4pVf\RV 24h\VRR4pV'd#V'd\V4Ve \V4V!\V4\V4X\ P!VP444VP'gVP'd\PP!4'd!\PP#4R#\%\R4'dG\P&P!4'd!\P&P#4R#R#R#R#)Nc_fp32_uint8zFunction not implemented: is_managedFxpu)r^rEfloat32getattrr rIc_floatuint8c_uint8NotImplementedErrorr*rzc_int64numelrmrF is_available synchronizehasattrr) func_namerwBvalueprefetchfunccvaluers&&&&& relementwise_funcrpsI Dww%--sa {%0$7E" EKK sa {&148E" |!$>yk"JKKL%0Jh = A WQZAGGI)>?zzzQZZZ :: " " $ $ JJ " " $ UE " "uyy'='='?'? II ! ! #(@ " rc"\RVRV4R#)fillNr)rwrrLrs&&&&rrrsVQe,rc \RW^4R#)_mulNr)rwrrLs&&&rrrsVQ1%rcV'dRMRp^V,pV'gV^8d"V'g ^V,M^V,^, p\P!VRV4pRVP4, pV^8XdV#VP4^,p\P!VRVP 4^.V,,WWRP 4,4#)?N)rElinspacerrtolist)signed total_bitsadd_zerosign total_valuesvaluesgapls&&& rcreate_linear_maprs4sDj=L:> -3q*}: 8I ^^D#| 4F   C ax LLNa ||F2AJ--/1#);fRj>O>O>QQRRrc(^RIHpT'd}TP\P !TR^ 4RR4P 4p^.^,pTP\P !TR^4RR4)P 4pM{TP\P !TR^4RR4P 4p^.^,pTP\P !TR^4RR4)P 4pYE,T,p\P!T4pTP4PpYP4,pTP4R8XgQhT# \dp\R4ThRp?ii;i)umCreate the NormalFloat (NF4) quantization map. Constructs a lookup table of 16 quantization values (stored in a 256-element tensor for indexing convenience) derived from quantiles of the standard normal distribution N(0, 1). Each bin has approximately equal probability mass under the normal distribution, which is optimal for normally-distributed data like neural network weights. Unlike floating-point types (FP4, FP8), NF4 is NOT a float encoding — the 4-bit index is simply a lookup into this table. There is no sign/exponent/mantissa decomposition. The values are generated by computing ``scipy.stats.norm.ppf()`` (inverse CDF) at evenly spaced quantile points, then normalizing to [-1, 1]. For more details, see: QLoRA: Efficient Finetuning of Quantized LLMs (https://arxiv.org/abs/2305.14314) Args: offset: The outermost quantile boundary, controlling the range of the normal distribution that is covered. ``norm.ppf(offset)`` gives the largest bin edge in standard deviations. The default (0.9677083) covers up to ~1.845 standard deviations and was empirically optimized to minimize quantization error for typical neural network weight distributions. use_extra_value: If True, creates an asymmetric type with 8 negative and 9 positive values (including zero), for 15 non-zero values total. If False, creates a symmetric type with 7 negative and 7 positive values (14 non-zero values total). Returns: A 256-element tensor where the first 16 values are the sorted NF4 quantization levels normalized to [-1, 1], and the remaining values are zero (padding for 8-bit indexing). )normzZScipy is required for `create_normal_map`. Install `bitsandbytes` with the `[test]` extra.Ng?rr)) scipy.statsr ImportErrorppfrErrrsortrmaxr) offsetuse_extra_valueriev1v2v3vrs && rcreate_normal_maprsN<$  XXennVS!4Sb9 : A A CSH xxvsA6s;<< D D F XXennVS!4Sb9 : A A CSH xxvsA6s;<< D D F " A \\!_F [[] ! !F jjlF <<>S  M/  h  sE55 F F  FcBTpTpV'd^M^pWE,W6, 8XgQh.p\\^W, ,)^W, ,^44FwrVP^V ,4K .p \\P !^^.VR74p ^V^, ,p \^V,4Fp V FpV ^8wd^M^p\\V44F%wppVV^V^,),,, pK' V ^8XdV^V ),,pMV^W, ^, ),,pV PV4V'gKV PV)4K K \ V 4^V,8XgQhV P4V^8d6R\ V 4, p\V4FpV P^4K V P4\P!V 4pVVP4,pV#)a Create a floating-point quantization map with configurable bit layout. Generates a lookup table for a custom floating-point format following IEEE 754-like encoding with configurable exponent and mantissa (precision) bits. Despite the name, this function handles any total bit width (including FP4 when called with ``total_bits=4``). The encoding uses: - Exponent bias: ``2^(exponent_bits - 1)`` - Normal values: ``(1 + mantissa) * 2^(exponent - bias - 1)`` - Subnormal values (exponent field = 0): ``mantissa * 2^(-bias)`` Note: The values in the returned tensor are normalized by dividing by the maximum value, so the actual represented range is [-1, 1]. For the FP4 type used in bitsandbytes (2 exponent bits, 1 mantissa bit, signed): ``create_fp8_map(signed=True, exponent_bits=2, precision_bits=1, total_bits=4)`` Args: signed: Whether the format includes a sign bit. exponent_bits: Number of bits for the exponent field. precision_bits: Number of bits for the mantissa (precision/fraction) field. total_bits: Total number of bits per value (must equal sign + exponent + precision). Returns: A 256-element tensor of sorted quantization levels normalized to [-1, 1]. For types with fewer than 8 bits, the remaining entries are zero-padded. )repeatr) enumeraterangeappendlist itertoolsproductlenrrEtensorr)r exponent_bitsprecision_bitsrephas_signevaluesivalrlstbiasevalue bit_patternrpvalrcodes&&&& rcreate_fp8_maprs8 AAqH 5J) )) )GEA-*B$C"DaMLdFeghijq#vkF y  !Q? @C " #Dm,-K1A!E$T+%674Ah//8{T7 *v}q'8%9 99 MM% v uf%. v;!Z- '' ' KKMA~CKsA MM!  KKM << DDHHJD KrcH.pV^, p^WA, ,^, p\V4EF p\V'd^Wd,V, ,^,M#^Wd,V, ^,,^,4p\P!R^V\PR7pVRRVR,,R, p V^ V^, )V,,V ,P 4, pV'gKV^ V^, )V,,)V ,P 4, pEK V^8d\P!R^V^,\PR7pVRRVR,,R, p V^ V^, )X,,V ,P 4, pV'd6V^ V^, )V,,)V ,P 4, pVP ^4VP R4\V4^V,8XgQhR\V4, p \V 4FpVP ^4K VP4\P!V\PR7#) a Creates the dynamic quantiztion map. The dynamic data type is made up of a dynamic exponent and fraction. As the exponent increase from 0 to -7 the number of bits available for the fraction shrinks. This is a generalization of the dynamic type where a certain number of the bits and be reserved for the linear quantization region (the fraction). n determines the maximum number of exponent bits. For more details see (8-Bit Approximations for Parallelism in Deep Learning)[https://arxiv.org/abs/1511.04561] g?)r^N:NN@rrr)) rintrErrrrrrr) rmax_exponent_bitsrdata non_sign_bitsadditional_itemsrfraction_items boundariesmeansrs &&& rcreate_dynamic_mapr(s" DNM]>?!C $ % !#&77 81 <q(+<1S8 ",q01A56%?GGII 6 r 1A 56:;1S8 ",q01A56%?GGII  r 1A 56:;\4pVFYpVfK \VRR4'dKVPVPPVPP 34K[ VR08XdR#RV9pV'dJ\ V4^8d:\RVUu.uF q"fKVPVP3NK" up 24h\ V4^8d:\RVUu.uF q"fKVPVP3NK" up 24hR#uupiuupi)aVerifies that the input tensors are all on the same device. An input tensor may also be marked as `paged`, in which case the device placement is ignored. CPU tensors are allowed and checked for consistency among themselves. Args: tensors (`Iterable[Optional[torch.Tensor]]`): A list of tensors to verify. Raises: `RuntimeError`: Raised when the verification fails. Returns: `Literal[True]` rmFTzfInput tensors need to be on the same device, but found the following tensor and device combinations: zcInput tensors need to be on the same GPU, but found the following tensor and device combinations: )cpuN) setraddrLtyperDrrr`)rdevicesr,has_cpus& r is_on_gpur_sw eG  =J!>!> KK7 8 =/!w&G3wQ>QRSS }}U"{{588<"++C,=,=>>>> g %\\%(@"ABEEfMF/226:!"45/033F;eW^%<= F"4F|,8$''/k*%((0%!1229'2B2N%**WW-.TX  MgDs JJJ$Jc<<V^8dQhRS[RS[S[S[3,/#)rQrr)boolrrr)rTr8s"rrUr!s#!!d!tCH~!rc ,RVPRVPRVPRVPR\ VP 4P R4RVPe\VP4MR/pVP'dVPR VPPR VPPR VPPP4R \ VPP 4P R4R VPP4/4V'dVPfV#VP4UUu/uF)wr4\!V\"P$4'gK'W4bK+ pppVP4UUu/uF)wr4\!V\"P$4'dK'W4bK+ pppRpVPeWpP, p\'V4WW&V#uuppiuuppi)z returns dict of tensors and strings to use in serialization via _save_to_state_dict() param: packed -- returns dict[str, torch.Tensor] for state_dict fit for safetensors saving rrrrr^ztorch.r`Nrrrrrzquant_state.bitsandbytes__)rrrrrr^stripr`tuplerrrcloneritemrrrErr )rrrr'rqs_packed_dictnon_tensor_dictkeys&& rrQuantState.as_dict!s $// dkk   S_**84 $***@U4::&d   ;;; NN#T[[%7%7& (=(=& (8(8(>(>(@"C (9(9$:$@$@$J#T[[%5%5%7  0N,3==?Z?41jELL>Y$!$?Z,3MMO_ODA:aQVQ]Q]C^414O_* ?? & ?? "C1/B [_s"$H  H '$HHcVPPV4VnVPPV4VnVP'dVPPV4VnVP PPV4VP nVP PPV4VP nR#R#r)rr%rrrr)rrLs&&rr% QuantState.toDsIILL( kknnV, ;;;++..0DK!%!3!3!6!6v!>DKK #{{//226:DKK  rcX\V\4'gR#\P!VPVPRR7;'Ed^VP VP 8H;'Ed<\P!VP VP RR7;'EdVPVP8H;'dVPVP8H;'dVPVP8H;'dVPe(VPeVPVP8HMVPVPJ;'dNVPe(VPeVPVP8H#VPVPJ#)Fgư>)atol) rrrEallcloserr`rr^rrrr)rothers&&r__eq__QuantState.__eq__MsL%,, NN4;; 4 @   ekk)  tyy%**4@   ekk)  %//1    5#3#33   ;;*u||/G u||+[[ELL0  ;;*u||/G u||+ [[ELL0 r) rrrr^rrrr`r)fp4nf4) rrrrrrrr^r`rrr)NNNNNNNr/)r0r1r2r3__doc__valid_quant_typesrr#rrr r4r*rr%r=r5r6)xr8s0@rrrsya&8IJ8I1N1#.8IJ M)* Z(11f!!F;  KKsArc V^8dQhR\PR\\P,R\\P,R\\P,R\\P\3,/#)rQrwrrrsrrErrr0r)rTs"rrUrUesnII ||I 5<< I U\\ "I %,,  I 5<< #$Irc V^8:d\RV 24hVP\P\P\P 39d\RVP 24hVfCR\ 9d+\4PVP4\ R&\ R,p\PPPPVVPVP4V4wrgV'd_VP4pWx,p\WtRR7wr\V VPVPRR7VVPVV R7p M2\WqPVPRR7W@PR 7p VeVP!V4MTpVe!VP!V P"4V nW;3#) aQuantize a tensor in blocks of values. The input tensor is quantized by dividing it into blocks of `blocksize` values. The the absolute maximum value within these blocks is calculated for scaling the non-linear quantization. Args: A (`torch.Tensor`): The input tensor. Supports `float16`, `bfloat16`, or `float32` datatypes. code (`torch.Tensor`, *optional*): A mapping describing the low-bit data type. Defaults to a signed 8-bit dynamic type. For more details, see (8-Bit Approximations for Parallelism in Deep Learning)[https://arxiv.org/abs/1511.04561]. absmax (`torch.Tensor`, *optional*): A tensor to use to store the absmax values. out (`torch.Tensor`, *optional*): A tensor to use to store the result. blocksize (`int`, *optional*): The size of the blocks. Defaults to 4096. Valid values are 64, 128, 256, 512, 1024, 2048, and 4096. nested (`bool`, *optional*): Whether to additionally quantize the absmax values. Defaults to False. Raises: ValueError: Raised when the input data type is not supported. Returns: `Tuple[torch.Tensor, QuantState]`: A tuple containing the quantization results. - `torch.Tensor`: The quantized tensor. - [`QuantState`]: The state object used to undo the quantization. blocksize must be positive, got z?Blockwise quantization only supports 16/32-bit floats, but got dynamicF)rrT)copy)rrrr^rrrrrr^)rr^rErfloat16bfloat16 name2qmaprr%rLops bitsandbytesquantize_blockwisedefaultmeanrcopy_r) rwrrrsrr_out_absmaxrqabsmaxrrs &&&&&& rrPrPesFA~;I;GHHwwu}}emmU^^DDZ[\[b[bZcdee | I %#5#7#:#:188#DIi #II**==EE  MD ,WRWX -''  !ggahhTg6R^govovw !_#))D/$C#\\+*<*<=   rc(V^8dQhR\PR\\,R\\P,R\\P,R\\P,R\R\P/#)rQrwrrrrsrrrErrrr)rTs"rrUrUsPP ||P*%P U\\ "P 5<< P %,,  P  P \\Prc VfVf \R4hVP\P8wd\RVP 24hVfGVfCR\9d+\ 4P VP4\R&\R,pVf\W#V\PR7pVP^8:d\RVP 24hVPpVP'db\VPVP4pW!P, pVP\P8wdVP!4pVes\P"P$PP'VVVP(P VP4VPVPVR7V#\P"P$PP+VVVP(P VP4VPVP4#)a|Dequantize a tensor in blocks of values. The input tensor is dequantized by dividing it into blocks of `blocksize` values. The the absolute maximum value within these blocks is used for scaling the non-linear dequantization. Args: A (`torch.Tensor`): The quantized input tensor. quant_state ([`QuantState`], *optional*): The quantization state as returned by [`quantize_blockwise`]. Required if `absmax` is not provided. absmax (`torch.Tensor`, *optional*): A tensor containing the scaling values. Required if `quant_state` is not provided and ignored otherwise. code (`torch.Tensor`, *optional*): A mapping describing the low-bit data type. Defaults to a signed 8-bit dynamic type. For more details, see (8-Bit Approximations for Parallelism in Deep Learning)[https://arxiv.org/abs/1511.04561]. Ignored when `quant_state` is provided. out (`torch.Tensor`, *optional*): A tensor to use to store the result. blocksize (`int`, *optional*): The size of the blocks. Defaults to 4096. Valid values are 64, 128, 256, 512, 1024, 2048, and 4096. Ignored when `quant_state` is provided. Raises: ValueError: Raised when the input data type is not supported. Returns: `torch.Tensor`: The dequantized tensor. The datatype is indicated by `quant_state.dtype` and defaults to `torch.float32`. z:dequantize_blockwise requires either quant_state or absmaxzA must be uint8, got rHrJrGrs)rr^rErrMrr%rLrrrrrdequantize_blockwiserrr$rNrOrsrrQ)rwrrrrsrrs&&&&&&&rr[r[sRv~UVVww%++0 :;; | + I %#5#7#:#:188#DIi # YV[VcVcd !;K > AHH%  rcVfRpRpVR8Xd.R OpM@VR8Xd.R OpM4VR8Xd.R OpM(VR8Xd"V^@8Xd.RORRR 1,pM \R4hVf\RV R24h\P!W1R 7pVPVP 4P 44VP 4^8XgQhV#)NrFr@r?int4af4z94-bit AbnormalFloats currently only support blocksize 64.z Typename z not supportedrL)rg 6GgfgTFٿgI4ҿg০ǿg Org__?g`\?g?g@g?g4?g`?g`v"?r)rg?g @g(@g@g@rg@rgg g(ggggr))rQrrrr)ii)rg|8geg:Kڞ׿gH2퓊cпg}Yu-ÿgQ #(DrgF?g`_?g 0E?gL_߹E?gƶ=?ga@?gкv -?r)rrErdiv_absrr)typenamerLrrs&&& r get_4bit_typerls ~ D5 $ U "l V G U  ?"d#D&&&ab b |!IhZ~"FGG << ,DIIdhhjnn ::<2   KrcV^8dQhR\PR\\P,R\\P,/#rQrwrrsrErr)rTs"rrUrU^A__ ||_ U\\ "_ %,, _rc  \WW#VRV4#r? quantize_4bitrwrrsrcompress_statistics quant_storages&&&&&&r quantize_fp4rx^ C4GP] ^^rcV^8dQhR\PR\\P,R\\P,/#rnro)rTs"rrUrUirprc  \WW#VRV4#r@rsrus&&&&&&r quantize_nf4r}iryrr?c V^8dQhR\PR\\P,R\\P,R\\P\3,/#)rQrwrrsrrE)rTs"rrUrUtsZUU ||U U\\ "U %,, U 5<< #$Urc Vf^@pVR 9d\RV 24hVR 9d\RV: 24hVP\P\P\P 39d\RVP 24hVP p\PPPPVVVV4wr\WPPR7p V'dEV P4p \W, RR7wr? \V VVPVV VV V R7pM\V VVPVV VR7pVeVP!V4MTpVe!VP!VP"4VnW.3#) aQuantize tensor A in blocks of 4-bit values. Quantizes tensor A by dividing it into blocks which are independently quantized. Args: A (`torch.Tensor`): The input tensor. Supports `float16`, `bfloat16`, or `float32` datatypes. absmax (`torch.Tensor`, *optional*): A tensor to use to store the absmax values. out (`torch.Tensor`, *optional*): A tensor to use to store the result. blocksize (`int`, *optional*): The size of the blocks. Defaults to 64. Valid values are 32, 64, 128, 256, 512, 1024, 2048, and 4096. compress_statistics (`bool`, *optional*): Whether to additionally quantize the absmax values. Defaults to False. quant_type (`str`, *optional*): The data type to use: `nf4` or `fp4`. Defaults to `fp4`. quant_storage (`torch.dtype`, *optional*): The dtype of the tensor used to store the result. Defaults to `torch.uint8`. Raises: ValueError: Raised when the input data type is not supported. Returns: Tuple[`torch.Tensor`, `QuantState`]: A tuple containing the quantization results. - `torch.Tensor`: The quantized tensor with packed 4-bit values. - [`QuantState`]: The state object used to undo the quantization. rinvalid blocksize 'quant_type must be 'nf4' or 'fp4', got zDBlockwise 4bit quantization only supports 16/32-bit floats, but got r_r)rr`r^rrrrr)rr`r^rrr @riiir@r?)rr^rErLrKrr`rNrOrtrQrlrLrRrPrrSr)rwrrsrrvrrw input_shaperTrUrrrVrstates&&&&&&& rrtrttskB AA-i[9::'B:.QRRwwu~~u}}emmDD_`a`g`g_hijj''KII**88@@  MD HH 5D,W-=M ''!  ''!  !_#))D/$C||ELL1 :rc V^8dQhR\PR\\,R\\P,R\\P,R\\,R\P/#rQrwrrrsrrrX)rTs"rrUrUoJJ ||J*%J U\\ "J %,,  J } J  \\ Jrc\WW#VR4#rrdequantize_4bitrwrrrsrs&&&&&rdequantize_fp4r 16 5 IIrc V^8dQhR\PR\\,R\\P,R\\P,R\\,R\P/#rrX)rTs"rrUrUrrc\WW#VR4#r|rrs&&&&&rdequantize_nf4rrrc V^8dQhR\PR\\,R\\P,R\\P,R\\,R\P/#rrX)rTs"rrUrUsoUU ||U*%U U\\ "U %,,  U } U \\Urc Vf^@pVf9VeVf \R4h\VVPVPVVR7pM VPpVP R9d\RVP 24hVP R9d\RVP : 24hVP\P\P\P39d\RVP 24hVP'db\VPVP4pW!P, pVP\P8wdVP4pVec\P P"P$P'WVP VP VPVPVR7M`\P P"P$P)VVVP VP VPVP4pVP^,^8XdVP+4#V#) a+Dequantizes a packed 4-bit quantized tensor. The input tensor is dequantized by dividing it into blocks of `blocksize` values. The absolute maximum value within these blocks is used for scaling the non-linear dequantization. Args: A (`torch.Tensor`): The quantized input tensor. quant_state ([`QuantState`], *optional*): The quantization state as returned by [`quantize_4bit`]. Required if `absmax` is not provided. absmax (`torch.Tensor`, *optional*): A tensor containing the scaling values. Required if `quant_state` is not provided and ignored otherwise. out (`torch.Tensor`, *optional*): A tensor to use to store the result. blocksize (`int`, *optional*): The size of the blocks. Defaults to 64. Valid values are 32, 64, 128, 256, 512, 1024, 2048, and 4096. quant_type (`str`, *optional*): The data type to use: `nf4` or `fp4`. Defaults to `fp4`. Raises: ValueError: Raised when the input data type or blocksize is not supported. Returns: `torch.Tensor`: The dequantized tensor. zMdequantize_4bit requires both absmax and out when quant_state is not provided)rr`r^rrrrzFBlockwise 4bit dequantization only supports 16/32-bit floats, but got rZrr)rrr`r^rrrrErLrKrrr[rrr$rNrOrrsrQr,)rwrrrsrrs&&&&&&rrrsF  >S[lm m ))))!  ##$MM-k.C.C-DEFF^3B;CYCYB\]^^ NNabmbsbsatuvv%k&8&8+:L:LM$$$ <<5== (\\^F  ..22 {,,k.D.DkFWFWYdYjYjps 3 ii$$44<<   ! !  " "         wwqzQuuw Jrc"0V^8dQhR\R\R\R\R\R\R\R\R \\ P,R \R \R \R \R\R\\ P,R\RR/#)rQoptimizer_namegrstate1beta1epssteplrrbeta2beta3alpha weight_decay gnorm_scale unorm_vec max_unormrN)rrr$rrrE)rTs"rrUrU8sVVV VV  V  V  V V V U\\ "V V V VVV %V !V$ %VrcRpVR8d/\P!VPP44p\ WW8V.4\P P PVVVVVVVVVV V V VV VVV V4R#)a Performs an inplace optimizer update with one or two optimizer states. Universal optimizer update for 32-bit state and 32/16-bit gradients/weights. Parameters ---------- optimizer_name : str The name of the optimizer: {adam}. g : torch.Tensor Gradient tensor. p : torch.Tensor Parameter tensor. state1 : torch.Tensor Optimizer state 1. beta1 : float Optimizer beta1. eps : float Optimizer epsilon. weight_decay : float Weight decay. step : int Current optimizer step. lr : float The learning rate. state2 : torch.Tensor Optimizer state 2. beta2 : float Optimizer beta2. beta3 : float Optimizer beta3. alpha : float Optimizer alpha. gnorm_scale : float The factor to rescale the gradient to the max clip value. unorm_vec : torch.Tensor The tensor for the update norm. max_unorm : float The maximum update norm relative to the weight norm. skip_zeros : bool Whether to skip zero-valued gradients or not (default: False). rN)rErrr$rrNrOoptimizer_update_32bit)rrrrrrrrrrrrrrrr skip_zeros param_norms&&&&&&&&&&&&&&&&& rrr8s|J3ZZ /  qVY/0 II11         %rc&rV^8dQhR\R\R\R\R\\P,R\R\R\R \R \R \ R \R \R\\P,R\R\\P,R\R\RR/#)rQrrrrrrrrrrrrqmap1qmap2absmax1absmax2rrrN)rrrrEr$r)rTs"rrUrUs+++ ++  + U\\ " +  + + + + + + + + ELL !++ ell #!+"#+$%+( )+rc\W!W4WW.4\PPP VVVVVVVVVV V V V V VVVVV4R#r)rrErNrOoptimizer_update_8bit_blockwise)rrrrrrrrrrrrrrrrrrrs&&&&&&&&&&&&&&&&&&&rrrsf*qVU7DE II::           'rzDThis function is deprecated and will be removed in a future release.)categoryc  \PP4'g\PP4VPV8wgVPV8wd&\ RVP RVP 24hVP pVP pTpTp Rp \V4^8XEd\V4^8XdV'g6V 'g.VP ^,VP ^,8wdRp EMV'd6V 'g.VP ^,VP ^,8wdRp EMV'd6V 'd.VP ^,VP ^,8wdRp EMbV'g4V 'd,VP ^,VP ^,8wdRp EM%\V4^8XEd\V4^8XdV'g6V 'g.VP ^,VP ^,8wdRp EMV'd6V 'g.VP ^,VP ^,8wdRp EMV'd6V 'd.VP ^,VP ^,8wdRp EMMV'g4V 'd,VP ^,VP ^,8wdRp EM\V4^8XEd\V4^8XdV'g5V 'g-VP ^,VP ^,8wdRp MV'd5V 'g-VP ^,VP ^,8wdRp MwV'd5V 'd-VP ^,VP ^,8wdRp M;V'g4V 'd,VP ^,VP ^,8wdRp VeVP p V 'gw\V4^8Xdg\V4^8XdWV ^,V^,8XdBV ^,V^,8Xd-V^,V^,8XdV^,V^,8XdRp EMI\V4^8Xd\V4^8XdV'gV 'gV^,V^,3p EMV'dV 'dV^,V^,3p EMV'dV 'gV^,V^,3p EMV'gV 'dV^,V^,3p EM\V4^8Xd\V4^8XdV'g%V 'gV^,V^,V^,3p EMMV'd%V 'dV^,V^,V^,3p EM!V'd$V 'gV^,V^,V^,3p MV'g#V 'dV^,V^,V^,3p M\V4^8Xd\V4^8XdV'g$V 'gV^,V^,V^,3p MV'd$V 'dV^,V^,V^,3p MUV'd$V 'gV^,V^,V^,3p M*V'g#V 'dV^,V^,V^,3p V 'g\RV RV RV RV R2 4hX #)z3Expected torch.int8 input tensors A and B, but got  and TFz?Tensor dimensions incorrect for matrix mulitiplication: A x B:  x z with transpose for A x B: r) rErFis_initializedinitr^ TypeErrorr`rr) rwrrs transposed_A transposed_B expected_typesAsBtAtBcorrectsouts &&&&&& r check_matmulrsz :: $ $ & & ww-177m#;MaggYV[\]\c\c[deff B B B BG 2w!|B1 "qwwqz!9G qwwqzQWWQZ7G B1771:3GqwwqzQWWQZ7G RA#b'Q,"qwwqz!9G qwwqzQWWQZ7G B1771:3GqwwqzQWWQZ7G RA#b'Q,"qwwqz!9G qwwqzQWWQZ7G B1771:3GqwwqzQWWQZ7G yy3r7aio supported for tensor contractions, but dims for A x B were: r)rrEzerosint32rLrr` batched_igemmstriderr:r%rKrr cigemmrIc_boolr|rz)rwrrsrrrrrnldbmr'ldaldcptrs&&&&& rigemmr9s c >D {kkt5;;qxxH 177|qS\Q. 771: # aggaj(@ s+ + B BB1 eRU^ #b'Q,eRUBqE "B1 eRU^ #b'Q,eRUBqE " 2w!| 88:a=AGGAJ & L XXZ]aggaj (L qww<1 xxz!} *$ A!''!*,# xxz!} *$ A!''!*,# r7a<1A((*,QA6C W\c"gl11 AQ%C qE qEhhj|!4e RA2w!||1A2a5BqE>_`b_ccfgifjk   qE qE qEBqEMe  % % ' 3 3AHH =CqSkJJ  , , 1  1  1     3 3 3  JrcfV^8dQhR\R\R\\P,/#rr)rTs"rrUrUs4]] ] ] %,, ]rc, \VP4^8Xd\VP4^8Xg&\RVP RVP 24h\WW#V4pVf2\P !V\P VPR7pVP4'dVP4^,pRpEM#VP4pV^,VP^,8wd)VP4pVP4^,pMV^,VP^,8XdRpVP4^,pMV^,^8Xd)VP4pVP4^,pM]V^,^8Xd)VP4pVP4^,pM'VP4pVP4^,pVP4'dVP4^,pRpMVP4pV^,VP^,8wd+VP4pVP4^,pRpMbV^,VP^,8XdVP4^,pRpM)VP4pVP4^,pRpVP^,p VP^,p VP^,p VP^,p T p VP^,VP^,,pVP^,VP^,,pVP^,VP^,,p\P4PVP4p\WV.4\P !V\"P$!V4\"P$!V4\"P&!V 4\"P&!V 4\"P&!V 4\)V4\)V4\)V4\"P&!V4\"P&!V4\"P&!V 4\"P*!V4\"P*!V4\"P*!V4\"P,!V 44V#)rdz@Expected 3-dimensional tensors for bmm, but got shapes A and B: rrFT)rr`rrrErrrL is_contiguousr contiguousr:r%rKrr cbatched_igemmrIrr|rzc_longc_uint32)rwrrsrrrrsr num_batchrrr'rstrideAstrideBstrideCrs&&&&& rrrso qww<1 CLA$5[\]\c\c[ddijkjqjqirstt c >D {kkt5;;qxxHhhjm HHJ Q41771:  A((*Q-C qTQWWQZ L((*Q-CtqyLLNhhjm1LLNhhjmLLNhhjmhhjm HHJ Q41771:  A((*Q-C L qTQWWQZ ((*Q-CL A((*Q-C L I  A  A  A Cggaj1771:%Gggaj1771:%Gggaj1771:%G  % % ' 3 3AHH =C qSk  , , 1  1  1     3 3 3 ' ' ' I!$ JrcV^8dQhR\PR\PR\\P,/#rro)rTs"rrUrUs7CC%,,C5<<Chu||>TCrcVe7\PPPP WV4V#\PPPP W4#)aPerforms an 8-bit integer matrix multiplication. A linear transformation is applied such that `out = A @ B.T`. When possible, integer tensor core hardware is utilized to accelerate the operation. Args: A (`torch.Tensor`): The first matrix operand with the data type `torch.int8`. B (`torch.Tensor`): The second matrix operand with the data type `torch.int8`. out (`torch.Tensor`, *optional*): A pre-allocated tensor used to store the result. dtype (`torch.dtype`, *optional*): The expected data type of the output. Defaults to `torch.int32`. Raises: `NotImplementedError`: The operation is not supported in the current environment. `RuntimeError`: Raised when the cannot be completed for any other reason. Returns: `torch.Tensor`: The result of the operation. )rErNrOint8_linear_matmulrsrQ)rwrrsr^s&&&&rrrsO&  1155aC@ 99 ! ! 4 4 < This function is useful for training, but for inference it is advised to use [`int8_vectorwise_quant`] instead. This implementation performs additional column-wise transposed calculations which are not optimized. Args: A (`torch.Tensor` with dtype `torch.float16`): The input matrix. col_stats (`torch.Tensor`, *optional*): A pre-allocated tensor to hold the column-wise quantization scales. row_stats (`torch.Tensor`, *optional*): A pre-allocated tensor to hold the row-wise quantization scales. out_col (`torch.Tensor`, *optional*): A pre-allocated tensor to hold the column-wise quantized data. out_row (`torch.Tensor`, *optional*): A pre-allocated tensor to hold the row-wise quantized data. threshold (`float`, *optional*): An optional threshold for sparse decomposition of outlier features. No outliers are held back when 0.0. Defaults to 0.0. Returns: `Tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, Optional[torch.Tensor]]`: A tuple containing the quantized tensor and relevant statistics. - `torch.Tensor` with dtype `torch.int8`: The row-wise quantized data. - `torch.Tensor` with dtype `torch.int8`: The column-wise quantized data. - `torch.Tensor` with dtype `torch.float32`: The row-wise quantization scales. - `torch.Tensor` with dtype `torch.float32`: The column-wise quantization scales. - `torch.Tensor` with dtype `torch.int32`, *optional*: A list of column indices which contain outlier features. zUrow_stats must be None. int8_double_quant() does not support pre-allocated row_stats.zUcol_stats must be None. int8_double_quant() does not support pre-allocated col_stats.zQout_col must be None. int8_double_quant() does not support pre-allocated out_col.zQout_row must be None. int8_double_quant() does not support pre-allocated out_row.) threshold)rrErNrOint8_double_quantrQ)rwrrrrrs&&&&&&rrr6sqNpqqpqqlmmlmm 99 ! ! 3 3 ; ;A ; SSrcXV^8dQhR\PR\P/#)rQrwstatsrS)rTs"rrUrUis& L Lu|| LELL Lrch\PPPP W4#)aADequantizes a tensor with dtype `torch.int8` to `torch.float32`. Args: A (`torch.Tensor` with dtype `torch.int8`): The quantized int8 tensor. stats (`torch.Tensor` with dtype `torch.float32`): The row-wise quantization statistics. Returns: `torch.Tensor` with dtype `torch.float32`: The dequantized tensor. )rErNrOint8_vectorwise_dequantrQ)rwrs&&rrris% 99 ! ! 9 9 A A! KKrc8V^8dQhR\P/#rvrS)rTs"rrUrUwsNNU\\Nrch\PPPP W4#)aGQuantizes a tensor with dtype `torch.float16` to `torch.int8` in accordance to the `LLM.int8()` algorithm. For more information, see the [LLM.int8() paper](https://arxiv.org/abs/2208.07339). Args: A (`torch.Tensor` with dtype `torch.float16`): The input tensor. threshold (`float`, *optional*): An optional threshold for sparse decomposition of outlier features. No outliers are held back when 0.0. Defaults to 0.0. Returns: `Tuple[torch.Tensor, torch.Tensor, Optional[torch.Tensor]]`: A tuple containing the quantized tensor and relevant statistics. - `torch.Tensor` with dtype `torch.int8`: The quantized data. - `torch.Tensor` with dtype `torch.float32`: The quantization scales. - `torch.Tensor` with dtype `torch.int32`, *optional*: A list of column indices which contain outlier features. )rErNrOint8_vectorwise_quantrQ)rwrs&&rrrws%$ 99 ! ! 7 7 ? ? MMrcPV^8dQhR\PR\R\/#)rQqweightrblock_n)rErrr)rTs"rrUrUs&3&3&ELL3&z3&\_3&rcVP\P8wd1VPVnVP \P4pVPVnVP VnVPVn VPR 4p\P!VP^,^,\PVPR7pV^,VR&V^, VR&VPVP4P\P4p\VP4^8XgQhVP^,VP^,reWR,^8XgQR4hV^,^8XgQR4hTp^ pWW,Wv^,^.p WV^,.p VPV 4p V P!R R 4P#4p V PR V^,4p V RVR13,p V RRV13,p V ^,V ,P\P4pVPV 4pVP 'd}\%VP&VP(4pVVP*, pVP\P,8wdVP/4pVVnRVn\1VR 4VP&PVP^,VP^,VP2,4P4P\P64P#4Vn\P6VnR VnW3#)z3 qweight: (K * N / 2) uint8 return: packed_weight )r^rLrNrQNNrQN must be divisible by block_nK must be even:NNNNFrTr)rfre)r^rEroriginal_storage_typerloriginal_dtyperoriginal_nestedr`original_qshapereshapeemptyrrLr%r transposerr[rrrrr$delattrrTrLpacking_format_for_cpu)rrr unpacked_w qweight_finalNKBLOCK_N BIT_COUNT new_shape out_shapeqwhighlowr final_qweightrs&&& r_convert_weight_packed_for_cpurs }} #,3MM ),,u{{+!,!2!2K"-"4"4K")--Koob!GW]]1-1W^^\J}JtlJsO&&{'8'89<>,,DDE O ##rcp\P!4pV# \\\3dRpT#i;i)zg Try calling native lib.has_avx512bf16_cpu(). Return False explicitly if symbol missing or call fails. F)r has_avx512bf16_cpurrOSError)support_avx_bf16s rhas_avx512bf16r)sB !113  L' 2!  !s 55g_@r/)T)NTr)TT)g+ew?T)TrbrQr*)Tr`r*)NNNrF)NNNNrF)Nr)NNNN)NNNNr?) NrrrrrNrF)rrF)NFFN)NFF)NN)NNNNr)r)r)Hcollections.abcrctypesrIrmathrtypingrrnumpyrhrErtyping_extensionsrbitsandbytes.utilsr r cextensionr rMr r:rLFIRST_CUDA_DEVICErF device_countrYr\rrtr*rrrrrrrrrrzrrPr[rlrrxr}rtrrrrr FutureWarningint8rrrrrrrrrrrr$r)Crrr9s" %  (I  ''0**2LLq1  ::q '(EMM2CK$<-&S&7tBJ43n'T X %~ ~ BIXPfWx&*"&++ _&*"&++ _&*"&++UpJJUpVr+\ R]jkFKjjQlQh"J R]jkblbJ R]jk]l]@X\chcncnC480Tf LN*3&lA$H  r