+ ja0t$Rt^RIHt^RIt^RIt^RIt^RIt^RIt^RI H t ^RI H t ^RI t ^RIHt^RIHt^RIHtHtHt^RIHtHt]P2!]4t] !R R 7!R R 44t]P:R Rl4t]P:RRl4t] P@PBRRl4t"] P@PBR:RRll4t#Rs$R]%R&] P@PLRRl4t'RRlt(RRlt)RRlt*R;RR llt+R!R"lt,R#R$lt-R%R&lt.R'R(lt/R)R*lt0R+R,lt1]!R-R.R/7RCurated entry points exposed by `kernels-community/deep-gemm`.rfp8_fp4_matmulgrouped_fp8_fp4_matmul_ntgrouped_fp8_fp4_matmul_nngrouped_bf16_matmul_ntgrouped_bf16_matmul_nnper_token_cast_to_fp8!transform_sf_into_required_layouttransform_weights_for_mega_moeget_symm_buffer_for_mega_moefp8_fp4_mega_moeint m_alignmentN)__name__ __module__ __qualname____firstlineno____doc____annotations____static_attributes__rc/home/luca/screen-checking/.venv/lib/python3.14/site-packages/transformers/integrations/deepgemm.pyrr6sIH''''$$$$##'//$,,"**r#rcV^8dQhRR/#)returnz str | Noner)formats"r$ __annotate__r)Ns r#c\PPR4;'g \PPR4pV'dV#\P!R4pV'd=\P P \P P V44#\P PR4'dR#R#)urResolve the CUDA toolkit root the way DeepGEMM's JIT does: ``CUDA_HOME`` → ``CUDA_PATH`` → dir of ``which nvcc`` → ``/usr/local/cuda`` (``None`` if none found). Mirrors DeepGEMM's own ``_find_cuda_home`` so we agree on the path it will actually use, rather than reusing ``torch.utils.cpp_extension.CUDA_HOME`` whose resolution inits a CUDA context (fork-unsafe). CUDA_HOME CUDA_PATHnvccz/usr/local/cudaN)osenvirongetshutilwhichpathdirnameisdir) cuda_homer-s r$_get_cuda_homer7Ms {+JJrzz~~k/JI << D wwrwwt455 ww}}&'' r#cV^8dQhRR/#)r&r'tuple[int, int] | Noner)r(s"r$r)r)as++1+r#c\4pVfR#\PPVR4p\PP V4'd\ V4;_uu_4p\ P!V4pRRR4XPRVPR/44PRR4pVPR4R,wrV\V4\V43#\PPVR 4p\PP V4'd\ V4;_uu_4p\P!R VP!44pRRR4X'd5\VP#^44\VP#^443#\PPVR R 4p \PP V 4'd\ V 4;_uu_4p\P!R VP!44pRRR4X'd4\VP#^44p V R,V R,^ ,3#R#R# +'giEL;i \\\3dELi;i +'giELG;i \\3dEL!i;i +'giL;i \\3dR#i;i)a{Version of the CUDA toolkit nvcc will use, as ``(major, minor)``, read off disk without a subprocess from (in order) ``{CUDA_HOME}/version.json``, ``version.txt``, or the ``CUDA_VERSION`` define in ``include/cuda.h``. ``None`` if unreadable. This is the compiler that builds the kernels, unlike ``torch.version.cuda`` (torch's bundled runtime, which never drives a JIT compile). Nz version.json cuda_nvcccudaversion.:Nr&Nz version.txtzCUDA Version (\d+)\.(\d+)includezcuda.hz#define CUDA_VERSION (\d+)i)r7r.r3joinisfileopenjsonloadr0splitrOSError ValueErrorAttributeErrorresearchreadgroup) r6 version_jsonf componentsr=majorminor version_txtmatchcuda_h cuda_versions r$_get_nvcc_versionrW`s3 I77<< >:L ww~~l## l##q!YYq\ $ nn[*..2LMQQR[]_`G"==-b1LEu:s5z) )'',,y-8K ww~~k"" k""a ">I#5;;q>*C A,??? WW\\)Y 9F ww~~f f "?J"5;;q>2 #t+lT.Ab-HHH =$## ^4    #""$    $    sJ *I7A)J 1J<&J(*J<:3J<6K' &K/K'?2K'7 J  J J%$J%( J9 3 J<<KK K$ K''K<;K<cV^8dQhRR/#)r&r'boolr)r(s"r$r)r)s77$7r#cR\PP4^,^ 8#)uWhether the current CUDA device is Blackwell (SM100+). `@assume_constant_result` makes dynamo evaluate it once at trace time and inline the bool — keeping the untraceable `torch.cuda.get_device_capability()` pybind out of the graph — so the branches on it (SF layout, cast kwargs, psum layout, and the arch guards) stay compile-safe. Re-queries each eager call, so a faked capability in tests is honoured. )torchr<get_device_capabilityrr#r$is_sm100r]s! :: + + -a 0B 66r#c V^8dQhRRRR/#)r& raise_errorrYr'r)r(s"r$r)r)sBBdBtBr#c\4'g\R\ 2VR7#\PP 4'g\RVR7#\PP 4wrVR9d\RV V R2VR7#V^ 8XdRMRp\4pVf#\RV^, RV^, R2VR7#\PP\PPVRR 44'g&\R V R V^, RV^, R 2VR7#\4pVf&\R V RV^, RV^, R2VR7#WS8dS\RV V RV^, RV^, RV^, RV^, RV RV^, RV^, R2VR7#R#)uWhether the DeepGEMM kernel can be loaded in this environment: `kernels` installed, a CUDA GPU on a supported arch (Hopper SM90 or Blackwell SM100), and a CUDA toolkit/nvcc new enough to JIT-compile it. A one-glance gate for callers — including external stacks — deciding whether to dispatch to DeepGEMM. With `raise_error=True` (used by the loader) it re-raises the specific `ImportError` explaining what's missing instead of returning `False`. `@assume_constant_result` makes dynamo evaluate this once at trace time and inline the bool (its probe is untraceable — an `lru_cache`'d import check + the `get_device_capability` pybind + nvcc lookup), so the check can sit in a compiled dispatch gate (e.g. `fp8_linear`) without breaking the graph. FP4 / Mega MoE additionally need Blackwell (SM100); the relevant forwards enforce that via `_assert_sm100_requirements` / `is_sm100()`, since this env check can't see which dtype path the caller will take. zDeepGEMM kernel unavailable: r_z9DeepGEMM kernel requires CUDA, but CUDA is not available.zJDeepGEMM requires Hopper (SM90) or Blackwell (SM100); current device is SMr?u(DeepGEMM's JIT needs a CUDA toolkit ≥ z8, but none was found. Set `CUDA_HOME` to a CUDA toolkit.binr-z:DeepGEMM's JIT compiles with nvcc, but none was found in `u,/bin`. Point `CUDA_HOME` at a full CUDA ≥ z& toolkit (not a runtime-only install).zDeepGEMM found nvcc in `u/bin` but could not read its CUDA version (no parseable `version.json`, `version.txt`, or `include/cuda.h`). Point `CUDA_HOME` at a complete CUDA ≥ z toolkit.zDeepGEMM on SMu needs a CUDA ≥ z toolkit, but nvcc z in `u.` is too old. Point `CUDA_HOME` at a CUDA ≥ T) ) rc)re) rrr r[r< is_availabler\r7r.r3rBrArW)r_rQrRmin_cudar6 nvcc_versions& r$is_deepgemm_loadablerjs ! !!$ABZA["\juvv :: " " $ $!"]kvww::335LE G!XY^X_`e_ffg h#   2+w7H I!6x{m1Xa[MR1 1#   77>>"'',,y%@ A A!H T..6qk]!HQK=Hn p#  %&L!&yk2!!)! Qx{m9 F$   !UGE7*> 15`, `>> 7`, `<< 1`) otherwise leak mantissa bits into adjacent UE8M0 byte slots and silently corrupt the SF. SM90 consumes raw fp32 SFs without going through this path. ii)viewr[int32 bitwise_and_float)rint_views& r$_ceil_to_ue8m0rZs<wwu{{#H  & 4 45E F K KEKK XXr#c(V^8dQhRRRRRRRR/#)r&rrr]rY expected_mnz int | Noner'r)r(s"r$r)r)gs)99l9d99_k9r#cVP\P8XdVeAVPR4V8d+W PR4,pVP VRR7pV'd/VP 4P \P4pMCVP4pM2VP\P8XdV'd \V4pVP4R9d\RVP4 R24hV'gVP 4#VPR4pVPR4p^VP4,pV)V,)V,pVP4^8Xd^V3M WW,^V3p\VP44V8XdV#\P !VP"WPVP$R7p V P'V4V #)uFLay out `sf` as DeepGEMM's dispatch expects, per arch. On SM100 the int-SF path only *checks* the SF (`tma_stride_check`) and never transforms it, so we hand it a TMA-aligned MN-major layout (`stride(-2) == 1`, `stride(-1) == align(mn, 16/esize)`). On SM90 DeepGEMM transforms SFA itself (`get_mn_major_tma_aligned_tensor`) and only *checks* SFB against `sm90_sfb_check`, which rejects TMA padding (`stride(-1)` must equal `size(-2)`, not `align(mn, …)`); a padded weight SF trips `layout.hpp` whenever `mn` isn't a multiple of `16/esize` (e.g. N=576 → mn=5). So on SM90 we return the raw row-major SF and let DeepGEMM lay it out. Inputs come in three flavors: - `float8_e8m0fnu` on SM100: raw UE8M0 bytes — pack 4 K-bytes → int32 (last dim /4) for the kernel's `(INT, 1, gran_k)` path. - `float8_e8m0fnu` on SM90: SM90 dispatch only accepts FP32 SFs, so cast UE8M0 → FP32 (exact upcast — UE8M0 is the biased-exponent half of a pow-of-2 FP32, so `.float()` rebuilds the original FP32 scale exactly). - `float32`: per-token / per-block SFs from `per_token_cast_to_fp8` or on-disk weights — round to UE8M0 on SM100 (see `_ceil_to_ue8m0`). - `int32`: already-packed UE8M0 — pass through. When `expected_mn` is set and the SF's M-dim is smaller (block-quantized UE8M0, e.g. DSv4-Flash compressor weights with `(N/128, K/128)` SFs), we repeat the SF on the M-axis to per-row before packing — the `(INT, 1, gran_k)` DeepGEMM kernel branch is the only UE8M0 path on SM100; for `gran_mn > 1` the kernel only handles FP32 SFs and would otherwise reject our INT SF here. dimz"DeepGEMM SF must be 2D or 3D, got Drdevice)r&rf)rr[float8_e8m0fnusizerepeat_interleave contiguousrrrrrrrH element_sizetuplestride empty_stridedshapercopy_) rr]rgran_mnmnkfalign_to aligned_mntarget_stridesouts &&& r$_coerce_sf_for_kernelrgsu8 xx5'''  "rwwr{['@!WWR[0G%%g2%6B %%ekk2BB U]] "x B  vvxv=bffhZqIJJ }} B BR__&&H3(?#h.J(*A a_BOQPZ;[N RYY[^+   bhhhhryy YCIIbM Jr#c ,V^8dQhRRRRRRRRRR /#) r&rrweight_scale_inv block_sizez tuple | Noner]rYr'dictr)r(s"r$r)r)s4// /,8/FR/^b/ /r#cVP\P8Xd RRR^ RR/#Vf \R4h\ V4pVR 9d\RV R24hVP\P 8XdV'd RRR^RR/#RRR^/#) u#Pick the `per_token_cast_to_fp8` kwargs from weight dtype + SF dtype + arch. Cases mirror the kernel's recipes: - FP4 weights (`int8`): gran_k=32 packed-UE8M0 SF. SM100+ only. - FP8 weights + UE8M0 SF on SM100: gran_k=128 packed-UE8M0 SF (DSv4). - FP8 weights + UE8M0 SF on SM90: gran_k=128 FP32 SF — the SM90 dispatch in `layout.hpp` only matches FP32 SFs, so we keep act SFs as FP32 (and float the weight SF in `_coerce_sf_for_kernel`; UE8M0 → FP32 is an exact upcast). - FP8 weights + float SF: gran_k=128 float SF (DSv3). use_ue8m0Tgran_kuse_packed_ue8m0z]DeepGEMM requires block-wise quantized FP8 weights, but the experts have no `block_size` set.u?DeepGEMM requires `block_size` ∈ {(128, 128), (1, 128)}, got r?F))r)r)rr[rrHrr)rrrr]s&&&&r$_select_fp8_cast_kwargsrs||uzz!T8R1CTJJ k  z"J//\]g\hhijkk!5!55(T8S2DdKK # ..r#c ,V^8dQhRRRRRRRRRR /#) r&expert_ids_sortedr num_expertsr alignmentuse_psum_layoutrYr'z&tuple[torch.Tensor, torch.Tensor, int]r)r(s"r$r)r)s4!?!?#!?25!?BE!?X\!?+!?r#c<VPpVP^4pVPR8XdVP4MVP 4p\ P !Wa^V^, R7P4pWr,^, V,V,pV\WQ4V^, ,,p W, p \ PPPV P^4R4p \ P!WTR7W,,p V'd!VP^4P 4p MS\ P!V 3RV\ PR7p \ P !W8VP 4R4W&WV 3#)avBuild the TMA-aligned grouped layout DeepGEMM expects. Returns `(sorted_to_padded, grouped_layout, total_padded_rows)`: - `grouped_layout` is per-row expert id (Hopper, with `-1` for padding / sentinels) or a cumsum of aligned per-expert counts (Blackwell). - EP sentinels (values == `num_experts`) are routed past the last expert block so DeepGEMM skips them. cpu)binsminmaxrrr)rr)rrtyperrr[histclongrnn functionalpadcumsumarangefullrwhere)rrrrr num_tokens histc_inputtokens_per_expertaligned_tokens_per_experttotal_padded_rowspadding_per_expertcumulative_paddingsorted_to_paddedgrouped_layouts&&&& r$!_build_deepgemm_contiguous_layoutrsR % %F"''*J06{{e/C#))+IZI^I^I`K Kqk\]o^cce"3"?!"C !QU^ ^"S%AYQR]%SS3F,,001C1J1J11MvV||J>ASAff299!<@@B%6$8"VSXS^S^_+0;;7H7VXiXmXmXoqs+t( -> >>r#c(V^8dQhRRRRRRRR/#)r&xrrrrr'r)r(s"r$r)r)s*Z]bnr#c\P!V.VPR,O5RVPRVP/pWV&V#)z;Pad a sorted tensor into the TMA-aligned contiguous layout.:rNNrr)r[emptyrrr)rrrpaddeds&&& r$_pad_for_deepgemmrsA [[* YQWWR[ Y YQRQXQX YF  Mr#c$V^8dQhRRRRRR/#)r&x_paddedrrr'r)r(s"r$r)r)s#&&\&Ua&fr&r#cW,#rr)rrs&&r$&_unpad_from_deepgemm_contiguous_layoutrs  %%r#c4V^8dQhRRRRRRRRRRRR R R /#) r& hidden_statesr top_k_index top_k_weightsrrrrrYr'rr)r(s"r$r)r)sN//// / /  /  / /r#cPVPR4pVPR4pVPR4p\P!V4wrW V,,p W,p \ WWE4wrpW8P R4pV P V^, R7V V V VV V VV3#)zSort tokens by expert id and build the M-grouped padded layout. Returns `(sorted_hidden_states_g, sample_weights_g, expert_ids_g, sentinel_mask, perm, sorted_to_padded, grouped_layout, total_padded_rows)`. )rr)rreshaper[sortr unsqueezeclamp_)rrrrrr num_top_k expert_idssample_weights expert_ids_gpermsorted_hidden_states_gsample_weights_grrr sentinel_masks&&&&&& r$_dispatch_routed_inputrs  $I$$R(J"**2.NJ/L*9+<=%+ ;\;;7&7"0;;B?MK!O,   r#c@V^8dQhRRRRRRRRRRRRR RR RR R R R/ #)r& out_paddedrsorted_weightsrrrrrr hidden_dim out_dtypez torch.dtyper'r)r(s"r$r)r)&sx___ _ _  _ # _  _____r#c \W4p WPV P4PR4,p V P VR4\ P !V4p \ P!VP^4V PR7W&W,PWVV4P^R7PV4#)uRUnpad → weighted multiply → mask sentinels → restore order → top-k reduce.grrr) rtorr masked_fill_r[ empty_likerrrrsum) rrrrrrrrrrweightedinv_perms &&&&&&&&& r$_combine_routed_outputr&s 1 NC&&syy1;;B??H --%H\\$))A,szzBHN   " ":* E I Ia I P S ST] ^^r# output_dtypezv5.16)r=c8V^8dQhRRRRRRRRRRR R R RR R/#) r&inputrrrbiasztorch.Tensor | Nonerr9rztorch.dtype | Noneactivation_scaler'r)r(s"r$r)r)AsX)) ) )#)  ) ' ) % )*))r#c Ve \R4hVP\P\P39d\ RVP 24h\ W4\4p\WV\44pVPR VPR ,4p VP!V 3/VBwr\P!V P^,VP^,VPVPR7p VPR4'd ^^VR,3MRp VP!V \#V \4V P%^4R73V\#V\4VP%^4R73V V R7V PVPRR VP^,3,4p VeV P'V4V #) uEnd-to-end DeepGEMM linear: per-token activation quant + FP8/FP4 matmul. Static (per-tensor) activation quantization is rejected — DeepGEMM needs per-row SFs. Callers should route static activations through the Triton fallback. Nz@DeepGEMM linear does not support static activation quantization.z7DeepGEMM linear requires FP16 or BF16 activations, got rrrr)reciper)rrr[bfloat16float16rHrrrr]rrrrrr0rrradd_)rrrr rrr deepgemm cast_kwargsinput_2d qinput_2dscale_2doutput sf_recipes&&&&&&& r$deepgemm_fp8_fp4_linearr@s#!"dee {{5>>5==99RSXS^S^R_`aav8#%H)&JPXPZ[Kzz"ekk"o.H"88Q[QI [[+V\\!_U\\Y^YdYd eF2=AS1T1TA{8,-Z^I  )(HJINN[\L]^_ &'7QWQ\Q\]^Q_`a  [[Sb)V\\!_,>> ?F  D Mr#c ,V^8dQhRRRRRRRRRR/#r&selftorch.nn.Modulerrrrr'r)r(s"r$r)r)ms:>> >>> >  >r#c VP\P8wd\RVP 24h\ 4pVP 'd VP M VPpVPpVPR4pVP^4pVPR4p \WW0PVP\44wp p p p ppppVP'd VPM VP pVP"pVP$'d+VP'd VP&M VP(MRpVP$'d VP*MRpVP 'dVP,R,MVP,^,p\/WV4p\P0!VVWaPR7pV!VVVV\4R7VP$'dVP3^VVV ,4VP'dVP5V4MVP7V4p\P0!VWVPR7pV!VVVV\4R7VP$'dVP3^VVV ,4\9VV V VVVVV VP4 #);DeepGEMM experts path requires bfloat16 hidden states, got Nr)rr)rr[rrHr is_transposedrrrrrrrr]has_gate gate_up_projup_proj down_projhas_biasgate_up_proj_bias up_proj_biasdown_proj_biasrrr index_add_ _apply_gateact_fnr)rrrrrgrouped_bf16_matmulrrrr sorted_hiddenrrrrrrr weight_up weight_downup_bias down_bias up_out_dimactproj_outrs&&&& r$deepgemm_bf16_experts_forwardr4msW enn,VWdWjWjVklmm#%H=A=O=O=O(99U]UtUt  ! !F  $I##A&J##B'J M3C3CXEYEY[c[e    &*]]]!! I..KRVR_R_R_t%%D!VV P@^,V\PR7pV!V\CV\14VR73V \CV \14V P!R 4R73VVV\14R 7VP"'dVPEV4MVPGV4pVP:!V3/VBwpp\P>!VW\PR7pV!V\CV\14VR73V \CV \14V P!R 4R73VVV\14R 7\IVVVVVVVV VP4 #) zDeepGEMM experts selected on a model spanning multiple CUDA devices in one process; its kernels are bound to a single CUDA context and corrupt across devices. Use `experts_implementation='grouped_mm'`, or run one device per process (TP/EP).staticzJDeepGEMM experts dispatch does not support static activation quantization.rrrNrr )r rrr)%_deepgemm_disabled RuntimeErrorrr#down_proj_scale_invractivation_schemerrr[rrHrrrrrr r!r"gate_up_proj_scale_invup_proj_scale_invrrr]rrrr0rrrrrr)r*r)rrrrrgrouped_fp8_fp4_matmulrrrrr-weight_scale_upr.weight_scale_downrr,r _expert_ids_grrrrrract_fp8 act_scalesr3proj_fp8 proj_scalesrs&&&& r$ deepgemm_fp8_fp4_experts_forwardrFs.   \  t~~t/G/GH#%H )!"nooenn,VWdWjWjVklmm.2.@.@.@**hFhFh ! !F  $I##A&J##B'J%)]]]!! I59]]]d11H^H^O..K00))dooW_WabK M3C3CXEYEY[c[e   2=AS1T1TA{8,-Z^I#88V+VGZ)9;LMG":/?ARSJ{{,iooa.@W\WeWefH ' HJL]^_ )/8:S\SaSabdSefg .2]]]t) H@UH%::8S{SHk ++'%.. YC (hjN_`a +,=xzWbWgWghjWklm   "    r#c V^8dQhRRRR/#)r&modulerr'rmr)r(s"r$r)r) s7R7R/7Rd7Rr#c\4pVPPpVPPpVPPP \ P4P4pVPPP \ P4P4pVPpVPpVPpV^ ,^8wgV^ ,^8wd\RV RV R24hVPVP4^V,VRVR7p VPVP4VVRVR7p VP!WI3WZ34wwrwr\ P"P%V RR7Vn\ P"P%V RR7Vn\ P"P%V RR7Vn \ P"P%V RR7VnR#) uOne-shot pack + permute of an FP8Experts module's L1/L2 weights into the Mega MoE UTCCP layout. Called lazily on the first megamoe forward; idempotent via the caller's ``_megamoe_transformed`` flag. Steps: 1. Cast UE8M0 SF → FP32 and call ``transform_sf_into_required_layout`` → packed int32 in MN-major TMA-aligned layout. 2. Run ``transform_weights_for_mega_moe``: interleaves gate/up on L1 and transposes both SFs for UTCCP. 3. Overwrite the loader-side parameters in place; the interleave preserves the ``[E_local, 2*I, *]`` leading dims so downstream ``.size(...)`` reads stay valid. Unwraps any ``DTensor`` wrappers FSDP2/EP may have placed around the loader- side Parameters — the kernel takes raw pointers. zwDeepGEMM Mega MoE requires `hidden_dim` and `intermediate_hidden` divisible by 32 (FP8 SF granularity); got hidden_dim=z, intermediate_hidden=r?)r  num_groupsF) requires_gradN)r )rr<datar:r!rr[rrr#intermediate_dimrrrHrrrr Parameter) rHrgate_up_sf_raw down_sf_raw gate_up_wdown_wintermediate_hiddennum_local_expertsr gate_up_sfdown_sfgate_updowns & r$setup_megamoe_weightsrZ s"$%H2277N,,11K##((--ejj9DDFI    " " ' ' 3 > > @F 11**""JB!2R71<44>PTU  44 ## $$r#)Fr)NNNN)7__conditional_annotations__r __future__r functoolsrDr.rJr1collections.abcr dataclassesrr[utilsrutils.deprecationrutils.import_utilsrrr hub_kernelsr r get_loggerrloggerrcacher7rW_dynamoassume_constant_resultr]rjrkr!allow_in_graphr}rrrrrrrrrrrr4rFrZrl)rms@r$r|s # $! / D   H %  $, $ ++\%%7&7%%B&BP" ?!DDN! H Y9x/>!?H&/d_41)2)X>BZz7RtV%V%r#