+ j2^RIt^RIHt^RIHt^RIHt^RIt^RIH t H t ^RI H t H t HtHtHtHtHtHtHtHtHtHtHt.ROt]!R R 4t!R R]P64tR ]P:3RRllt!RR]4t!RR ]4t R s!Rt"R s#Rt$R#)N) namedtuple)CallableAny))sparse_semi_structured_from_dense_cutlass'sparse_semi_structured_to_dense_cutlass) fallback_dispatchersemi_sparse_addmmsemi_sparse_clonesemi_sparse_detachsemi_sparse_indicessemi_sparse_linearsemi_sparse_mmsemi_sparse_scaled_mm semi_sparse_tsemi_sparse_tosemi_sparse_to_copysemi_sparse_valuessemi_sparse_viewSparseSemiStructuredTensor!SparseSemiStructuredTensorCUTLASS$SparseSemiStructuredTensorCUSPARSELT_SEMI_STRUCTURED_SPARSE_CONFIGz=sparse_min_rows sparse_min_cols dense_min_rows dense_min_colscna]tRt^*toRt^tRtRtRt.ROt ] RV3RlRll4t V3RlRlt V3RlRlt ]V3R lR l4t]P"P$t]V3R lR l4t]RV3RlRll4t]V3RlRl4tRt]]3V3RlRll4tRR /V3RlRlltV3RltRtVtR #)ra This class implements semi-structured sparsity as a Tensor subclass. Semi-structured sparsity describes a sparsity pattern where n in every 2n elements are sparse, depending on the datatype. It is also referred to as 2:4 sparsity or fine-grained structured sparsity. There are two backends available for semi_structured sparsity, either cuSPARSELt or CUTLASS. This class is meant to serve as a base class for both implementations. SparseSemiStructuredCUTLASS and SparseSemiStructuredCUSPARSELT both inherit from this class and define three backend-specific items. Note that as such, this class cannot be instantiated directly. -`_DTYPE_SHAPE_CONSTRAINTS` - A dictionary holding backend specific dense/sparse min shape constraints - `def from_dense()` - backend specific compression routines - `def _mm()` - backend specific mm op (either torch._cslt_sparse_mm or torch._sparse_semi_structured_(mm|addmm)) Fc<V^8dQhRS[PRS[PR,RS[PR,RS[PR,RS[PR,RS[PR,RS[R S[R S[/ #) shapepackedNmetapacked_tmeta_tcompressed_swizzled_bitmaskfuse_transpose_cusparseltalg_id_cusparselt requires_grad)torchSizeTensorboolint)format __classdict__s"]/home/luca/screen-checking/.venv/lib/python3.14/site-packages/torch/sparse/semi_structured.py __annotate__'SparseSemiStructuredTensor.__annotate__PsRRzzR t#RllT! R ,,% R  t# R&+\\D%8R$(RRRc  VP'gT\P!R\^R7RVnVP 4\ P PV4VeTp MVeTp M \R4h\ PPVVV PV PV PV R7p W+nW;nWKnW[nWknW{nWnV #)a Create a new instance of the tensor subclass from the compressed sparse representation. We have the option to create the subclass with the compressed representations of both X and X', for training. For inference, we only need a single representation (either X or X'), while the corresponding other set will be None. Depending on the backend selected, certain fields will be set to None. (CUSPARSELT vs CUTLASS) Args: shape: The shape of the original dense tensor packed: The compressed representation of the original dense tensor meta: The metadata of the original dense tensor, if it is stored separately packed_t: The compressed representation of the transposed original dense tensor meta_t: The metadata of the transposed original dense tensor, if it is stored separately compressed_swizzled_bitmask: The masks used by the CUTLASS backend to determine which threads should participate in the computation. Used for pointwise ops. fuse_transpose_cusparselt: When running with cuSPARSELt, we have the option to fuse a transposition with a matmul, which is useful in the case of 2:4 sparse training. alg_id_cusparselt: The algorithm id to use when using cuSPARSELT, will have an effect on performance Returns: torch.Tensor: A torch.Tensor wrapper subclass. Raises: ValueError: If all of the tensor arguments are None. zThe PyTorch API of SparseSemiStructuredTensor is in prototype stage and will change in the near future. Please open a Github issue for features requests and see our documentation on the torch.sparse module for further information about the project. stacklevelTz3At least one of packed or packed_t must be provided)devicedtypelayoutr%)_PROTOTYPE_WARNING_SHOWNwarningswarn UserWarning_load_dispatch_tabler&_dynamoallow_in_graph ValueErrorr(_make_wrapper_subclassr4r5r6rrr r!r"r#r$) clsrrrr r!r"r#r$r%previous_tensortensors &&&&&&&&&& r-__new__"SparseSemiStructuredTensor.__new__OsN+++ MMH  ,0C (  $ $ & MM ( ( -  $O  !&ORS S44  "))!''"))' 5   " -H*+D(#4  r0c <V^8dQhRS[/#rreturn)str)r+r,s"r-r.r/s@@#@r0c\VR4'g \R4hVPP RVP R2#)rztensor has no shape attributez(shape=))hasattrAssertionError __class____name__r)selfs&r-__repr__#SparseSemiStructuredTensor.__repr__s=tW%% !@A A..))*'$**Q??r0cx<V^8dQhRS[S[S[,S[S[PS[S[S[3,3,/#rF)tuplelistrHr&r'r)r*)r+r,s"r-r.r/s5 * * tCy% D#t ;<< = *r0ca\\V3RlSP44pSPSPSP SP 3pW3#)c <\SV4RJ#N)getattr)xrOs&r-?SparseSemiStructuredTensor.__tensor_flatten__..sWT1-T9r0)rTfilter __slots__rr#r$r%)rO inner_tensors tensor_metasf r-__tensor_flatten__-SparseSemiStructuredTensor.__tensor_flatten__sT 94>> J  JJ  * *  " "     ))r0cl<V^8dQhRS[S[PS[S[S[3,RS[P /#)rr_rG)rSr&r'r)r*r()r+r,s"r-r.r/s7  5::tS$67   r0c VwrVrxV!VVPRR4VPRR4VPRR4VPRR4VPRR4VVVR7 #)rNrr r!r" rrrr r!r"r#r$r%)get) r@r^r_ outer_size outer_striderr#r$r%s &&&&& r-__tensor_unflatten__/SparseSemiStructuredTensor.__tensor_unflatten__sNYJ*; $$Xt4""640"&&z48 $$Xt4(5(9(9-t)'@/'  r0c <V^8dQhRS[/#rFr)r+r,s"r-r.r/sTTcTr0cVPVP9d&\VP RVP R24hVPVP,!WW44#)zI only supports a specific set of operations, can't perform requested op (rJ)_overloadpacketSPARSE_DISPATCHNotImplementedErrorrN)r@functypesargskwargss&&&&&r-__torch_dispatch__-SparseSemiStructuredTensor.__torch_dispatch__sb   s':': :%<<.!//3}}oQ@ ""4#7#78dSSr0Nc<V^8dQhRR/#)rrGN)r+r,s"r-r.r/sBBBr0c *\VRR4Ef\PPP\ \PPP \\PPP\\PPP\\PPP\\PPP\\PPP\ \PPP"\$\PPP&\$\PPP(\*\PPP,\.\PPP0\2\PPP4\6\PPP8\:\PPP<\>/Vn VeVP@PCV4R#R#R#)zD Loads the op overload sparse dispatch table for the current class. rmN)"rXr&opsatenvaluesrindicesr is_same_sizer detach_detachr trviewrmmrmatmuladdmmr linearr_to_copyr _scaled_mmrcloner torrmupdate)r@custom_dispatch_tables&&r-r;/SparseSemiStructuredTensor._load_dispatch_tablesm 3)4 0 8 %%'9 &&(; ++-@ &&(; %%'9   - ##%5 !!> %%~ $$&7 %%'9 '')< ))+@ $$&7 !!>#C "%0##**+@A1% 9r0c8<V^8dQhRS[PRR/#)roriginal_tensorrGNr&r()r+r,s"r-r.r/s ((u||(PT(r0c  VP'g\RVP R24hVP4^8wd\RVP4 R24hVP 4'g \R4hVP VP 9d\RVP RV R24hVPwr#VP VP ,PpVP VP ,PpW$8g#W$,'gW58gW5,'d \R VP R V R V R 24hR #)zO Assert that the given tensor is valid for semi-structured sparse compression. zError original_tensor.device= z= is not supported! Only CUDA tensors are currently supported.zError original_tensor.dim = z; is not supported! Only 2d tensors are currently supported.zXError original_tensor is not contiguous!Only contiguous tensors are currently supported.zError original_tensor.dtype z is not a supported dtype for !zError original_tensor.shape zV is not supported! Both dimensions must be larger than or equal to and a multiple of (z, rJN) is_cuda RuntimeErrorr4dim is_contiguousr5_DTYPE_SHAPE_CONSTRAINTSrsparse_min_rowssparse_min_cols)r@rmnmin_rowsmin_colss&& r- _validate_device_dim_dtype_shape;SparseSemiStructuredTensor._validate_device_dim_dtype_shapesr &&&01G1G0HI==     A %./B/B/D.EF;;  ,,..C   (D(D D./D/D.EEcdgchhij  $$//0E0EFVV//0E0EFVV <1<<1<1<<./D/D.EFVV^U__abjakkln <V^8dQhRS[PRS[RR/#)rralg_idrGrr&r(r*)r+r,s"r-r.r/$s-"""" & "r0c\hrWrnr@rrs&&&r- from_dense%SparseSemiStructuredTensor.from_dense#s "!r0biascv<V^8dQhRS[PRS[PR,RS[P/#)rBrNrGr)r+r,s"r-r.r/+s:"" <<"llT! "  "r0c \hrWr)rOrrrrs&&$,r-_mmSparseSemiStructuredTensor._mm+s "!r0c<V^8dQh/S[;R&S[S[PS[3,;R&S[;R&S[;R&S[;R&S[;R&S[S[S[3,;R&S[PR,;R &S[PR,;R &S[PR,;R &S[PR,;R &S[PR,;R &S[;R&S[;R&#)r_DEFAULT_ALG_IDr_FORCE_CUTLASS_FUSE_TRANSPOSEr7BACKENDrmNrrr r!r"r#r$) r*dictr&r5rr)rHrr()r+r,s"r-r.r/*s$%&#5;;0N#NOO'( )*!+,#*-0L12(H,--36 LL4 78 ,, 9:llT!!;< LL4 =>"'!44?@ $#ABCr0rv)rrr r!r")FrFrW)rN __module__ __qualname____firstlineno____doc__rrrr7r] staticmethodrCrPr` classmethodrhr&_C_disabled_torch_function_impl__torch_function__rsr;rrrr__annotate_func____static_attributes____classdictcell__r,s@r-rr*s"O N!O%*XIRRRh@@ * *  .??TTBBB2((TT&""""%) ""Cr0Fc\V^8dQhR\PR\R\R\/#)rr transposedrrG)r&r(r)r*r)r+s"r-r.r.5s@DFDF\\DFDF DF DFr0cV'd\P!R\^R7\P'd\ P PM\ P PpVPWR7#)a This function converts a dense tensor into a sparse semi-structured tensor. It will return a SparseSemiStructuredTensor, a subclass of torch.Tensor. This function will check to ensure the dense tensor has the right dtype, size, dims, and device. We currently only support semi-structured sparse tensors for 2d CUDA tensors. Additionally, your tensor must be a positive multiple of the minimum sparse block size, given in `_DTYPE_TO_SHAPE_CONSTRAINTS` for each dtype (float32, float16, bfloat16, int8). Args: original_tensor (Tensor): the dense tensor to convert transposed (bool, optional): deprecated arg to be removed in another release. Do not use. alg_id (int, optional): the algorithm id to use for cuSPARSELt matmul. Defaults to 0. Can be obtained via ``torch._cslt_sparse_mm_search``. Returns: SparseSemiStructuredTensor: A sparse semi-structured tensor created from the given original_tensor Raises: None Example: >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_CUDA) >>> A = torch.Tensor([0, 0, 1, 1]).tile((128, 32)).half().cuda() tensor([[0., 0., 1., ..., 0., 1., 1.], [0., 0., 1., ..., 0., 1., 1.], [0., 0., 1., ..., 0., 1., 1.], ..., [0., 0., 1., ..., 0., 1., 1.], [0., 0., 1., ..., 0., 1., 1.], [0., 0., 1., ..., 0., 1., 1.]], device='cuda:0', dtype=torch.float16) >>> A_sparse = to_sparse_semi_structured(A) SparseSemiStructuredTensor(shape=torch.Size([128, 128])) >>> A_sparse.values() tensor([[1., 1., 1., ..., 1., 1., 1.], [1., 1., 1., ..., 1., 1., 1.], [1., 1., 1., ..., 1., 1., 1.], ..., [1., 1., 1., ..., 1., 1., 1.], [1., 1., 1., ..., 1., 1., 1.], [1., 1., 1., ..., 1., 1., 1.]], device='cuda:0', dtype=torch.float16), >>> A_sparse.indices() tensor([[-4370, -4370, -4370, ..., -4370, -4370, -4370], [-4370, -4370, -4370, ..., -4370, -4370, -4370], [-4370, -4370, -4370, ..., -4370, -4370, -4370], ..., [-4370, -4370, -4370, ..., -4370, -4370, -4370], [-4370, -4370, -4370, ..., -4370, -4370, -4370], [-4370, -4370, -4370, ..., -4370, -4370, -4370]], device='cuda:0', dtype=torch.int16)) zSetting transpose from `to_sparse_semi_structured` is deprecated and will be removed in a future release. `SparseSemiStructuredTensor` only support contiguous input tensors.r2)r) r8r9 FutureWarningrrr&sparserrr)rrrSPARSE_SUBCLASSs&&& r-to_sparse_semi_structuredr5sgh  R    & 4 4 4  66 \\ > >  % %o % EEr0c Zaa]tRtRtoRtRt]P]!^^^^4]P]!^ ^@^^4]P]!^ ^@^^4]P]!^ ^ ^^4/t ] ]P3V3RlRll4tV3Rlt] RV3RlRll4tR R R R /V3R lRlltRtVtV;t#)ri|a This class implements semi-structured sparsity for the CUTLASS backend. In this implementation, the specified elements and metadata are stored separately, in packed and meta respectively. When _FORCE_CUTLASS is set, or when cuSPARSELt is not available, this subclass calls into _sparse_semi_structured_(mm|addmm) and sparse_semi_structured_from_dense for conversion to the compressed format. cutlassc><V^8dQhRS[PRS[RR/#)rrrrGrr)r+r,s"r-r..SparseSemiStructuredTensorCUTLASS.__annotate__s-     -  r0c VPV4\V4wppV!VPVVRRRVPR7#)Nrrr r!r"r%)rrrr%)r@rrsparse_tensor_cutlassmeta_tensor_cutlasss&&& r-r,SparseSemiStructuredTensorCUTLASS.from_densesW ,,_= 6o F !   ! !($(,)77  r0c<VPeVPf \R4hVPP^8Xd!\ VPVP4#\ SV`4#)Nz meta and packed must not be None)rrrLndimrsuperr)rOrMs&r-r*SparseSemiStructuredTensorCUTLASS.to_densesc 99  3 !CD D yy~~" 4    !#  r0c8<V^8dQhRS[PRR/#rrrGrr)r+r,s"r-r.rs%K K #llK %K r0c  p\P!WRR7wpppppV!VPVVVVVRR7#)a This function takes in a unpruned dense tensor and runs a (branchless) static sort across a 4x4 tile. It greedily picks the largest values in the tile, upholding the 2:4 sparsity constraint across both rows and columns. The algorithm used to prune the matrix is implemented in `_sparse_semi_structured_tile`. Then it creates the packed and meta tensors for the compressed sparse representation of the pruned dense tensor. It also calculates the packed_t and meta_t tensors for the compressed sparse representation of the transposed pruned dense tensor. Since we cannot transpose the compressed representations, we store both for the fw/bw pass respectively. Finally, this function also computes a compressed swizzled bitmask that encodes the sparsity pattern. This can be used in the backward pass to mask the gradients. :: [9 1 7 4] [9 0 7 0] [1 2 3 0] [0 2 0 0] [8 3 5 4] -> prune 4x4 tile -> [8 0 0 4] -> pack to CUTLASS semi-structured -> packed [1 2 6 2] [0 0 6 2] -> metadata -> pack to transposed CUTLASS -> packed_t semi-structured representation -> metadata_t -> compute swizzled bitmask -> compressed_swizzled_bitmask The equivalent PyTorch code to create the same five outputs from the dense tensor can be found below:: from torch.sparse import SparseSemiStructuredTensorCUTLASS from torch.sparse._semi_structured_conversions import ( _sparse_semi_structured_tile, _compute_compressed_swizzled_bitmask, ) pruned = _sparse_semi_structured_tile(dense) packed_cutlass, meta_cutlass = sparse_semi_structured_from_dense_cutlass( pruned ) packed_t_cutlass, meta_t_cutlass = ( sparse_semi_structured_from_dense_cutlass(pruned.t().contiguous()) ) bitmask = _compute_compressed_swizzled_bitmask(pruned) SparseSemiStructuredTensorCUTLASS( dense.shape, packed_cutlass, meta_cutlass, packed_t_cutlass, meta_t_cutlass, bitmask, ) T algorithm use_cutlassFr)r&_sparse_semi_structured_tilerr@rrrrr r!r"s&&& r-prune_dense_static_sort9SparseSemiStructuredTensorCUTLASS.prune_dense_static_sortsV~  . . d      '   ! !(C  r0rNshould_transpose_denseFc|<V^8dQhRS[PRS[PR,RS[RS[P/#rrrNrrGr&r(r))r+r,s"r-r.rsD!! <<!llT! ! !% ! !r0c H\V\4'd \R4hVPPpVP ^8wgVP ^8wd\ RV R24hVPeVPf\ RV R24h\4VPVP,p\PPPVVPVPVVP ^,VP"VP$V4#)Z`SparseSemiStructuredTensor @ SparseSemiStructuredTensor` is not supported by the hardware`)` matmul: Broadcasting is not implemented$` matmul: operation is not supported) isinstancerr>rMrNrrnrr_ensure_cutlass_mm_registeredrr5r&rxsemi_structured cutlass_mmrdense_min_rowsdense_min_cols)rOrrrrrcls_name constraintss&&$$, r-r%SparseSemiStructuredTensorCUTLASS._mms a3 4 4l >>** 99>QVVq[%H:FG  ;; $))"3%H:AB  * +77@K99,,77   1 ****&  r0rv)rNrrrrrr&int8rfloat16bfloat16float32rrrrrrrrrr __classcell__)rMr,s@@r-rr|s G 22sBC 5b"aC 6r2q!D 5b"aC  1@@   *  K K K Z!%) ! (- !!!r0c Ha]tRtRtoRtRt]P]!^ ^ ^^4]P]!^ ^ ^^4]P]!^^^^4]P]!^^^^4/t ] ]P3V3RlRll4t] RV3RlRll4tRR R R /V3R lR lltRtVtR #)ri$ab The cuSPARSELt backend expects the specified elements and the metadata to be stored in a single tensor: packed = [ specified elements of original tensor | metadata ] For an original tensor of size (m, k) we expect the first m * k // 2 elements to be the kept elements The rest of the tensor is metadata. Since there is only one tensor, we only use the packed and packed_t attributes respectively. cuSPARSELt also supports transposition fusion, which is necessary for performant 2:4 sparse training, as well as specifying alg_id, a config that affects the performance of the matmul depending on matmul sizes. cusparseltc><V^8dQhRS[PRS[RR/#)rrrrGrr)r+r,s"r-r.1SparseSemiStructuredTensorCUSPARSELT.__annotate__9s-     0  r0c VPV4V!VP\P!V4RRRR\P VVP R7 #)Nrd)rrr&_cslt_compressrrr%rs&&&r-r/SparseSemiStructuredTensorCUSPARSELT.from_dense8sW ,,_=!''''8(,&@&P&P$)77  r0c8<V^8dQhRS[PRR/#rr)r+r,s"r-r.rMs!> > #ll> %> r0c  \P!WRR7wpppppVPVP^,R4pVPVP^,R4pV!VPVVVVVRR7#)a This function does the same thing as described in SparseSemiStructuredCUTLASS, but uses the cuSPARSELt metadata layout and sparse matmul. The only functional difference is that cuSPARSELt stores `metadata` and `packed` together into a single tensor. :: [9 1 7 4] [9 0 7 0] [1 2 3 0] [0 2 0 0] [8 3 5 4] -> prune 4x4 tile -> [8 0 0 4] -> pack to cuSPARSELT semi-structured -> packed [1 2 6 2] [0 0 6 2] -> pack to transposed cuSPARSELt -> packed_t semi-structured representation -> compute swizzled bitmask -> compressed_swizzled_bitmask The equivalent PyTorch code to create the same three outputs from the dense tensor can be found below:: from torch.sparse import SparseSemiStructuredTensorCUSPARSELT from torch.sparse._semi_structured_conversions import ( _sparse_semi_structured_tile, _compute_compressed_swizzled_bitmask, ) pruned = _sparse_semi_structured_tile(dense) packed_cusparselt = torch._cslt_compress(pruned) packed_t_cusparselt = torch._cslt_compress(pruned.t().contiguous()) bitmask = _compute_compressed_swizzled_bitmask(pruned) SparseSemiStructuredTensorCUSPARSELT( dense.shape, packed_cutlass, None, packed_t_cutlass, None, bitmask ) Frrr)r&rrrrs&&& r-r  ! !(C  r0rNrFc|<V^8dQhRS[PRS[PR,RS[RS[P/#rr)r+r,s"r-r.rsD44 <<4llT! 4 !% 4 4r0c `\V\4'd \R4hVP^8wgVP^8wd$\ RVP P R24hVPVP8wdj\ RVP P R\VP4 R\VP4 RVP RVP R2 4hVeVPVP8wdj\ RVP P R\VP4 R\VP4 RVP R VP R 2 4hVP\P8Xd]\ RVP P R\VP4 R\VP4 R VP R 2 4hVPf$\ RVP P R 24h\4VPVP,p\PP P#VVPVVP^,VP$VP&VVP(V4 #)rrrz` matmul: trying to do `A=z @ B=z`, with A.dtype=z and B.dtype=zH. This operation is only supported when A and B have the same data type.z + C`, with A.dtype=B.dtype=z and C.dtype=zK. This operation is only supported when A, B and C have the same data type.z`, with A.dtype=B.dtype=zO. mm is not supported for float8_e4m3fn, please use `torch._scaled_mm` instead.r)rrr>rrnrMrNr5rSrr& float8_e4m3fnr _ensure_cusparselt_mm_registeredrrxr cusparselt_mmrrr$)rOrrrrrrs&&$$, r-r(SparseSemiStructuredTensorCUSPARSELT._mmst a3 4 4l  99>QVVq[%DNN++,,UV  77djj %DNN++,,FuTZZGXFYY^_defelel_m^no $ |= BYY    djj 8%DNN++,,FuTZZGXFYY^_defelel_m^no((, |= J\\  ::,, ,%DNN++,,FuTZZGXFYY^_defelel_m^no((, |4``  ;; %DNN++,,PQ  - .77@K99,,::  1 ****&&&&  r0rvr)rNrrrrrr&r rrrrrrrrrrrrrrs@r-rr$s G ;BBK 22r2rB 5b"aC 6r2q!D  1@@   &> > > @4%) 4 (- 444r0caa\'dR#Rs^RIHpRRloV!RR R7!S4pVPRR l4p^R IHoHpVV3R lpVPW44R#) zLazily register the cutlass_mm custom op. Registration is deferred to avoid importing torch.library at module load time, since torch.sparse is imported early during ``import torch``. NT custom_opcV^8dQhR\PR\PR\PR\PR,R\R\R\R \R \P/ #) rdenserrrN out_featuresrrrrGr&r(r*r))r+s"r-r.3_ensure_cutlass_mm_registered..__annotate__sGG||G GllGllT! G  G  GG!%G Gr0cVPwrV)V,p V )V,p V ^8g;'gV ^8gp Tp V 'd/\PPP V^V ^V 34p V'dV P 4MT pVf\P !WV4pM\P!W1W.4pV 'dYV'dTMT pVRVP^^V4pV'dVP 4P4#VP4#V'dVP 4P4#T#)rN) rr&nn functionalpadr_sparse_semi_structured_mm_sparse_semi_structured_addmmnarrow contiguous)rrrrrrrrrrto_pad_mto_pad_nneed_pad dense_paddedmm_inputresout_colss&&&&&&&& r-_cutlass_mm_impl7_ensure_cutlass_mm_registered.._cutlass_mm_impls{{B(?B(?q=11HM  88..2251h8:TUL'=<>>#< <226JC55dDSC 2qHm|$++Aq(;C+A3557%%' Ws~~GW W'=suuw!!#F3Fr0zsemi_structured::cutlass_mm mutates_argscV^8dQhR\PR\PR\PR\PR,R\R\R\R \R \P/ #) rrrrrNrrrtranspose_denserGr)r+s"r-r.rs{  ||   ll llT!          r0cV'dVP^,MVP^,pV'dW3MWH3p \P!V VPVPR7#rrrr&emptyr5r4) rrrrrrrr*r$ output_shapes &&&&&&&& r-_cutlass_mm_fake7_ensure_cutlass_mm_registered.._cutlass_mm_fakesS&55;;q>%++a.(7X $l=U {{ ++<<  r0)_has_unrecognized_tensor_typesFunctionalTensorModec<S!V4'd\#V;_uu_4S!V/VBuuRRR4# +'giR#;irW)NotImplemented)mode_oprprqrrr%r2s&&&&&r-_cutlass_mm_decomp9_ensure_cutlass_mm_registered.._cutlass_mm_decomps2 )% 0 0! ! T#T4V4TTTs 3 A rv)_cutlass_mm_registered torch.libraryr register_fake#torch._subclasses.functional_tensorr2r3register_torch_dispatch)rrr0r3r8r%r2s @@r-rrsi!'G88rJJ  (5 &&';Pr0c\'dR#Rs^RIHpV!RR R7R RRll4pVPRR l4pR#) z,Lazily register the cusparselt_mm custom op.NTrzsemi_structured::cusparselt_mmr'cV^8dQhR\PR\PR\PR,R\R\R\R\R \R \R \P/ # rrrrNrrrfuse_transposerrrGr)r+s"r-r.6_ensure_cusparselt_mm_registered..__annotate__)s  ||   llT!        !%   r0c VPwrV )V,p V )V,p V ^8g;'gV ^8gp TpV 'd/\PPP V^V ^V 34pV'dVP 4MTp\P !VVVVVR7pV 'dWV'dT MT pV'd"VP^^V4P4#VP^^V4P4#V#)r)rtranspose_resultr) rr&rrrr_cslt_sparse_mmrr)rrrrrrrBrrrrrrr r!r"r#r$s&&&&&&&&& r-r 7_ensure_cusparselt_mm_registered..cusparselt_mm(s{{B(?B(?q=11HM  88..2251h8:TUL'=<>>#<##  +   2qHzz!Q1<<>>zz!Q1<<>> r0cV^8dQhR\PR\PR\PR,R\R\R\R\R \R \R \P/ #rAr)r+s"r-r.rCLs  ||   llT!         !%   r0c V'dVP^,MVP^,p V'dW3MW93p \P!V VPVPR7#r,r-) rrrrrrrBrrr$r/s &&&&&&&&& r-_cusparselt_mm_fake=_ensure_cusparselt_mm_registered.._cusparselt_mm_fakeKsS&<5;;q>Q(6X $\rSs"$ " ",$C" H"H"Z,<<DFNe(BeP]+E]@RQj"@ r0