This repository has been archived by the owner on Oct 11, 2024. It is now read-only.
Semi-structured 2:4 sparsity via SparseSemiStructuredTensor #4
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…anch safe_expose_semi_structured_sparse_tensor
afeldman-nm
requested review from
mgoin,
LucasWilkinson and
alexm-neuralmagic
mgoin
reviewed
Feb 8, 2024
vllm/model_executor/layers/sparsity/semi_structured_sparse_w16a16.py
Outdated
Show resolved
Hide resolved
vllm/model_executor/layers/sparsity/semi_structured_sparse_w16a16.py
Outdated
Show resolved
Hide resolved
vllm/model_executor/layers/sparsity/sparse_w16a16_linear_method.py
Outdated
Show resolved
Hide resolved
|
We shouldn't add |
Yes I believe this is how I have it set up right now, the user does manual install |
|
The most recent commit adds bfloat16 as a supported dtype for vLLM 2:4 sparsity profile. Note #1: no changes to magic_wand were required for bfloat16 + 2:4 support: turns out I parameterized the magic_wand 2:4 implementation by dtype. So bfloat16 works automatically (and I can see that inference produces a slightly different result for bfloat16 than float16, which makes sense; the token probabilities are probably being tweaked due to differences in rounding errors between the two datatypes.) Note #2: the BE sparsity profile does not list bfloat16 as a supported dtype; I will not investigate bfloat16 support in BE in this PR owing to the potential complexity involved in i.e. making the bitmask expand process generalize to different dtypes beyond float16. @LucasWilkinson could potentially investigate this if it is a priority. |
| if bias is not None: | ||
| output = F.linear(x, sparse_weight, bias) | ||
| else: | ||
| output = F.linear(x, sparse_weight, bias=None) |
There was a problem hiding this comment.
we do we need this if? if bias is None, why not just pass bias?
mgoin
reviewed
Feb 12, 2024
vllm/model_executor/layers/sparsity/semi_structured_sparse_w16a16.py
Outdated
Show resolved
Hide resolved
tlrmchlsmth
pushed a commit
that referenced
this pull request
Feb 13, 2024
magic_wand semi_structured_sparse_tensor_linear branch integrates 2:4 semi-structured sparsity into SparseTensor. This PR adds a new sparsity config for 2:4 sparsity to neuralmagic-vllm, using the SparseTensor 2:4 support. This PR also refactors the sparse linear method into a separate file, vllm/model_executor/layers/sparsity/sparse_w16a16_linear_method.py, which supports all sparsity formats.
robertgshaw2-neuralmagic
pushed a commit
that referenced
this pull request
Feb 20, 2024
magic_wand semi_structured_sparse_tensor_linear branch integrates 2:4 semi-structured sparsity into SparseTensor. This PR adds a new sparsity config for 2:4 sparsity to neuralmagic-vllm, using the SparseTensor 2:4 support. This PR also refactors the sparse linear method into a separate file, vllm/model_executor/layers/sparsity/sparse_w16a16_linear_method.py, which supports all sparsity formats.
robertgshaw2-neuralmagic
pushed a commit
that referenced
this pull request
Feb 20, 2024
magic_wand semi_structured_sparse_tensor_linear branch integrates 2:4 semi-structured sparsity into SparseTensor. This PR adds a new sparsity config for 2:4 sparsity to neuralmagic-vllm, using the SparseTensor 2:4 support. This PR also refactors the sparse linear method into a separate file, vllm/model_executor/layers/sparsity/sparse_w16a16_linear_method.py, which supports all sparsity formats.
robertgshaw2-neuralmagic
pushed a commit
that referenced
this pull request
Feb 21, 2024
magic_wand semi_structured_sparse_tensor_linear branch integrates 2:4 semi-structured sparsity into SparseTensor. This PR adds a new sparsity config for 2:4 sparsity to neuralmagic-vllm, using the SparseTensor 2:4 support. This PR also refactors the sparse linear method into a separate file, vllm/model_executor/layers/sparsity/sparse_w16a16_linear_method.py, which supports all sparsity formats.
tlrmchlsmth
pushed a commit
that referenced
this pull request
Feb 21, 2024
magic_wand semi_structured_sparse_tensor_linear branch integrates 2:4 semi-structured sparsity into SparseTensor. This PR adds a new sparsity config for 2:4 sparsity to neuralmagic-vllm, using the SparseTensor 2:4 support. This PR also refactors the sparse linear method into a separate file, vllm/model_executor/layers/sparsity/sparse_w16a16_linear_method.py, which supports all sparsity formats.
robertgshaw2-neuralmagic
pushed a commit
that referenced
this pull request
Feb 21, 2024
magic_wand semi_structured_sparse_tensor_linear branch integrates 2:4 semi-structured sparsity into SparseTensor. This PR adds a new sparsity config for 2:4 sparsity to neuralmagic-vllm, using the SparseTensor 2:4 support. This PR also refactors the sparse linear method into a separate file, vllm/model_executor/layers/sparsity/sparse_w16a16_linear_method.py, which supports all sparsity formats.
robertgshaw2-neuralmagic
pushed a commit
that referenced
this pull request
Feb 22, 2024
magic_wand semi_structured_sparse_tensor_linear branch integrates 2:4 semi-structured sparsity into SparseTensor. This PR adds a new sparsity config for 2:4 sparsity to neuralmagic-vllm, using the SparseTensor 2:4 support. This PR also refactors the sparse linear method into a separate file, vllm/model_executor/layers/sparsity/sparse_w16a16_linear_method.py, which supports all sparsity formats.
robertgshaw2-neuralmagic
pushed a commit
that referenced
this pull request
Feb 22, 2024
magic_wand semi_structured_sparse_tensor_linear branch integrates 2:4 semi-structured sparsity into SparseTensor. This PR adds a new sparsity config for 2:4 sparsity to neuralmagic-vllm, using the SparseTensor 2:4 support. This PR also refactors the sparse linear method into a separate file, vllm/model_executor/layers/sparsity/sparse_w16a16_linear_method.py, which supports all sparsity formats.
Closed
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
magic_wand semi_structured_sparse_tensor_linear branch integrates 2:4 semi-structured sparsity into SparseTensor. This PR adds a new sparsity config for 2:4 sparsity to neuralmagic-vllm, using the SparseTensor 2:4 support.
This PR also refactors the sparse linear method into a separate file,
vllm/model_executor/layers/sparsity/sparse_w16a16_linear_method.py, which supports all sparsity formats.