Co-authored-by: Zhuohan Li <[email protected]>

Co-authored-by: zhaoyang <[email protected]>
Co-authored-by: Zhuohan Li <[email protected]>

Co-authored-by: zhaoyang-star <[email protected]>

Co-authored-by: roy <[email protected]>

Co-authored-by: chen shen <[email protected]>

…e-ray` (vllm-project#2664)

* fix: engine-useray complain

* fix: typo

…uld respect prefix_len (vllm-project#2688)

Signed-off-by: Tao He <[email protected]>

…ect#2697)

Co-authored-by: Pernekhan Utemuratov <[email protected]>

…ct#2503)

Co-authored-by: Jiang Li <[email protected]>
Co-authored-by: Kunshang Ji <[email protected]>

…ct#2696)

…oject#2741)

Co-authored-by: shujunhua1 <[email protected]>

Co-authored-by: yangrui1 <[email protected]>

… of the model. (vllm-project#2489)

… kv-cache support (vllm-project#2790)

…ntion (vllm-project#2768)

Co-authored-by: Chunan Zeng <[email protected]>

…ct#2851)

This reverts commit 5c976a7.

…lm-project#2854)

Co-authored-by: Roy <[email protected]>

* add mixtral lora support

* formatting

* fix incorrectly ported logic

* polish tests

* minor fixes and refactoring

* minor fixes

* formatting

* rename and remove redundant logic

* refactoring

* refactoring

* minor fix

* minor refactoring

* fix code smell

Co-authored-by: Roy <[email protected]>

…lm-project#2863)

…vllm-project#2880)

* Fix AttributeError: MixtralModel object has no attribute org_vocab_size.

* Make LoRA logic for Mistral and Mixtral the same

---------

Co-authored-by: Pernekhan Utemuratov <[email protected]>

If the SamplingParams object passed to LLMEngine.add_request() is mutated after it returns, it could affect the async sampling process for that request.

Suggested by @Yard1 vllm-project#2514 (comment)

how to serve the loras (mimicking the [multilora inference example](https://github.com/vllm-project/vllm/blob/main/examples/multilora_inference.py)):
```terminal
$ export LORA_PATH=~/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/
$ python -m vllm.entrypoints.api_server \
 --model meta-llama/Llama-2-7b-hf \
 --enable-lora \
 --lora-modules sql-lora=$LORA_PATH sql-lora2=$LORA_PATH
```
the above server will list 3 separate values if the user queries `/models`: one for the base served model, and one each for the specified lora modules. in this case sql-lora and sql-lora2 point to the same underlying lora, but this need not be the case. lora config values take the same values they do in EngineArgs

no work has been done here to scope client permissions to specific models

…2892)

…ner (vllm-project#2905)

…2858)

This version is for more model support. Add support for Gemma models (vllm-project#2964) and OLMo models (vllm-project#2832).

magic_wand semi_structured_sparse_tensor_linear branch integrates 2:4 semi-structured sparsity into SparseTensor. This PR adds a new sparsity config for 2:4 sparsity to neuralmagic-vllm, using the SparseTensor 2:4 support.

This PR also refactors the sparse linear method into a separate file, vllm/model_executor/layers/sparsity/sparse_w16a16_linear_method.py, which supports all sparsity formats.

Summary:

Initial integration for the sparse-fused gemm. To achieve this, we need
to ensure that we compress the weight matrix only once and never
decompress it, as decompression is currently unsupported.

Before this change, using `SparseParameter(SparseTensor)` meant that in
`MergedColumnParallelLinear` and `QKVParallelLinear` every time a new
shard was loaded by the `weight_loader` (e.g., the "q" portion of
`QKVParallelLinear`), we would decompress the tensor in-order to use
narrow to update the appropriate section of the weight tensor. With this
change, `SparseParameter(SparseTensor)` is replaced with
`LazyCompressedParameter`, which allows us to operate on
`uncompressed_data` until we explicitly compress it. At that point, the
`uncompressed_data` is compressed into `compressed_data` and freed.
Currently, the detection of when to call compress is somewhat hacky. For
`QKVParallelLinear`, we compress only after inserting "q", "k", and "v"
shard ids, and for `MergedColumnParallelLinear`, we compress once we've
inserted the same number of shards as outputs (determined by
`len(output_sizes)`), which implicitly assumes one shard per output.

Moving away from `SparseParameter(SparseTensor)` means that
`SparseTensor` no longer handles dispatching to the custom ops; instead,
this is handled by `SparseW16A16LinearMethod`. I believe this is a
positive change overall. `SparseTensor` was an unnecessary extra layer
of abstraction/indirection originally designed for the SLoRA work, not
vLLM.

This did result in the 2:4 sparse implementation breaking. However, it
turns out it was already broken (i.e., it was decompressing and running
dense within `SparseTensor`), so we "disable" it for now ("disable"
meaning decompress and run dense instead).

We should revisit all of this infrastructure post-MVP.

---------

Co-authored-by: Andrew Feldman <[email protected]>

SUMMARY:
- Fix bug whereby 2:4 is not being invoked
- Eschew SparseTensor based implementation

TESTING:
- examples/offline_inference_semi_structured_sparse.py

---------

Co-authored-by: Lucas Wilkinson <[email protected]>

SUMMARY
* add callable seed workflow for initial boundary testing

Co-authored-by: marcella-found <[email protected]>