Integrate fused Mixtral MoE with Marlin kernels

[ElizaWszola]

This PR's functionality has been implemented through PRs #8217 and #8032. I'm closing it.

---

Reimplement quantized Mixtral to combine Marlin kernels with fused MoE.

This PR rewrites the Mixtral model to run a modified Marlin kernel that takes advantage of fused_moe functionality.

The C++ code takes in all expert data and topk_ids tensor. It runs a kernel to compute sorted_ids offsets related to each expert, and then feeds them to the Marlin kernels. The Marlin kernels are run multiple times per each expert, using current expert number to figure out the current position inside sorted_ids and the number of tokens to process in each particular call. The values of sorted_ids are then used to indirectly access the rows of input/output A/C tensors. If the the rows of input A are identical for each of topk experts that access them (first MMM of fused MoE), tensor A consists of M x K elements, with each row being accessed topk times by the relevant experts. Otherwise (second MMM of fused MoE), A consists of M x topk x K elements, with each row being accessed once.

Unit testing:

pytest tests/kernels/test_moe.py -k test_fused_marlin_moe

End-to-end testing:
Run offline_inference.py with

// 4-bit quantized moe without act order
llm = LLM(model="TheBloke/Mixtral-8x7B-v0.1-GPTQ")

// 4-bit quantized moe with act order
llm = LLM(model="TheBloke/Mixtral-8x7B-v0.1-GPTQ",
          revision="gptq-4bit-128g-actorder_True")

// 8-bit quantized moe with act order
llm = LLM(model="TheBloke/Mixtral-8x7B-v0.1-GPTQ",
          revision="gptq-8bit-128g-actorder_True")

Sonnet benchmark results (no act order, 4-bit):

TTFT	TPOT

Sonnet benchmark results (with act order, 8-bit):

TTFT	TPOT

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which consists a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of default ones by unblocking the steps in your fast-check build on Buildkite UI.

Once the PR is approved and ready to go, please make sure to run full CI as it is required to merge (or just use auto-merge).

To run full CI, you can do one of these:

• Comment /ready on the PR
• Add ready label to the PR
• Enable auto-merge.

🚀

[alexm-neuralmagic]

@robertgshaw2-neuralmagic @mgoin @tlrmchlsmth

[ElizaWszola]

/unready

[ElizaWszola]

@dsikka I've added some compressed_tensors compatibility code (tested only offline inference with W4A16)

[dsikka]

We'd want to use the weight loading functionality already present.

[dsikka]

we'd want mixtral_quant by default

[dsikka]

Can we leverage what already exists?

[ElizaWszola]

Closing this PR because its entire functionality has been implemented through PRs #8217 and #8032.