[Feature]: Support Mixtral-8x22B-v0.1

[yh-yao]

🚀 The feature, motivation and pitch

Do we support running Mixtral-8x22B-v0.1 now? It takes very long for compiling on my side.
https://huggingface.co/mistral-community/Mixtral-8x22B-v0.1

Alternatives

No response

Additional context

No response

[simon-mo]

vLLM should be able to support it as-is without modification. I have heard report people successfully running it. The out of the box performance should be ok but there are rooms for improvement on tuning the MoE kernels. cc @pcmoritz @richardliaw if your team have bandwidth to help tune.

[ywang96]

Can confirm https://huggingface.co/mistral-community/Mixtral-8x22B-v0.1 already works on latest PyPI release.

I've created #4002 to add the configs for the moe kernels - feel free to test them out.

[yh-yao]

It is working now. vLLM is so good, although it requires more GPU memory than the normal one. Thank you very much for the help!!!

[chenliverantos]

Hi guys,

For the mixtral X8 models, do you know if the VLLM implementation supports the full sparse matmul optimization designed for MoE models, like the original white paper:
https://arxiv.org/pdf/2401.04088
has specified. More specifically, using the recommended MEGABLOCKS sparse matrix computation described here:
https://arxiv.org/pdf/2211.15841
in the form of Blocked Compressed Sparse Row method ?

Does the VLLM implementation have everything enabled for sparse matmul optimization, or would it need additional infrastructure steps to enable it, or would it not be possible to work with it in the exact manner that MEGABLOCKS parallel optimization has specified?

We have not been able to find sufficiently specific documentation anywhere in VLLM to make this clear. Would appreciate any information you guys can furnish.

Best