[Usage]: How to decrease VRAM Usage when specifying FP8 arguments

[AnyISalIn]

Your current environment

Thank you for the teamwork on FP8 quantization. However, I am currently loading FP16 using the arguments --quantization fp8 --kv-cache-dtype fp8, and it appears that the VRAM usage is the same as with FP16. How can we decrease VRAM usage when specifying FP8 arguments?

How would you like to use vllm

I believe that FP8 should require less than 50% of the VRAM usage compared to FP16.

[pcmoritz]

Currently with #4118, the weights are loaded in FP16 and then converted to FP8. So you will still need enough VRAM to fit the FP16 weights.

This will change soon once we support FP8 checkpoints :)

[AnyISalIn]

Currently with #4118, the weights are loaded in FP16 and then converted to FP8. So you will still need enough VRAM to fit the FP16 weights.

This will change soon once we support FP8 checkpoints :)

Thanks for your reply!