SqueezeLLM Support

[chooper1]

This PR adds support for the SqueezeLLM quantization method, which is described in the following preprint: https://arxiv.org/abs/2306.07629, and which has open-source GPU inference code and quantization code available at: https://github.com/SqueezeAILab/SqueezeLLM. SqueezeLLM is a post-training quantization framework that allows for high-accuracy and runtime-efficient quantization at low bit precision. SqueezeLLM leverages non-uniform quantization to better represent the underlying distribution by shifting the quantization signposts to the optimal positions. This PR contains the kernels and quantization configurations files in order to run the 4-bit dense-only non-uniform quantization scheme outlined in the preprint.

[WoosukKwon]

Hi @chooper1, thanks for submitting the PR! Before getting into review, could you check the code format? Please run the following and upstream the changes:

pip install -r requirements-dev.txt
./format.sh

[casper-hansen]

This is super interesting work, especially after the release of the quantization code to produce newly quantized models.

I am curious if Woosuk or the author could run benchmarks/benchmark_throughput.py to check the thoughput of FP16 versus SqueezeLLM?

EDIT: I am getting very low tokens/s at low batch sizes at around 14.35 tokens/s. Is this the expected performance?

[WoosukKwon]

@chooper1 Sorry for the late review. The PR looks good to me! I've updated it with the latest main branch and modified QuantizationConfig as I found that the original interface of QuantizationConfig was overfitted to AWQ. Thanks again for the great work!

As the next step, I hope we can see more SqueezeLLM models, especially Mistral and Falcon. Also, please consider optimizing the matmulu kernel.

[gesanqiu]

Hi @casper-hansen, It seems there are some issues in SqueezeLLM-gradients? Have you produced SqueezeLLM-gradients for Llama-2-13B, I modified the _model.set_devices() to _model.cuda() and _model.num_linear_layers to 40, and met OOM problem even I set 2 A40(48GB) devices.