Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use HuggingFace's Quanto library KV Cache Quantization for any Transformers-based loader #6126

Closed
Interpause opened this issue Jun 15, 2024 · 2 comments
Closed

Use HuggingFace's Quanto library KV Cache Quantization for any Transformers-based loader #6126

Interpause opened this issue Jun 15, 2024 · 2 comments
Labels
enhancement New feature or request stale

Comments

@Interpause
Copy link

Description

HuggingFace's Quanto has implemented 4 bit & 2 bit KV cache quantization compatible with Transformers. See: https://huggingface.co/blog/kv-cache-quantization

I may PR when I've time to experiment.
@Interpause Interpause added the enhancement New feature or request label Jun 15, 2024
@Interpause
Copy link
Author

Seems definitely possible: Vahe1994/AQLM#85 (comment)

But man they made me project lead for something in uni so im in a time crunch

@dinerburger dinerburger mentioned this issue Dec 7, 2024
Merged
1 task
@github-actions github-actions bot added the stale label Dec 25, 2024
@github-actions GitHub Actions
Copy link

This issue has been closed due to inactivity for 6 months. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Interpause