Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update int4pack related in torchchat gguf #1404

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Update int4pack related in torchchat gguf #1404

wants to merge 2 commits into from

Conversation

yanbing-j
Copy link
Contributor

@yanbing-j yanbing-j commented Dec 9, 2024 •
edited
Loading

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Dec 9, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchchat/1404

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

❌ 4 New Failures, 1 Cancelled Job

As of commit b884e29 with merge base 326c1fe (image):

NEW FAILURES - The following jobs have failed:

  • pull / compile-gguf (macos-14) (gh)
    ImportError: cannot import name 'TORCH_VERSION_AT_LEAST_2_6' from 'torchao.utils' (/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/torchao/utils.py)
  • pull / test-cpu-eval-sanity-check (aarch64, stories15M) (gh)
    huggingface_hub.errors.HfHubHTTPError: 504 Server Error: Gateway Time-out for url: https://huggingface.co/api/datasets/EleutherAI/wikitext_document_level/revision/3ad9e13102af0ea3a47d6b0907694d908f01be05
  • pull / test-gguf-util (macos-14) (gh)
    ImportError: cannot import name 'TORCH_VERSION_AT_LEAST_2_6' from 'torchao.utils' (/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/torchao/utils.py)
  • pull / test-tinystories-executorch (macos-14-xlarge) (gh)
    ImportError: cannot import name 'TORCH_VERSION_AT_LEAST_2_6' from 'torchao.utils' (/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/torchao/utils.py)

CANCELLED JOB - The following job was cancelled. Please retry:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label Dec 9, 2024
@yanbing-j yanbing-j mentioned this pull request Dec 9, 2024
Open
@Jack-Khuu
Copy link
Contributor

Jack-Khuu commented Dec 9, 2024 •
edited
Loading

I'll bump the pin real quick

#1407

@Jack-Khuu Jack-Khuu mentioned this pull request Dec 9, 2024
Open
Jack-Khuu
Jack-Khuu approved these changes Dec 9, 2024
View reviewed changes
torchchat/utils/gguf_loader.py
@@ -24,6 +24,9 @@
pack_scales_and_zeros,
)

from torchao.dtypes.utils import is_device
from torchao.utils import TORCH_VERSION_AT_LEAST_2_6
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

torchchat locks onto a specific torch version, so we don't need to check

Assume > 2.6

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CI failures seem that torchao version is not that new, because TORCH_VERSION_AT_LEAST_2_6 is a new one. And I saw you pin pytorch nightly to 20241013, which is also not new, and this nightly does not have pytorch/pytorch#139611 inside. This is my question, because the nightly used in the CI is 20241126.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yup, working on the bump here: #1367

We'll test your fixes on there

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

torchchat/utils/gguf_loader.py
weight = torch.empty(
(
out_features,
in_features // 2,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice

torchchat/utils/gguf_loader.py
@@ -623,7 +655,7 @@ def load_model_and_state_dict(
in_features=in_features,
out_features=out_features,
bias=False,
device="meta",
device="cpu",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's keep this as a meta device as long as we can

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now only CPU acts different from cuda and meta. https://github.com/pytorch/torchchat/pull/1404/files/b884e295a164fa0b8cd172196e4409e51315567b#diff-28cab20c48af32e561f6e95cec7d029fa076708223a00d64afa80ad62b9b52a4R192 Use device meta here cannot tell the right shape of weight.

@Jack-Khuu Jack-Khuu added the Quantization Issues related to Quantization or torchao label Dec 10, 2024
@mikekgfb mikekgfb mentioned this pull request Dec 11, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Jack-Khuu Jack-Khuu Jack-Khuu approved these changes

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Meta Open Source bot. Quantization Issues related to Quantization or torchao
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Working around new int4wo weight packing
3 participants
@yanbing-j @Jack-Khuu @facebook-github-bot