Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python3Packages.accelerate: Bump version 1.0.0 -> 1.1.0 and fix few things #353184

Merged
merged 3 commits into from
Nov 3, 2024

Conversation

mikatammi
Copy link
Contributor

python3Packages.accelerate

  • Add missing huggingface-hub dependency, it caused package being broken already before version bump
  • Bump version 1.0.0 -> 1.1.0
  • On Darwin: Disable some tests, which started failing after enabling MPS in pytorch package

Changelog can be found here: https://github.com/huggingface/accelerate/releases

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)

Tested some of the basic functionality.

I tested that now running binaries don't give straight ImportError about missing huggingface-hub anymore, but didn't test the functionality otherwise than trying to run binaries does print out --help to the terminal. I'm able to use accelerate in my own Python code whch runs StableDiffusion.

  • 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

Add huggingface-hub to the dependencies, this should fix various import
errors.

Signed-off-by: Mika Tammi <[email protected]>
darwin: Disable some tests, which started failing after enabling MPS in
pytorch package.

Signed-off-by: Mika Tammi <[email protected]>
@mikatammi
Copy link
Contributor Author

mikatammi commented Nov 2, 2024

Here's log of accelerate tests failing, before I disabled the failing tests:

accelerate-test-fails.txt

@mikatammi
Copy link
Contributor Author

Two questions i'm mostly concerned about:

  • After doing version bump, should we check if some tests started working again?
  • Should we report about some of the failing tests to accelerate upstream?

@GaetanLepage
Copy link
Contributor

nixpkgs-review result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 353184


x86_64-linux

⏩ 4 packages marked as broken and skipped:
  • khoj
  • khoj.dist
  • tts
  • tts.dist
❌ 18 packages failed to build:
  • openllm
  • openllm.dist
  • private-gpt
  • private-gpt.dist
  • python311Packages.cltk
  • python311Packages.cltk.dist
  • python311Packages.k-diffusion
  • python311Packages.k-diffusion.dist
  • python311Packages.llama-index-embeddings-huggingface
  • python311Packages.llama-index-embeddings-huggingface.dist
  • python311Packages.private-gpt
  • python311Packages.private-gpt.dist
  • python311Packages.txtai
  • python311Packages.txtai.dist
  • python312Packages.private-gpt
  • python312Packages.private-gpt.dist
  • python312Packages.txtai
  • python312Packages.txtai.dist
✅ 46 packages built:
  • libretranslate (python312Packages.libretranslate)
  • libretranslate.dist (python312Packages.libretranslate.dist)
  • open-webui
  • open-webui.dist
  • python311Packages.accelerate
  • python311Packages.accelerate.dist
  • python311Packages.argos-translate-files
  • python311Packages.argos-translate-files.dist
  • python311Packages.argostranslate
  • python311Packages.argostranslate.dist
  • python311Packages.langchain-huggingface
  • python311Packages.langchain-huggingface.dist
  • python311Packages.libretranslate
  • python311Packages.libretranslate.dist
  • python311Packages.manifest-ml
  • python311Packages.manifest-ml.dist
  • python311Packages.peft
  • python311Packages.peft.dist
  • python311Packages.sentence-transformers
  • python311Packages.sentence-transformers.dist
  • python311Packages.stanza
  • python311Packages.stanza.dist
  • python311Packages.translatehtml
  • python311Packages.translatehtml.dist
  • python312Packages.accelerate
  • python312Packages.accelerate.dist
  • python312Packages.argos-translate-files
  • python312Packages.argos-translate-files.dist
  • python312Packages.argostranslate
  • python312Packages.argostranslate.dist
  • python312Packages.k-diffusion
  • python312Packages.k-diffusion.dist
  • python312Packages.langchain-huggingface
  • python312Packages.langchain-huggingface.dist
  • python312Packages.llama-index-embeddings-huggingface
  • python312Packages.llama-index-embeddings-huggingface.dist
  • python312Packages.manifest-ml
  • python312Packages.manifest-ml.dist
  • python312Packages.peft
  • python312Packages.peft.dist
  • python312Packages.sentence-transformers
  • python312Packages.sentence-transformers.dist
  • python312Packages.stanza
  • python312Packages.stanza.dist
  • python312Packages.translatehtml
  • python312Packages.translatehtml.dist

aarch64-linux

⏩ 12 packages marked as broken and skipped:
  • khoj
  • khoj.dist
  • open-webui
  • open-webui.dist
  • private-gpt
  • private-gpt.dist
  • python311Packages.private-gpt
  • python311Packages.private-gpt.dist
  • python312Packages.private-gpt
  • python312Packages.private-gpt.dist
  • tts
  • tts.dist
❌ 6 packages failed to build:
  • openllm
  • openllm.dist
  • python311Packages.txtai
  • python311Packages.txtai.dist
  • python312Packages.txtai
  • python312Packages.txtai.dist
✅ 50 packages built:
  • libretranslate (python312Packages.libretranslate)
  • libretranslate.dist (python312Packages.libretranslate.dist)
  • python311Packages.accelerate
  • python311Packages.accelerate.dist
  • python311Packages.argos-translate-files
  • python311Packages.argos-translate-files.dist
  • python311Packages.argostranslate
  • python311Packages.argostranslate.dist
  • python311Packages.cltk
  • python311Packages.cltk.dist
  • python311Packages.k-diffusion
  • python311Packages.k-diffusion.dist
  • python311Packages.langchain-huggingface
  • python311Packages.langchain-huggingface.dist
  • python311Packages.libretranslate
  • python311Packages.libretranslate.dist
  • python311Packages.llama-index-embeddings-huggingface
  • python311Packages.llama-index-embeddings-huggingface.dist
  • python311Packages.manifest-ml
  • python311Packages.manifest-ml.dist
  • python311Packages.peft
  • python311Packages.peft.dist
  • python311Packages.sentence-transformers
  • python311Packages.sentence-transformers.dist
  • python311Packages.stanza
  • python311Packages.stanza.dist
  • python311Packages.translatehtml
  • python311Packages.translatehtml.dist
  • python312Packages.accelerate
  • python312Packages.accelerate.dist
  • python312Packages.argos-translate-files
  • python312Packages.argos-translate-files.dist
  • python312Packages.argostranslate
  • python312Packages.argostranslate.dist
  • python312Packages.k-diffusion
  • python312Packages.k-diffusion.dist
  • python312Packages.langchain-huggingface
  • python312Packages.langchain-huggingface.dist
  • python312Packages.llama-index-embeddings-huggingface
  • python312Packages.llama-index-embeddings-huggingface.dist
  • python312Packages.manifest-ml
  • python312Packages.manifest-ml.dist
  • python312Packages.peft
  • python312Packages.peft.dist
  • python312Packages.sentence-transformers
  • python312Packages.sentence-transformers.dist
  • python312Packages.stanza
  • python312Packages.stanza.dist
  • python312Packages.translatehtml
  • python312Packages.translatehtml.dist

x86_64-darwin

⏩ 50 packages marked as broken and skipped:
  • khoj
  • khoj.dist
  • libretranslate
  • libretranslate.dist
  • open-webui
  • open-webui.dist
  • private-gpt
  • private-gpt.dist
  • python311Packages.argos-translate-files
  • python311Packages.argos-translate-files.dist
  • python311Packages.argostranslate
  • python311Packages.argostranslate.dist
  • python311Packages.langchain-huggingface
  • python311Packages.langchain-huggingface.dist
  • python311Packages.libretranslate
  • python311Packages.libretranslate.dist
  • python311Packages.llama-index-embeddings-huggingface
  • python311Packages.llama-index-embeddings-huggingface.dist
  • python311Packages.manifest-ml
  • python311Packages.manifest-ml.dist
  • python311Packages.private-gpt
  • python311Packages.private-gpt.dist
  • python311Packages.sentence-transformers
  • python311Packages.sentence-transformers.dist
  • python311Packages.translatehtml
  • python311Packages.translatehtml.dist
  • python311Packages.txtai
  • python311Packages.txtai.dist
  • python312Packages.argos-translate-files
  • python312Packages.argos-translate-files.dist
  • python312Packages.argostranslate
  • python312Packages.argostranslate.dist
  • python312Packages.k-diffusion
  • python312Packages.k-diffusion.dist
  • python312Packages.langchain-huggingface
  • python312Packages.langchain-huggingface.dist
  • python312Packages.libretranslate
  • python312Packages.libretranslate.dist
  • python312Packages.llama-index-embeddings-huggingface
  • python312Packages.llama-index-embeddings-huggingface.dist
  • python312Packages.manifest-ml
  • python312Packages.manifest-ml.dist
  • python312Packages.private-gpt
  • python312Packages.private-gpt.dist
  • python312Packages.sentence-transformers
  • python312Packages.sentence-transformers.dist
  • python312Packages.translatehtml
  • python312Packages.translatehtml.dist
  • python312Packages.txtai
  • python312Packages.txtai.dist
❌ 6 packages failed to build:
  • openllm
  • openllm.dist
  • python311Packages.cltk
  • python311Packages.cltk.dist
  • python311Packages.k-diffusion
  • python311Packages.k-diffusion.dist
✅ 12 packages built:
  • python311Packages.accelerate
  • python311Packages.accelerate.dist
  • python311Packages.peft
  • python311Packages.peft.dist
  • python311Packages.stanza
  • python311Packages.stanza.dist
  • python312Packages.accelerate
  • python312Packages.accelerate.dist
  • python312Packages.peft
  • python312Packages.peft.dist
  • python312Packages.stanza
  • python312Packages.stanza.dist

aarch64-darwin

⏩ 28 packages marked as broken and skipped:
  • khoj
  • khoj.dist
  • libretranslate
  • libretranslate.dist
  • open-webui
  • open-webui.dist
  • private-gpt
  • private-gpt.dist
  • python311Packages.cltk
  • python311Packages.cltk.dist
  • python311Packages.k-diffusion
  • python311Packages.k-diffusion.dist
  • python311Packages.libretranslate
  • python311Packages.libretranslate.dist
  • python311Packages.llama-index-embeddings-huggingface
  • python311Packages.llama-index-embeddings-huggingface.dist
  • python311Packages.private-gpt
  • python311Packages.private-gpt.dist
  • python312Packages.k-diffusion
  • python312Packages.k-diffusion.dist
  • python312Packages.libretranslate
  • python312Packages.libretranslate.dist
  • python312Packages.llama-index-embeddings-huggingface
  • python312Packages.llama-index-embeddings-huggingface.dist
  • python312Packages.private-gpt
  • python312Packages.private-gpt.dist
  • tts
  • tts.dist
❌ 6 packages failed to build:
  • openllm
  • openllm.dist
  • python311Packages.txtai
  • python311Packages.txtai.dist
  • python312Packages.txtai
  • python312Packages.txtai.dist
✅ 36 packages built:
  • python311Packages.accelerate
  • python311Packages.accelerate.dist
  • python311Packages.argos-translate-files
  • python311Packages.argos-translate-files.dist
  • python311Packages.argostranslate
  • python311Packages.argostranslate.dist
  • python311Packages.langchain-huggingface
  • python311Packages.langchain-huggingface.dist
  • python311Packages.manifest-ml
  • python311Packages.manifest-ml.dist
  • python311Packages.peft
  • python311Packages.peft.dist
  • python311Packages.sentence-transformers
  • python311Packages.sentence-transformers.dist
  • python311Packages.stanza
  • python311Packages.stanza.dist
  • python311Packages.translatehtml
  • python311Packages.translatehtml.dist
  • python312Packages.accelerate
  • python312Packages.accelerate.dist
  • python312Packages.argos-translate-files
  • python312Packages.argos-translate-files.dist
  • python312Packages.argostranslate
  • python312Packages.argostranslate.dist
  • python312Packages.langchain-huggingface
  • python312Packages.langchain-huggingface.dist
  • python312Packages.manifest-ml
  • python312Packages.manifest-ml.dist
  • python312Packages.peft
  • python312Packages.peft.dist
  • python312Packages.sentence-transformers
  • python312Packages.sentence-transformers.dist
  • python312Packages.stanza
  • python312Packages.stanza.dist
  • python312Packages.translatehtml
  • python312Packages.translatehtml.dist

@GaetanLepage
Copy link
Contributor

  • After doing version bump, should we check if some tests started working again?

Ideally yes. You can try re-enabling them all and adding them back only if needed.

Should we report about some of the failing tests to accelerate upstream?

Also yes, if the failures are not due to our testing setup (example, no need to report tests that are failing because of lacking internet access).

Copy link
Contributor

@GaetanLepage GaetanLepage left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wegank wegank added the 12.approvals: 1 This PR was reviewed and approved by one reputable person label Nov 3, 2024
@GaetanLepage GaetanLepage merged commit 6ee487a into NixOS:master Nov 3, 2024
27 checks passed
@mikatammi mikatammi deleted the fix_accelerate_pr branch November 3, 2024 15:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants