-
Notifications
You must be signed in to change notification settings - Fork 22.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cyclic dependency created between torch 2.0 and triton #99622
Comments
Unfortunately this is unavoidable. As long as the two libraries don't try to use each other on import (which they shouldn't) this is fine and won't break anything. |
Can this cyclic dependency not be removed, or solved differently? For some environments like pip, the cyclic dependency is not a problem but for other enviroments it is. For example, Bazel cannot handle the cyclic dependency in the pip package. |
The easiest fix will be to split pytorch into two packages, one which is traditional eager mode pytorch, and the other which has inductor (which is what has the triton dep). I'm not familiar enough with Bazel to say more about how to do this. |
Wouldn't you still have a cycle of pytorch->inductor->triton->pytorch? IMO this is a bug in Bazel. |
the idea is inductor would depend on pytorch but not vice versa |
I saw a change to Triton here that moved the torch dependency only to the "test" rule: triton-lang/triton#1389. This is not in the current release, but would just updating the release solve this? |
We cannot update pytorch 2.0.0 to next triton release, but on nightlies triton has been updated |
I just tried the nightly version and the cyclic dependency is gone with the new triton release. So everything should be good with the 2.1.0 version |
Will this fix make it into torch 2.0.1? |
No, we cannot update triton version used for 2.0.1 as 2.0.1 is just a bugfix release, and updating triton version is more risk and requires more significant changes. |
@milutter what was your solution for bazel? Using 2.1 nightly is tricky with some of the package constraints (senetence-transformers etc) |
As far as I know there is no solution other than waiting. I stayed on @ngimel |
As a gross hack, I have patched the wheel for our local usage at Stripe and it works. We are heavy Bazel users. YMMV. I spent a few days trying to build the wheel from source. It was a nightmare. I ran out of disk space on the root partition trying to install system packages, so I had to bring up a custom instance. Then I ran out of space on the main partition, trying to compile, so I had to bring up another custom instance. Then I realized I had installed CUDA 12.1 and couldn't install CUDA 11.8 over it, so yet another instance. Then a long list of other problems. I was eventually able to get Then I had a brainwave: what if I patch the official wheel and remove the requirement on Triton? That worked. Here are my notes. Manually patching PyTorch wheel to remove dependency on TritonSee Binary distribution format for details on unzip -d torch201.2 torch-2.0.1-cp38-cp38-manylinux1_x86_64.whl
cd torch201.2
# Rename the `dist-info` directory to include '+stripe.2' as a suffix for `2.0.1`
mv torch-2.0.1{,+stripe.2}.dist-info/
cd torch-2.0.1+stripe.2.dist-info/ Update
|
this PR has been merged a while ago. PR #95896 has updated triton from thus i would assume this issue should be closed ? |
Issue description
Installing torch 2.0 creates cyclic dependency between torch and triton. However, earlier torch versions (1.x) work fine.
Code example
System Info
cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @desertfire
The text was updated successfully, but these errors were encountered: