cycles between torch 2.0.0 and triton 2.0.0

[qzmfranklin]

Thanks a lot for all the great work put in rules_python.

I really enjoyed the multi-version and vendored toolchain support.

Ran into a cycled dependency issue, as follows:

bazel: 6.1.1 (though I don' this this is relevant)

rules_python: a recent HEAD (1e869d8)

repro:

WORKSPACE:

load("@rules_python//python:repositories.bzl", "python_register_multi_toolchains")
load("@rules_python//python/pip_install:repositories.bzl", "pip_install_dependencies")

pip_install_dependencies()
python_register_multi_toolchains(
    name = "python",
    default_version = "3.10",
    python_versions = [
        "3.8",
        "3.9",
        "3.10",
        "3.11",
    ],
    register_coverage_tool = True,
)

load("@python//:pip.bzl", "multi_pip_parse")
load("@python//3.10:defs.bzl", interpreter_3_10 = "interpreter")
load("@python//3.11:defs.bzl", interpreter_3_11 = "interpreter")
load("@python//3.8:defs.bzl", interpreter_3_8 = "interpreter")
load("@python//3.9:defs.bzl", interpreter_3_9 = "interpreter")

multi_pip_parse(
    name = "pypi",
    default_version = "3.10",
    python_interpreter_target = {
        "3.10": interpreter_3_10,
        "3.11": interpreter_3_11,
        "3.8": interpreter_3_8,
        "3.9": interpreter_3_9,
    },
    requirements_lock = {
        "3.10": "//third_party/python/common:requirements.lock.3_10.txt",
        "3.11": "//third_party/python/common:requirements.lock.3_11.txt",
        "3.8": "//third_party/python/common:requirements.lock.3_8.txt",
        "3.9": "//third_party/python/common:requirements.lock.3_9.txt",
    },
    extra_pip_args = [
        "--index-url",
        "https://pypi.tuna.tsinghua.edu.cn/simple",
        # "--no-dependencies",
    ],
)

requirements.in:

torch==2.0.0

Run

bazel build @pypi_3_8_torch//:pkg

yields the following error:

ERROR: /home/zhongmingqu/.cache/bazel/_bazel_zhongmingqu/9dac6aec585d38dc753dcfb84402be22/external/pypi_3_8_triton/BUILD.bazel:22:11: in py_library rule @pypi_3_8_triton//:pkg: cycle in dependency graph:
.-> @pypi_3_8_triton//:pkg (4204372a7bb0ec55b382995a9796bae6a21218e5607b1f1ed1019218df840129)
|   @pypi_3_8_torch//:pkg (4204372a7bb0ec55b382995a9796bae6a21218e5607b1f1ed1019218df840129)
`-- @pypi_3_8_triton//:pkg (4204372a7bb0ec55b382995a9796bae6a21218e5607b1f1ed1019218df840129)
WARNING: errors encountered while analyzing target '@pypi_3_8_triton//:pkg': it will not be built
ERROR: /home/zhongmingqu/.cache/bazel/_bazel_zhongmingqu/9dac6aec585d38dc753dcfb84402be22/external/pypi_3_8_triton/BUILD.bazel:22:11: in py_library rule @pypi_3_8_triton//:pkg: cycle in dependency graph: target depends on an already-reported cycle
WARNING: errors encountered while analyzing target '@pypi_3_8_triton//:pkg': it will not be built

A quick fix for me would be to remove the dependency of torch from triton. Afterall, triton only depends on torch for testing. It's optional.

I looked around and found package_annotation. But it does not allow removing an edge.

I also found that the wheel_installer.py already removed self-edge. But it does not remove cycles in the general case. Understandable, it's a wheel_installer, not a wheels_installer.

I think the fundamental issue is that wheel_installer includes all dependencies instead of just the core dependencies.

What would be a suggested fix from rules_python's authors for the situation I'm in?

[aignas]

This is potentially a duplicate of #1076.

[chrislovecnm]

@qzmfranklin do we have a circular dependency? I know they are allowed in Python, but bazel does not, at this point, allow for them. I am thinking you may want to see if torch and triton are addressing this.

[ph03]

Had the same issue with pytorch2.0 / triton and resolved this by patching the triton wheel to remove the pytorch-dependency from it (resolving the circular dependency affecting the bazel-side)

[groodt]

Closing this as a duplicate of #1076

For what it's worth, there are workarounds for the torch / triton situation that may be suitable for anyone on this issue: pytorch/pytorch#99622 (comment)