incompatible_use_python_toolchains: The Python runtime is obtained from a toolchain rather than a flag

[brandjon]

Flag: --incompatible_use_python_toolchains
Available since: 0.25
Will be flipped in: 0.27
Feature tracking issue: #7375

FAQ (common problems)

I'm getting Python 2 vs 3 errors

This flag fixes #4815 on non-Windows platforms, so your code might now be running under a different version of Python than it was in previous Bazel versions. You may notice this as a Python stack trace complaining about bad print syntax, problems with bytes vs str (encode/decode), unknown imports, etc.

In order for your code to run under the proper version of Python, make sure Python 2 binaries and tests have the attribute python_version = "PY2" (the default is PY3).

For targets that are built in the host configuration (for example, genrule tools in particular), python_version has no effect. It is currently impossible for PY2 and PY3 host-configured targets to co-exist in the same build; they will always be overridden to one or the other, depending on the value of --host_force_python. This incompatible change does not affect how the host config works, it just makes it so targets actually run with the version the host config specifies. If you (or your dependencies) have host-configured tools that require Python 2, and which are now failing because they're running under Python 3, add --host_force_python=PY2 to your bazelrc (the default value is PY3).

Bazel 0.27 introduces a diagnostic message when a host-configured tool fails at run time (non-zero exit code), alerting you when it may be necessary to set this flag.

The default Python toolchain can't find the interpreter

If you get an error like this:

Error: The default python toolchain (@bazel_tools//tools/python:autodetecting_toolchain) was unable to locate a suitable Python interpreter on the target platform at execution time. Please register an appropriate Python toolchain. [...]
Failure reason: Cannot locate 'python3' or 'python' on the target platform's PATH, which is: [...]

Determine whether you have python2, python3, and/or python on your shell PATH. For py_test targets, and for py_binary targets used as tools (in genrules, etc.), also check whether your PATH is being manipulated by the flags --incompatible_strict_action_env and/or --action_env=PATH=[...]. For instance, the strict action environment does not include /usr/local/bin in PATH by default, which is where python3 is typically located on Mac, if it is installed at all. See also #8536.

If modifying your PATH is not feasible, try defining and registering your own Python toolchain as described at the bottom of this post.

I don't have Python 3 installed (e.g. default Mac environment)

Previously, if you didn't have a Python 3 interpreter but all your code was compatible with Python 2, Bazel would happily analyze it as PY3 and execute it using a Python 2 python command. Now this breaks because the autodetecting toolchain validates that python is actually Python 3.

The ideal solution is to not depend on Python 3 code, or else install a Python 3 environment on the target system. The practical workaround is to opt out of version checking by using the non-strict autodetecting toolchain. The error message tells you how: Add to your bazelrc

build --extra_toolchains=@bazel_tools//tools/python:autodetecting_toolchain_nonstrict

Note that you will not benefit from the fix to #4815 as long as you are using this toolchain.

If you're using a custom Python toolchain (using py_runtime_pair, as described at the bottom of this post), you can have the py3_runtime attribute point to a py_runtime that declares itself as PY3 but in actuality references a Python 2 interpreter. This abuse of version information achieves the same result: PY3-analyzed targets get run with a Python 2 interpreter.

Neither of these approaches is recommended for anyone but end-users, since they affect how Python targets get run globally throughout the build.

I'm a rule author and I want my target to run regardless of whether the downstream user has Python 2 or 3

See this comment.

Did the behavior of toolchains change between 0.26 and 0.27

This incompatible change was available since 0.25 and flipped to true by default in 0.27. Bazel 0.27 introduces some bug fixes in the behavior of the autodetecting toolchain, better diagnostic messages, and the non-strict toolchain.

Motivation

For background on toolchains, see here.

Previously, the Python runtime (i.e., the interpreter used to execute py_binary and py_test targets) could only be controlled globally, and required passing flags like --python_top to the bazel invocation. This is out-of-step with our ambitions for flagless builds and remote-execution-friendly toolchains. Using the toolchain mechanism means that each Python target can automatically select an appropriate runtime based on what target platform it is being built for.

Change

Enabling this flag triggers the following changes.

1. Executable Python targets will retrieve their runtime from the new Python toolchain.

2. It is forbidden to set any of the legacy flags --python_top, --python2_path, or --python3_path. Note that the last two of those are already no-ops. It is also strongly discouraged to set --python_path, but this flag will be removed in a later cleanup due to Remove --python_path flag #7901.

3. The python_version attribute of the py_runtime rule becomes mandatory. It must be either "PY2" or "PY3", indicating which kind of runtime it is describing.

For builds that rely on a Python interpreter installed on the system, it is recommended that users (or platform rule authors) ensure that each platform has an appropriate Python toolchain definition.

If no Python toolchain is explicitly registered, on non-Windows platforms there is a new default toolchain that automatically detects and executes an interpreter (of the appropriate version) from PATH. This resolves longstanding issue #4815. A Windows version of this toolchain will come later (#7844).

Migration

See the above FAQ for common issues with the autodetecting toolchain.

If you were relying on --python_top, and you want your whole build to continue to use the py_runtime you were pointing it to, you just need to follow the steps below to define a py_runtime_pair and toolchain, and register this toolchain in your workspace. So long as you don't add any platform constraints that would prevent your toolchain from matching, it will take precedence over the default toolchain described above.

If you were relying on --python_path, and you want your whole build to use the interpreter located at the absolute path you were passing in this flag, the steps are the same, except you also have to define a new py_runtime with the interpreter_path attribute set to that path.

Otherwise, if you were only relying on the default behavior that resolved python from PATH, just enjoy the new default behavior, which is:

1. First try python2 or python3 (depending on the target's version)
2. Then fall back on python if not found
3. Fail-fast if the interpreter that is found doesn't match the target's major Python version (PY2 or PY3), as per the python -V flag.

On Windows the default behavior is currently unchanged (#7844).

Example toolchain definition

# In your BUILD file...
load("@bazel_tools//tools/python/toolchain.bzl", "py_runtime_pair")
py_runtime(
    name = "my_py2_runtime",
    interpreter_path = "/system/python2",
    python_version = "PY2",
)
py_runtime(
    name = "my_py3_runtime",
    interpreter_path = "/system/python3",
    python_version = "PY3",
)
py_runtime_pair(
    name = "my_py_runtime_pair",
    py2_runtime = ":my_py2_runtime",
    py3_runtime = ":my_py3_runtime",
)
toolchain(
    name = "my_toolchain",
    target_compatible_with = [...],  # optional platform constraints
    toolchain = ":my_py_runtime_pair",
    toolchain_type = "@bazel_tools//tools/python:toolchain_type",
)

# In your WORKSPACE...
register_toolchains("//my_pkg:my_toolchain")

Of course, you can define and register many different toolchains and use platform constraints to restrict them to appropriate target platforms. It is recommended to use the constraint settings @bazel_tools//tools/python:py2_interpreter_path and [...]:py3_interpreter_path as the namespaces for constraints about where a platform's Python interpreters are located.

The new toolchain-related rules and default toolchain are implemented in Starlark under @bazel_tools. Their source code and documentation strings can be read here.

[nlopezgi]

I'm trying this flag for rules_k8s in the context of bazelbuild/rules_k8s#305.
If I run a build with an explicit --python_top my build / test succeeds (bazelbuild/rules_k8s#306)
However if I use the new --incompatible_use_python_toolchains I get the following error:

ERROR: .../src/rules_k8s/fork/k8s/BUILD:58:1: Building par file //k8s:resolver.par failed (Exit 1) compiler.par failed: error executing command bazel-out/host/bin/external/subpar/compiler/compiler.par --manifest_file bazel-out/k8-fastbuild/bin/k8s/resolver.par_SOURCES --outputpar bazel-out/k8-fastbuild/bin/k8s/resolver.par --stub_file ... (remaining 4 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
src/main/tools/linux-sandbox-pid1.cc:427: "execvp(bazel-out/host/bin/external/subpar/compiler/compiler.par, 0x1830290)": No such file or directory

Note I did not experience this type of error when trying this flag out with rules_docker (bazelbuild/rules_docker#787)

See also bazelbuild/rules_k8s#305 for context

[brandjon]

Unfortunately this broke a large number of downstream projects, mostly because it actually enforces that the Python interpreter has the requested version. Looks like switching the default Python version in Bazel 0.25 from PY2 to PY3 was too easy precisely because it wasn't being enforced at execution time. :(

We'll need to do some downstream fixing before we can flip again.

[tmc]

I'm trying this flag for rules_k8s in the context of bazelbuild/rules_k8s#305.
If I run a build with an explicit --python_top my build / test succeeds (bazelbuild/rules_k8s#306)
However if I use the new --incompatible_use_python_toolchains I get the following error:

ERROR: .../src/rules_k8s/fork/k8s/BUILD:58:1: Building par file //k8s:resolver.par failed (Exit 1) compiler.par failed: error executing command bazel-out/host/bin/external/subpar/compiler/compiler.par --manifest_file bazel-out/k8-fastbuild/bin/k8s/resolver.par_SOURCES --outputpar bazel-out/k8-fastbuild/bin/k8s/resolver.par --stub_file ... (remaining 4 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
src/main/tools/linux-sandbox-pid1.cc:427: "execvp(bazel-out/host/bin/external/subpar/compiler/compiler.par, 0x1830290)": No such file or directory

Note I did not experience this type of error when trying this flag out with rules_docker (bazelbuild/rules_docker#787)

See also bazelbuild/rules_k8s#305 for context

FWIW I think this is google/subpar#98

[nlopezgi]

Documentation here has a small typo. Currently says:
load("@bazel_tools//tools/python/toolchain.bzl", "py_runtime_pair")
when it should be
load("@bazel_tools//tools/python:toolchain.bzl", "py_runtime_pair")
Note /toolchain.bzl vs :toolchain.bzl

[brandjon]

Fixed in 331c84b. :)

[brandjon]

Downstream buildkite run here. Cataloging the failures:

• Tests that run on mac workers fail because python3 is not on the PATH propagated to actions.

• Android Testing failures are tracked here

• Bazel's own failures are android-related so they're probably the same cause as the above.

• The Bazel Toolchains failure is due to a PY2 host tool running as PY3.

• The Cloud Robotics Core failure is also a host config issue.

• Remote execution and Rules jvm external have some android issues.

• rules_k8s has the same issue as the above host-config stuff with Bazel Toolchains

[brandjon]

We're now targeting 0.27 for flipping this flag. In the absence of the execution transition feature, all projects that require PY2 host tools should set --host_force_python=PY2.

[bjacklyn]

@brandjon can we remove this diagnostic, or add a way to silence it? We're on bazel 5.0.0 now, I think it has been enough time since 0.27 that this is probably never the reason that a python rule fails (and it has never been the case in my org, confusing devs), but this is still always printed:

If this error started occurring in Bazel 0.27 and later, it may be because the Python toolchain now enforces that targets analyzed as PY2 and PY3 run under a Python 2 and Python 3 interpreter, respectively. See https://github.com/bazelbuild/bazel/issues/7899 for more information.

Also python2 was EOL in 2020