py_binary doesn't automatically pick a Python runtime matching the requested version

[emcfarlane]

Description of the problem / feature request:

Running py_test() and py_binary() rules under bazel doesn't select python3 when only srcs_version="PY3", default_python_version="PY3" attributes are set on the rules. How do you specify python2/3 builds?

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I wrote an integration test: emcfarlane@5a6feef

bazel.log

What operating system are you running Bazel on?

Mac

What's the output of bazel info release?

release 0.10.0-homebrew

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

[email protected]:afking/bazel.git
d3e102c9fb62f4416099dc9f097a5f207aa7d442
5a6feef7953d2fe664e7a3444a36c9a875416913

Have you found anything relevant by searching the web?

#3871
#3517
bazelbuild/rules_python#62

[buchgr]

@lberki can you take this please?

[brandjon]

A proper solution would probably involve allowing py_runtime to be used to specify both py2 and py3 interpreters, and allowing individual py_binary rules to declare their runtimes (probably by exposing the hidden py_interpreter attribute).

But in the meantime, a quicker and simpler solution would probably be to make the default behavior (where py_interpreter / --python_top is not given) simply choose shell commands "python2" or "python3" depending on the python mode.

[brandjon]

The proper fix for this bug is to implement py_toolchain [edit: now py_runtime_pair] and to create a default toolchain that sensibly locates the system's installed Python 2 and Python 3 runtimes.

Until that's implemented, the workaround is to dispatch to the system interpreters in a py_runtime rule using select(), and point to that py_runtime in --python_top. The best way to do this from 0.23 onward is as follows:

py_runtime(
    name = "myruntime",
    interpreter_path = select({
        # Update paths as appropriate for your system.
        "@bazel_tools//tools/python:PY2": "/usr/bin/python2",
        "@bazel_tools//tools/python:PY3": "/usr/bin/python3",
    }),
    files = [],
)

Then run your build with the flag --python_top=//path/to/the:myruntime.

[Toxicable]

I could be totally missing something about how Python works here, so feel free to correct me.
But is there a reason why we're relying on the system having python installed at all?
Couldn't you just have a python binary downloaded similar to how rules_nodejs does it.
https://github.com/bazelbuild/rules_nodejs#installation-with-a-manually-specified-version-of-nodejs-and-yarn
This will make it much easier to use the python rules across platforms or even the same systems
Also allowing you to specific exactly what python version is being used by the repository rather than just going with whatever the system has installed

[brandjon]

At the moment we don't bundle a specific standard Python interpreter with Bazel. We would need an interpreter for each platform that Bazel supports, and we'd have to worry about coordinating upgrades to this version.

We don't tend to bundle tools with Bazel and its rule sets in other languages either, e.g. there's no gcc included in Bazel.

However, as a user you can include a Python interpreter in your repo and reference it via a py_runtime rule. Then you'd pass --python_top=//path/to/that:py_runtime to make the Python rules all use that interpreter instead of the system default. (This mechanism will soon change when toolchains are implemented, so that different runtimes can be used for different platforms.) In this way, you can have your py_binary target executing using a vendored (built-from-source or checked-in) Python interpreter, rather than the system one. However, a system interpreter is still needed to execute the Python stub script that launches your py_binary target.

Having a workspace rule fetch a Python interpreter sounds like a reasonable feature off the top of my head. Created #7467.

[Toxicable]

Ah yes, I didn't mean that bazel should ship an interpreter itself.
But instead as you mentioned; a way to fetch it.

Your plan for fetching it should work perfect for us, we have a lot of systems that don't necessarily have Python on them and would prefer avoiding any transient issues due to version skews.

[depthwise]

This is especially bad when building Python programs (or Subpar *.par's) for e.g. RaspberryPi. You build and copy your program, and then it barfs because it uses Python2 instead of 3 even though everything in the build tells it to use Python3.

[jayeye]

bump

This has been open for over three years; the workarounds specify deprecated flags.

[haberman]

The current docs for the python_version attribute directly link to this bug with the following warning:

Bug warning: This attribute sets the version for which Bazel builds your target, but due to #4815, the resulting stub script may still invoke the wrong interpreter version at runtime. See this workaround, which involves defining a py_runtime target that points to either Python version as needed, and activating this py_runtime by setting --python_top.

This leads me to believe that python_version might be totally broken (eg. py_binary() might run my program under the wrong version of Python). It is surprising then that this bug is prioritized at P3.

Furthermore, the linked workaround uses deprecated rules and flags (py_runtime() and --python_top) which Bazel will not accept unless you pass --incompatible_use_python_toolchains=false. The currently blessed solution appears to be described here, which is to use py_runtime_pair(), toolchain(), and register_toolchains().

It also appears according to this update that this bug may currently be fixed for all platforms except Windows.

If all of this is correct, some updates that might help are:

1. Update the docs to be clear that python_version does in fact work properly except on Windows.
2. Update the docs to point to the non-deprecated workaround.
3. Update the title of this bug to reflect the fact that it only applies to Windows.

[github-actions]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 14 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

[github-actions]

This issue has been automatically closed due to inactivity. If you're still interested in pursuing this, please reach out to the triage team (@bazelbuild/triage). Thanks!