incompatible_load_python_rules_from_bzl: Python rules now need to be loaded from @rules_python

[brandjon]

Flag: --incompatible_load_python_rules_from_bzl
Available since: 0.29
Will be flipped in: 2.0
Feature tracking issue: #8893

Motivation

The "core" Python rule logic is currently bundled with Bazel as a combination of native symbols and symbols loaded from @bazel_tools//tools/python. Eventually the native rules will be ported to Starlark, and all symbols will live in bazelbuild/rules_python. In the meantime, we're making it so the user loads these symbols from the rules_python repository, even though they're largely still implemented inside of Bazel itself. This will help make future migration of rule logic to rules_python less painful.

Change

Enabling this flag causes the four native Python rules,

• py_library
• py_binary
• py_test
• py_runtime

to fail at analysis time when they are used directly instead of via rules_python.

Migration

Any usage of the these rules as a built-in symbol should be replaced with a use of the macros defined in rules_python as follows. (All four macros are defined in defs.bzl.)

# My BUILD file

py_binary(
    ...
)

should become

# My BUILD file

load("@rules_python//python:defs.bzl", "py_binary")

py_binary(
    ...
)

In the case of macros in .bzl files,

# My .bzl file

def my_macro(...):
    native.py_binary(
        ...
    )

should become

# My .bzl file

load("@rules_python//python:defs.bzl", "py_binary")

def my_macro(...):
    py_binary(
        ...
    )

To access the @rules_python repo you'll need to make sure you have the following in your WORKSPACE file:

# My WORKSPACE file

load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")

git_repository(
    name = "rules_python",
    remote = "https://github.com/bazelbuild/rules_python.git",
    commit = "4b84ad270387a7c439ebdccfd530e2339601ef27",  # (2019-08-02 or later)
)

load("@rules_python//python:repositories.bzl", "py_repositories")
py_repositories()

If you're using rules_python for pip support (which is separate from the core Python rules), you'll also need to call the pip_repositories() function.

Note that until recently, @rules_python went by the workspace name @io_bazel_rules_python. If you have a definition for the old name in your WORKSPACE file then you'll need to rename it, and ensure your commit is updated to a recent version.

[Yannic]

What's the status of this flag? Is it going to be flipped eventually or has it been abandoned? If it's the former, can we add migration-X labels for 2.x and 3.x so it'll be detected by Bazelisk and tested on Buildkite?

[wchargin]

+1 to @Yannic’s question: this flag was supposed to be flipped in Bazel
2.0, but still isn’t flipped in Bazel 3.7, which released almost a year
later. Is it still realistically planned? This matters for downstream
rules_* maintainers; we need to know whether we have to go to the
trouble of pulling in a rules_python dependency for helper scripts.

[lberki]

Ow, this bug is still open? After careful consideration, we decided that this would be change that causes way too much breakage, so we decided to take another approach instead: we'll teach Bazel to be able to carry Starlark rules with itself just like it does ones implemented in Java now. We'll have to figure out how to decouple these rules from Bazel, of course, but that's a next step. (My crystal ball says that it will involve some magic connection between BUILD files and the WORKSPACE file, for example, by adding implicit load() statements to the beginning of every BUILD file, but that's just a guess)

[cpsauer]

If the plan is to reverse course, could it be a good idea to delete this flag?
(Genuinely not sure; just wanted to float.)

[Context: I, like others it seems, got here because someone using a tool I'd build for bazel had it flipped, breaking things. Probably via --all_incompatible_changes, so it's removal will certainly help when it ships. Maybe there's a need for this to still be around; not sure.]