feat: support parsing whl METADATA on any python version

[aignas]

With this PR we can deterministically parse the METADATA and generate a
BUILD.bazel file using the config settings introduced in #1555. Let's
imagine we had a requirements.txt file that used only wheels, we could
use the host interpreter to parse the wheel metadata for all the target
platforms and use the version aware toolchain at runtime. This potentially
unlocks more clever layouts of the bzlmod hub repos explored in #1625
where we could have a single whl_library instance for all versions within
a single hub repo.

Work towards #1643.

[rickeylev]

Mostly just small things. Otherwise LGTM.

[aignas]

I got this this example to work by setting the default toolchain/python version using this and then version specific selects start working in practice with regular py_binary and py_test rules.

@rickeylev , what do you think about how we should set the default python_version value? I think there are several improvements (outside this PR) that we could make:

1. Allow users to set 3.9 as the value to python_version. This would requires us to change the toolchain definition so that it uses select.match_any(3.9, 3.9.18) in the target_settings and update the is_python generation code.
2. Move the definition of the python_version to the pythons_hub under bzlmod with setting the default value the same as the default toolchain. We can make an alias if we are using bzlmod to the pythons_hub string value. That way I can keep the default settings in the MODULE.bazel and would not need this line here.

[rickeylev]

Allow users to set 3.9 as the value to python_version. This would requires us to change the toolchain definition so that it uses select.match_any(3.9, 3.9.18) in the target_settings and update the is_python generation code.

Yeah, I think that's fine. I was going to suggest a slightly different structure: two separate toolchain registrations, with 3.9 after any 3.9.X toolchains. This would allow multiple micro-versions. But, the rest of the everything doesn't really support a micro-version, so perhaps that is premature.

I'm mixed on allowing "3.9" as a flag value. It does seem very natural, so I do like that part. But with how flags work, "3.9" has no relation to "3.9.18", so I worry that's going to rear its head somehow. I guess our public interface here are the "is_XX" settings, though, so we have some freedom about the flag input format (e.g. "latest" as a possible value sounds convenient).

Anyways, yeah, i guess allowing 3.9 as valid value is fine.

I've heard rumor that config_common.FeatureFlagInfo can be used to allow a flag be implemented as a rule, and have the resulting value seen by config_setting. I haven't tried it myself, though, and was told this was "deprecated", so I'm not sure about that. In any case, my point in mentioning this is that would allow taking 3.9 as the input value and then transform it to 3.8.18 (by looking at MINOR_MAPPING), and everything should just work as before.

Move the definition of the python_version to the pythons_hub under bzlmod with setting the default value the same as the default toolchain.

What about just loading the default and setting it instead? e.g.

# python/config_settings/BUILD
load("@pythons_hub//:interpreters.bzl", "DEFAULT_PYTHON_VERSION")

define_python_version_flag(default=DEFAULT_PYTHON_VERSION)

Regardless, something that isn't clear to me is how to do the same for workspace builds. Is that even possible? I think python_register_multi_toolchains would need to generate something with the default value in it. Except that function isn't always called, so we can't guarantee it is called. Yeah, I dunno.

[jvolkman]

FWIW I'm currently using python_version directly since I can mix it with constraints in a single config_setting, like

config_setting(
  name = "python_3.10.12_aarch64-apple-darwin_config",
  flag_values = {"@rules_python//python/config_settings:python_version": "3.10.12"},
  constraint_values = ["@platforms//os:macos", "@platforms//cpu:aarch64"],
)

Otherwise, to use the is_XX settings, I guess I'd have to use skylib's selects.config_setting_group?

[aignas]

Yeah, that is what I noticed and using config_setting_group is a little unwieldy. I have created #1720 as a POC to potentially allow having "@rules_python//python/config_settings:python_version": "3.10" instead, but I am missing a few things there to get it fully working.

[aignas]

FYI, I had to move the rendering of the config settings outside the main loop because it was getting complex. However, this PR can generate the right select for pylint:

    deps = [
        "@pip_39_astroid//:pkg",
        "@pip_39_dill//:pkg",
        "@pip_39_isort//:pkg",
        "@pip_39_mccabe//:pkg",
        "@pip_39_platformdirs//:pkg",
        "@pip_39_tomlkit//:pkg",
    ] + select(
        {
            "@@rules_python~override//python/config_settings:is_python_3.10": ["@pip_39_tomli//:pkg"],
            "@@rules_python~override//python/config_settings:is_python_3.9": [
                "@pip_39_tomli//:pkg",
                "@pip_39_typing_extensions//:pkg",
            ],
            ":is_cp310_windows_anyarch": [
                "@pip_39_colorama//:pkg",
                "@pip_39_dill//:pkg",
                "@pip_39_tomli//:pkg",
            ],
            ":is_cp311_windows_anyarch": [
                "@pip_39_colorama//:pkg",
                "@pip_39_dill//:pkg",
            ],
            ":is_cp39_windows_anyarch": [
                "@pip_39_colorama//:pkg",
                "@pip_39_dill//:pkg",
                "@pip_39_tomli//:pkg",
                "@pip_39_typing_extensions//:pkg",
            ],
            "//conditions:default": [],
        },
    ),

Disregard the fact that the conditions and the target repos should match - this can be done as a followup item. I am not sure yet what design would be best. This versioned select is only enabled if the user specifies multiple target python versions.

[rickeylev]

Sorry, removed from queue -- just want to double check something about the unit tests.