Non-tagged Python builds use non-PEP 440 versions

[jaraco]

Sorry if this should be a new issue. I have a problem where the version of Python itself is causing an Invalid Version error. I install Python from the tip of 3.12, then:

% .tox/anypy/bin/python -c "import pkg_resources as p; p.load_entry_point('coverage', 'console_scripts', 'coverage')()"
Traceback (most recent call last):
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2711, in _dep_map
    return self.__dep_map
           ^^^^^^^^^^^^^^
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2826, in __getattr__
    raise AttributeError(attr)
AttributeError: _Distribution__dep_map

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/__init__.py", line 522, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2855, in load_entry_point
    return ep.load()
           ^^^^^^^^^
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2467, in load
    self.require(*args, **kwargs)
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2489, in require
    reqs = self.dist.requires(self.extras)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2746, in requires
    dm = self._dep_map
         ^^^^^^^^^^^^^
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2713, in _dep_map
    self.__dep_map = self._filter_extras(self._build_dep_map())
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2728, in _filter_extras
    invalid_marker(marker) or not evaluate_marker(marker)
    ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/__init__.py", line 1415, in invalid_marker
    evaluate_marker(text)
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/__init__.py", line 1433, in evaluate_marker
    return marker.evaluate()
           ^^^^^^^^^^^^^^^^^
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/_vendor/packaging/markers.py", line 245, in evaluate
    return _evaluate_markers(self._markers, current_environment)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/_vendor/packaging/markers.py", line 151, in _evaluate_markers
    groups[-1].append(_eval_op(lhs_value, op, rhs_value))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/_vendor/packaging/markers.py", line 109, in _eval_op
    return spec.contains(lhs, prereleases=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/_vendor/packaging/specifiers.py", line 565, in contains
    normalized_item = _coerce_version(item)
                      ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/_vendor/packaging/specifiers.py", line 36, in _coerce_version
    version = Version(version)
              ^^^^^^^^^^^^^^^^
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/_vendor/packaging/version.py", line 197, in __init__
    raise InvalidVersion(f"Invalid version: '{version}'")
pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: '3.12.0a5+'

% .tox/anypy/bin/python -m pip freeze --all
attrs==22.2.0
colorama==0.4.6
-e git+ssh://[email protected]/nedbat/coveragepy.git@1edd0608fac9ffadf24da72820572bec43d0c8fd#egg=coverage
distlib==0.3.6
exceptiongroup==1.1.0
execnet==1.9.0
filelock==3.9.0
flaky==3.7.0
hypothesis==6.68.0
importlib-metadata==6.0.0
iniconfig==2.0.0
packaging==23.0
pip==23.0
platformdirs==3.0.0
pluggy==1.0.0
pytest==7.2.1
pytest-xdist==3.2.0
setuptools==67.2.0
sortedcontainers==2.4.0
tomli==2.0.1
typing_extensions==4.4.0
virtualenv==20.19.0
wheel==0.38.4
zipp==3.12.1

Why is setuptools trying to parse the Python version?

There's a corresponding coverage.py issue about this.

Originally posted by @nedbat in pypa/setuptools#3772 (comment)

[jaraco]

I have a problem where the version of Python itself is causing an Invalid Version error.

Looking at the implementation of _build_dep_map, I suspect that some library has a dependency with a python_full_version marker, and when packaging attempts to parse the Python version (platform.python_version()), it's invalid (at least per PEP 440).

You can probably replicate the issue by installing packaging, then running:

>>> import packaging
>>> packaging.markers.Marker('python_full_version > "3.0"').evaluate()
True

I'm unsure if Python should use PEP 440 version numbers or if packaging should be updated to honor Python versions that aren't PEP 440 compliant, but this issue only happens to be loosely affiliated with this issue, which is about package versions.

[uranusjr]

Python has never formally followed PEP 440 (although most of the version numbers it used are compatible). Even if we start mandating PEP 440 for it, there are still much too many problematic version strings to deal with (considering redistributors probably have their own scheme). A separate parser for Python versions is probably more viable; IIRC Python has documentation on the rules somewhere, but we can ask the core devs for a concrete ruling if not.

[pradyunsg]

This was flagged to me on a private Discord today, and jotting down a summary/my takeaways from that discussion. There's three directions we can take here:

• Make packaging parse Python versions slightly differently.
• Change the definition of python_full_version to always generate valid PEP 440 versions.
• Change how Python picks in-development Python versions.

Broadly though, the underlying reason of why this was surfaced is the same as all other PEP 440 issues -- #530

---

For the first bullet:

I don't think this deviates much from PEP 440 TBH, and it's not clear that it's worth a compete new concept/mental model. Plus, it only affect development builds and patched copies of CPython. It's unclear that there's much value to changing that.

---

For the second bullet:

platform.python_version is documented as:

Returns the Python version as string 'major.minor.patchlevel'.

I think the expectation with PEP 508 is that all segments in the documentation would be numbers. We could just strip the trailing + and utilize that in this context and be fine.

---

For the third bullet:

There's also valid concern that people might be splitting the Python version on '.', so using a .devN suffix instead of the + suffix (i.e. 3.12.0a5+ vs 3.12.0a5.dev0). Folks seem to be open to the idea of changing the 3.12.0a5+ to 3.12.0a5+dev or something.

---

Adding cross-references mentioned by @hugovk and @Yhg1s:

• Python version ending with plus sign not compatible with PEP 508 python_full_version marker python/cpython#99968
• Poetry is incompatible with unstable-tagged Python builds (Invalid PEP 440 version: '3.Y.Z+') python-poetry/poetry#6925
• Python nightly tests fail: "InvalidVersion: Invalid version: '3.12.0a4+'" nedbat/coveragepy#1556

[CAM-Gerlach]

Not to bikeshed too early, but I would suggest something other than dev for the local identifier as +dev could be mistaken for the more conventional .devN (which would be the most "canonical"/semantically accurate way to do this, though +local is still syntactically valid of course which is the primary concern). So perhaps a different name should be preferred.

[sbidoul]

I propose a fix in #802

[jaraco]

I ran into this issue again today. It seems it's not fixed:

I'm using pipx with the latest packaging version, and the Python 3.13.0rc1+ is triggering a failure:

 🐚 pip-run pipx -- -m pipx install build
⚠️ Found a space in the home path. We heavily discourage this, due to multiple incompatibilities. Please check our docs for more information on
    this, as well as some pointers on how to migrate to a different home path.

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-dq9xt44k/pipx/__main__.py", line 14, in <module>
    sys.exit(cli())
             ~~~^^
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-dq9xt44k/pipx/main.py", line 1149, in cli
    return run_pipx_command(parsed_pipx_args, subparsers)
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-dq9xt44k/pipx/main.py", line 267, in run_pipx_command
    return commands.install(
           ~~~~~~~~~~~~~~~~^
        None,
        ^^^^^
    ...<13 lines>...
        python_flag_passed=python_flag_passed,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-dq9xt44k/pipx/commands/install.py", line 95, in install
    venv.install_package(
    ~~~~~~~~~~~~~~~~~~~~^
        package_name=package_name,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...
        suffix=suffix,
        ^^^^^^^^^^^^^^
    )
    ^
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-dq9xt44k/pipx/venv.py", line 265, in install_package
    self.update_package_metadata(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        package_name=package_name,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...
        suffix=suffix,
        ^^^^^^^^^^^^^^
    )
    ^
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-dq9xt44k/pipx/venv.py", line 363, in update_package_metadata
    venv_package_metadata = self.get_venv_metadata_for_package(package_name, get_extras(package_or_url))
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-dq9xt44k/pipx/venv.py", line 348, in get_venv_metadata_for_package
    venv_metadata = inspect_venv(package_name, package_extras, self.bin_path, self.python_path, self.man_path)
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-dq9xt44k/pipx/venv_inspect.py", line 285, in inspect_venv
    app_paths_of_dependencies, man_paths_of_dependencies = _dfs_package_resources(
                                                           ~~~~~~~~~~~~~~~~~~~~~~^
        root_dist,
        ^^^^^^^^^^
    ...<3 lines>...
        man_paths_of_dependencies,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-dq9xt44k/pipx/venv_inspect.py", line 146, in _dfs_package_resources
    dependencies = get_package_dependencies(dist, package_req.extras, venv_inspect_info.env)
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-dq9xt44k/pipx/venv_inspect.py", line 61, in get_package_dependencies
    if req.marker.evaluate(eval_env):
       ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-dq9xt44k/packaging/markers.py", line 325, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-dq9xt44k/packaging/markers.py", line 225, in _evaluate_markers
    groups[-1].append(_eval_op(lhs_value, op, rhs_value))
                      ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-dq9xt44k/packaging/markers.py", line 183, in _eval_op
    return spec.contains(lhs, prereleases=True)
           ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-dq9xt44k/packaging/specifiers.py", line 552, in contains
    normalized_item = _coerce_version(item)
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-dq9xt44k/packaging/specifiers.py", line 28, in _coerce_version
    version = Version(version)
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-dq9xt44k/packaging/version.py", line 202, in __init__
    raise InvalidVersion(f"Invalid version: '{version}'")
packaging.version.InvalidVersion: Invalid version: '3.13.0rc1+'

When I dive into the debugger, I see current_environment still has a python_full_version='3.13.0rc1+':

> /var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-pdyel8eu/packaging/markers.py(325)evaluate()
-> return _evaluate_markers(self._markers, current_environment)
(Pdb) self._markers
[(<Variable('python_full_version')>, <Op('<')>, <Value('3.10.2')>)]
(Pdb) current_environment
{'implementation_name': 'cpython', 'implementation_version': '3.13.0c1', 'os_name': 'posix', 'platform_machine': 'arm64', 'platform_release': '23.6.0', 'platform_system': 'Darwin', 'platform_version': 'Darwin Kernel Version 23.6.0: Mon Jul 29 21:14:30 PDT 2024; root:xnu-10063.141.2~1/RELEASE_ARM64_T6030', 'python_full_version': '3.13.0rc1+', 'platform_python_implementation': 'CPython', 'python_version': '3.13', 'sys_platform': 'darwin', 'extra': ''}

It seems pipx is passing in their own eval_env and that env doesn't get the repair treatment.

[jaraco]

I see now the fix in #802 specifically retained the "InvalidVersion" behavior for user-provided environments. Does that mean that this issue needs to be addressed by pipx and every other upstream caller that's passing in an environment? That seems rather unsustainable.

[brettcannon]

Does that mean that this issue needs to be addressed by pipx and every other upstream caller that's passing in an environment? That seems rather unsustainable.

I think it was just an oversight in the PR and not intentional.