Preserve original PKG-INFO contents when creating wheel (instead of calling wheel.metadata.pkginfo_to_metadata)

[abravalheri]

This seems to be a necessary step towards implementing new versions of metadata (e.g. PEP 685).

As pointed out in #4698, even if we modify setuptools to emit new versions of the metadata, pypa/wheel still hardcodes version 2.1, which can be problematic.

The good news is that recently setuptools has started emitting valid and complete 2.1 metadata on its own (see setuptools/_core_metadata.py). This PR assumes that it is OK just to reuse the file generated by setuptools without any need to post-process it in bdist_wheel.

Summary of changes

• Strengthening of test_core_metadata to ensure the generated PKG-INFO is compatible with standards for METADATA.
  • Add tests for the generated PKG-INFO of real-world configuration scenarios, makimg sure they are compatible with wheel.metadata.pkginfo_to_metadata.
  • Add tests for the generated PKG-INFO to make sure they "round-trip" when subject to the parsing and serialisation methods used by pypa/wheel.
• Modify setuptools.command.bdist_wheel to simply re-use the already generated PKG-INFO.

Closes

Limitations

There are "minor" differences between METADATA generated by pypa/wheel and PKG-INFO generated by setuptools:

• The order of some headers (trivial)
• The normalisation of requirements

I believe that an acceptable level of normalisation, fully compliant with the spec, will be "automatically" achieved once pypa/packaging#845 and pypa/packaging#644 (comment) are handled¹.

Pull Request Checklist

• Changes have tests
• News fragment added in newsfragments/.
  (See documentation for details)

Footnotes

1. Right now, the specs mandate the extras to be normalised, but not the name. Setuptools relies on packaging for the normalisation, so once the potential bug in https://github.com/pypa/packaging/issues/845 is solved, it will be compliant to the spec. If packaging introduces names normalisation we will also adopt it. ↩

[abravalheri]

@agronholm do you have any reservations regarding this approach?

[agronholm]

If I understood this correctly, then it's fine by me. It's about time we start ditching the awkward conversion code from egg-info metadata. The only thing preventing me from advancing this myself is mental exhaustion.

[abravalheri]

I will wait to see the assessment on #4698 before merging the PR.

[agronholm]

Given that the bdist_wheel implementation on pypa/wheel is no longer actually being used (unless you have an old version of setuptools but upgrade only wheel), wouldn't it make sense to just skip the PKG-INFO creation altogether and just emit the proper 2.4 metadata directly?

[abravalheri]

Given that the bdist_wheel implementation on pypa/wheel is no longer actually being used (unless you have an old version of setuptools but upgrade only wheel), wouldn't it make sense to just skip the PKG-INFO creation altogether and just emit the proper 2.4 metadata directly?

There are 2 aspects to this question. The first one is the existence of .egg-info and all files that exist within it... At this point in time .egg-info is very intertwined with the rest of the setuptools implementation, and although there is ambition to remove it eventually, for the time being it seems to difficult/risky.

The second aspect is that this PR in some way is a step towards generating proper 2.4 metadata directly. What I plan in #4698 is to change setuptools._core_metadata.write_pkg_file so that it is produces proper 2.4 metadata and can be used to generate both PKG-INFO and METADATA. We need to change bdist_wheel to avoid the conversion and the hardcoded parts, so that we can use the proper metadata that is generated directly.

[abravalheri]

I will go ahead and merge this so that the follow up PRs can be rebased.

[abravalheri]

I plan to create a release that consists mostly of this change so that we can receive feedback and see if there is an unintended consequence.