Deprecate bdist_dumb, bdist_rpm, bdist_dmg, bdist_msi, bdist_wininst

[hugovk]

Warehouse is planning on removing upload support for bdist_dumb, bdist_rpm, bdist_dmg, bdist_msi, and bdist_wininst:

• https://www.python.org/dev/peps/pep-0527/
• Fully implement PEP 527 pypi/warehouse#6792

CPython's distutils deprecated bdist_wininst in 3.8:

• bpo-37481: Deprecate distutils bdist_wininst command python/cpython#14553
• https://discuss.python.org/t/deprecate-bdist-wininst/1929

And is planning on removing bdist_wininst in 3.9:

• bpo-39541: Remove distutils bdist_wininst command python/cpython#18329
• https://discuss.python.org/t/remove-distutils-bdist-wininst-command/3115

In this latter discussion, @pganssle said:

I think the right thing to do here would be to propose removing this [bdist_wininst] in setuptools, and drop the idea of removing it in distutils. The same goes for bdist_rpm and all the other weird non-wheel binary distributions.

---

What's necessary to deprecate these in setuptools?

PR #1764 deprecates bdist_egg. A similar pattern could be followed to add warnings in run() in bdist_rpm.py and bdist_wininst.py. Does this sound right?

But I'm less clear on bdist_dumb, bdist_dmg and bdist_msi. Are these handled by setuptools, and if so, where?

---

• bdist_dmg
• bdist_dumb
• bdist_msi
• bdist_rpm Deprecate bdist_rpm #2780
• bdist_wininst Deprecate bdist_wininst #2040

[pganssle]

Thanks for kicking off this process, Hugo!

I think we can follow the road map from setup.py test and setup.py upload/register:

1. Add deprecation warnings for 9-12 months (with tests).
2. Document the deprecation.
3. During removal time, add new sub-commands that actively raise errors (rather than simply removing the commands from setuptools).

But I'm less clear on bdist_dumb, bdist_dmg and bdist_msi. Are these handled by setuptools, and if so, where?

setuptools is a wrapper around distutils (and hopefully eventually distutils will move into setuptools), so anything that's setuptools doesn't modify / monkey-patch just is passed through to distutils. The two options we have for deprecating bdist_dumb et al would be creating our own sub-command that raises a deprecation warning or monkey-patching the relevant commands in distutils. Normally monkey-patching is somewhat abhorrent to me, but in this case I'm inclined towards monkey-patching so that the warnings hit the greatest number of people (e.g. people who for some reason or another are subclassing distutils.command.dumb or whatever).

It would probably be nice for us to prepare a blog post or some other similarly "loud" announcement of these forthcoming removals. Probably we can do that during the deprecation period, though.

[Cimbali]

Should these commands be removed from setuptools/distutils? I understand that these formats are not going to be supported on PyPI anymore − however this is based on PyPI download statistics.

Arguably, building binary releases in non-wheel binary formats is still being used for different distribution channels.

In fact, I find centralising all the build meta-data in setup.cfg to build on all platforms with different commands extremely useful, rather than needing different processes, configurations, etc. for every targeted release platform.

bdist_wininst seemed to have additional implementation issues from what I read in the linked discussions. The other formats (rpm, dmg, msi) are not discussed. What’s the rationale for deprecating those?

[hugovk]

PEP 527 has rationale for removing those formats from PyPI, and not only based on downloads.

As of today, PyPI has now fully implemented PEP 527:

• Issue Fully implement PEP 527 pypi/warehouse#6792
• PR Fully implement PEP 527 pypi/warehouse#7529

[Cimbali]

The rationale of PEP 527 is:

• bdist_dumb is not usable
• maintaining the non-wheel binary formats on PyPI is undesirable
  • as it has a significant bandwidth cost,
  • and because other distribution channels do it better.
• the impact of removing such formats from PyPI should be low, because
  • most of these are barely used,
  • the only format significant download traffic on these formats (bdist_wininst) actually is due at 90% from mirrorring

and then a discussion of non-binary format and extensions.

Since the commands associated with these formats are also used to build packages that are not shipped through PyPI, it makes the impact of such a deprecating these commands harder to gauge than centralized download statistics. I see nothing in there that straightforwardly implicates commands should be removed from setuptools, except for bdist_dumb.

---

In the end, I’m only asking to be enlightened on the pros and cons of removing the bdist_rpm, bdist_dmg, bdist_msi, bdist_wininst commands from setuptools.

• Are they a significant maintenance overhead? Is there another cost in keeping them around?

• Am I missing the point and is setuptools only meant to be an interface to/from PyPI? (The host of other commands such as build_sphinx or flake8 would indicate not.)

• Is there available data on how widely they are (un)used?

• Is it likely useful alternatives can be found?

  For example to build rpms, the only alternative short of writing spec files by hand is pyp2rpm, which is both much less equipped (can’t even specify dependencies outside of other python packages) and apparently in need of new maintainers. I haven’t had the time to take a deeper look at pynsist yet, but that seems an option to replace bdist_wininst.

• etc.

[jaraco]

* Am I missing the point and is setuptools only meant to be an interface to/from PyPI? (The host of other commands such as `build_sphinx` or `flake8` would indicate not.)

The primary motivation here is that Setuptools should no longer be a general-purpose packaging tool, but should focus on being a best-in-class build backend for the Python Packaging ecosystem, based on the standards set forth by the PyPA. To that end, Setuptools is seeking to deprecate setup.py <anything> and instead support the PEP 517 build backends, which only target building wheels. So yes, I would expect to discourage/deprecate usage of distutils commands such as build_sphinx and flake8.

The essential problem with supporting bdist_rpm is that it requires every Python project to adopt Setuptools to have RPM support.

Instead, probably what needs to happen is for someone to build a tool to convert wheels to RPMs. Then, that tool could generate an RPM from any wheel, including those built by other backends. Maybe that's what pyp2rpm does.

[Cimbali]

Thanks for the answer @jaraco

Setuptools should no longer be a general-purpose packaging tool

That makes a lot more sense than quoting PEP 527. I’m not keen on that direction, as I really valued the extensible single-entry-point interface that setuptools provided, but so be it.

Note that to your point about bdist_rpm, this deprecation doesn’t make it more possible to have RPM support without setuptool’s bdist_rpm. We go from 3 options: do it manually / use setuptools / roll your own tool, to just 2: do it manually / roll your own tool.

Have you also considered what this implies for packages that extend setuptools to provide packaging support, e.g. cx_Freeze? These are doing the work you’re no longer wanting to do with bdist_msi etc. so I think they are valuable people to talk to and give advance notice of what will happen when the deprecations are enacted.

• Will they still be able to extend setup() with their own commands? Or will their whole process have to be changed in the future to build from wheels?

If the extensibility remains, such packages will likely have to duplicate all the bdist_msi and similar code they’re extending now to take ownership of maintaining that. That’s a ton of work that probably shouldn’t be asked if in the end it’s useless and they should write a wheel-to-something converter from scratch instead.

[abravalheri]

It's been 2 years since we have start issuing deprecation warnings for bdist_rpm.

Should we remove it for good in November?

CC @jaraco

[twmr]

What are the plans for setuptools once the bdist_* commands are removed from the code-base? Does it make sense that versions of the bdist_* commands are maintained by the community outside of setuptools repo? I mean will setuptools keep providing the infrastructure for such out-of-tree maintained bdist commands?

[abravalheri]

Hi @Thisch, the plan in setuptools for the last (many) years is to phase out any CLI tool capabilities. This means that the intended public API is mostly related to standardised build backend hooks and the entry_points meant for plugins. We haven't fully discussed the future of the public API, but the items I mentioned are what I would call the planned A-tier support.