deprecate and remove the upload_docs command

[jaraco]

In #1988, we are exploring the deprecation of many of the setup.py invocations. Based on your experience and mine and that of RTD, I’d like to consider deprecating and removing the build_sphinx command.

Originally posted by @jaraco in #2761 (comment)

[kloczek]

If may I ask to leave that part until some replacement will be not developed.

IMO it is big difference between build_sphinx and other build commands.
I've described most of the details about that subject in my comment #2761 (comment)

IMO it is kid of need to have generic way to not only build but install documentation (depends on chosen rendered format it should be different dest location).
Other build_* commands have some generic approaches or tooling however that is not the case (still) with documentation.

[jaraco]

it is kid of need to have generic way to not only build but install documentation

Does Setuptools currently provide a way to install documentation?

I've described most of the details about that subject in my comment #2761 (comment)

If I try to distill the requirements, they boil down to:

• docs builder should be able to reference metadata from the local project.
• docs builder should allow customization of the docs source directory.
• docs builder should allow selection of backends (sphinx, mkdocs, etc) to invoke the build.

I'd also say:

• docs builder should be able to reference behavior from the local project (be installed optionally).
• docs builder should solicit dependencies needed to build, possibly from project extras.

In my experience, a simple tox environment does all of these things, while simultaneously being largely decoupled from specifics of the implementation. I can even imagine using {posargs} to allow the build to specify different formats (html/man) and publishing targets.

Admittedly, having those lines in each project is a bit of a burden. It might be nice for a tool to solicit a small subset of behaviors into declarative options in pyproject.toml (with defaults).

I contend that the tox approach, while not perfect, easily obviates the features offered by setup.py build_sphinx. Can you describe any feature that's not obviated by that approach?

[abravalheri]

Recently @webknjaz has brought to my attention that the sphinx team has already deprecated build_sphinx.

So I think there is not much left to do from setuptools perspective?

It is likely that users seeking replacement will have to call sphinx-build directly.

[kloczek]

It is likely that users seeking replacement will have to call sphinx-build directly.

Issue is that sphinx it is not only tooling used to generate documentation.

[abravalheri]

Issue is that sphinx it is not only tooling used to generate documentation.

In that scenario, I am afraid it will be necessary to analyse package by package. Hopefully most of them will have something like tox -e docs or nox -s docs available to be used, but it is currently impossible to guarantee all the packages will use the same tools and processes.

In fact, it is already currently impossible to guarantee all the packages rely on build_sphinx -- what might be happening is that most of them work "by chance" (e.g. if they follow the convention of having a docs folder at the top level). sphinx-build can be used as a replacement for those scenarios.

If you are after improving compatibility and the standards for distributing and building the docs, I think the best course of action is to bring this discussion to https://discuss.python.org/c/packaging/14. The topic https://discuss.python.org/t/graceful-cooperation-between-external-and-python-package-managers-pep-668/10302 might also interest you.

[kloczek]

Hopefully most of them will have something like tox -e docs or nox -s docs available to be used, but it is currently impossible to guarantee all the packages will use the same tools and processes

Amongst +750 packaged module as rpm package probably only 3-5 have something like that.
All tox -e docs targets have fixed generate only html output. Every such example is done with noon declarative description which is hard to propagate across more modules and allows define that in every possible way. In case at least one of those modules when I've been pointing that on generate man output there are some sphinx warning related to documentation definition maintainer refused to accept that as something possible to work on it because tox -e docs generating html output had no such warnings.
IMO using tox here is one of the worse possible solutions ..
tox is to handle something to do in venv an that is its main goal. Fact that it could be (and is actually) used to generate documentation is IMO only coincidence.

As I'm building my rpm packages in isolated envs in which is installed set of build dependencies I don't need tox wrapping on testing. Result:

[tkloczko@ss-desktop SPECS]$ echo tox; grep 'python3dist(tox)' python-*; echo nox ; grep 'python3dist(nox)' python-*;  echo; ls -1 python-*|wc -l
tox
python-dpkt.spec:BuildRequires: python3dist(tox)
python-gitlab.spec:BuildRequires:       python3dist(tox)
python-jedi.spec:BuildRequires: python3dist(tox)
python-lazy-object-proxy.spec:BuildRequires:    python3dist(tox)
python-nox.spec:BuildRequires:  python3dist(tox)
python-tox-current-env.spec:BuildRequires:      python3dist(tox)
python-twisted.spec:BuildRequires:      python3dist(tox)
python-zope-interface.spec:BuildRequires:       python3dist(tox)
nox
python-googleapis-common-protos.spec:BuildRequires:     python3dist(nox)
python-pytest-harvest.spec:#BuildRequires:      python3dist(nox)
python-yaxmldiff.spec:BuildRequires:    python3dist(nox)

760

So at the moment I'm using tox in only about 1% of all build procedures descriptions written as rpm spec file.
My target is to remove all those tox use cases because I want time to time be able perform test build for example with installed 'pytest-randomly'. I would be able do that as I've already managed to switch to use pytest in as many cases as possible. tox AFAIK does not allow to inject additional module into venv specified as tox command line parameter (please correct me if I'm wrong).

[tkloczko@ss-desktop SPECS]$ grep 'python3dist(pytest)' python-*|wc -l
643

Most of the time and other resources consuming by tox is used to prepare actual venv. With my current almost 4k rpm packages I have days when my automation on top of production builds performs even few thousands additional builds only to test that sill other packages are correctly building. From that point of view I'm carefully choosing what I'm implementing in build procedures.

And again .. tox build procedure description will be always functional/non-declarative which in longer time will be as always harder to maintain. Result sooner or later will be that most of those tox -e docs procedure will be not working.

[abravalheri]

I understand your position here, and I do value your effort in trying to bring documentation accessible via terminal (I am a huge fan of offline docs...).

However, after seeing your last comment, I am convinced that currently there is no silver bullet for your use case (build_sphinx is definitely not it...)

I encourage you to engage the community in https://discuss.python.org/c/packaging/14, hopefully the discussion can result in a new PEP that will make it easier for you to achieve your objectives.

Meanwhile, build_sphinx (which is not directly implemented by setuptools) has been deprecated by its maintainers. Hopefully you can devise a workaround based on the strategies we have discussed in #2995 (auxiliary script to find the docs folder + sphinx-build).

[webknjaz]

Action items:

• Drop calling build_sphinx from upload_docs (which only seems to be deprecated for uploading the docs to PyPI/pythonhosted but not other places):

  setuptools/setuptools/command/upload_docs.py

  Lines 45 to 62 in a4dbe34

  	def has_sphinx(self):
  	if self.upload_dir is None:
  	for ep in iter_entry_points('distutils.commands', 'build_sphinx'):
  	return True
  	sub_commands = [('build_sphinx', has_sphinx)]
  	def initialize_options(self):
  	upload.initialize_options(self)
  	self.upload_dir = None
  	self.target_dir = None
  	def finalize_options(self):
  	upload.finalize_options(self)
  	if self.upload_dir is None:
  	if self.has_sphinx():
  	build_sphinx = self.get_finalized_command('build_sphinx')
  	self.target_dir = dict(build_sphinx.builder_target_dirs)['html']

• Close this issue as the actual command implementation exists in a plugin, not here

Not sure if this needs to be documented anywhere — setuptools is a framework for producing packages, not docs.

[abravalheri]

Thank you very much @webknjaz for summarising the future steps.

Should we deprecate upload_docs entirely?

[webknjaz]

Should we deprecate upload_docs entirely?

I believe there's been an attempt in the past that was followed by some of the users giving feedback that they use it outside of the PyPI index that lead to it being kept. Maybe @jaraco remembers better.

[kloczek]

However, after seeing your last comment, I am convinced that currently there is no silver bullet for your use case (build_sphinx is definitely not it...)

That is true because still I have no full idea how to provide that kind functionality in first place :P

I'm only more or less aware that producing documentation using some standard procedure may not only simplify what I'm personally doing but it may generally improve accessibility and quality of the python modules documentation for end users/developers.
This is why I'm throwing some random (almost) ideas about how could be working from end-user point of view. If we would be able to reach some agreement how to provide that functionality we would be able to move to proposal and than implementation steps.
Just pleas don't get me wrong. I'm not assuming that I'm 100% right. All what I expect is kind of free conversation about how that kind of functionalities could be provided some civilised way :)

Good is that build_sphinx still is only marked as deprecated. It gives all sides of this talk time to prepare some alternative solution which may provide 100% actual functionality like current setuptools<>sphinx integration and could be stretched on beyond that pair of modules to provide transparent procedure to build documentation when flit or poetry is used and when different documentation generators/renderers are used.

So please don't stop whole conversation at current point. Please bring to the table some criticism or please ask/invite some people who you know that may provide some high level critique (all that without trying to write even single line of code ..)

Summarising ..
So far IMO the best point to hook that kind of functionality would be build module with probably install for installing generated documentation. From point of view declarative properties so far my idea would be additional project.tom [docs] section with content like

[docs]
source-dir = <some/doc/subdirectory>    # default should be docs
generator = {sphinx|mkdoc|whatever}

and other properties which are already documented in setuptools<>sphinx integration https://www.sphinx-doc.org/en/master/usage/advanced/setuptools.html

Is that looks valid/doable/not to much disturbing module code which will be necessary to modify?

[CAM-Gerlach]

@kloczek As several of the developers here have explained, standardizing a way to invoke building, and perhaps even packaging/installing docs (beyond just Sphinx, many projects nowadays use mkdocs or myST, and a handful use other builders) is a potentially valuable endevour worthy of discussion and listening to the needs of stakeholders like yourself (which, indeed, needs to happen before something is written and standardized). However, as they've also stated, Setuptools is moving toward focusing more tightly on being a PEP 517 build backend, rather than a jack of all trades (and master of none), and many projects are evolving to not rely on Setuptools at all (as you mentioned).

Therefore, as it likely isn't something that would be implemented by Setuptools, and if it were, it would need standardization first, the best place to have this discussion is the packaging section of the Python Discourse, the authoritative place to discuss new packaging-related standards. We'd love to see a proposal on this topic over there, and if there was sufficient interest, could help guide you toward either a PEP or perhaps other ideas/tools to achieve this. Thanks!

[kloczek]

Therefore, as it likely isn't something that would be implemented by Setuptools

I know. I'm aware of that .. 😄
In meantime trying to package all stuff around whey (yet another pep517 backend) I found sphinx-pyproject module which IMO at least could be used as some example or staring point to implement something which I've sketched above.
However IMO that kind of functionality should be integrated at least into build module, and when it comes to installation install.
Probably even better place would be pep517 module but wheel format AFAIK does not say anything about documentation so probably better point would be the build module.
If someone have any suggestions/free thoughts please drop those comments here 😄

[CAM-Gerlach]

The goal of build is to be a simple, correct PEP 517 build frontend, and installer a simple, correct wheel (PEP 427) installer, whereas this repo is that of Setuptools, a PEP 517 build backend. In order to implement it there (and in other frontends and backends), you'd first need to extend the wheel standard to support including and installing documentation (perhaps as a .data category, with a corresponding docs path added to sysconfig), and either define a new PEP or extent PEP 517/518/621 to introduce a mechanism for declaring and calling a project's docs builder, etc.

And in order to do that, you need to start a discussion and eventually propose a PEP on the Packaging section of the Python Discourse, which is a much more appropriate place to discuss this than this tangentially related issue, so I strongly suggest you take your further replies there. Thanks.

[jaraco]

I don’t see or recall any attempts to deprecate/remove upload docs, but it does seem like the right thing to do.

[pradyunsg]

I think this issue can be retitled to "Deprecate and drop upload_docs command", which is reasonably the only bit that setuptools can action on.

[CAM-Gerlach]

I can't seem to find the associated PR(s), but it appears the setuptools-actionable task may have been completed given the above-linked commits.

Also, @kloczek , you might be interested in this Packaging Discourse discussion on this topic, on which I mention this issue (among others) in my latest post.

[webknjaz]

I don’t see or recall any attempts to deprecate/remove upload docs, but it does seem like the right thing to do.

@jaraco
Well, this commit is yours: 995d309. The previous deprecation attempt was 7f81445.

There's a comment there pointing at https://bitbucket.org/hpk42/devpi/issues/388/support-rtd-model-for-building-uploading#comment-34292423 which has moved to devpi/devpi#388.

devpi/devpi#619 seems to indicate that there's an alternative way of uploading the docs to devpi through their client so this shouldn't be considered a blocker anymore.

it appears the setuptools-actionable task may have been completed given the above-linked commits.

@CAM-Gerlach
The command still exists: https://github.com/pypa/setuptools/blob/main/setuptools/command/upload_docs.py.

The deprecation in the command definition seems to only be issued for PyPI upload attempts. I think it should first become unconditional and then removed.