intersphinx overrides for local inventories w/o modifying source code

[warsaw]

There are many build environments where downstream consumers of upstream projects are constrained from accessing the internet. Examples include building projects for Debian and Ubuntu. The build daemons for these projects have no access to the internet.

When a project built in these environments, remote inventories are inaccessible, which can cause a variety of problems, including broken builds, long timeouts, and incorrect documentation.

The solutions for this have been ad-hoc, but generally involve patching the upstream project's conf.py file to point the intersphinx_mapping to inventories on the local file system. For example, in Debian, there is a package that provides the objects.inv for Python's documentation, and the Debian version of packages which reference this can carry deltas against upstream that point to the local objects.inv file instead of trying to get them from docs.python.org.

This is suboptimal for a variety of reasons. Because the solution is ad-hoc and cannot be applied globally, many packages just don't do it. It also means that the packages that do make this adjustment have to carry patches against upstream, and those patches can easily get out of date for new upstream releases. They are a maintenance burden and a tech debt.

I'd like to find a more general solution to the problem, one that would cover downstream build environments without requiring every affected package to be patched. I'm very open to discussing solutions which make sense for Sphinx itself, and downstream consumers. I might even be able to provide a PR if we can come up with a good design.

We need to handle multiple keys, but generally each key will point to the same local file. E.g. if the original file contains:

intersphinx_mapping = {'python': ('http://docs.python.org/', None)}

we'll want the effect of the override to be say

intersphinx_mapping = {'python': ('http://docs.python.org/', '/usr/share/docs/python3.5/html/objects.inv')}

for all packages. Similarly if it were

intersphinx_mapping = {
    'python': ('http://docs.python.org/dev', None),
    'kombu': ('http://kombu.readthedocs.org/en/latest/', None),
    'djcelery': ('http://django-celery.readthedocs.org/en/latest', None),
    'cyme': ('http://cyme.readthedocs.org/en/latest', None),
    'amqp': ('http://amqp.readthedocs.org/en/latest', None),
}

we'd want the global effects to be equivalent to

intersphinx_mapping = {
    'python': ('http://docs.python.org/dev', '/usr/share/docs/python3.5/html/objects.inv'),
    'kombu': ('http://kombu.readthedocs.org/en/latest/', '/usr/share/doc/python-kombu-doc/html/objects.inv'),
    'djcelery': ('http://django-celery.readthedocs.org/en/latest', '/usr/share/doc/python-djcelery-doc/html/objects.inv'),
    'cyme': ('http://cyme.readthedocs.org/en/latest', '/usr/share/doc/python-cyme-doc/html/objects.inv'),
    'amqp': ('http://amqp.readthedocs.org/en/latest', '/usr/share/doc/python-amqp-doc/html/objects.inv'),
}

It may also be the case that some of those mappings don't exist locally, so we'd want to be able to remove keys from the intersphinx_mapping (or at least tell Sphinx "don't try to hit the internet to resolve the objects.inv for this key).

So maybe an environment variable that is something like INTERSPHINX_MAPPINGS_FILE that we could point to a file like debian/intersphinx_conf.py. That file would contain the override mapping above.

Thoughts?

[tk0miya]

How about replacing conf.py with debian's? conf.py is simply python script. So you can override specific variables in conf.py of debian's:

# read settings from original conf.py
with open('/path/to/original/conf.py`) as original_config:
    exec(original_config.read())

# override intersphinx_mapping
intersphinx_mapping = {
   ...
}

And then, sphinx-build allows to specify the location of conf.py with -c option. I think this might work fine.

[tk0miya]

I'm closing this now because it seems no updates.
Feel free to reopen this if you still have this problem.

Thanks,