mirrors: Make targets_path and metadata_path optional

[jku]

Now clients can leave out targets_path or metadata_path if the
client knows the mirror does not have that type of targets.

This is backwards compatible: old mirror configs continue to work.

Fixes #1079.

Signed-off-by: Jussi Kukkonen [email protected]

---

PR notes:

This is a minor improvement in the warehouse case where one mirror has metadata and targets, and the other mirror has only targets.

Note that this leaves the issue of using os.path.join() on URLs as is: Fixing #1077 in a provably backwards compatible way seems tricky.

I believe this is a good if minor change... but I'll make the case against merging as well:

• the usefulness in pip/warehouse is minimal: the directory confinement is not expressive enough to be useful so pip has to use two separate mirror configurations anyway -- this should mean that in normal operation no requests will be made to the 'wrong' mirror: I believe they would only happen if the metadata mirror does not respond to a request for e.g. a specific targets bin file. In that case the wrong mirror would be asked as well -- this would be an annoyance, not a security issue
• it can be argued that the correct solution is to make directory confinement more powerful instead:
  • it should work on both metadata and targets
  • it should be expressive enough to say "allow everything under this directory"

Unfortunately improving directory confinement is likely to be a breaking change.

Please verify and check that the pull request fulfills the following
requirements:

• The code follows the Code Style Guidelines
• Tests have been added for the bug fix or new feature
• Docs have been added for the bug fix or new feature

[jku]

In case it's relevant, these are the configs I keep currently switching in pip depending on what I'm downloading:

config 1 (for downloading index file targets from pypi.org):

        {
            'https://pypi.org': {
                'url_prefix': 'https://pypi.org',
                'metadata_path': 'tuf/',
                'targets_path': 'simple/',
                'confined_target_dirs': ['']
            }
        }

config 2 (for downloading distribution target files from pythonhosted.org):

        {
            'https://pypi.org': {
                'url_prefix': 'https://pypi.org',
                'metadata_path': 'tuf/',
                'targets_path': 'not_real_path/', # TODO should not exist
                'confined_target_dirs': []
            },
            'https://files.pythonhosted.org/packages': {
                'url_prefix': 'https://files.pythonhosted.org/packages',
                'metadata_path': 'not_real_path', # TODO should not exist
                'targets_path': '',
                'confined_target_dirs': ['']
            }
        }

[jku]

directory confinement is not expressive enough to be useful so pip has to use two separate mirror configurations anyway

Actually, it's possible that even a very expressive directory confinement might not be able to handle the warehouse setup in a single config... I'd have to really think the solution through (but maybe it's not needed for this PR)

[MVrachev]

Аside from my comment LGTM.

[joshuagl]

Prior to this change a check_match() call will ensure that both metadata_path and targets_path are set. But following this change it's possible to pass an object conformant to MIRROR_SCHEMA with neither value set.

I'm not sure whether we should worry too much about this, it should be fairly obvious to a client if they misconfigure by not setting either value?

[jku]

This is true, does not seem like a major issue to me either.

[joshuagl]

LGTM, thanks!