Goal 'update-admin-pip-requirements' fails on Ubuntu 20.04

[gonzalo-bulnes]

Description

When updating development dependencies, the guidelines recommend running make update-pip-requirements, but the goal update-admin-pip-requirements fails on Ubuntu 20.04.

Steps to Reproduce

From a fresh Ubuntu 20.04:

1. Install the development dependencies following the Documentation Guidelines
2. Imagine that you need to upgrade a Python dependency (no need to actually do it, but it is a plausible scenario)
3. Follow the instructions to upgrade Python dependencies, that is: run make update-pip-requirements
4. Alternatively, you can run make update-admin-pip-requirements because that's where the error happens (relevant code: main Makefile, admin Makefile)

Expected Behavior

A requirements file is updated (if you actually upgraded dependencies), or at least there are no errors.

Actual Behavior

cd securedrop
make update-admin-pip-requirements

███ Updating admin pip requirements...
Traceback (most recent call last):
  File "/opt/admin/.venv3/bin/pip-compile", line 5, in <module>
    from piptools.scripts.compile import cli
  File "/opt/admin/.venv3/lib/python3.5/site-packages/piptools/scripts/compile.py", line 11, in <module>
    from .._compat import InstallCommand, install_req_from_line, parse_requirements
  File "/opt/admin/.venv3/lib/python3.5/site-packages/piptools/_compat/__init__.py", line 7, in <module>
    from .pip_compat import (
  File "/opt/admin/.venv3/lib/python3.5/site-packages/piptools/_compat/pip_compat.py", line 40, in <module>
    PackageFinder = do_import("index", "PackageFinder")
  File "/opt/admin/.venv3/lib/python3.5/site-packages/piptools/_compat/pip_compat.py", line 24, in do_import
    return getattr(imported, package)
AttributeError: module 'pip._internal.index' has no attribute 'PackageFinder'
make[1]: *** [Makefile:10: update-pip-requirements] Error 1
make: *** [Makefile:26: update-admin-pip-requirements] Error 2

Comments

More context

Two things happen successively when running make update-admin-pip-requirements:

1. A Docker image is built that contains Python dependencies. Specifically: pip and pip-tools (that provides pip-compile)
2. A container is created from that image and a pip-compile command is run that updates a requirements file (that is the goal of make update-admin-pip-requirements)

Note: there are two requirements files to update, so all this happens twice, but it is not relevant to the issue.

Troubleshooting

The pip-tools package apparently relies on some APIs that are internal to pip and because of that is sensitive to the version of pip that is installed.

Currently, pip v20.1.1 and pip-tools v4.0.0 are installed in the Docker image.

💡 I can confirm that adding RUN pip3 install --upgrade pip-tools line 22 of the admin/Dockerfile solves the issue.
And I can see that the issue seems to have happened before? :P

Next steps

I may be mistaken, but I am under the impression that these may be production dependencies. I am happy to look into how the issue was resolved last time (I see that the code that fixed the issue I linked to was removed since), but I'd like more context on the side-effects to watch for:

• Are these indeed production dependencies?
• pip3 install --upgrade pip-tools is fairly blunt as an upgrade, what do we need to take into account to decide on a version?
• Does this indeed seem like the plausible root cause, or does anyone think there may be more to it?

[gonzalo-bulnes]

This may be relevant to #4768.