pip failure on Tails 4.5 for securedrop-admin

[kushaldas]

Description

On a fresh Tails 4.5, I was trying to run the following command (branch is 1.2.0), ./securedrop-admin setup, it failed as pip could not connect to the pypi server.

Steps to Reproduce

• create a fresh tails 4.5, remove any previous admin/.vevn3 path
• ./securedrop-admin setup

Expected Behavior

Setup should finish with all the dependencies installed in admin/.venv3/ virtual environment.

Actual Behavior

Processing triggers for man-db (2.8.5-2) ...
Processing triggers for libc-bin (2.28-10) ...
[INFO] Examining package changes
[INFO] New packages manually installed: {'python3-pip', 'ccontrol', 'libffi-dev', 'python3-virtualenv', 'libpython3-dev', 'virtualenv', 'libssl-dev'}
INFO: Setting up virtualenv
INFO: Checking Python dependencies for securedrop-admin
ERROR: Failed to install pip dependencies. Check network connection and try again.

This kept happening till manually ran the pip command with torrify.

Comments

Suggestions to fix, any other relevant information.

[redshiftzero]

possibly related to #4751

[kushaldas]

Closing as the issue was in my side.

[zenmonkeykstop]

I'm seeing this now - how did you resolve it on your side?

[zenmonkeykstop]

running ./securedrop-admin -v setup i 1.3.0-rc1 I see:

...
INFO: Setting up virtualenv
DEBUG: b"Already using interpreter /usr/bin/python3\nUsing base prefix '/usr'\nNew python executable in /home/amnesia/Persistent/securedrop/admin/.venv3/bin/python3\nAlso creating executable in /home/amnesia/Persistent/securedrop/admin/.venv3/bin/python\nInstalling setuptools, pkg_resources, pip, wheel...done.\n"
INFO: Checking Python dependencies for securedrop-admin
DEBUG: b'WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by \'ReadTimeoutError("HTTPSConnectionPool(host=\'pypi.org\', port=443): Read timed out. (read timeout=15)")\': /simple/ansible/\nCollecting ansible==2.9.7\n  Using cached ansible-2.9.7.tar.gz (14.2 MB)\nCollecting asn1crypto==0.24.0\n  Using cached asn1crypto-0.24.0-py2.py3-none-any.whl (101 kB)\nCollecting cffi==1.11.4\n  Using cached cffi-1.11.4.tar.gz (436 kB)\nWARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by \'ReadTimeoutError("HTTPSConnectionPool(host=\'pypi.org\', port=443): Read timed out. (read timeout=15)")\': /simple/cryptography/\n1588186507 ERROR torsocks[10515]: Unable to resolve. Status reply: 4 (in socks5_recv_resolve_reply() at socks5.c:677)\nWARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by \'NewConnectionError(\'<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7eb778fd2cf8>: Failed to establish a new connection: [Errno -4] Non-recoverable failure in name resolution\')\': /simple/cryptography/\nCollecting cryptography==2.7\n  Downloading cryptography-2.7-cp34-abi3-manylinux1_x86_64.whl (2.3 MB)\nERROR: Exception:\nTraceback (most recent call last):\n  File "/home/amnesia/Persistent/securedrop/admin/.venv3/lib/python3.7/site-packages/pip/_vendor/urllib3/response.py", line 425, in _error_catcher\n    yield\n  File "/home/amnesia/Persistent/securedrop/admin/.venv3/lib/python3.7/site-packages/pip/_vendor/urllib3/response.py", line 507, in read\n    data = self._fp.read(amt) if not fp_closed else b""\n  File "/home/amnesia/Persistent/securedrop/admin/.venv3/lib/python3.7/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 62, in read\n    data = self.__fp.read(amt)\n  File "/usr/lib/python3.7/http/client.py", line 457, in read\n    n = self.readinto(b)\n  File "/usr/lib/python3.7/http/client.py", line 501, in readinto\n    n = self.fp.readinto(b)\n  File "/usr/lib/python3.7/socket.py", line 589, in readinto\n    return self._sock.recv_into(b)\n  File "/usr/lib/python3.7/ssl.py", line 1052, in recv_into\n    return self.read(nbytes, buffer)\n  File "/usr/lib/python3.7/ssl.py", line 911, in read\n    return self._sslobj.read(len, buffer)\nsocket.timeout: The read operation timed out\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "/home/amnesia/Persistent/securedrop/admin/.venv3/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 188, in _main\n    status = self.run(options, args)\n  File "/home/amnesia/Persistent/securedrop/admin/.venv3/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 185, in wrapper\n    return func(self, options, args)\n  File "/home/amnesia/Persistent/securedrop/admin/.venv3/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 333, in run\n    reqs, check_supported_wheels=not options.target_dir\n  File "/home/amnesia/Persistent/securedrop/admin/.venv3/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 179, in resolve\n    discovered_reqs.extend(self._resolve_one(requirement_set, req))\n  File "/home/amnesia/Persistent/securedrop/admin/.venv3/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 362, in _resolve_one\n    abstract_dist = self._get_abstract_dist_for(req_to_install)\n  File "/home/amnesia/Persistent/securedrop/admin/.venv3/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 314, in _get_abstract_dist_for\n    abstract_dist = self.preparer.prepare_linked_requirement(req)\n  File "/home/amnesia/Persistent/securedrop/admin/.venv3/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 412, in prepare_linked_requirement\n    hashes=hashes,\n  File "/home/amnesia/Persistent/securedrop/admin/.venv3/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 198, in unpack_url\n    hashes=hashes,\n  File "/home/amnesia/Persistent/securedrop/admin/.venv3/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 124, in get_http_url\n    link, downloader, temp_dir.path, hashes\n  File "/home/amnesia/Persistent/securedrop/admin/.venv3/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 220, in _download_http_url\n    for chunk in download.chunks:\n  File "/home/amnesia/Persistent/securedrop/admin/.venv3/lib/python3.7/site-packages/pip/_internal/cli/progress_bars.py", line 166, in iter\n    for x in it:\n  File "/home/amnesia/Persistent/securedrop/admin/.venv3/lib/python3.7/site-packages/pip/_internal/network/utils.py", line 39, in response_chunks\n    decode_content=False,\n  File "/home/amnesia/Persistent/securedrop/admin/.venv3/lib/python3.7/site-packages/pip/_vendor/urllib3/response.py", line 564, in stream\n    data = self.read(amt=amt, decode_content=decode_content)\n  File "/home/amnesia/Persistent/securedrop/admin/.venv3/lib/python3.7/site-packages/pip/_vendor/urllib3/response.py", line 529, in read\n    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)\n  File "/usr/lib/python3.7/contextlib.py", line 130, in __exit__\n    self.gen.throw(type, value, traceback)\n  File "/home/amnesia/Persistent/securedrop/admin/.venv3/lib/python3.7/site-packages/pip/_vendor/urllib3/response.py", line 430, in _error_catcher\n    raise ReadTimeoutError(self._pool, None, "Read timed out.")\npip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host=\'files.pythonhosted.org\', port=443): Read timed out.\n'
ERROR: Failed to install pip dependencies. Check network connection and try again.

[zenmonkeykstop]

Also resolved locally (by bouncing Tor repeatedly). Closing, but if access to pypi.org over Tor is getting flaky it might be worth increasing the number of attempts and or length of timeouts for these requests.