Upgrade to a pre-19 pip version

[AlanCoding]

SUMMARY

Connect #3968

There is a lot of history here.

Previously, an upgrade to the latest pip was attempted, that was reverted as of #4114 and #4136, but the labeling there is confusing. Those weren't clean reverts, but eventually we settled back to the very old versions.

I have confirmed an error due to the latest pip & setuptools version which looks like:

10:30:37     ERROR: Exception:
10:30:37     Traceback (most recent call last):
10:30:37       File "/builddir/build/BUILD/ansible-tower-3.7.0/venv/ansible/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 153, in _main
10:30:37         status = self.run(options, args)
10:30:37       File "/builddir/build/BUILD/ansible-tower-3.7.0/venv/ansible/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 382, in run
10:30:37         resolver.resolve(requirement_set)
10:30:37       File "/builddir/build/BUILD/ansible-tower-3.7.0/venv/ansible/lib/python2.7/site-packages/pip/_internal/legacy_resolve.py", line 201, in resolve
10:30:37         self._resolve_one(requirement_set, req)
10:30:37       File "/builddir/build/BUILD/ansible-tower-3.7.0/venv/ansible/lib/python2.7/site-packages/pip/_internal/legacy_resolve.py", line 365, in _resolve_one
10:30:37         abstract_dist = self._get_abstract_dist_for(req_to_install)
10:30:37       File "/builddir/build/BUILD/ansible-tower-3.7.0/venv/ansible/lib/python2.7/site-packages/pip/_internal/legacy_resolve.py", line 313, in _get_abstract_dist_for
10:30:37         req, self.session, self.finder, self.require_hashes
10:30:37       File "/builddir/build/BUILD/ansible-tower-3.7.0/venv/ansible/lib/python2.7/site-packages/pip/_internal/operations/prepare.py", line 224, in prepare_linked_requirement
10:30:37         req, self.req_tracker, finder, self.build_isolation,
10:30:37       File "/builddir/build/BUILD/ansible-tower-3.7.0/venv/ansible/lib/python2.7/site-packages/pip/_internal/operations/prepare.py", line 49, in _get_prepared_distribution
10:30:37         abstract_dist.prepare_distribution_metadata(finder, build_isolation)
10:30:37       File "/builddir/build/BUILD/ansible-tower-3.7.0/venv/ansible/lib/python2.7/site-packages/pip/_internal/distributions/source/legacy.py", line 37, in prepare_distribution_metadata
10:30:37         self._setup_isolation(finder)
10:30:37       File "/builddir/build/BUILD/ansible-tower-3.7.0/venv/ansible/lib/python2.7/site-packages/pip/_internal/distributions/source/legacy.py", line 90, in _setup_isolation
10:30:37         reqs = backend.get_requires_for_build_wheel()
10:30:37       File "/builddir/build/BUILD/ansible-tower-3.7.0/venv/ansible/lib/python2.7/site-packages/pip/_vendor/pep517/wrappers.py", line 152, in get_requires_for_build_wheel
10:30:37         'config_settings': config_settings
10:30:37       File "/builddir/build/BUILD/ansible-tower-3.7.0/venv/ansible/lib/python2.7/site-packages/pip/_vendor/pep517/wrappers.py", line 255, in _call_hook
10:30:37         raise BackendUnavailable(data.get('traceback', ''))
10:30:37     BackendUnavailable
10:30:37     make: *** [requirements_ansible] Error 2

This is related to pypa/pip#6164 and pypa/setuptools#1644 which I do not want to go within a mile of.

Pip version 19.0.0 was called out specifically in those issues. As such, dipping down to latest 18.x seems reasonable. Versions selected by inspecting history:

pip
19.0    Jan 22, 2019
18.1    Oct 5, 2018

setuptools
40.7.0  Jan 27, 2019
40.6.3  Dec 11, 2018

So I'm going to upgrade, pretending that today is Jan 21, 2019.

This is still almost exactly what the partial rollback did in #4114. However, there are several notable differences from there to here:

• the versions in Makefile are set to the same (conservative) versions
• setuptools has a slightly lower version, and there is reason to think it might play better
• the pip version in one of the requirements file was still >19

Importantly, I have not been able to replicate any of the fallout with this PR. It fixed issues I saw with the higher versions, so it would be reasonable to speculate that the issue was caused by the versions in the Makefile.

The hope is that this gets us past the old error:

[ec2-user@ip-10-0-14-230 ~]$ sudo /var/lib/awx/venv/awx/bin/pip --help
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/bin/pip", line 7, in <module>
    from pip import main
ImportError: cannot import name 'main'

But I still have yet to double and triple check that, and we should hold off merge until that time.

ISSUE TYPE

• Feature Pull Request

COMPONENT NAME

• API

AWX VERSION

9.0.1

[softwarefactory-project-zuul]

Build succeeded.

• awx-api-lint : SUCCESS in 9m 13s
• awx-api : SUCCESS in 15m 54s
• awx-ui : SUCCESS in 12m 13s
• awx-ui-next : SUCCESS in 13m 31s
• awx-swagger : SUCCESS in 18m 08s
• awx-detect-schema-change : SUCCESS in 15m 47s (non-voting)
• awx-ansible-modules : SUCCESS in 14m 06s

[AlanCoding]

Unfortunately it is still giving the

cannot import name main

error.

[AlanCoding]

I think we need to follow some form of instructions from:

pypa/pip#5599

whatever is doing any variation pip3 install pip must be stopped. This is not a supported way to do things.

Testing locally, python -m pip install --force-reinstall pip is promising.

[softwarefactory-project-zuul]

Build succeeded.

• awx-api-lint : SUCCESS in 11m 43s
• awx-api : SUCCESS in 19m 28s
• awx-ui : SUCCESS in 12m 38s
• awx-ui-next : SUCCESS in 12m 58s
• awx-swagger : SUCCESS in 17m 17s
• awx-detect-schema-change : FAILURE in 20m 12s (non-voting)
• awx-ansible-modules : SUCCESS in 12m 26s

[softwarefactory-project-zuul]

Build succeeded.

• awx-api-lint : SUCCESS in 12m 02s
• awx-api : SUCCESS in 15m 39s
• awx-ui : SUCCESS in 13m 28s
• awx-ui-next : SUCCESS in 14m 34s
• awx-swagger : SUCCESS in 17m 28s
• awx-detect-schema-change : FAILURE in 19m 56s (non-voting)
• awx-ansible-modules : SUCCESS in 11m 05s

[softwarefactory-project-zuul]

Build succeeded.

• awx-api-lint : SUCCESS in 3m 55s
• awx-api : SUCCESS in 17m 50s
• awx-ui : SUCCESS in 12m 49s
• awx-ui-next : SUCCESS in 7m 49s
• awx-swagger : SUCCESS in 9m 33s
• awx-detect-schema-change : SUCCESS in 19m 58s (non-voting)
• awx-ansible-modules : SUCCESS in 11m 31s

[softwarefactory-project-zuul]

Build succeeded.

• awx-api-lint : SUCCESS in 4m 47s
• awx-api : SUCCESS in 11m 34s
• awx-ui : SUCCESS in 6m 43s
• awx-ui-next : SUCCESS in 8m 28s
• awx-swagger : SUCCESS in 10m 54s
• awx-detect-schema-change : SUCCESS in 13m 16s (non-voting)
• awx-ansible-modules : SUCCESS in 4m 46s

[AlanCoding]

with current approaches this is no longer necessary