Always install the latest pip version

[stsewd]

Currently, the pip version comes from our docker images, since we don't release then too often, the pip version gets outdated very fast.

Here @humitos proposed to install the latest pip as a build step #4808 (comment)

I already have some code that makes that real (a description here #4808 (comment))

Are you agree with this? Should we add a feature flag first? I don't think any project depends on a specific pip version, or at least to install the packages if they depend on a pip version to build their docs, specifying that on their requirements will work.

[humitos]

I don't think any project depends on a specific pip version

I agree with this.

if they depend on a pip version to build their docs, specifying that on their requirements will work.

This is not true. If I need (for some reason) pip==9.0.0 and I pin it in my requirements, all the requirements will be installed with the pip version already installed in the virtualenv (18.0.0, for example).

At the end of the command execution, all my packages were installed with pip==18.0.0 and now I have pip==9.0.0 available on the virtualenv, but it was not used to install anything.

[humitos]

Should we add a feature flag first?

I don't see too much value here but a precaution. I mean, depending on a not pinned package is kind of dangerous because it's something that we don't control and could be anything.

Also, if the new release of pip breaks, all of our builds will break. I think it's preferable to add a Feature flag here that will install the latest pip by default for all the projects. If we notice that this fails for some reason, we can easily go back by picking those packages that are failing.

[stsewd]

At the end of the command execution, all my packages were installed with pip==18.0.0 and now I have pip==9.0.0 available on the virtualenv, but it was not used to install anything.

What I mean, the project itself may depend on pip, as a dependency, not to install their packages

[stsewd]

Also, if the new release of pip breaks, all of our builds will break. I think it's preferable to add a Feature flag here that will install the latest pip by default for all the projects. If we notice that this fails for some reason, we can easily go back by picking those packages that are failing.

What about pin pip and then updated it on each release?

[humitos]

What about pin pip and then updated it on each release?

Yes. I think this is the best approach.

It has been easier updating some Python code than the docker image (I would like to release more frequently, though).

So, I propose to run

pip install -U pip==18.1

as the first step before creating the virtualenv (this can be added in the VirtualEnvironment class as a method and run it first). That 18.1 can be updated just by changing the code (similar to other reqs that we install in the venv after creating it)

Also, I'd add a feature flag around that pinning so we can mark some project to always upgrade to the latest without pinning for example.

Once there, we can take another different approach. Maybe better, but we have something already working that does not require tons of work.

[stsewd]

as the first step before creating the virtualenv

I wasn't able to do that #4808 (comment)

I wasn't able to install the latest version of pip before the virtualenv creation, because the command will need to be run with sudo (we don't have sudo anyway...). We could use this option https://virtualenv.pypa.io/en/stable/userguide/#the-extra-search-dir-option or more easy, create the virtual env with the old pip version, but update the pip version after the virtualenv creation, that way we are able to install the core and user requirements using the latest pip.

[humitos]

Either way works for me.

Using --extra-search-dir is more robust to me, but there is a small problem with it that is we will need to download first the new version of pip/setuptools first. So, it will be,

pip download pip setuptools

and then,

python -m virtualenv --extra-search-dir .

Upgrading pip after creating the virtualenv env is easier, but if the first pip is broken for whatever reason, we won't be able to upgrade it.

[stsewd]

Upgrading pip after creating the virtualenv env is easier, but if the first pip is broken for whatever reason, we won't be able to upgrade it.

If the first pip is broken all docs will be broken anyway and --extra-search-dir wouldn't work because pip download shouldn't work either p:

I can say that it will be easier to see what's going on for the user if we install pip after the env creation.