Builds failing with NamespacePath has no attribute sort

[jaraco]

Details

• Read the Docs project URL: https://readthedocs.org/projects/jaracoclipboard/
• Build URL (if applicable): https://readthedocs.org/projects/jaracoclipboard/builds/7996994/
• Read the Docs username (if applicable): jaraco

Expected Result

Build should succeed without error.

Actual Result

Build is failing during the virtualenv creation.

This issue started this evening sometime after this commit (which succeeded).

The code for configuring RTD is derived from the same code as many other similar projects.

I suspected that something was cached incorrectly, as this error message is due to a known issue that was fixed with later versions of pip, so I deleted and recreated the project, but the docs build is still failing.

Can you suggest a way to work around the issue?

[jaraco]

I see another project exhibiting the same error.

[stsewd]

If this is a problem with pip 9.0.1 then I'm not sure what we can do to fix this soon :/ pip comes from the docker images, so it can't be changed now, (I think we will be able to change this with the v2 of the configuration file).

So, our docker images are as follow:

• v2 (the one that you are using), comes with pip 9.0.1
• latest, comes with 9.0.3

So, if this bug is fixed on that version of pip, you can solve this problem by adding this in your rtd confifguration file:

build:
  image: latest

[humitos]

I think we will be able to change this with the v2 of the configuration file

IMHO, I think that we should force to install the latest pip version always the initial step by running:

pip install -U pip

So, if we are going to update the Docker versions more frequently, this won't have any effect. On the other hand, if we tend to make our Docker images stable and do not upgrade them very often (as we are doing now), it will always upgrade to the latest version which is the maintained one.

In the meanwhile, we could add a dated Feature flag for this so all new project start installing the latest pip.

[humitos]

If this is a problem with pip 9.0.1 then I'm not sure what we can do to fix this soon :/ pip comes from the docker images, so it can't be changed now

I was able to reproduce this issue by importing jaracocollections project into my local instance and running two builds of latest with readthedocs/build:3.0 image. So, it seems that it's not a problem of the pip version itself but with "something else".

If you take a closer look: the first command from this build https://readthedocs.org/projects/jaracocollections/builds/7963873/ is git remote set-url which means that the builder already have the code and the virtualenv is already created.

BUT after all the git commands are executed, the python -mvirtualenv command is executed again but the virtualenv is already there. I think it does fail because of this.

[humitos]

BUT after all the git commands are executed, the python -mvirtualenv command is executed again but the virtualenv is already there. I think it does fail because of this.

It shouldn't fail because of this.

We don't have any kind of check to run the command if the venv doesn't exist or to not run it if it does.

Actually, I did a simple local test:

$ python -mvirtualenv --no-site-packages --no-download /tmp/testvenv
Using base prefix '/home/humitos/.pyenv/versions/3.6.6'
New python executable in /tmp/testvenv/bin/python
Installing setuptools, pip, wheel...done.
Time: 0h:00m:03s                                                                                                                                                                              

$ python -mvirtualenv --no-site-packages --no-download /tmp/testvenv
Using base prefix '/home/humitos/.pyenv/versions/3.6.6'
New python executable in /tmp/testvenv/bin/python
Installing setuptools, pip, wheel...done.
Time: 0h:00m:03s                                                                                                                                                                              

$ 

[humitos]

In fact, by upgrading pip before running the command that creates the venv doesn't work either. I added this command

python3.6 -mpip install -U pip

and it still fail with the same problem and in the log I can see that it's still using pip 9.0.3 which I don't know why yet.

[jaraco]

I’m following loosely. Let me just say, thanks so much for the in-depth investigation.

[stsewd]

I think I found the problem.

Pip does support reading configurations from a pyprojec.toml file https://pip.pypa.io/en/stable/reference/pip/#pep-518-support, when creating the virtualenv pip installs some packages and the pyproject.toml file is read, by renaming/removing the file, I was able to build the project without issues.

So, I'm not sure what we can do here, maybe the new deps requires a new version of pip? The dep are incompatible somehow?

[stsewd]

This is weird, I'm installing the latest version of pip before the virtualenv is created, the build pass. BUT, if I build it again, the build fails... Also I noticed that the first time the latest version of pip is only downloaded, in the second time, the old pip version is uninstalled :_

[stsewd]

I think we are messing up this #4808 (comment) with the env variables

https://github.com/rtfd/readthedocs.org/blob/9b2b17c0fc603267b4b20f5923862c2db82602da/readthedocs/projects/tasks.py#L576-L583

https://github.com/rtfd/readthedocs.org/blob/9b2b17c0fc603267b4b20f5923862c2db82602da/readthedocs/doc_builder/environments.py#L353-L365

[stsewd]

So, I have som code that "fixes" this AND install the latest version of pip.

For the pyproject problem we can change the cwd of the command to other than the project's root (probably /).

I wasn't able to install the latest version of pip before the virtualenv creation, because the command will need to be run with sudo (we don't have sudo anyway...). We could use this option https://virtualenv.pypa.io/en/stable/userguide/#the-extra-search-dir-option or more easy, create the virtual env with the old pip version, but update the pip version after the virtualenv creation, that way we are able to install the core and user requirements using the latest pip.

[humitos]

I'm a little lost here. Can you explain in simple words/steps where is the current issue/bug and what's your proposal to fix it? It seems there are more than one specific problem (considering the toml case).

[stsewd]

The problem here is that we create the virtualenv with the cwd on the project's root, when creating the virtualenv, pip is triggered, pip reads the .toml file from the repo and everything explodes, as we don't need to create the virtualenv from the project's root, I'm proposing to change the cwd to / here

https://github.com/rtfd/readthedocs.org/blob/9b2b17c0fc603267b4b20f5923862c2db82602da/readthedocs/doc_builder/python_environments.py#L207-L220

(instead of checkout_path, just /)

[stsewd]

My other solution is for installing the latest pip always, but that isn't part of the solution

[humitos]

The problem here is that we create the virtualenv with the cwd on the project's root, when creating the virtualenv, pip is triggered, pip reads the .toml file from the repo and everything explodes, as we don't need to create the virtualenv from the project's root, I'm proposing to change the cwd to / here

If the project has a pyproject.toml in the root of the repository, I assume that the project wants us to read it and use it, right?

If we change the CWD we will be fixing this particular problem for this particular repository but breaking the general rule that it's to read that file (as the pip's documentation says).

Still, I don't understand why this fails. It seems an incompatibility with the pip + virtualenv version installed on the docker image + the dependencies required from the toml file.

On the other hand, if the problem is just that CWD path, we should be able to reproduce it outside RTD and I didn't find a way yet (I tried installing different versions of pip, virtualenv and setuptools)

[stsewd]

If the project has a pyproject.toml in the root of the repository, I assume that the project wants us to read it and use it, right?

Yeah, but when installing the packages, not when creating the virtualenv

If we change the CWD we will be fixing this particular problem for this particular repository but breaking the general rule that it's to read that file (as the pip's documentation says).

Only for the virtualenv creation, installation of packages are still running the cwd on the project's root

[humitos]

@stsewd you are right! It seems that you have been trying to say that in many comments but I wasn't getting it :)

OK, so I suppose that a good place to point the CWD when creating the venv is the HOME directory instead of / because if the command needs to write a temp file or whatever, it will fail on /.

[stsewd]

Pointing to home sounds better, not sure how to do that (yet), I think we don't allow shell expansions, I'll see what I can do

[jaraco]

\o/ Thanks all!