Builds broken with poetry 1.8

[etianen]

Following the install dependencies with poetry guide, I get the following error with poetry 1.8:

no theme named 'furo' found (missing theme.conf?)

However, the theme is clearly installed!

- Installing furo (2024.1.29)

Specifically installing poetry 1.7 with fixes the issue:

- pip install 'poetry~=1.7.0'

I don't see anything in the poetry 1.8 release notes to as a cause for the breakage. If you can pin down what the problem is, I'll gladly follow-up with an issue on the poetry issue tracker. 🙇

Details

• Read the Docs project URL: https://readthedocs.org/projects/logot/
• Build URL (if applicable): https://readthedocs.org/projects/logot/builds/23555282/
• Read the Docs username (if applicable): https://readthedocs.org/profiles/etianen/

Expected Result

The build to pass with poetry 1.8 👍

Actual Result

The build fails with poetry 1.8 👎

[ewjoachim]

Looks like we have the same issue: #11149

[ewjoachim]

I'm noting that 1.8.0 has been released in december, but it only started failing recently, so something changed, that is different in Poetry 1.8.0 and 1.7.0 but chances are it's not poetry itself.

[ewjoachim]

Hmpf, I completely misread. Poetry 1.8 was released 5 hours ago.

[ewjoachim]

So there's python-poetry/poetry#9025 for tracking the issue in Poetry.

So far, it seems it's a consequence of python-poetry/poetry#8970 that makes detection of venv more strict. Because RTD doesn't expose VIRTUAL_ENV, Poetry doesn't consider this a virtualenv, and installs to the base Python instead of the one in the venv.

Poetry folks indicate that RTD should not advocate using poetry config virtualenvs.create false, it only worked because of a bug.

So what now ?

• Should we use a different technique for installing poetry deps ? (e.g. poetry export + pip install ?)
• Should we make it so that the build sets the VIRTUAL_ENV envvar as part of the yaml instructions ?
• Should the VIRTUAL_ENV variable be set by RTD itself for every build ?

Quoting the Poetry discord:

I don't think there is an official way to use a specific python, which is not an activated venv. You can only fake an activated venv by setting PATH and VIRTUAL_ENV.

[etianen]

I'd vote for (3). I always set this when activating a venv programmatically.

…

On Sun, 25 Feb 2024 at 18:49, Joachim Jablon ***@***.***> wrote: So there's python-poetry/poetry#9025 <python-poetry/poetry#9025> for tracking the issue in Poetry. So far, it seems it's a consequence of python-poetry/poetry#8970 <python-poetry/poetry#8970> that makes detection of venv more strict. Because RTD doesn't expose VIRTUAL_ENV, Poetry doesn't consider this a virtualenv, and installs to the base Python instead of the one in the venv. Poetry folks indicate that RTD should not advocate using poetry config virtualenvs.create false, it only worked because of a bug. So what now ? - Should we use a different technique for installing poetry deps ? (e.g. poetry export + pip install ?) - Should we make it so that the build sets the VIRTUAL_ENV envvar as part of the yaml instructions ? - Should the VIRTUAL_ENV variable be set by RTD itself for every build ? — Reply to this email directly, view it on GitHub <#11150 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABEKCFDMACV6UZ55N5B74DYVOBTPAVCNFSM6AAAAABDZAMGXWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRTGAZDOMBXGU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[dimbleby]

Should the VIRTUAL_ENV variable be set by RTD itself for every build ?

If that means that the variable is also set at the moment that you execute pip install poetry - then this also is not correct. Poetry must be in a different environment than the project under management: else installing the project dependencies can break the environment for poetry.

I do not see a solution that can avoid changing the instructions.

(This probably means that both poetry and readthedocs will be dealing with duplicate reports of this for some time to come: until fixed instructions eventually work their way through the system...)

[etianen]

How about making pipx be available on runners (maybe it is already?!). Installing poetry via pipx is how it's done in GitHub actions, and it works like a charm!

…

On Sun, 25 Feb 2024 at 19:34, David Hotham ***@***.***> wrote: Should the VIRTUAL_ENV variable be set by RTD itself for every build ? If that means that the variable is also set at the moment that you execute pip install poetry - then this also is not correct. Poetry must be in a different environment than the project under management: else installing the project dependencies can break the environment for poetry. I do not see a solution that can avoid changing the instructions. (This probably means that both poetry and readthedocs will be dealing with duplicate reports of this for some time to come: until fixed instructions eventually work their way through the system...) — Reply to this email directly, view it on GitHub <#11150 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABEKCBSA4BROAK3O5FM5ADYVOG35AVCNFSM6AAAAABDZAMGXWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRTGAZTQNRRHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[humitos]

How about making pipx be available on runners (maybe it is already?!).

It's not, but it can be installed via

python3 -m pip install --user pipx
python3 -m pipx ensurepath

(from https://pipx.pypa.io/stable/#on-linux)

If that's the best way to install and execute poetry, we can update the documentation to reflect that.

[humitos]

Poetry folks indicate that RTD should not advocate using poetry config virtualenvs.create false

This is required because there are other Python commands executed by Read the Docs (e.g. sphinx-build) that need to be executed using the current virtualenv where all the requirements were installed --that's why poetry and the requirements for building the documentation has to be installed in the same virtualenv.

These commands are executed by Read the Docs,

python -m pip install --upgrade --no-cache-dir pip setuptools              
python -m pip install --upgrade --no-cache-dir sphinx readthedocs-sphinx-ext
python -m sphinx -T -b html -d _build/doctrees -D language=en . $READTHEDOCS_OUTPUT/html

and these should work without any modification. Note that we cannot prepend them with pipx run or similar.

[etianen]

Installing via pipx is one of the recommended ways to install poetry: https://python-poetry.org/docs/ (Sorry I can't link more specifically on mobile, docs seem to be a SPA.) So a pipx installation coupled with the additional venv environmental variable would work. It would be lovely to have pipx already installed. It's very mainstream and opens the door to installing all sorts of CLI tools without polluting the main venv. GHA comes with it installed, for example.

…

On Sun, 25 Feb 2024 at 20:27, Manuel Kaufmann ***@***.***> wrote: How about making pipx be available on runners (maybe it is already?!). It's not, but it can be installed via python3 -m pip install --user pipx python3 -m pipx ensurepath *(from https://pipx.pypa.io/stable/#on-linux <https://pipx.pypa.io/stable/#on-linux>)* If that's the best way to install and execute poetry, we can update the documentation to reflect that. — Reply to this email directly, view it on GitHub <#11150 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABEKCDBLQFA2YLJLRW2TJDYVONCJAVCNFSM6AAAAABDZAMGXWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRTGA2TCNJYGQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[humitos]

I'm happy to update our documentation with a different solution if you have one that does work for this particular use case.

It would be lovely to have pipx already installed

Unfortunately, we can't do this in the short term at least; but it's easy to install it as I showed before.

[humitos]

@ewjoachim

Poetry folks indicate that RTD should not advocate using poetry config virtualenvs.create false, it only worked because of a bug.

BTW, do you know what's the bug that they fixed that changed this behavior? Their documentation for virtualenvs.create (https://python-poetry.org/docs/configuration#virtualenvscreate) still says that it should work in the way we expect to work: don't create a new virtualenv when installing dependencies --but it seems it doesn't work like that anymore.

[etianen]

Would it be possible to set that extra env var? That's all that's needed for poetry to detect the venv correctly. And I think it makes the venv activation for correct for other tools that might rely on it. Then the docs can be updated to install poetry via pipx, and lose the poetry config call.

…

On Sun, 25 Feb 2024 at 20:48, Manuel Kaufmann ***@***.***> wrote: @ewjoachim <https://github.com/ewjoachim> Poetry folks indicate <python-poetry/poetry#9025 (comment)> that RTD should not advocate using poetry config virtualenvs.create false, it only worked because of a bug. BTW, do you know what's the bug that they fixed that changed this behavior? Their documentation for virtualenvs.create ( https://python-poetry.org/docs/configuration#virtualenvscreate) still says that it should work in the way we expect to work: don't create a new virtualenv when installing dependencies --but it seems it doesn't work like that anymore. — Reply to this email directly, view it on GitHub <#11150 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABEKCFTJ2KVGWPQGHXOMI3YVOPTBAVCNFSM6AAAAABDZAMGXWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRTGA2TMOBVGA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ewjoachim]

BTW, do you know what's the bug that they fixed that changed this behavior? Their documentation for virtualenvs.create (python-poetry.org/docs/configuration#virtualenvscreate) still says that it should work in the way we expect to work: don't create a new virtualenv when installing dependencies --but it seems it doesn't work like that anymore.

Not really. They don't create a virtualenv, so it works as intended. The behaviour is "if we're in a venv, use the venv. If not, install to the system python."
Because RTD doesn't set VIRTUAL_ENV (as virtualenv and python -m venv do when using source bin/activate), they don't detect they're in a virtualenv, so they don't do that. I suspect that they used to do something along the lines of python -m pip ... and they now do $(readlink python) -m pip ... (if they don't detect a venv). I don't think it contradicts the doc.

[ewjoachim]

For the record, I got a running build in RTD with:

version: 2

build:
  os: "ubuntu-22.04"
  tools:
    python: "3.10"
  jobs:
    post_create_environment:
      - python -m pip install poetry
    post_install:
      - VIRTUAL_ENV=$READTHEDOCS_VIRTUALENV_PATH python -m poetry install --with docs

I'll open a PR for this to become "the way". When we can improve this, we will.

[jfagoagas]

For the record, I got a completed build in RTF with:

version: 2

build:
  os: "ubuntu-22.04"
  tools:
    python: "3.11"
  jobs:
    post_create_environment:
      # Install poetry and the export plugin for pip
      - python -m pip install poetry poetry-plugin-export
    post_install:
      - poetry export -f requirements.txt --without-hashes --only docs -o only-docs.txt
      - pip install --requirement only-docs.txt

I prefer you workaround @ewjoachim since is simpler.

Thanks!