A PyInfo cache populated under v20.28.1 cannot be loaded and used under v20.29.0 -- AttributeError on 'free_threaded'

[sirosen]

Issue

Unfortunately, I still don't have a reasonable MRE to show.
This originally surfaced for me in a closed-source build process built on Poetry -- my initial bug report here.

However, I eventually found that my project was installing virtualenv==20.28.1 into an environment where 20.29.0 was already running. (Full detail in this comment, same issue.)

I'm fairly certain that I'm getting a v20.28.1 PyInfo object loaded from the cache while running v20.29.0, but as I'm unable to reproduce it reliably, I don't yet know how this could happen. In the original case, I believe that poetry is running virtualenv.cli_run in threads, and multiple versions could be executing simultaneously.
If I can reproduce, obviously that would help. If threaded invocations are not supported, that may carry back to poetry as a consideration.

Environment

Provide at least:

• OS: Dockerized Debian -- python3.10 slim-bookworm

• pip list of the host python where virtualenv is installed -- cannot provide from the closed source original (still working on a MRE)

  # sorry, still working on the MRE

Output of the virtual environment creation

Make sure to run the creation with -vvv --with-traceback:

# also needs to wait for the MRE

---

I really don't like that I'm opening a ticket without an MRE, but since this is new in the today's release, I think it's worth reporting.
If this cache load is something which should be supported, we just need to populate free_threaded explicitly if it's missing in PyInfo._from_dict -- I would be happy to open a PR with that change if appropriate.

[sirosen]

That trace looks like poetry output. Make sure to read the last replies on the linked issue -- you might have the same poetry usage error that my team had.

For the virtualenv devs: the original issue is closed as it was tracked down (for me and for the other reporter on that thread) to an incorrect usage of poetry.

Given that I have some level of cause established, I'm not sure how much more effort I want to pour into trying to recreate this scenario. (I'm several hours in and no closer.) I think what it needs is threaded invocations of cli_run and for one of those threads to downgrade virtualenv in the current environment.

[gaborbernat]

Cc @robsdedude

[robsdedude]

I've tried to reproduce this now for an hour or so, without success. I'd need more information on what exactly has been done. However, during my investigation, I found out that virtualenv already had a way to version cache files that I was not aware of:

virtualenv/src/virtualenv/app_data/via_disk_folder.py

Line 82 in caf03a2

return self.lock / "py_info" / "1"

At least I assume that that "1" there is meant to be able to store different cache file versions.

@gaborbernat if you think it'd be cleaner and/or has potential to fix this issue, I'm happy to craft a PR to revert my cache file version change and instead bump this path segment.

[gaborbernat]

@gaborbernat if you think it'd be cleaner and/or has potential to fix this issue, I'm happy to craft a PR to revert my cache file version change and instead bump this path segment.

Yes please 🙏

[pedrurrego]

I am experiencing a very similar issue with virtualenv-20.29.0.

The same code ran using virtualenv-20.28.1.

AttributeError
'PythonSpec' object has no attribute 'free_threaded'
at .tox/py/lib/python3.12/site-packages/virtualenv/discovery/py_info.py:335 in satisfies
331│ return False
332│ return True
333│
334│ _current_system = None
→ 335│ _current = None
336│
337│ @classmethod
338│ def current(cls, app_data=None):
339│ """
Cannot install peewee.

I solved the issue pinning virtualenv@^20.29.0.

[sirosen]

I still have my closed source reproducer, which I tried with #2827. I thought it would solve it, and I was surprised to find that it did not. (I still think #2827 looks good? But it's not a fix for this.)

I've done more legwork and managed to get a usable reproducer which uses poetry. I wish I had something which used virtualenv only, but this is a start. I've shared it in my reproducer repo here: https://github.com/sirosen/repro/tree/main/poetry-and-virtualenv

Notably, I incorporated this fix: https://github.com/sirosen/repro/blob/b3a37f75d81c9061526cfeeb6c86e615a4df9365/poetry-and-virtualenv/Dockerfile#L8
And here's an unredacted log from it: https://github.com/sirosen/repro/blob/main/poetry-and-virtualenv/sample.clean.log

---

I see a few paths to resolution of this issue and I am -- as the reporter -- flexible and happy to follow the maintainers' discretion.

One is to close this as not a bug, at least for now. poetry causes something weird in this situation, but this only happens in unsupported poetry usage. Maybe all we're seeing is that bad usage of poetry produces a bad usage of virtualenv.

Or we could treat the reproducer as something to look into. We could try to experiment on it without trying to read poetry source for a full understanding of the scenario, or we can try to pick it apart to figure out what happens -- to see if there is maybe a broader issue buried in all of this.

[gaborbernat]

Please check with https://github.com/pypa/virtualenv/releases/tag/20.29.1

[sirosen]

I updated my reproducer and ran it again -- still gets a failure with the same trace.

[gaborbernat]

As workaround, just delete the virtualenv cache?

[sirosen]

This is starting from a fresh state -- no cache.

The whole situation has a vague whiff of a race condition to me. I'm sure poetry has parallel invocations of virtualenv.cli_run, and virtualenv gets downgraded in-place in the middle of it.

I think that the virtualenv posture towards this bug should be "changing the installed version of virtualenv while it is running is not supported". Since that's the only way I can make this happen, I'm inclined to self-close this if you agree.