Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(cache): ensure installation and cache works with any environment #2839

Merged
merged 1 commit into from
Apr 20, 2024

Conversation

noirbizarre
Copy link
Member

@noirbizarre noirbizarre commented Apr 19, 2024

Pull Request Checklist

  • A news fragment is added in news/ describing what is new.
  • Test cases added for changed code.

Describe what you have changed in this PR.

Since a few versions I was having a lot of sync/install failures due to some cache file collisions (especially the .referrers file).

This PR ensure installers (with or without cache) are tested against both PythonEnvironement (virtualenv) and PythonLocalEnvironment (PEP582) because installation and caching might differ between both. (Only the PythonLocalEnvironment was tested).

It also tests cached installations across multiple projects with those environments allowing to reproduce my .referrers case:
When installing with cache in a virtualenv env, the referrer files was not linked on first install (because created after installation) but it was on the 2nd one.
As a side-effect, when installing 2 already cached packages in the same virtualenv, the 2nd one was failing because the .referrers file was already linked (root file), produced the following error:

pdm.termui: Error occurs:
Traceback (most recent call last):
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/noirbizarre/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/installers/synchronizers.py", line 282, in install_candidate
    self.manager.install(can)
  File "/home/noirbizarre/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/installers/manager.py", line 32, in install
    dist_info = install_wheel(
                ^^^^^^^^^^^^^^
  File "/home/noirbizarre/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/installers/installers.py", line 189, in install_wheel
    dist_info_dir = install(source, destination=destination, additional_metadata=additional_metadata)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/noirbizarre/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/installers/installers.py", line 206, in install
    _install(source, destination, additional_metadata=additional_metadata or {})
  File "/home/noirbizarre/.local/pipx/venvs/pdm/lib/python3.11/site-packages/installer/_core.py", line 109, in install
    record = destination.write_file(
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/noirbizarre/.local/pipx/venvs/pdm/lib/python3.11/site-packages/installer/destinations.py", line 207, in write_file
    return self.write_to_fs(scheme, path_, stream, is_executable)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/noirbizarre/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/installers/installers.py", line 129, in write_to_fs
    os.symlink(src_path, target_path)
FileExistsError: [Errno 17] File exists: '/home/noirbizarre/.cache/pdm/packages/termcolor-2.4.0-py3-none-any.whl.cache/.referrers' -> '/tmp/pytest-of-noirbizarre/pytest-48/test_apply_fastapi_3_12_0/project/.venv/lib/python3.12/site-packages/.referrers'
pdm.termui: Error occurs
Traceback (most recent call last):
  File "/home/noirbizarre/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/termui.py", line 257, in logging
    yield logger
  File "/home/noirbizarre/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/cli/actions.py", line 238, in do_sync
    synchronizer.synchronize()
  File "/home/noirbizarre/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/installers/synchronizers.py", line 464, in synchronize
    raise InstallationError("Some package operations are not complete yet")
pdm.exceptions.InstallationError: Some package operations are not complete yet

Fixed by storing a list of cache metadata files on the CachedPackage class (avoiding to search/know where it is used in the codebase each time a new metadata file is created).

@noirbizarre noirbizarre force-pushed the fix/cache-files-collision branch from 88f2958 to 97d3674 Compare April 19, 2024 16:47
@frostming frostming merged commit e3784d8 into pdm-project:main Apr 20, 2024
20 checks passed
@noirbizarre noirbizarre deleted the fix/cache-files-collision branch April 20, 2024 11:24
@exherb
Copy link

exherb commented Apr 22, 2024

when it's this going to release? our CI is not happy with this, 😂

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants