uv 0.4.11 failing to read distribution cache in github actions #7485

winwinashwin · 2024-09-18T06:13:42Z

Our Github Actions CI started failing with the following error when running uv pip install pyspark==3.2.1

Using Python 3.8.18 interpreter at: /__w/_tool/Python/3.8.18/x64/bin/python3
Creating virtualenv with seed packages at: venv
warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance.
         If the cache and target directories are on different filesystems, hardlinking may not be supported.
         If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning.
 + pip==24.2
 + setuptools==75.1.0
 + wheel==0.44.0
error: Failed to download and build `pyspark==3.2.1`
  Caused by: Failed to read from the distribution cache
  Caused by: failed to read directory `/__w/_temp/setup-uv-cache/built-wheels-v3/pypi/pyspark/3.2.1/IStJ-q5m0o-G-eruFFpEK/pyspark-3.2.1.tar.gz`
  Caused by: No such file or directory (os error 2)

This started happening after the latest 0.4.11 release.

Workflow Declaration

- name: Setup Python
  uses: actions/setup-python@v5
  with:
    python-version: '3.8'

- name: Install uv
  uses: astral-sh/setup-uv@v2
  with:
    enable-cache: true

- run: |
    uv venv venv --seed
    uv pip install -r requirements.txt pyarrow pyspark==3.2.1 --python venv  # <--- FAILS HERE
  shell: bash

astral-sh/setup-uv logs

> Run astral-sh/setup-uv@v2
  with:
    enable-cache: true
    version: latest
    github-token: ***
  env:
    pythonLocation: /__w/_tool/Python/3.8.18/x64
    PKG_CONFIG_PATH: /__w/_tool/Python/3.8.18/x64/lib/pkgconfig
    Python_ROOT_DIR: /__w/_tool/Python/3.8.18/x64
    Python2_ROOT_DIR: /__w/_tool/Python/3.8.18/x64
    Python3_ROOT_DIR: /__w/_tool/Python/3.8.18/x64
    LD_LIBRARY_PATH: /__w/_tool/Python/3.8.18/x64/lib:/opt/oracle/instantclient_21_8/
/usr/bin/docker exec  b1f1b19e402573bba171b6a6d208b411385eed518d4f720827a31a93cb1a6f88 sh -c "cat /etc/*release | grep ^ID"
Downloading uv from "https://github.com/astral-sh/uv/releases/latest/download/uv-x86_64-unknown-linux-gnu.tar.gz" ...
/usr/bin/tar xz --warning=no-unknown-keyword --overwrite -C /__w/_temp/5138b5d3-39fd-4adf-abe8-9a6b348864e2 -f /__w/_temp/ef8159c7-bfe3-439b-b565-b6a11cf5126c
Added /__w/_tool/uv/0.4.11/x86_64 to the path
Successfully installed uv version latest
Set UV_CACHE_DIR to /__w/_temp/setup-uv-cache
Trying to restore uv cache from GitHub Actions cache with key: setup-uv-1-x86_64-unknown-linux-gnu-0.4.11
-no-dependency-glob
Received 0 of 301099792 (0.0%), 0.0 MBs/sec
Received 33554432 of 301099792 (11.1%), 16.0 MBs/sec
Received 67108864 of 301099792 (22.3%), 21.3 MBs/sec
Received 109051904 of 301099792 (36.2%), 26.0 MBs/sec
Received 150994944 of 301099792 (50.1%), 28.8 MBs/sec
Received 184549376 of 301099792 (61.3%), 29.3 MBs/sec
Received 213909504 of 301099792 (71.0%), 29.1 MBs/sec
Received 251658240 of 301099792 (83.6%), 30.0 MBs/sec
Received 288516880 of 301099792 (95.8%), 30.5 MBs/sec
Cache Size: ~287 MB (301099792 B)
/usr/bin/tar -xf /__w/_temp/92d12651-008f-4643-8d55-d87145b39532/cache.tgz -P -C /__w/<PRIVATE-REPOSITORY> -z
Received 301099792 of 301099792 (100.0%), 28.7 MBs/sec
Cache restored successfully
uv cache restored from GitHub Actions cache with key: setup-uv-1-x86_64-unknown-linux-gnu-0.4.11

uv pip install ... complete logs

> Run uv venv venv --seed
  uv venv venv --seed
  uv pip install -r requirements.txt pyarrow pyspark==3.2.1 --python venv
  shell: bash --noprofile --norc -e -o pipefail {0}
  env:
    pythonLocation: /__w/_tool/Python/3.8.18/x64
    PKG_CONFIG_PATH: /__w/_tool/Python/3.8.18/x64/lib/pkgconfig
    Python_ROOT_DIR: /__w/_tool/Python/3.8.18/x64
    Python2_ROOT_DIR: /__w/_tool/Python/3.8.18/x64
    Python3_ROOT_DIR: /__w/_tool/Python/3.8.18/x64
    LD_LIBRARY_PATH: /__w/_tool/Python/3.8.18/x64/lib:/opt/oracle/instantclient_21_8/
    UV_CACHE_DIR: /__w/_temp/setup-uv-cache
    JAVA_HOME: /__w/_tool/Java_Corretto_jdk/8.422.05-1/x64
    JAVA_HOME_8_X64: /__w/_tool/Java_Corretto_jdk/8.422.05-1/x64
    PNPM_HOME: /github/home/setup-pnpm/node_modules/.bin
Using Python 3.8.18 interpreter at: /__w/_tool/Python/3.8.18/x64/bin/python3
Creating virtualenv with seed packages at: venv
warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance.
         If the cache and target directories are on different filesystems, hardlinking may not be supported.
         If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning.
 + pip==24.2
 + setuptools==75.1.0
 + wheel==0.44.0
error: Failed to download and build `pyspark==3.2.1`
  Caused by: Failed to read from the distribution cache
  Caused by: failed to read directory `/__w/_temp/setup-uv-cache/built-wheels-v3/pypi/pyspark/3.2.1/IStJ-q5m0o-G-eruFFpEK/pyspark-3.2.1.tar.gz`
  Caused by: No such file or directory (os error 2)

Tried reproducing this on my local machine with uv self update followed by a uv pip install pyspark==3.2.1 but couldn't reproduce the failure.

CI Runner Info

All our actions run inside our private CI docker image which is based on Ubuntu 22.04 LTS

Host OS

Distribution: Amazon Linux release 2023.5.20240805 (Amazon Linux)
Architecture: x86_64

The text was updated successfully, but these errors were encountered:

winwinashwin · 2024-09-18T06:26:28Z

Seems to be related? #7479

winwinashwin · 2024-09-18T07:33:30Z

After going through the release notes for 0.4.11, I was able to reproduce this locally. This seem to be linked to the newly added uv cache prune --ci option.

NOTE - We don't explicitly invoke uv cache prune --ci in our CI. Just adding the below MRE here if it helps as it was the only way I could reproduce this in my local.

$ uv cache clean
Clearing cache at: /home/ashwin/.cache/uv
Removed 14 files (269.7MiB)

$ uv venv venv-1 --python 3.8
Using Python 3.8.19
Creating virtualenv at: venv-1
Activate with: source venv-1/bin/activate

$ uv pip install pyspark==3.2.1 --python venv-1
Resolved 2 packages in 36.30s
   Built pyspark==3.2.1
Prepared 2 packages in 17.55s
Installed 2 packages in 25ms
 + py4j==0.10.9.3
 + pyspark==3.2.1

$ tree -L 2 `uv cache dir`/built-wheels-v3/pypi/pyspark/3.2.1/
/home/ashwin/.cache/uv/built-wheels-v3/pypi/pyspark/3.2.1/
├── MlQraUk4JIaysmTs9mj2e
│   ├── metadata.msgpack
│   ├── pyspark-3.2.1-py2.py3-none-any -> /home/ashwin/.cache/uv/archive-v0/LbVUP3RZg8fwvcArKRmt8
│   ├── pyspark-3.2.1-py2.py3-none-any.whl
│   └── pyspark-3.2.1.tar.gz
└── revision.http

3 directories, 3 files

$ uv cache --prune ci
Pruning cache at: /home/ashwin/.cache/uv
Removed 2816 files (887.9MiB)

$ uv venv venv-2 --python 3.8
Using Python 3.8.19
Creating virtualenv at: venv-2
Activate with: source venv-2/bin/activate

$ uv pip install pyspark==3.2.1 --python venv-2
⠙ pyspark==3.2.1                                                                                                                                                                                                 error: Failed to download and build `pyspark==3.2.1`
  Caused by: Failed to read from the distribution cache
  Caused by: failed to read directory `/home/ashwin/.cache/uv/built-wheels-v3/pypi/pyspark/3.2.1/MlQraUk4JIaysmTs9mj2e/pyspark-3.2.1.tar.gz`
  Caused by: No such file or directory (os error 2)

$ tree -L 2 `uv cache dir`/built-wheels-v3/pypi/pyspark/3.2.1/
/home/ashwin/.cache/uv/built-wheels-v3/pypi/pyspark/3.2.1/
├── MlQraUk4JIaysmTs9mj2e
│   ├── metadata.msgpack
│   └── pyspark-3.2.1-py2.py3-none-any.whl
└── revision.http

1 directory, 3 files

Kazy · 2024-09-18T10:12:34Z

This happened to us as well. Reverting to 0.4.10 and busting the cache solve the issue. If you re-use a cache done in 0.4.11 with uv cache prune --ci called but with a previous version of uv (like if you're in a CI and you just change the version of uv installed), it will also fail. It seems that uv cache prune --ci in 0.4.11 corrupts the cache since uv sync or uv lock --locked don't work after that.

zanieb · 2024-09-18T12:23:27Z

Thanks for the report! We'll look into this right away cc @charliermarsh

MRigal · 2024-09-18T12:38:43Z

I can confirm a similar issue with Gitlab, also using prune --ci. Breaks installs on CI

MRigal · 2024-09-18T12:41:26Z

I think this comes from #7446 as it happens for me with the package https://pypi.org/project/pytest-clarity/#files which has no wheels

zanieb · 2024-09-18T12:43:39Z

#7446 is the very likely cause yeah

MRigal · 2024-09-18T12:58:30Z

Here some output to maybe ease spotting the error:

(.venv) ➜  dataworker git:(main) uv cache prune --ci                                                                                                                                                   git:(main|⚑7)
Pruning cache at: /Users/mrigal/Library/Caches/uv
Removed 18847 files (587.2MiB)

(.venv) ➜  dataworker git:(main) uv pip uninstall pytest-clarity                                                                                                                                       git:(main|⚑7)
Uninstalled 1 package in 26ms
 - pytest-clarity==1.0.1
 
(.venv) ➜  dataworker git:(main) uv sync --frozen                                                                                                                                                      git:(main|⚑7)
error: Failed to prepare distributions
  Caused by: Failed to fetch wheel: pytest-clarity==1.0.1
  Caused by: /Users/mrigal/Library/Caches/uv/built-wheels-v3/pypi/pytest-clarity/1.0.1/yE42-NVZgq4lK-UrXRJnS/pytest-clarity-1.0.1.tar.gz does not appear to be a Python project, as neither `pyproject.toml` nor `setup.py` are present in the directory

(.venv) ➜  dataworker git:(main) uv sync                                                                                                                                                               git:(main|⚑7)
⠙ faktory==1.1.0                                                                                                                                                                                                      error: Failed to download and build `pytest-clarity==1.0.1`
  Caused by: Failed to read from the distribution cache
  Caused by: failed to read directory `/Users/mrigal/Library/Caches/uv/built-wheels-v3/pypi/pytest-clarity/1.0.1/yE42-NVZgq4lK-UrXRJnS/pytest-clarity-1.0.1.tar.gz`
  Caused by: No such file or directory (os error 2)

And verbose:

(.venv) ➜  dataworker git:(main) uv sync -v                                                                                                                                                            git:(main|⚑7)
DEBUG uv 0.4.11
DEBUG Found project root: `/Users/mrigal/dev/prospect-server/dataworker`
DEBUG No workspace root found, using project root
DEBUG The virtual environment's Python version satisfies `Python >=3.12`
DEBUG Using request timeout of 30s
DEBUG Found static `pyproject.toml` for: dataworker @ file:///Users/mrigal/dev/prospect-server/dataworker
DEBUG No workspace root found, using project root
DEBUG Ignoring existing lockfile due to mismatched dev dependencies for: `dataworker==0.1.0`
[... noisy output of dependencies listing]
DEBUG Starting clean resolution
DEBUG Found static `pyproject.toml` for: dataworker @ file:///Users/mrigal/dev/prospect-server/dataworker
DEBUG No workspace root found, using project root
DEBUG Solving with installed Python version: 3.12.3
DEBUG Solving with target Python version: >=3.12
DEBUG Narrowed `requires-python` bound to: >=3.12
DEBUG Narrowed `requires-python` bound to: >=3.12
DEBUG Solving split python_full_version == '3.12.*' (requires-python: RequiresPython { specifiers: VersionSpecifiers([VersionSpecifier { operator: GreaterThanEqual, version: "3.12" }]), range: RequiresPythonRange(LowerBound(Included("3.12")), UpperBound(Excluded("3.13"))) })
DEBUG Adding direct dependency: dataworker*
DEBUG Searching for a compatible version of dataworker @ file:///Users/mrigal/dev/prospect-server/dataworker (*)
DEBUG Adding transitive dependency for dataworker==0.1.0: arrow>=1.3.0
[... similar lines]
DEBUG Adding transitive dependency for dataworker==0.1.0: pytest-clarity>=1.0.1
DEBUG Searching for a compatible version of faktory @ file:///Users/mrigal/dev/prospect-server/dataworker/faktory-1.1.0-py3-none-any.whl (*)
DEBUG Found stale response for: https://pypi.org/simple/arrow/
DEBUG Sending revalidation request for: https://pypi.org/simple/arrow/
[... similar lines from pypi]
DEBUG Acquired lock for `/Users/mrigal/Library/Caches/uv/built-wheels-v3/pypi/pytest-clarity/1.0.1`
-------> Maybe also interesting
DEBUG No cache entry for: https://files.pythonhosted.org/packages/e2/14/c8a7d861262139688fa465d2e27ff7113764d6fa03b15b9c7b666729ea2e/dynaconf-3.2.6-py2.py3-none-any.whl.metadata
DEBUG No cache entry for: https://files.pythonhosted.org/packages/8a/0e/0f755db36f47f96464463385552f8f132a981731356837c9a30a11ab2d35/psycopg-3.2.1-py3-none-any.whl.metadata
DEBUG No cache entry for: https://files.pythonhosted.org/packages/6a/3e/b68c118422ec867fa7ab88444e1274aa40681c606d59ac27de5a5588f082/python_dotenv-1.0.1-py3-none-any.whl.metadata
DEBUG No cache entry for: https://files.pythonhosted.org/packages/24/d3/00e575657422055c4ea220b2f80e8cc6026ab7130372b7067444d1b0ac10/pytest_randomly-3.15.0-py3-none-any.whl.metadata
DEBUG No cache entry for: https://files.pythonhosted.org/packages/dd/49/de869130028fb8d90e25da3b7d8fb13e40f5afa4c4af1781583eb1ff3839/pandas-2.2.2-cp312-cp312-macosx_10_9_x86_64.whl.metadata
----->>> See here
DEBUG Found fresh response for: https://files.pythonhosted.org/packages/52/5c/cafa97944de55738a6a2c5a7cee00d073cb80495032d2b112c4546525eca/pytest-clarity-1.0.1.tar.gz
DEBUG No static `pyproject.toml` available for: pytest-clarity==1.0.1 (MissingPyprojectToml)
DEBUG No static `PKG-INFO` available for: pytest-clarity==1.0.1 (MissingPkgInfo)
DEBUG Released lock at `/Users/mrigal/Library/Caches/uv/built-wheels-v3/pypi/pytest-clarity/1.0.1/.lock`

error: Failed to download and build `pytest-clarity==1.0.1`
  Caused by: Failed to read from the distribution cache
  Caused by: failed to read directory `/Users/mrigal/Library/Caches/uv/built-wheels-v3/pypi/pytest-clarity/1.0.1/yE42-NVZgq4lK-UrXRJnS/pytest-clarity-1.0.1.tar.gz`
  Caused by: No such file or directory (os error 2)

charliermarsh · 2024-09-18T13:03:12Z

Thanks, I’ll take a look.

charliermarsh · 2024-09-18T13:03:32Z

We can also just revert that change and re-release, it’s not a critical change.

…7446)" This reverts commit 9f7d9da. This is meant to be available as a quick fix if we can't otherwise find a quick fix to the problem reported in #7485. Fixes #7485

MRigal · 2024-09-18T13:07:00Z

Actually, the root cause is possibly deeper. After reverting to 0.4.10, it fixes the behaviour for uv sync but not for uv pip install:

(.venv) ➜  dataworker git:(main) ✗ uv self update 0.4.10                                                                                                                                              git:(main|✚1⚑7
info: Checking for updates...
success: Upgraded uv from v0.4.11 to v0.4.10! https://github.com/astral-sh/uv/releases/tag/0.4.10
(.venv) ➜  dataworker git:(main) ✗ uv pip install pytest-clarity                                                                                                                                      git:(main|✚1⚑7
⠙ pytest-clarity==1.0.1                                                                                                                                                                                               error: Failed to download and build `pytest-clarity==1.0.1`
  Caused by: Failed to read from the distribution cache
  Caused by: failed to read directory `/Users/mrigal/Library/Caches/uv/built-wheels-v3/pypi/pytest-clarity/1.0.1/yE42-NVZgq4lK-UrXRJnS/pytest-clarity-1.0.1.tar.gz`
  Caused by: No such file or directory (os error 2)
(.venv) ➜  dataworker git:(main) ✗ uv sync --frozen                                                                                                                                                   git:(main|✚1⚑7
Prepared 1 package in 3ms
Installed 1 package in 1ms
 + pytest-clarity==1.0.1

charliermarsh · 2024-09-18T13:12:54Z

I wouldn’t expect downgrading to help. Once the cache is in a bad state, any version would fail.

…7498) ## Summary Closes #7485. ## Test Plan ``` $ cargo run cache clean $ cargo run venv $ cargo run pip install django-allauth==0.51.0 $ cargo run venv $ cargo run pip install django-allauth==0.51.0 ```

## Summary See: #7485. The test was using `uv pip sync` which doesn't require fetching metadata, and the failure was in fetching metadata.

MRigal · 2024-09-18T14:55:07Z

Thanks for the quick action! I confirm that 0.4.12 fixes the issue!

zanieb added bug Something isn't working cache Caching of packages and metadata labels Sep 18, 2024

zanieb assigned BurntSushi and charliermarsh Sep 18, 2024

BurntSushi mentioned this issue Sep 18, 2024

Revert "Prune unzipped source distributions in uv cache prune --ci (#7446)" #7495

Closed

charliermarsh mentioned this issue Sep 18, 2024

Avoid fatal error when searching for egg-info with missing directory #7498

Merged

charliermarsh closed this as completed in #7498 Sep 18, 2024

charliermarsh mentioned this issue Sep 18, 2024

Add test for uv cache prune --ci with source distributions #7500

Merged

charliermarsh added a commit that referenced this issue Sep 18, 2024

Add test for uv cache prune --ci with source distributions (#7500)

039b683

## Summary See: #7485. The test was using `uv pip sync` which doesn't require fetching metadata, and the failure was in fetching metadata.

nijel mentioned this issue Sep 19, 2024

uv 0.4.12 failing to read distribution cache in github actions #7543

Closed

zanieb mentioned this issue Sep 27, 2024

Can install pypi package pmtiles #7728

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

uv 0.4.11 failing to read distribution cache in github actions #7485

uv 0.4.11 failing to read distribution cache in github actions #7485

winwinashwin commented Sep 18, 2024 •

edited

Loading

winwinashwin commented Sep 18, 2024

winwinashwin commented Sep 18, 2024 •

edited

Loading

Kazy commented Sep 18, 2024 •

edited

Loading

zanieb commented Sep 18, 2024

MRigal commented Sep 18, 2024

MRigal commented Sep 18, 2024

zanieb commented Sep 18, 2024

MRigal commented Sep 18, 2024

charliermarsh commented Sep 18, 2024

charliermarsh commented Sep 18, 2024

MRigal commented Sep 18, 2024

charliermarsh commented Sep 18, 2024

MRigal commented Sep 18, 2024

uv 0.4.11 failing to read distribution cache in github actions #7485

uv 0.4.11 failing to read distribution cache in github actions #7485

Comments

winwinashwin commented Sep 18, 2024 • edited Loading

CI Runner Info

winwinashwin commented Sep 18, 2024

winwinashwin commented Sep 18, 2024 • edited Loading

Kazy commented Sep 18, 2024 • edited Loading

zanieb commented Sep 18, 2024

MRigal commented Sep 18, 2024

MRigal commented Sep 18, 2024

zanieb commented Sep 18, 2024

MRigal commented Sep 18, 2024

charliermarsh commented Sep 18, 2024

charliermarsh commented Sep 18, 2024

MRigal commented Sep 18, 2024

charliermarsh commented Sep 18, 2024

MRigal commented Sep 18, 2024

winwinashwin commented Sep 18, 2024 •

edited

Loading

winwinashwin commented Sep 18, 2024 •

edited

Loading

Kazy commented Sep 18, 2024 •

edited

Loading