Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

opam does not wait for the Software Heritage fallback archives to be "cooked" #5721

Open
kit-ty-kate opened this issue Nov 8, 2023 · 3 comments · May be fixed by #6036
Open

opam does not wait for the Software Heritage fallback archives to be "cooked" #5721

kit-ty-kate opened this issue Nov 8, 2023 · 3 comments · May be fixed by #6036
Assignees
@rjbou
Labels
KIND: BUG
Milestone
2.4.0~alpha1

Comments

@kit-ty-kate
Copy link
Member

When using the software heritage fallback feature, it is understood that regenerating (as noted in #5720) the archive from a given SWHID (called "cooking" by Software Heritage) might take some time. The code currently denotes this by a 60 seconds timer:

opam/src/repository/opamDownload.ml

Line 299 in c4961ba

Unix.sleep 10;

However in practice, trying it locally, leads only a short wait then the following failure:

OpamSolution.Fetch_fail("\"/opt/homebrew/bin/wget -X POST --content-disposition -t 3 -O - -U opam/2.2.0~alpha2 -- https://archive.softwareheritage.org/api/1/vault/directory/6f61d745641e701d5cad73a9e918d5e888d5ab64/\" exited with code 8 \"2023-11-08 16:00:08 ERROR 404: Not Found.\"")

Calling the same command again when the archive has finished cooking, succeeds.
@kit-ty-kate
Copy link
Member Author

The software heritage fallback was disabled by default in #5899 so moving this issue off the 2.2 milestone

@kit-ty-kate
Copy link
Member Author

After further investigation it looks like wget does not work to fetch the expected data, but curl does:

$ curl https://archive.softwareheritage.org/api/1/vault/directory/6f61d745641e701d5cad73a9e918d5e888d5ab64/
{"fetch_url": ...}

$ wget -O - https://archive.softwareheritage.org/api/1/vault/directory/6f61d745641e701d5cad73a9e918d5e888d5ab64/
<!DOCTYPE html>...

@rjbou rjbou self-assigned this May 27, 2024
@rjbou
Copy link
Collaborator

rjbou commented Jun 13, 2024

Some more output

Skipping 121 bytes of body: [{"exception":"NotFoundExc","reason":"Cooking of swh:1:dir:45b87e1ca127914b936884dc73c9f788df5b7abd was never requested."}] done.
2024-06-13 18:04:26 ERROR 404: Not Found.

@rjbou rjbou linked a pull request Jun 19, 2024 that will close this issue
Software Heritage fallback fixes #6036
Draft
@rjbou rjbou linked a pull request Jun 19, 2024 that will close this issue
Draft
@rjbou rjbou added this to the 2.3 milestone Jun 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rjbou rjbou

Labels
KIND: BUG
Projects
None yet
Milestone
2.4.0~alpha1
Development

Successfully merging a pull request may close this issue.

Software Heritage fallback fixes rjbou/opam
2 participants
@kit-ty-kate @rjbou