cabal new-update fails for head.hackage

[AndreasPK]

Either I get an invalid response from the server, or cabal fails with a file locking issue.

Andi@Horzube MINGW64 /e/bench_hc_libs/libs/perf-xml/build
$ cabal v2-update 'hackage.haskell.org,2019-02-19T16:58:55Z'
Downloading the latest package list from hackage.haskell.org
To revert to previous state run:
    cabal v2-update 'hackage.haskell.org,2019-02-20T16:58:55Z'

Andi@Horzube MINGW64 /e/bench_hc_libs/libs/perf-xml/build
$ cabal v2-update 'head.hackage,2019-02-19T16:58:55Z'
Downloading the latest package list from head.hackage
C:\ghc\msys64\tmp\tra2CC0: DeleteFile "\\\\?\\C:\\ghc\\msys64\\tmp\\tra2CC0": permission denied (The process cannot access the file because it is being used by another process.)

Andi@Horzube MINGW64 /e/bench_hc_libs/libs/perf-xml/build
$ cabal v2-update
Downloading the latest package lists from:
- hackage.haskell.org
- head.hackage
To revert to previous state run:
    cabal v2-update 'hackage.haskell.org,2019-02-19T16:58:55Z'
C:\ghc\msys64\tmp\tra3EEA: DeleteFile "\\\\?\\C:\\ghc\\msys64\\tmp\\tra3EEA": permission denied (The process cannot access the file because it is being used by another process.)

Andi@Horzube MINGW64 /e/bench_hc_libs/libs/perf-xml/build
$ cabal v2-update
Downloading the latest package lists from:
- hackage.haskell.org
- head.hackage
Unexpected response 416for http://head.hackage.haskell.org/01-index.tar.gz

The unexpected response might not be a Cabal bug. But at least the failing DeleteFile seems like one.

This was with cabal-install built from commit 8e90301
After updating cabal-install to HEAD I only got unexpected response.

After cleaning ~/.cabal I was able to update successfully.

[23Skidoo]

/cc @hvr

[23Skidoo]

This is probably a server issue. HTTP 416 means "range not satisfiable".

[gbaz]

debugging the delete file issue would require knowing which transport was used, which can be detected with a verbose flag.

But if the delete file issue doesn't occur with HEAD, then I guess it doesn't need to be debugged?

[mpickering]

This happened to me as well.

[AndreasPK]

andreask@ghc-dev:~/layout_bench/containers$ cabal new-update --verbose
this build was affected by the following (project) config files:
- /home/andreask/layout_bench/containers/cabal.project
Downloading the latest package lists from:
- hackage.haskell.org
- head.hackage
Warning: The package list for 'head.hackage' is 131 days old.
Run 'cabal update' to get the latest list of available packages.
Selected mirror http://head.hackage.haskell.org/
Downloading timestamp
/usr/bin/curl 'http://head.hackage.haskell.org/timestamp.json' --output /tmp/transportAdapterGet31387-1 --location --write-out '%{http_code}' --user-agent 'cabal-install/3.0.0.0 (linux; x86_64)' --silent --show-error --dump-header /tmp/curl-headers31387-2.txt --header 'Cache-Control: no-transform'
Warning: Parsing the index cache failed (Unknown encoding for constructor).
Trying to regenerate the index cache...
Downloading snapshot
/usr/bin/curl 'http://head.hackage.haskell.org/snapshot.json' --output /tmp/transportAdapterGet31387-4 --location --write-out '%{http_code}' --user-agent 'cabal-install/3.0.0.0 (linux; x86_64)' --silent --show-error --dump-header /tmp/curl-headers31387-5.txt --header 'Cache-Control: no-transform'
Downloading mirrors
/usr/bin/curl 'http://head.hackage.haskell.org/mirrors.json' --output /tmp/transportAdapterGet31387-7 --location --write-out '%{http_code}' --user-agent 'cabal-install/3.0.0.0 (linux; x86_64)' --silent --show-error --dump-header /tmp/curl-headers31387-8.txt --header 'Cache-Control: no-transform'
Updating index
/usr/bin/curl 'http://head.hackage.haskell.org/01-index.tar.gz' --output /tmp/transportAdapterGet31387-10 --location --write-out '%{http_code}' --user-agent 'cabal-install/3.0.0.0 (linux; x86_64)' --silent --show-error --dump-header /tmp/curl-headers31387-11.txt --header 'Cache-Control: no-transform' --header 'Range: bytes=152495-127581'
Unexpected response 416for http://head.hackage.haskell.org/01-index.tar.gz

After updating cabal on another box.

[dpwiz]

This is probably a server issue. HTTP 416 means "range not satisfiable".

...
Updating index
/usr/bin/curl ... --header 'Range: bytes=152495-127581'

Huh? The start is larger than the end of the range. It is unsatisfiable indeed. But not for the server reasons.

[gbaz]

I can't reproduce this. I wonder if it has to do with how head.hackage is configured/built? Perhaps it occasionally "resets" and confuses cabal's range update logic.

[andreabedini]

There is no way we can reproduce this now.

Most likely, what had happened is that head.hackage index file http://head.hackage.haskell.org/01-index.tar.gz had been updated but not incrementally.

Cabal was trying to fetch an incremental update, computing the range from the length of its current 01-index.tar.gz to the length of http://head.hackage.haskell.org/01-index.tar.gz as stored in the TUF snapshot.json metadata). Given the remove file had been entirely rewritten, this failed badly.

I think nowadays the story is different, cabal will try a couple of times the incremental update, checking the hash of the resulting 01-index.tar.gz matches snapshot.json. Failing that cabal will download the index from scratch, surviving the issue.

This logic is actually in hackage-security not in Cabal itself, see https://github.com/haskell/hackage-security/blob/0ca094698ef50f126b0575922a0d2d33970c87a5/hackage-security/src/Hackage/Security/Client/Repository/Remote.hs#L220

I have been playing with situations like this myself and I do rely on this "fail-path".

[gbaz]

I think the above PR resolved the underlying problem, so will close optimistically.

[andreabedini]

I agree. I think it would nice to have some guarantees about how updates are performed but that's subject for another discussion. 👍

[arguri]

If anyone stumbles upon this issue, as I just have due to failing cabal update under Windows today with an 85 days old index: the workaround was to remove the content under c:\cabal\packages (or where ever you installed cabal)