confusing error-message "thread blocked indefinitely in an MVar operation" when downloading tarball fails

[hvr]

when download a source-tarball fails due to missing file we currently get a somewhat unfortunate error message:

$ cabal new-build -j1 exe:cabal -v3
File monitor 'improved-plan' unchanged.
In order, the following will be built:
HTTP-4000.3.3 (lib:HTTP) (requires download & build)
cabal-install-1.24.0.0 (exe:cabal) (configuration changed)
creating /stuff3/work/GitHub/cabal/dist-newstyle/build
creating /stuff3/work/GitHub/cabal/dist-newstyle/tmp
Downloading HTTP-4000.3.3...
writing
/home/hvr/.cabal/packages/hackage.haskell.org/HTTP/4000.3.3/HTTP-4000.3.3.tar.gz
Ready to install HTTP-4000.3.3
Selected mirror http://hackage.haskell.org/
Waiting for install task to finish...
Waiting for download of HTTP-4000.3.3 to finish
Downloading package HTTP-4000.3.3
Sending:
GET /package/HTTP-4000.3.3.tar.gz HTTP/1.1
Host: hackage.haskell.org
User-Agent: cabal-install/1.24.0.0 (linux; x86_64)
Cache-Control: no-transform

Creating new connection to hackage.haskell.org
Received:
HTTP/1.1 404 Not Found
Server: nginx/1.10.0 (Ubuntu)
Date: Mon, 02 May 2016 09:23:40 GMT
Content-Type: text/html
Content-Length: 178
Connection: keep-alive

Exception Unexpected response 404for
http://hackage.haskell.org/package/HTTP-4000.3.3.tar.gz when using mirror
http://hackage.haskell.org/
Selected mirror http://ftp.osuosl.org/pub/hackage
Downloading package HTTP-4000.3.3
Sending:
GET /pub/hackage/package/HTTP-4000.3.3.tar.gz HTTP/1.1
Host: ftp.osuosl.org
User-Agent: cabal-install/1.24.0.0 (linux; x86_64)
Cache-Control: no-transform

Creating new connection to ftp.osuosl.org
Received:
HTTP/1.1 404 Not Found
Date: Mon, 02 May 2016 09:24:35 GMT
Server: Apache
Content-Length: 302
Content-Type: text/html; charset=iso-8859-1

thread blocked indefinitely in an MVar operation

/cc @dcoutts

[dcoutts]

Related to #3416