Optionally go to hackage.haskell.org instead of hackage.fpcomplete.com for source .gz files.

[philipcraig]

General summary/comments (optional)

Allow stack to use hackage.haskell.org instead of hackage.fpcomplete.com for source files.

The reason is that hackage.haskell.org doesn't mirror / gateway the link into an amazon s3 link, which is banned from big company firewalls. hackage,fpcomplete.com does.

Steps to reproduce

For example:

stack install clock

Wanted

Have an option such that stack downloads clock from https://hackage.haskell.org/package/clock-0.7.2/clock-0.7.2.tar.gz

Actual

Downloads clock from https://hackage.fpcomplete.com/package/clock-0.7.2.tar.gz which turns into an s3 request from amazon, which is banned by big companies.

Stack version

$ stack --version
Version 1.1.2, Git revision c6dac65e3174dea79df54ce6d56f3e98bc060ecc (3647 commits) x86_64 hpack-0.14.0

### Method of installation

* Official binary, downloaded from stackage.org

[Blaisorblade]

The docs is unclear enough I had to check the sources and I'm still not sure, but we might be supporting this already. Based on https://github.com/commercialhaskell/stack/blob/master/doc/yaml_configuration.md#package-indices, can you adding to your config in ~/.stack/config.yaml the following?

package-indices:
- name: Hackage
  download-prefix: https://hackage.haskell.org/package/
  git: https://github.com/commercialhaskell/all-cabal-hashes.git

If that works, this is a documentation bug.

[philipcraig]

Hi,

Thanks for the info, but no joy yet.

I've tried adding that in three places:

STACK_ROOT/config.yaml
STACK_ROOT/global-project/stack.yaml
project level stack.yaml

In each, it seems to make no difference to behaviour, so, e.g.

"stack install clock"

or adding a clock dependency results in a failed trip to s3, as before.

[Blaisorblade]

Can you post a complete log of stack install -v clock?

[philipcraig]

C:\hsdev\foo>stack install -v clock
Version 1.1.2, Git revision c6dac65e3174dea79df54ce6d56f3e98bc060ecc (3647 commits) x86_64 hpack-0.14.0
2016-08-18 20:09:26.459270: [debug] Checking for project config at: C:\hsdev\foo\stack.yaml @(stack_9kewhubnl5WIl89fhd1ea2:Stack.Config src/Stack\Config.hs:811:9)
2016-08-18 20:09:26.461270: [debug] Loading project config file stack.yaml @(stack_9kewhubnl5WIl89fhd1ea2:Stack.Config src/Stack\Config.hs:829:13)
2016-08-18 20:09:26.465270: [debug] Trying to decode C:\Stack\build-plan-cache\x86_64-windows\lts-6.12.cache @(stack_9kewhubnl5WIl89fhd1ea2:Data.Binary.VersionTagged src/Data\Binary\VersionTagged.hs:55:5)
2016-08-18 20:09:26.474271: [debug] Success decoding C:\Stack\build-plan-cache\x86_64-windows\lts-6.12.cache @(stack_9kewhubnl5WIl89fhd1ea2:Data.Binary.VersionTagged src/Data\Binary\VersionTagged.hs:64:13)
2016-08-18 20:09:26.475271: [debug] Getting system compiler version @(stack_9kewhubnl5WIl89fhd1ea2:Stack.Setup src/Stack\Setup.hs:341:17)
2016-08-18 20:09:26.477272: [debug] Run process: ghc --info @(stack_9kewhubnl5WIl89fhd1ea2:System.Process.Read src/System\Process\Read.hs:283:3)
2016-08-18 20:09:26.669291: [debug] Asking GHC for its version @(stack_9kewhubnl5WIl89fhd1ea2:Stack.Setup.Installed src/Stack\Setup\Installed.hs:94:13)
2016-08-18 20:09:26.670291: [debug] Run process: ghc --numeric-version @(stack_9kewhubnl5WIl89fhd1ea2:System.Process.Read src/System\Process\Read.hs:283:3)
2016-08-18 20:09:26.786302: [debug] Getting Cabal package version @(stack_9kewhubnl5WIl89fhd1ea2:Stack.GhcPkg src/Stack\GhcPkg.hs:165:5)
2016-08-18 20:09:26.787303: [debug] Run process: ghc-pkg --no-user-package-db field --simple-output Cabal version @(stack_9kewhubnl5WIl89fhd1ea2:System.Process.Read src/System\Process\Read.hs:283:3)
2016-08-18 20:09:26.985322: [debug] Resolving package entries @(stack_9kewhubnl5WIl89fhd1ea2:Stack.Setup src/Stack\Setup.hs:221:5)
2016-08-18 20:09:26.987323: [debug] Getting global package database location @(stack_9kewhubnl5WIl89fhd1ea2:Stack.GhcPkg src/Stack\GhcPkg.hs:48:5)
2016-08-18 20:09:26.987323: [debug] Run process: ghc-pkg --no-user-package-db list --global @(stack_9kewhubnl5WIl89fhd1ea2:System.Process.Read src/System\Process\Read.hs:283:3)
2016-08-18 20:09:27.137338: [debug] Run process: ghc-pkg --global --no-user-package-db dump --expand-pkgroot @(stack_9kewhubnl5WIl89fhd1ea2:System.Process.Read src/System\Process\Read.hs:283:3)
2016-08-18 20:09:27.345358: [debug] Ignoring package haskeline due to wanting version 0.7.2.3 instead of 0.7.2.1 @(stack_9kewhubnl5WIl89fhd1ea2:Stack.Build.Installed src/Stack\Build\Installed.hs:189:5)
2016-08-18 20:09:27.346358: [debug] Ignoring package Cabal due to wanting version 1.22.8.0 instead of 1.22.5.0 @(stack_9kewhubnl5WIl89fhd1ea2:Stack.Build.Installed src/Stack\Build\Installed.hs:189:5)
2016-08-18 20:09:27.349359: [debug] Run process: ghc-pkg --user --no-user-package-db --package-db C:\Stack\snapshots\2bcf57b6\pkgdb dump --expand-pkgroot @(stack_9kewhubnl5WIl89fhd1ea2:System.Process.Read src/System\Process\Read.hs:283:3)
2016-08-18 20:09:27.494373: [debug] Run process: ghc-pkg --user --no-user-package-db --package-db C:\hsdev\foo\.stack-work\install\cf5a45a3\pkgdb dump --expand-pkgroot @(stack_9kewhubnl5WIl89fhd1ea2:System.Process.Read src/System\Process\Read.hs:283:3)
2016-08-18 20:09:27.686392: [debug] Trying to decode C:\Stack\indices\Hackage\00-index.cache @(stack_9kewhubnl5WIl89fhd1ea2:Data.Binary.VersionTagged src/Data\Binary\VersionTagged.hs:55:5)
2016-08-18 20:09:27.880412: [debug] Success decoding C:\Stack\indices\Hackage\00-index.cache @(stack_9kewhubnl5WIl89fhd1ea2:Data.Binary.VersionTagged src/Data\Binary\VersionTagged.hs:64:13)
2016-08-18 20:09:27.898414: [debug] Getting global package database location @(stack_9kewhubnl5WIl89fhd1ea2:Stack.GhcPkg src/Stack\GhcPkg.hs:48:5)
2016-08-18 20:09:27.899414: [debug] Run process: ghc-pkg --no-user-package-db list --global @(stack_9kewhubnl5WIl89fhd1ea2:System.Process.Read src/System\Process\Read.hs:283:3)
2016-08-18 20:09:28.039428: [debug] Precompiled cache input = ["--dependency=base=base-4.8.2.0-14035a44a8b95c6832da6dae1420f59e"] @(stack_9kewhubnl5WIl89fhd1ea2:Stack.Build.Cache src/Stack\Build\Cache.hs:261:13)
2016-08-18 20:09:28.043428: [debug] Downloading /hackage.fpcomplete.com/package/clock-0.7.2.tar.gz @(stack_9kewhubnl5WIl89fhd1ea2:Network.HTTP.Download.Verified src/Network\HTTP\Download\Verified.hs:231:9) ProxyConnectException "s3.amazonaws.com" 443 (Right (StatusCodeException (Status {statusCode = 403, statusMessage = "Forbidden"}) [] (CJ {expose = []})))

[Blaisorblade]

Thanks! I fixed formatting using ```.

[Blaisorblade]

Thanks. I could reproduce this, and it basically seems that beyond changing the setting, you'll need to remove C:\Stack\build-plan-cache\x86_64-windows\lts-6.12.cache, since it caches even the URL to use!

To be safer, I'd also rename the package index by changing the snippet to:

package-indices:
- name: HackageOrig
  download-prefix: https://hackage.haskell.org/package/
  git: https://github.com/commercialhaskell/all-cabal-hashes.git

You want this only in STACK_ROOT/config.yaml (that is for you, IIUC, C:\stack\config.yaml).

[Blaisorblade]

EDIT: that's not enough it seems... I'll try more carefully.

[Blaisorblade]

So I think I managed. The config I needed was the following:

package-indices:
- name: Hackage
  download-prefix: https://hackage.haskell.org/package/
  http: https://hackage.haskell.org/00-index.tar.gz

The problem is that the all-cabal-hashes repository already contains two resolved URLs, listing both mirrors (from Amazon and from Hackage).

EDIT: the evidence I have is this log entry:

2016-08-18 22:09:30.351111: [debug] Downloading /package/clock-0.7.2.tar.gz @(stack_JAcs0EuAGbK5rtqRdoxQaL:Network.HTTP.Download.Verified src/Network/HTTP/Download/Verified.hs:231:9)

which prints the correct path (hostnames aren't included, see below).

Also, sorry @philipcraig for fooling myself so often, I hope I didn't again.

Technical details

For instance

$ git ls-tree current-hackage clock/0.7.2/clock.json
100644 blob 8d43ab2cc7a4cd4c7cccde2f00386390d9d5a13f    clock/0.7.2/clock.json
$ git cat-file -p 8d43ab2cc7a4cd4c7cccde2f00386390d9d5a13f
{"package-hashes":{"MD5":"3bb91ede556cde92a64c635da68699cd","Skein512_512":"98cdeff213b26ec5ad28249558df25446e1811bc421bf1ce9589035a6754ff57a03863cd7792b15d78ea70a85b6bae977009c2f4616a9dcc576550705db8a6cd","SHA1":"9ab44e86f04cd76eaf9cb0acea2223c1361ba931","SHA512":"cf44dfca6c0825a003018c68266ef0fe503a9b147aaf784435f6297b365c6d18fc71b63b77a23ee2a3298ace638a35de0c6118e361d43b418a45d4fdd24d57cf","SHA256":"886601978898d3a91412fef895e864576a7125d661e1f8abc49a2a08840e691f"},"package-locations":["https://hackage.haskell.org/package/clock-0.7.2/clock-0.7.2.tar.gz","https://s3.amazonaws.com/hackage.fpcomplete.com/package/clock-0.7.2.tar.gz"],"package-size":7465}

Of the URLs, stack insists using https://s3.amazonaws.com/hackage.fpcomplete.com/package/clock-0.7.2.tar.gz. The log says Downloading /hackage.fpcomplete.com/package/clock-0.7.2.tar.gz, but that's because there's no good way to print the URLs we parse (see #2431 and #2267).

[philipcraig]

It works! Thanks so much.

I'm not quite good to go yet though -- even with this config, if I want stack to setup ghc via stack setup, it still goes to s3 to download ghc.

If I give it a ghc that I already have, it's fine. The test is, I was able to stack install clock.

I can use --stack-setup-yaml to give it a customised stack-setup.yaml file, which doesn't point to s3 even for the setup ghc files.

However, in general use, I then get bitten by #2248. And I can't do a stack upgrade --git, as the stack upgrade process itself gets hit by the same issue.

[Blaisorblade]

Is there a way of persuading stack setup to get ghc from [...]

It seems yes, but I'm not sure how it exactly works (ahem, still new here). As long as that's a standard GHC bindist.

Looking at stack setup --help I see a --ghc-bindist which is made for you, but I'm not 100% sure how to use it. It mentions --ghc-variant (a global option, see stack --help). Maybe there are docs explaining this better?

Something is explained here, in "Support for custom GHC binary distributions". Maybe that helps?
https://github.com/commercialhaskell/stack/blob/master/ChangeLog.md#0150

(Alternatively, one could --stack-setup-yaml with the URL for a variant of https://raw.githubusercontent.com/fpco/stackage-content/master/stack/stack-setup-2.yaml, using the right URL, but that might be more annoying to do).

[Blaisorblade]

BTW, I consider even the part of the "solution" that worked as a hack.

[philipcraig]

Thanks. What would you consider not to be a hack? Removing the need to delete the cache file, or something else?

Also note that I still have #2248 in real-world use, and that stack upgrade --git when using hackage via the above settings also doesn't work due to, er, #2248.

Net result: can't yet use stack.

[Blaisorblade]

Also note that I still have #2248 in real-world use, and that stack upgrade --git when using hackage via the above settings also doesn't work due to, er, #2248.

Ah thanks, I've just seen that error too. I assume that would also happen if you use stack build from a git checkout, with a binary stack. It looks like you'd need a (64bit) stack binary. I have a Win10 64bit VM, but no clue about code signing.

Thanks. What would you consider not to be a hack? Removing the need to delete the cache file, or something else?

That might have been a shot in the dark—I'm not yet sure of the entire logic, and not sure of which steps are needed.

A problem is that, IIUC, you can't use git: https://github.com/commercialhaskell/all-cabal-hashes.git at all.
Hence, you download the index not incrementally but from scratch each time (like cabal does, so it's not so bad).

Why? You can't use that repo because it embeds actual URLs (an example is above in https://github.com/commercialhaskell/stack/issues/2509#issuecomment-240844481)—though it contains multiple URLs for each package, both Hackage and FPComplete. And for some reason it picks the FPComplete one, apparently always. One could maybe allow picking between them at least. But that's like a denormalized database, because it hardcodes a list of two mirrors. And that makes me unhappy.

Looking at the spec for that repo (https://github.com/commercialhaskell/commercialhaskell/wiki/Git-backed-Hackage-index-signing-and-distribution), It seems we'd want to restrict it to relative URLs.

[philipcraig]

You are right on all points.

[Blaisorblade]

So here's an unofficial Stack binary, for Win64, from current master (d5b8c6e), built on a MS Windows 10 VM. I suppose this is not up to code, and it's not signed, but it should avoid #2248 and get you going. You might want to do a stack upgrade --git to be safe.

https://www.dropbox.com/s/ti108qrvx1w83vb/stack-d5b8c6ef6740e280e73604a4e04d1d58c5b14413-win64.zip?dl=0

Note: I'll take this down soon.

Version output:

$ /c/Users/IEUser/Documents/stack/.stack-work/install/a604a5e3/bin/stack --version
Version 1.1.3, Git revision d5b8c6ef6740e280e73604a4e04d1d58c5b14413 (dirty) (4014 commits) x86_64 hpack-0.14.0

@sjakobi should we have nightlies instead? (I'm also happy with more common releases).

[sjakobi]

should we have nightlies instead? (I'm also happy with more common releases).

Nightly builds might be a good way to get more beta testers who could help us test new features and detect regressions.

@borsboom, your thoughts on this as stack's release manager?!

[philipcraig]

Thanks @Blaisorblade . That worked.

[Blaisorblade]

Regarding custom GHC URLs, we do have decent docs:
https://docs.haskellstack.org/en/latest/yaml_configuration/#ghc-variant
https://docs.haskellstack.org/en/latest/yaml_configuration/#setup-info

[philipcraig]

So, when could we expect nightlies? @borsboom

cheers.

[Blaisorblade]

FTR: @snoyberg pointed out we could fallback to Hackage when Stackage downloads fail, without changes in the repos.

[borsboom]

Nightlies would be nice; it's a matter of putting in the effort to get them set up (and decide which platforms they should be built for). A separate issue would make sense.

[mpilgrem]

I am going to close this given (a) the passage of time and (b) the fact that 'fpcomplete' does not appear in Stack's or Pantry's source code.