Rebuilding packages from the global database with modified dependencies

[snoyberg]

This applies to Stack master, specifically commit 9900d99. Repro (on a non-Windows system). With the following stack.yaml:

resolver: ghc-8.6.3
packages: []
extra-deps:
- time-1.8.0.1

This replaces the global time package with a different version. Running stack build unix causes both time and unix to be compiled, which makes sense, since unix depends on time.

Running stack build unix a second time should be a no-op, but instead, it rebuilds:

$ stack build unix
unix-2.7.2.2: configure
^C

This appears to be because the SourceMap knows nothing about unix (since it's a global package and not present at all in snapshot), and therefore Stack is ignoring the already-present copy in the snapshot database.

Possible solution: update the SourceMap to contain information about global packages.

CC @qrilka

[dbaynard]

Could #4508 require the same fix?

[qrilka]

@dbaynard that one looks to be quite a different issue

[snoyberg]

After some further discussion with @qrilka and a reminder that global packages may not match what's on Hackage, I have a different proposal: https://www.snoyman.com/blog/2019/01/mismatched-global-packages.

[qrilka]

It looks like we could have troubles going that way but correct me if there are any flaws in my logic:
Among other GHC boot libraries there's also ghc which is a bit special - it is said to be wired-in but at the same time it depends on a number of "independent" packages, e.g. directory, process, containers. We are OK as far as none of these packages appears in a Stackage snapshot but if it does we could have problems with packages depending on ghc, one example of such a package could be https://www.stackage.org/package/sbv
So I think we shouldn't deal specially with wired-in packages and global packages pruning should be done starting in Stackage curator so problems could be caught earlier.

[snoyberg]

@qrilka I don't think this finding is anything new. Stackage already ensures that packages from the global database, or at least those which are transitive dependencies of ghc, don't get included in a snapshot, exactly for this purpose.

[qrilka]

I didn't know that there's a special check for that, I guess it needs to be added into the new curator?

[snoyberg]

Yes, I guess so. I didn't think about that previously.