Stack rebuilds pretty-show and few more packages with no apparent reason

[gromakovsky]

General summary/comments (optional)

If I build one package, modify one file there and built it again, for some reason more than one package is rebuilt.

Steps to reproduce

Can be reproduced using this code: https://github.com/serokell/cardano-sl/tree/vanya (serokell/cardano-sl repository, vanya branch).

1. Build the cardano-sl-txp package:

stack build --fast cardano-sl-txp

2. Modify some file:

echo "\n -- aa" >> txp/src/Pos/Txp/Base.hs

3. Build cardano-sl-txp again:

stack build --fast cardano-sl-txp

Expected

Only one file is recompiled and the command finishes quickly.

Actual

Stack decides to unregister cardano-sl-core, pretty-show, cardano-sl-crypto-test and some other packages. Then it builds these packages:

cardano-sl-core-1.2.0: unregistering
cardano-sl-crypto-test-1.1.0: unregistering
cardano-sl-db-1.2.0: unregistering
cardano-sl-infra-1.2.0: unregistering
cardano-sl-txp-1.2.0: unregistering
hedgehog-0.6: unregistering
pretty-show-1.6.16: unregistering
pretty-show-1.6.16: configure
pretty-show-1.6.16: build
pretty-show-1.6.16: copy/register
hedgehog-0.6: configure (lib)
hedgehog-0.6: build (lib) 
hedgehog-0.6: copy/register
cardano-sl-crypto-test-1.1.0: configure (lib)
cardano-sl-crypto-test-1.1.0: build (lib) 
cardano-sl-crypto-test-1.1.0: copy/register
cardano-sl-core-1.2.0: configure (lib)    
cardano-sl-core-1.2.0: build (lib)        
cardano-sl-core-1.2.0: copy/register
cardano-sl-db-1.2.0: configure (lib)
cardano-sl-db-1.2.0: build (lib)   
cardano-sl-db-1.2.0: copy/register
cardano-sl-infra-1.2.0: configure (lib)
cardano-sl-infra-1.2.0: build (lib) 
cardano-sl-infra-1.2.0: copy/register
cardano-sl-txp-1.2.0: configure (lib)
Configuring cardano-sl-txp-1.2.0... 
cardano-sl-txp-1.2.0: build (lib) 
Preprocessing library for cardano-sl-txp-1.2.0..
Building library for cardano-sl-txp-1.2.0..
[23 of 37] Compiling Pos.Txp.Base     ( src/Pos/Txp/Base.hs, .stack-work/dist/x86_64-linux-tinfo6/Cabal-2.0.1.0/build/Pos/Txp/Base.o )
cardano-sl-txp-1.2.0: copy/register
Installing library in /home/gromak/Work/cardano-sl/.stack-work/install/x86_64-linux-tinfo6/lts-11.2/8.2.2/lib/x86_64-linux-ghc-8.2.2/cardano-sl-txp-1.2.0-kSN1N3EEYBFSpz2b5PzUX
Registering library for cardano-sl-txp-1.2.0..
Completed 7 action(s).            

Here is the output in --verbose mode: https://gist.github.com/gromakovsky/a23b77cb9bcf3cc0e415b9a65f5f422d

Stack version

$ stack --version
Version 1.7.1, Git revision 681c800873816c022739ca7ed14755e85a579565 (5807 commits) x86_64 hpack-0.28.2

Method of installation

I am using version 1.7.1 downloaded from https://github.com/commercialhaskell/stack/releases.

[nh2]

We'll be looking into this.

[snoyberg]

Note that it's a better idea for repros to give a commit than a branch. Right now, the vanya branch is pointing at 6c109584a6a7f182334eedc79c063a591626cca9.

[snoyberg]

Also, as a side note: due to the extremely large number of extra-deps in this stack.yaml, I'd recommend moving over to a custom snapshot, which will allow for more precompiled cache usage.

[snoyberg]

I'm going to start a debug log in this comment. I'll be editing it as I move along.

My first step is to confirm that this bug reproduces with the Stack on master. Also, in order to avoid wasting lots of CPU cycles, I'm going to be using the following command as a replacement for step 3 above:

stack build --fast cardano-sl-txp --dry-run

This way I'll keep the same broken state between attempts to play with this bug. I'm also almost certainly going to throw in a --verbose, and then begin adding a bunch of logInfo calls to the Stack codebase to get more info on why it's decided to unregister the packages in question.

I'm still building the absolute latest master, but I've tested with:

Version 1.8.0, Git revision d20833a5c0d3b2b62495fb126ecfd8fdbf1f6864 (5895 commits) x86_64 hpack-0.28.2

I get the following output, indicating that the bug no longer exists:

$ stack build --fast cardano-sl-txp --dry-run

Warning: There were multiple candidates for the Cabal entry " hlint.hs 
         * /Users/michael/Documents/cardano-sl/.stack-work/downloaded/XTTklt4ROs4T/hlint.hs 
         picking:
         /Users/michael/Documents/cardano-sl/.stack-work/downloaded/XTTklt4ROs4T/tests/hlint.hs
Would unregister locally:
cardano-sl-txp-1.2.0 (local file changes: src/Pos/Txp/Base.hs)

Would build:
cardano-sl-txp-1.2.0: database=local, source=/Users/michael/Documents/cardano-sl/txp/

No executables to be installed.

However, building with Stack 1.7.1 produces the same bug as reported here:

$ stack-1.7.1 build --fast cardano-sl-txp --dry-run

Warning: There were multiple candidates for the Cabal entry " hlint.hs 
         * /Users/michael/Documents/cardano-sl/.stack-work/downloaded/XTTklt4ROs4T/hlint.hs 
         picking: /Users/michael/Documents/cardano-sl/.stack-work/downloaded/XTTklt4ROs4T/tests/hlint.hs
Would unregister locally:
cardano-crypto-1.0.0 (missing dependencies: cryptonite, cryptonite-openssl)
cardano-report-server-0.4.10 (missing dependencies: log-warper, wreq)
cardano-sl-binary-1.2.0 (missing dependencies: serokell-util)
cardano-sl-core-1.2.0 (missing dependencies: cardano-sl-binary, cardano-sl-crypto, cardano-sl-crypto-test, cardano-sl-networking, cardano-sl-u...)
cardano-sl-crypto-1.2.0 (missing dependencies: cardano-crypto, cardano-sl-binary, cardano-sl-util, cryptonite, cryptonite-openssl, ed25519, pvss...)
cardano-sl-crypto-test-1.1.0 (missing dependencies: cardano-sl-binary, cardano-sl-crypto, cardano-sl-util, cardano-sl-util-test, cryptonite, hedgehog)
cardano-sl-db-1.2.0 (missing dependencies: cardano-sl-binary, cardano-sl-core, cardano-sl-crypto, cardano-sl-util, rocksdb-haskell-ng, serok...)
cardano-sl-infra-1.2.0 (missing dependencies: cardano-report-server, cardano-sl-binary, cardano-sl-core, cardano-sl-crypto, cardano-sl-db, card...)
cardano-sl-networking-1.2.0 (missing dependencies: cardano-sl-util, cryptonite, kademlia, log-warper, network-transport-tcp, serokell-util)
cardano-sl-txp-1.2.0 (missing dependencies: cardano-sl-binary, cardano-sl-core, cardano-sl-crypto, cardano-sl-crypto-test, cardano-sl-db, car...)
cardano-sl-util-1.2.0 (missing dependencies: cryptonite, log-warper, serokell-util)
cardano-sl-util-test-1.1.0 (missing dependencies: cardano-sl-util, cryptonite)
connection-0.2.8 (missing dependencies: tls, x509, x509-store, x509-system, x509-validation)
cryptonite-0.24 (local file changes: benchs/Bench.hs benchs/Number/F2m.hs cbits/aes/block128.h cbits/aes/generic.h cbits/aes/gf.h cbit...)
cryptonite-openssl-0.7 (missing dependencies: cryptonite)
dns-3.0.2 (missing dependencies: cryptonite)
ed25519-0.0.5.0 (local file changes: .travis.yml AUTHORS.txt CHANGELOG.md CONTRIBUTING.md README.md benchmarks/bench.hs ed25519.cabal ...)
ekg-wai-0.1.0.2 (missing dependencies: wai-app-static)
hedgehog-0.6 (local file changes: /Users/michael/Documents/cardano-sl/.stack-work/downloaded/27pmaoPEDNVl/CHANGELOG.md /Users/micha...)
http-client-tls-0.3.5.3 (missing dependencies: connection, cryptonite, tls)
kademlia-1.1.0.1 (missing dependencies: cryptonite)
log-warper-1.8.10.1 (local file changes: CHANGES.md README.md examples/HowTo.md examples/PureLogging.md log-warper.cabal test/Spec.hs test...)
network-transport-tcp-0.6.0 (local file changes: ChangeLog network-transport-tcp.cabal tests/TestQC.hs tests/TestTCP.hs)
plutus-prototype-0.1.0.0 (missing dependencies: cardano-crypto, cryptonite, ed25519)
pvss-0.2.0 (missing dependencies: cryptonite, cryptonite-openssl)
rocksdb-haskell-ng-0.0.0 (local file changes: bench/Main.hs rocksdb-haskell-ng.cabal test/Spec.hs)
serokell-util-0.8.0 (missing dependencies: log-warper)
tls-1.4.1 (missing dependencies: cryptonite, x509, x509-store, x509-validation)
wai-app-static-3.1.6.2 (missing dependencies: cryptonite)
wreq-0.5.2.1 (missing dependencies: cryptonite, http-client-tls)
x509-1.7.3 (missing dependencies: cryptonite)
x509-store-1.6.6 (missing dependencies: cryptonite, x509)
x509-system-1.6.6 (missing dependencies: x509, x509-store)
x509-validation-1.6.10 (missing dependencies: cryptonite, x509, x509-store)

Would build:
cardano-crypto-1.0.0: database=local, source=/Users/michael/Documents/cardano-sl/.stack-work/downloaded/Du3of51NERkg/, after: cryptonite-0.24,cryptonite-openssl-0.7
cardano-report-server-0.4.10: database=local, source=/Users/michael/Documents/cardano-sl/.stack-work/downloaded/HxVyEV0NYAPr/, after: log-warper-1.8.10.1,wreq-0.5.2.1
cardano-sl-binary-1.2.0: database=local, source=/Users/michael/Documents/cardano-sl/binary/, after: serokell-util-0.8.0
cardano-sl-core-1.2.0: database=local, source=/Users/michael/Documents/cardano-sl/core/, after: cardano-sl-binary-1.2.0,cardano-sl-crypto-1.2.0,cardano-sl-crypto-test-1.1.0,cardano-sl-networking-1.2.0,cardano-sl-util-1.2.0,cardano-sl-util-test-1.1.0,cryptonite-0.24,log-warper-1.8.10.1,plutus-prototype-0.1.0.0,serokell-util-0.8.0
cardano-sl-crypto-1.2.0: database=local, source=/Users/michael/Documents/cardano-sl/crypto/, after: cardano-crypto-1.0.0,cardano-sl-binary-1.2.0,cardano-sl-util-1.2.0,cryptonite-0.24,cryptonite-openssl-0.7,ed25519-0.0.5.0,pvss-0.2.0,serokell-util-0.8.0
cardano-sl-crypto-test-1.1.0: database=local, source=/Users/michael/Documents/cardano-sl/crypto/test/, after: cardano-sl-binary-1.2.0,cardano-sl-crypto-1.2.0,cardano-sl-util-1.2.0,cardano-sl-util-test-1.1.0,cryptonite-0.24,hedgehog-0.6
cardano-sl-db-1.2.0: database=local, source=/Users/michael/Documents/cardano-sl/db/, after: cardano-sl-binary-1.2.0,cardano-sl-core-1.2.0,cardano-sl-crypto-1.2.0,cardano-sl-util-1.2.0,rocksdb-haskell-ng-0.0.0,serokell-util-0.8.0
cardano-sl-infra-1.2.0: database=local, source=/Users/michael/Documents/cardano-sl/infra/, after: cardano-report-server-0.4.10,cardano-sl-binary-1.2.0,cardano-sl-core-1.2.0,cardano-sl-crypto-1.2.0,cardano-sl-db-1.2.0,cardano-sl-networking-1.2.0,cardano-sl-util-1.2.0,dns-3.0.2,ekg-wai-0.1.0.2,http-client-tls-0.3.5.3,kademlia-1.1.0.1,log-warper-1.8.10.1,network-transport-tcp-0.6.0,serokell-util-0.8.0
cardano-sl-networking-1.2.0: database=local, source=/Users/michael/Documents/cardano-sl/networking/, after: cardano-sl-util-1.2.0,cryptonite-0.24,kademlia-1.1.0.1,log-warper-1.8.10.1,network-transport-tcp-0.6.0,serokell-util-0.8.0
cardano-sl-txp-1.2.0: database=local, source=/Users/michael/Documents/cardano-sl/txp/, after: cardano-sl-binary-1.2.0,cardano-sl-core-1.2.0,cardano-sl-crypto-1.2.0,cardano-sl-crypto-test-1.1.0,cardano-sl-db-1.2.0,cardano-sl-infra-1.2.0,cardano-sl-networking-1.2.0,cardano-sl-util-1.2.0,cardano-sl-util-test-1.1.0,log-warper-1.8.10.1,plutus-prototype-0.1.0.0,rocksdb-haskell-ng-0.0.0,serokell-util-0.8.0
cardano-sl-util-1.2.0: database=local, source=/Users/michael/Documents/cardano-sl/util/, after: cryptonite-0.24,log-warper-1.8.10.1,serokell-util-0.8.0
cardano-sl-util-test-1.1.0: database=local, source=/Users/michael/Documents/cardano-sl/util/test/, after: cardano-sl-util-1.2.0,cryptonite-0.24
connection-0.2.8: database=local, source=package index, after: tls-1.4.1,x509-1.7.3,x509-store-1.6.6,x509-system-1.6.6,x509-validation-1.6.10
cryptonite-0.24: database=local, source=/Users/michael/Documents/cardano-sl/.stack-work/downloaded/lnvg6q-bhfcw/
cryptonite-openssl-0.7: database=local, source=package index, after: cryptonite-0.24
dns-3.0.2: database=local, source=package index, after: cryptonite-0.24
ed25519-0.0.5.0: database=local, source=/Users/michael/Documents/cardano-sl/.stack-work/downloaded/XTTklt4ROs4T/
ekg-wai-0.1.0.2: database=local, source=package index, after: wai-app-static-3.1.6.2
hedgehog-0.6: database=local, source=/Users/michael/Documents/cardano-sl/.stack-work/downloaded/27pmaoPEDNVl/hedgehog/
http-client-tls-0.3.5.3: database=local, source=package index, after: connection-0.2.8,cryptonite-0.24,tls-1.4.1
kademlia-1.1.0.1: database=local, source=/Users/michael/Documents/cardano-sl/.stack-work/downloaded/uiwbEc8tbIi9/, after: cryptonite-0.24
log-warper-1.8.10.1: database=local, source=/Users/michael/Documents/cardano-sl/.stack-work/downloaded/DOh1FXO6uJXI/
network-transport-tcp-0.6.0: database=local, source=/Users/michael/Documents/cardano-sl/.stack-work/downloaded/17t1IvOFATIc/
plutus-prototype-0.1.0.0: database=local, source=/Users/michael/Documents/cardano-sl/.stack-work/downloaded/KQSGhWYbOHSL/, after: cardano-crypto-1.0.0,cryptonite-0.24,ed25519-0.0.5.0
pvss-0.2.0: database=local, source=package index, after: cryptonite-0.24,cryptonite-openssl-0.7
rocksdb-haskell-ng-0.0.0: database=local, source=/Users/michael/Documents/cardano-sl/.stack-work/downloaded/WoYdLUpErLYh/
serokell-util-0.8.0: database=local, source=package index, after: log-warper-1.8.10.1
tls-1.4.1: database=local, source=package index, after: cryptonite-0.24,x509-1.7.3,x509-store-1.6.6,x509-validation-1.6.10
wai-app-static-3.1.6.2: database=local, source=package index, after: cryptonite-0.24
wreq-0.5.2.1: database=local, source=package index, after: cryptonite-0.24,http-client-tls-0.3.5.3
x509-1.7.3: database=local, source=package index, after: cryptonite-0.24
x509-store-1.6.6: database=local, source=package index, after: cryptonite-0.24,x509-1.7.3
x509-system-1.6.6: database=local, source=package index, after: x509-1.7.3,x509-store-1.6.6
x509-validation-1.6.10: database=local, source=package index, after: cryptonite-0.24,x509-1.7.3,x509-store-1.6.6

No executables to be installed.

Meaning I'm fairly certain that this bug is already resolved on master. It will take more investigation to determine what change on master could have resolved this.

[gromakovsky]

I couldn't fetch d20833a5c0d3b2b62495fb126ecfd8fdbf1f6864, so I tried using its parent: dbc74ecce2e31b47626d87a1cf3ea6129fed1e99.
d20833a5c0d3b2b62495fb126ecfd8fdbf1f6864 has only cosmetic changes, so it shouldn't matter.

For me it still unregisters packages that it shouldn't, here's the output of stack build --fast cardano-sl-txp --dry-run: https://gist.github.com/gromakovsky/7ca7bac5b4f5fedaf269665985eadb5a

[dbaynard]

It might be helpful to have a minimum example, though it seems that might be tricky. Would you try to produce one, @gromakovsky?

[gromakovsky]

I was successfully minimizing this example while keeping the bug reproducible, but then I did stack clean --full and since then I can't reproduce this bug anymore, even when I return to where I started (6c109584a6a7f182334eedc79c063a591626cca9) and use the same version of stack as before (681c800) 🤷‍♂️

Probably something bad appeared in .stack-work, I don't know how :(

[gromakovsky]

I suceeded to reproduce it using version 1.6.5 (revision 24ab0d6).
I also minimized it as much as I could.
Commit 571fd554b3e1613fc27e26563f4d9dee823b4de6 is the minimal example where the bug is present. It's a bit different though. Steps:

1. stack clean --full.
2. stack build --fast cardano-sl-core.
3. stack build --fast cardano-sl-core again rebuilds hedgehog and others.

Commit adb0e36ab6be0d0f4c907d1b9fc866e67bcdf673 is the next one in the vanya branch and there doing stack build --fast cardano-sl-core again doesn't rebuild anything.
I stopped at that point.

Once again, after stack clean --full the bug reproduces only with stack revision 24ab0d6. I can't reproduce it using latest release or master anymore (I could before probably because I didn't do stack clean --full).

[dbaynard]

OK, it's good that there's a resolution. Thank you for digging around (I editied your commit refs to links so I could follow them). I suspect you're right about something in the .stack-work folder being a culprit — and the extra-deps appear to have contributed, too.

I'd recommend @snoyberg's advice: it is worth using a custom snapshot.

I'm going to close — do get in touch if there's anything else you'd need, on this.