Get rid of persistent build errors due to non-atomic file writes in GHC and stack

[nh2]

GHC as of writing does not ensure that files are written atomically.

This means that a Ctrl+C, kill or reboot at the right time can result in truncated files.
GHC does not detect this, so that resuming/rerunning the build with ghc --make continues to show up error messages.

In such situation, the only workaround is to wipe all files (e.g. stack/cabal/make clean).

Specifically, we've observed the following to happen:

• object files being written half-way, resulting in persistent linker errors
• executable files being written half-way, so they can start executing but then crash
• users reporting downstream tooling bug reports that only a build directory wipe helped

There may be situations where stack has this problem too, but so far we believe all occurrences of this that we see are in GHC.

GHC issue about this: #14533 - Make GHC more robust against PC crashes by using atomic writes

Repro

@lehins made a repro script that shows the problem in GHC at https://github.com/lehins/exec-kill-loop

Planned solution

• GHC should use atomic writes (write to temp file, then rename() syscall).
  • Make repro script that shows the problem (done)
  • Decrease repro waiting time from hours to a few minutes by killing just at the right time, doing syscall interception (done with hatrace test)
  • Find all places where GHC does non-atomic file writes, and fix them (ghc issue)
  • If possible, add this as a system test to GHC CI

Related issues

• Missing .dyn_o file does not trigger module rebuild #1842 (and GHC #9121 - Presence of dyn_o files not checked upon recompilation)

Issues for which this may be the reason

• Linker error when compiling with --trace/--profile #2422 (comment)
• stack build failing with new project on MacOS #4399 (comment)
• https://stackoverflow.com/questions/54156236/how-to-resolve-however-the-given-installed-package-instance-does-not-exist-in

[nh2]

In case we need to add atomic writes for stack, in commercialhaskell/rio#138 @roman introduced atomic+durable file writes that we can use.

However, as per this:

There is no function exported by this module that provides /only/ atomicity.

We probably want to add that since for many cases, atomic-but-not-durable is enough, and durability (fsync) makes things a lot slower.

[nh2]

New task done:

• Decrease repro waiting time from hours to a few minutes by killing just at the right time, doing syscall interception

Here comes hatrace, a Haskell-API for strace-style syscall incerception and scripting.

I made it for my work on https://phabricator.haskell.org/D42, but it's very useful for this problem, too.

[nh2]

My GHC patch for writing .o files atomically got merged:

https://gitlab.haskell.org/ghc/ghc/merge_requests/391

This should fix the biggest source for these errors, but the other files GHC and its subprograms write need to be done.

[snoyberg]

@nh2 I don't think this issue belongs on the Stack issue tracker, as it's entirely upstream. Any objection to closing it?

[nh2]

The main GHC bug has been fixed, but I still wanted to do an investigation with hatrace on

There may be situations where stack has this problem too

to ensure that there's no case where stack may write non-atomic files.

I'd like to write a hatrace filter like --find-nonatomic-writes that listens for all open()+close()s without subsequent rename(), so that it can print a report like

The following files were written nonatomically by the program:
  - path/to/file.o
  - ...

[nh2]

In nh2/hatrace#9 @qrilka implemented a --find-nonatomic-writes mode for hatrace that can point out non-atomic writes happening.

In nh2/hatrace#9 (comment) we recoded its output for a stack build invocation. It yields some paths that are written non-atomically, and we should check whether those are benign or should be improved.

Especially interesting are the entries

 - "/home/niklas/src/haskell/call-haskell-from-anything/.stack-work/install/x86_64-linux/lts-8.22/8.0.2/lib/x86_64-linux-ghc-8.0.2/call-haskell-from-anything-1.1.0.0-6Xcajdv7jFLAvwMt571tKe/st4joiWL/DataKinds.o"
 - "/home/niklas/src/haskell/call-haskell-from-anything/.stack-work/install/x86_64-linux/lts-8.22/8.0.2/lib/x86_64-linux-ghc-8.0.2/call-haskell-from-anything-1.1.0.0-6Xcajdv7jFLAvwMt571tKe/st4joiWL/Msgpack.o"
 - "/home/niklas/src/haskell/call-haskell-from-anything/.stack-work/install/x86_64-linux/lts-8.22/8.0.2/lib/x86_64-linux-ghc-8.0.2/call-haskell-from-anything-1.1.0.0-6Xcajdv7jFLAvwMt571tKe/st4joiWL/TH.o"
 - "/home/niklas/src/haskell/call-haskell-from-anything/.stack-work/install/x86_64-linux/lts-8.22/8.0.2/lib/x86_64-linux-ghc-8.0.2/call-haskell-from-anything-1.1.0.0-6Xcajdv7jFLAvwMt571tKe/st4joiWL/TypeUncurry.o"

which disappear after stack build completes.

[nh2]

I think the fastest way to investigate it would be to add a command like hatrace --kill-after-write /path/to/Myfile.o, and then we can run it on each file shown by --find-nonatomic-writes, and afterwards run another stack build or stack repl and see if it recovers or crashes.

[snoyberg]

This is referring to writes in GHC itself, not Stack, correct?

[qrilka]

@snoyberg the most of non-atomically written files are form GHC, I think we see some from Cabal e.g. from the copy operation but also there are some produced by Stack, I think at least these:

 - "/home/niklas/src/haskell/call-haskell-from-anything/.stack-work/dist/x86_64-linux/Cabal-1.24.2.0/stack-config-cache"
 - "/home/niklas/src/haskell/call-haskell-from-anything/.stack-work/install/x86_64-linux/lts-8.22/8.0.2/flag-cache/call-haskell-from-anything-1.1.0.0-6Xcajdv7jFLAvwMt571tKe"

And during our call Niklas was able to reproduce Stack failure by cutting one of those

[snoyberg]

Awesome, that's good to know @qrilka. I believe both of those are going away with @borsboom's changes to remove store.

[qrilka]

We were testing 1.9.3 so it makes total sense to do proper testing with master

[borsboom]

Awesome, that's good to know @qrilka. I believe both of those are going away with @borsboom's changes to remove store.

That is correct.

[snoyberg]

Possibly relevant?

• Power failure during stack build results in corruption to global Stack directory #3248
• (windows) stack build fails due to encoding of setup-shim-******.hs #3599

[nh2]

Yes, those look relevant.

The last one suffers from non-atomic writes to .hi files, which is something I haven't fixed yet in GHC (only object files).