Garbled output due to per-component parallelism

[23Skidoo]

When building the log package with cabal new-build and GHC 7.8.4, I saw this:

[...]
[11 of 11] Compiling Log.Backend.ElasticSearch ( src/Log/Backend/ElasticSearch.hs, /home/refold/code/scrive/log/dist-newstyle/build/x86_64-linux/ghc-7.8.4/log-0.5.5/build/Log/Backend/ElasticSearch.o )
Configuring component test:log-test-integration from log-0.5.5
Configuring component test:log-test from log-0.5.5
Warning: The pWarning: aThcekage  has a verspackage hiaosn a  versriaonge fnor  arange  for da depenepdenedncye onn an interncaly on an in
ltibrarye:r nlaolg -any && ==0.5.5
, logl -any && ==i0.5.5. Tbhis rvaerrys:ion range  has nloo
semangtic meani ng an-da ncyan be r emoved.&
& ==0.5.5, log -any && ==0.5.5. This version range has no
semantic meaning and can be removed.
Preprocessing test suite 'log-test' for log-0.5.5...
Preprocessing test suite 'log-test-integration' for log-0.5.5...
[1 of 2] Compiling Test.ElasticSearch ( test/Test/ElasticSearch.hs, /home/refold/code/scrive/log/dist-newstyle/build/x86_64-linux/ghc-7.8.4/log-0.5.5/c/log-test/build/log-test/log-test-tmp/Test/ElasticSearch.o )
[...]

Looks like a synchronisation fail somewhere.

[23Skidoo]

Also interesting that the warning seems to be bogus: log-test and log-test-integration test suites don't have any constraint on the log library.

[ezyang]

This is going to be difficult to "debug" without a reproduceable test case. Perhaps there is some architectural rethink of how we do output in cabal-install that can eliminate this class of bugs entirely? IIUC we do put some effort into avoiding interleaving; how can we make sure it's impossible to forget?

[23Skidoo]

It happens relatively often on my machine when building the log package, but maybe it's just me.

[BardurArantsson]

It would obviously be a lot of quite tedious work, but my dream scenario would be https://hackage.haskell.org/package/concurrent-output

I imagine the build process output looking a lot like what's shown in https://buckbuild.com/static/buck-build-15fps.gif (perhaps one line per dependency currently building rather than per-file)

[BardurArantsson]

OK, so I've had a little think about this and looked a little bit into the code... I'm not sure the concurrent-output is really feasible as it stands[1], but I think we could make a little headway if we were to consider using a custom monad as the base monad instead of IO. To avoid having to do a huge refactor in one go we could have that monad be an instance of MonadIO, but require our custom OutputM (let's say) monad for anything that calls "warn" (etc.). I'm basically thinking something MTL-like such that explicit lifting could be avoided as much as possible. We could start with a simple "lock-around-every-output" type thing to at least ensure that there are no mangled single lines, at least. How does this sound? Does it sound reasonable?

One thing that I haven't thought about, but which may end up causing trouble would be any "bracket"-type constructs (and exception handling in general) since unfortunately doesn't really work with MonadIO. (At least without monad-control or similar. Even then it can make type signatures pretty horrendous.). OTOH, if all bracket usage is really just wrapping some short-lived IO-action (and not complicated stuff with callbacks), we might be able to get away with just having a liftIO around all usages of bracket. I'll try to a little bit investigation and have a look at a random sample of bracket usages in the current code to see if this might be a problem.

EDIT: [1] I imagine that any refactor to use Shake as per @ezyang's recent issue would perhaps make it a lot more feasible, but I imagine such a refactor would be... highly non-trivial? :)

[BardurArantsson]

Oh, and just as a little aside using a custom monad instead of IO has an additional nice little benefit of being able to elide the Verbosity parameter (almost) everywhere. That'd be a pretty nice little perk.

[BardurArantsson]

I'm just assigning myself for now. I'll do a little more investigating...

[ezyang]

@BardurArantsson If you do IO with environment, bracket/etc functions can be implemented perfectly. I think it's a worthwhile refactor but it's worth thinking about what things you are going to put in the environment. Might need multiple monads (at least, GHC does!)

[phadej]

@BardurArantsson do you still plan to investigate this?
I'm remilestoning this to 3.4, as I don't expect this happen for 3.2

[BardurArantsson]

No, I've given up on trying to contribute to Cabal at this point. (Details available upon request, but I'll avoid giving them here to avoid needless drama.)

I believe the issue is still relevant, but I have no idea if anything has happened recently to address the problem, so it may have been solved. (Unlikely, but it might...)