-j should build package components in parallel

[kolmodin]

cabal install -j foo will build and install foo and its dependencies in parallel, it works on the package level.

$ cabal build -j doesn't use the -j flag (or maybe just passes it to GHC), but it doesn't build the different package components in parallel. It'd be great if it would build all your executables, tests and benchmarks in parallel instead of in sequence as today.

The extra parallelism would be very useful when you're developing a package and you constantly recompile your single package (not its dependencies).

[RubenAstudillo]

IIRC on cabal install -j foo ghc builds each package on it's own thread. But as cabal build -j only builds only one package it only can use on thread.

[phadej]

In cabal build -j4 the -j flag is passed to ghc, so ghc builds modules in parallel. Seems that is something which is changed between 1.18 and current master:

1.18.1.5:

$ cabal build -j4
...
/usr/local/bin/ghc --make -fbuilding-cabal-package -O -static -dynamic-

master:

/usr/local/bin/ghc --make -fbuilding-cabal-package -O -j4 -static

---

Dunno if it makes sense to build different components in parallel if the build of single component is already parallel.

---

btw. you can cabal build component component1 so you don't need to rebuild tests when you are running benchmarks.

[23Skidoo]

Dunno if it makes sense to build different components in parallel if the build of single component is already parallel.

Can be useful if you have a package with a bazillion executables/components.

[phadej]

Could this be done by generalising how cabal treats components internally (more separate from package, in the way it treats package now, discussed in #2716 or related reddit thread). That would help with #960 too?

[kolmodin]

Dunno if it makes sense to build different components in parallel if the build of single component is already parallel.

Can be useful if you have a package with a bazillion executables/components.

You don't need a lot of executables/components before it would be faster than ghc -j. GHC's compiles the modules in parallel, which is better than nothing, but worse than running multiple instances of GHC in parallel.

[ezyang]

Hey guys, so I'm working on a refactor for the Cabal library that distinguishes components from packages. Once this is done, we can push the change up so that cabal-install creates a graph of components rather than packages, and then this issue will be solved. There might be some potholes along the way though.

[angerman]

@ezyang what's the status of this? building cabal takes awfully long, and utilizes only one of the 8 available cores. Could you elaborate on what is done and what still needs to be done? Also if related, how this ties in with the using shake ticket?

[ezyang]

With cabal-install HEAD (soon to be 2.0), if you new-build and your components are in non-Custom packages, they will be built in parallel.

This doesn't help much with building a single library, since Cabal makes a single call to GHC to build it, and thus does not have any opportunities for parallelism. I think we have another ticket on this.

[angerman]

I believe the other ticket you are referring to is #976.

[ezyang]

OK, I'm going to close this ticket as it is thinking about component-level parallelism specifically.