rdmd should compile package at a time

[dlang-bugzilla-migration]

Transferred from: https://issues.dlang.org/show_bug.cgi?id=14654

User @andralex reported (2015-Jun-05 16:10:40):

rdmd builds the given file and all of its dependents in one shot. For large projects, this becomes problematic because of many files are built unnecessarily if one file is touched.

One emerging pattern is to build one package (= directory) at a time, Haskell/Rust style. That allows users to organize dependencies in a scalable manner.

rdmd would collect the dependency graph, "color" the files have been affected by the changed files since last build, and then carve the entire file set by directory. If at least one file has been colored in a directory, that directory needs to be rebuilt.

This is a high impact project because it would instantly improve the lot of all rdmd users, and scales rdmd up for larger builds.

User @CyberShadow responded (2015-Jun-06 16:16:41):

If we want to improve build times with rdmd, the best thing we can do is get rid of rdmd (and only leave it as a thin wrapper around dmd). This is because rdmd needs to run dmd -o- to get the dependency graph, and then run dmd again to actually build the program. This means that lexing, parsing, semantics - everything except code gen and linking - must happen twice. For template or mixin-heavy code, this nearly doubles compilation time.

User @andralex responded (2015-Jun-06 16:46:38):

(In reply to Vladimir Panteleev from comment 1)

If we want to improve build times with rdmd, the best thing we can do is get
rid of rdmd (and only leave it as a thin wrapper around dmd). This is
because rdmd needs to run dmd -o- to get the dependency graph, and then run
dmd again to actually build the program. This means that lexing, parsing,
semantics - everything except code gen and linking - must happen twice. For
template or mixin-heavy code, this nearly doubles compilation time.

This improvement stands.

User @CyberShadow responded (2015-Jun-06 16:48:24):

Yes, they are orthogonal. But:

1. It is potentially higher-impact. Even with per-package compilation, DMD will still need to lex/parse/etc. ALL modules.

2. Any improvements we make now to rdmd may be harder to migrate into dmd

User @andralex responded (2015-Jun-06 16:58:03):

(In reply to Vladimir Panteleev from comment 3)

Yes, they are orthogonal. But:

1. It is potentially higher-impact. Even with per-package compilation, DMD
   will still need to lex/parse/etc. ALL modules.

It's almost surprising rdmd is so immensely useful, eh :o).

BTW there was work on collecting dependencies while actually building. One nice solution would be for dmd -v to generate a line at the end of imports to mean "done". Then rdmd can run dmd concurrently and terminate if no need to generate code.

2. Any improvements we make now to rdmd may be harder to migrate into dmd

I see it as the polar opposite. Improvements in rdmd clarify what sort of primitives and interfaces we need for highly modular compilation.

User @aG0aep6G responded (2015-Jun-14 22:52:52):

I implemented an alternative mentioned in the forum discussion: Compile all outdated files, and only those, with one compiler invocation to multiple object files.

#170