Parallelize precompiles generation

[petvana]

This PR parallelizes generate_precompile.jl. It saves half a minute building Julia, and, thanks to using Channel, the order of the precompiles should be still fully deterministic.

PR

...
    JULIA usr/lib/julia/sys-o.a
Generating REPL precompile statements... 40/40
Executing precompile statements... 1750/1754
Precompilation complete. Summary:
Total ─────── 112.036051 seconds
...

Master

...
    JULIA usr/lib/julia/sys-o.a
Generating REPL precompile statements... 40/40
Executing precompile statements... 1748/1752
Precompilation complete. Summary:
Generation ── 121.136666 seconds 80.6719%
Execution ───  29.023100 seconds 19.3281%
Total ─────── 150.159765 seconds
...

The next optimization would be to read precompile_file on the fly, that would save several seconds extra.

[IanButterworth]

Seems like a good thing to do, but it could make the precompilation process harder to debug. It might be good to be able to fallback to the current serial approach for debugging purposes. I guess putting optional waits on the @async tasks could be a simple way to do that?

Also I think the progress printing needs figuring out. It's currently a bit chaotic and hard to interpret.

[vchuravy]

These should have errormonitor on them so that their failure is loud

[petvana]

Thank you, should be loud now.

[petvana]

@vchuravy errormonitor causes CI to fail for some reason. Is the current approach with fetch(step1) == :ok || throw("Step 1 of collecting precompiles failed.") sufficient?

[petvana]

I've tried to improve the printing (link to asciinema below) and add PARALLEL_PRECOMPILATION = false option for serial debugging

[petvana]

Not sure, where the current CI problem comes from. I cannot reproduce it locally.

[IanButterworth]

Try rebasing?

[IanButterworth]

The REPL precomp generation script includes intentionally throwing errors. Perhaps that's why errormonitor etc was surfacing errors that were previously ignored?

[petvana]

The REPL precomp generation script includes intentionally throwing errors. Perhaps that's why errormonitor etc was surfacing errors that were previously ignored?

Perhaps, but I'm unable to confirm locally.

[petvana]

Besides errormonitor, the PR seems ready. Suggestions for improving the progress printing welcomed.

[IanButterworth]

Looks great.
For formatting maybe this, where Waiting etc. are colored and the counts for the categories are colored appropriately. Also perhaps "Basic" instead of "normal"?

Precompile statements (Waiting, Running, Finished): 
└ Collect 1748 (Basic: 729, REPL 40/40: 1653) => Execute 1744 (4 failed)
Precompilation complete. Summary:
Total ─────── 114.175765 seconds
Outputting sysimage file...
Output ────── 162.738189 seconds

[IanButterworth]

Looking great. Disabling the cursor is best with a try-finally because it can be annoying to lose it if the build process fails

[IanButterworth]

Great!

[IanButterworth]

I realized the 4 we're reporting as failures are just comments being skipped, so (I hope you don't mind) I pushed a fix for that

[IanButterworth]

The next optimization would be to read precompile_file on the fly, that would save several seconds extra.

It seems like --trace-compile=$precompile_file is populated at the end of the process. Making that more parallel might take a lot of internals work

[KristofferC]

We could flush the strean for every statement printed maybe.

[IanButterworth]

I actually think I might be wrong, and we're already flushing. Investigating..

[petvana]

It seems like --trace-compile=$precompile_file is populated at the end of the process. Making that more parallel might take a lot of internals work

Actually, they are populated during the process. I've already tried a very naive approach and it seems to work.
petvana/julia@pv/parallel-repl-statements-v2...petvana:julia:pv/parallel-repl-statements-on-the-fly-v2

[IanButterworth]

Nice. I've been trying an approach that doesn't re-read and unique the statements each loop, though it's not working yet

[petvana]

I propose to leave the asynchronous reading to a different PR because it adds one more level of complexity (and space for possible dead-locks).

[IanButterworth]

Sounds good to me

[IanButterworth]

This seems to work without re-reading the total file each time IanButterworth@5ee671a?w=1

[petvana]

This seems to work without re-reading the total file each time IanButterworth@5ee671a?w=1

Nice, but I'm slightly worried about readline if it may split the line before it's fully written. I'd rather use readavailable and split manually.

Moreover, we should wait until precompile_file exists.

[IanButterworth]

Builds are passing, but win 32 tests are reporting failure without an obvious reason. Updating from master just to check

[vtjnash]

readline will never do that (if reading from a proper stream/pipe), but readavailable will often do that

[KristofferC]

Wow, I didn't mean to merge this. I tried to merge it locally using gh merge and it merged it here instead 😆

[IanButterworth]

Wasn't even a squash merge. That seems to easy to do accidentally..!

[KristofferC]

I think it asked me and since I thought it was local I thought whatever. The last step said "submit" which in hind sight probably should have made me more alert...

[petvana]

readline will never do that (if reading from a proper stream/pipe), but readavailable will often do that

It seems possible if someone flushes (that is a correct think to do). The following MWE

t = tempname()
w = open(t, "w")
r = open(t, "r")
tr = @async begin
    for i = 1:30
        sleep(0.1)
        line = readline(r)
        line == "" && continue
        println(line)
    end
end

print(w, "a+")
flush(w)
sleep(0.3)
print(w, "b\n")
close(w)

wait(tr)

prints only

a+

but it is not a full line.

[vtjnash]

A file is not a stream/pipe (unless it was created with mkfifo)