Compiled mode: Run tests with custom system image

[maleadt]

Initial stab at #79. Uses PackageCompiler to generate a custom systemimage that includes the package we're interested in (albeit without any precompilation statements, but I don't think that matters). It then uses that system image to run tests, but in a slightly different container (Arch Linux based) and with a different depot.

TODO:

• Print some statistics on the generated image
• Change the user / home folder.

[maleadt]

And some results: https://github.com/maleadt/packagecompiler_report/blob/08cc1a7a4aa7d4a8ae12d2b03d11369716c08bd7/REPORT.md

Summarized:

4680 packages tested on Julia 1.5.3
1656 failed tests, or were not considered
2435 passed tests in both cases (with and without PackageCompiler)
582 packages failed only after being compiled in the system image

Not too bad, I guess?

cc @KristofferC

[ViralBShah]

@aviks @StefanKarpinski

[ViralBShah]

Most of the binary_dependency issues are just that - old packages that do not use Yggdrasil provided binaries, or packages that need commercial binaries.

[KristofferC]

A lot of them are also for packages using some local file. JuliaLang/julia#38696 could be a way to help with that.

[ViralBShah]

@maleadt Do you move the compiled system image to a different path and perhaps even different user account before running? A lot of the images fail when they are run in a different place.

In fact it would be nice to run in-place and then a different place and see which ones fail.

[maleadt]

Do you move the compiled system image to a different path and perhaps even different user account before running?

The system image is built on Ubuntu, with the depot in ~/.julia and Julia in /opt/julia. Evaluation happens in a fresh Arch Linux-based container (so system libraries are different), using ~/.another_julia as depot and Julia in /usr/local/julia exactly to flush out these path issues.

I don't put the system image itself in another path, but that doesn't matter (no package is going to contain hard-coded references to the active system image).

I currently don't use a different user account, maybe it would be valuable to add that too in case there's hard-coded references to the users home folder -- but we already catch many instances of that because packages usually refer to existing files there (and by using a fresh containers there are no files at those paths).

[ViralBShah]

I have heard complaints about stuff not working when the system image is moved to a different path, because packages sometimes hardcode paths at image building time. So when those absolute paths are not present on a different user's computer, things break.

[maleadt]

because packages sometimes hardcode paths at image building time

Paths to the system image itself? How would that happen?
Also, the newly-created system image doesn't exist when it's being generated, so how could that path end up hard-coded?

[ViralBShah]

Can we note somewhere the size of the generated system image?

[ViralBShah]

Would be nice to get this merged so we can test if packages can reliably be compiled into system images.

[maleadt]

There isn't a real UI though, and it's not really suited for Nanosoldier (which compares multiple versions of Julia). Maybe a special mode?

[DilumAluthge]

Yeah, the comparison we really want here is, for the same version of Julia, to compare regular package tests to the "compiled mode", and list out packages that pass on the regular package tests but fail on the "compiled mode".

[maleadt]

OK, this has been rebased and updated to work on Sandbox.jl now. I'll have a look at integrating with Nanosoldier.