Feature request: --pkgimages=existing #51474
Labels
packages
Package management and loading
Comments
KristofferC
pushed a commit
that referenced
this issue
Dec 19, 2023
Equivalent of #50586; implements #51474. With `--pkgimages=existing`, it's possible to disable the (often slow) generation of package images, without losing the ability to use existing ones. That's important now that we're moving more and more packages outside of the system image, e.g., running with `--pkgimages=no` otherwise takes close to 30s here before the Pkg REPL is usable. The main motivation for this is PkgEval, where generating package images is not very useful, yet disabling generation of them makes each job (which requires Pkg to drive the test process) take a significantly longer time. For example, `--pkgimages=yes` vs `no`: ```julia ❯ JULIA_DEBUG=loading ./julia --project=Example.jl --pkgimages=yes # no precompilation of REPL.jl ┌ Debug: Loading object cache file /Users/tim/Julia/src/julia/build/dev/usr/share/julia/compiled/v1.11/REPL/u0gqU_XmENM.dylib for REPL [3fa0cd96-eef1-5676-8a61-b3b8758bbffb] └ @ Base loading.jl:1116 julia> using Example # short time precompiling + pkgimg generation for Example.jl ┌ Debug: Loading object cache file /Users/tim/.julia/compiled/v1.11/Example/lLvWP_tJaso.dylib for Example [7876af07-990d-54b4-ab0e-23690620f79a] └ @ Base loading.jl:1116 ``` ```julia ❯ JULIA_DEBUG=loading ./julia --project=Example.jl --pkgimages=no ┌ Debug: Rejecting cache file /Users/tim/Julia/src/julia/build/dev/usr/share/julia/compiled/v1.11/REPL/u0gqU_XmENM.ji for REPL [3fa0cd96-eef1-5676-8a61-b3b8758bbffb] since the flags are mismatched │ current session: use_pkgimages = false, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2 │ cache file: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2 └ @ Base loading.jl:3289 # long time precompiling REPL.jl ┌ Debug: Loading cache file /Users/tim/.julia/compiled/v1.11/REPL/u0gqU_CWvWI.ji for REPL [3fa0cd96-eef1-5676-8a61-b3b8758bbffb] └ @ Base loading.jl:1119 julia> using Example # short time precompiling Example ┌ Debug: Loading cache file /Users/tim/.julia/compiled/v1.11/Example/lLvWP_CWvWI.ji for Example [7876af07-990d-54b4-ab0e-23690620f79a] └ @ Base loading.jl:1119 ``` With the new `--pkgimages=existing`: ```julia ❯ JULIA_DEBUG=loading ./julia --project=Example.jl --pkgimages=existing # no precompilation of REPL.jl ┌ Debug: Loading object cache file /Users/tim/Julia/src/julia/build/dev/usr/share/julia/compiled/v1.11/REPL/u0gqU_XmENM.dylib for REPL [3fa0cd96-eef1-5676-8a61-b3b8758bbffb] └ @ Base loading.jl:1116 julia> using Example # short time precompiling Example ┌ Debug: Loading cache file /Users/tim/.julia/compiled/v1.11/Example/lLvWP_CWvWI.ji for Example [7876af07-990d-54b4-ab0e-23690620f79a] └ @ Base loading.jl:1119 ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Generating package images is relatively expensive, but for most users that's worth it as packages are more frequently imported than compiled. Not so for PkgEval, which generally only imports a package once after precompiling it. That's why PkgEval runs with
--pkgimages=no, taking about 12h to test all ecosystem packages.However, now that more stdlibs are being excised from the system image, running with
--pkgimages=noresults in e.g. the REPL and package manager being recompiled over and over, regressing PkgEval times to ~20h. As a stop-gap solution, we're running with the default--pkgimages=yesnow, making sure the stdlib pkgimages can be used. PkgEval now takes ~15h to complete.Seeing how this is still a 25% regression, and many more packages time-out now, it would be great if we had something like #50586 for pkgimages. With
--pkgimages=existing, stdlibs would be loaded from precompiled pkgimages, but we wouldn't be generating new images for the package we're testing.cc @vchuravy
The text was updated successfully, but these errors were encountered: