Regression in Julia 1.12: Creating fresh base sysimage (incremental=false) fails with error during bootstrap

[DilumAluthge]

The error

Here is what the error message itself looks like:

✔ [01m:33s] PackageCompiler: creating compiler .ji image (incremental=false)
⡆ [00m:06s] PackageCompiler: compiling fresh sysimage (incremental=false)error during bootstrap:
LoadError(at "/var/folders/lp/2fm0z_wn3wqf__vj6lc1tcgh0000gq/T/jl_YBjqXM/sysimage_packagecompiler_a12bf01a-97ce-11ef-0ebb-19685e72afd2.jl" line 3: LoadError(at "Base.jl" line 287: ErrorException("File "/var/folders/lp/2fm0z_wn3wqf__vj6lc1tcgh0000gq/T/jl_YBjqXM/sysimage_packagecompiler_a12bf01a-97ce-11ef-0ebb-19685e72afd2.jlbuild_h.jl" not found")))

Here is the full stacktrace (click to expand):

Precompiling all packages...
  4 dependencies successfully precompiled in 3 seconds. 26 already precompiled.
+ ./julia --startup-file=no --project=../PackageCompiler.jl -e 'import Pkg; Pkg.precompile()'
+ ./julia --startup-file=no --project=../PackageCompiler.jl ../mwe.jl
✔ [01m:33s] PackageCompiler: creating compiler .ji image (incremental=false)
⡆ [00m:06s] PackageCompiler: compiling fresh sysimage (incremental=false)error during bootstrap:
LoadError(at "/var/folders/lp/2fm0z_wn3wqf__vj6lc1tcgh0000gq/T/jl_YBjqXM/sysimage_packagecompiler_a12bf01a-97ce-11ef-0ebb-19685e72afd2.jl" line 3: LoadError(at "Base.jl" line 287: ErrorException("File \"/var/folders/lp/2fm0z_wn3wqf__vj6lc1tcgh0000gq/T/jl_YBjqXM/sysimage_packagecompiler_a12bf01a-97ce-11ef-0ebb-19685e72afd2.jlbuild_h.jl\" not found")))
⠇ [00m:06s] PackageCompiler: compiling fresh sysimage (incremental=false)ijl_errorf at /mydir/bisect-pc/julia/src/rtutils.c:77
jl_file_content_as_string at /mydir/bisect-pc/julia/src/toplevel.c:1161 [inlined]
ijl_load_ at /mydir/bisect-pc/julia/src/toplevel.c:1177
include at ./boot.jl:436 [inlined]
include at ./Base.jl:24
include at ./Base.jl:26
unknown function (ip: 0x1048f805f)
jl_apply at /mydir/bisect-pc/julia/src/julia.h:2185 [inlined]
do_call at /mydir/bisect-pc/julia/src/interpreter.c:127
eval_stmt_value at /mydir/bisect-pc/julia/src/interpreter.c:175
eval_body at /mydir/bisect-pc/julia/src/interpreter.c:678
jl_interpret_toplevel_thunk at /mydir/bisect-pc/julia/src/interpreter.c:889
top-level scope at Base.jl:287
jl_toplevel_eval_flex at /mydir/bisect-pc/julia/src/toplevel.c:1005
jl_eval_module_expr at /mydir/bisect-pc/julia/src/toplevel.c:215 [inlined]
jl_toplevel_eval_flex at /mydir/bisect-pc/julia/src/toplevel.c:807
jl_parse_eval_all at /mydir/bisect-pc/julia/src/toplevel.c:1132
ijl_load_ at /mydir/bisect-pc/julia/src/toplevel.c:1179
include at ./boot.jl:436
unknown function (ip: 0x1047b405b)
jl_apply at /mydir/bisect-pc/julia/src/julia.h:2185 [inlined]
do_call at /mydir/bisect-pc/julia/src/interpreter.c:127
eval_stmt_value at /mydir/bisect-pc/julia/src/interpreter.c:175
eval_body at /mydir/bisect-pc/julia/src/interpreter.c:678
jl_interpret_toplevel_thunk at /mydir/bisect-pc/julia/src/interpreter.c:889
top-level scope at /var/folders/lp/2fm0z_wn3wqf__vj6lc1tcgh0000gq/T/jl_YBjqXM/sysimage_packagecompiler_a12bf01a-97ce-11ef-0ebb-19685e72afd2.jl:3
jl_toplevel_eval_flex at /mydir/bisect-pc/julia/src/toplevel.c:1005
jl_parse_eval_all at /mydir/bisect-pc/julia/src/toplevel.c:1132
ijl_load_ at /mydir/bisect-pc/julia/src/toplevel.c:1179
ijl_load at /mydir/bisect-pc/julia/src/toplevel.c:1192
exec_program at /mydir/bisect-pc/julia/src/jlapi.c:858
true_main at /mydir/bisect-pc/julia/src/jlapi.c:915
jl_repl_entrypoint at /mydir/bisect-pc/julia/src/jlapi.c:1059

✖ [00m:06s] PackageCompiler: compiling fresh sysimage (incremental=false)
ERROR: LoadError: failed process: Process(setenv(`/mydir/bisect-pc/julia/usr/bin/julia --color=yes --startup-file=no --pkgimages=no --cpu-target generic --sysimage=/var/folders/lp/2fm0z_wn3wqf__vj6lc1tcgh0000gq/T/jl_YBjqXM/corecompiler.ji --output-o=/var/folders/lp/2fm0z_wn3wqf__vj6lc1tcgh0000gq/T/jl_YBjqXM/sys-o.a /var/folders/lp/2fm0z_wn3wqf__vj6lc1tcgh0000gq/T/jl_YBjqXM/sysimage_packagecompiler_a12bf01a-97ce-11ef-0ebb-19685e72afd2.jl`,[...environment variables...]), ProcessExited(1)) [1]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:598 [inlined]
  [2] read(cmd::Cmd)
    @ Base ./process.jl:482
  [3] macro expansion
    @ /mydir/bisect-pc/PackageCompiler.jl/src/PackageCompiler.jl:270 [inlined]
  [4] (::PackageCompiler.var"#12#17"{String, Cmd, Vector{String}, String, String, String, String})()
    @ PackageCompiler /mydir/bisect-pc/PackageCompiler.jl/ext/TerminalSpinners.jl:157
  [5] spin(f::PackageCompiler.var"#12#17"{String, Cmd, Vector{String}, String, String, String, String}, s::PackageCompiler.TerminalSpinners.Spinner{Base.TTY})
    @ PackageCompiler.TerminalSpinners /mydir/bisect-pc/PackageCompiler.jl/ext/TerminalSpinners.jl:164
  [6] macro expansion
    @ /mydir/bisect-pc/PackageCompiler.jl/ext/TerminalSpinners.jl:157 [inlined]
  [7] (::PackageCompiler.var"#10#15"{String, Cmd, Vector{String}, String, String, String, String, String})()
    @ PackageCompiler /mydir/bisect-pc/PackageCompiler.jl/src/PackageCompiler.jl:256
  [8] cd(f::PackageCompiler.var"#10#15"{String, Cmd, Vector{String}, String, String, String, String, String}, dir::String)
    @ Base.Filesystem ./file.jl:112
  [9] create_fresh_base_sysimage(stdlibs::Vector{String}; cpu_target::String, sysimage_build_args::Cmd)
    @ PackageCompiler /mydir/bisect-pc/PackageCompiler.jl/src/PackageCompiler.jl:243
 [10] top-level scope
    @ /mydir/bisect-pc/mwe.jl:3
 [11] include
    @ ./Base.jl:580 [inlined]
 [12] exec_options(opts::Base.JLOptions)
    @ Base ./client.jl:324
 [13] _start()
    @ Base ./client.jl:532
in expression starting at /mydir/bisect-pc/mwe.jl:3

Here is an example CI log that shows the error: https://github.com/JuliaLang/PackageCompiler.jl/actions/runs/11573962346/job/32217198440?pr=987

MWE

Here is an MWE:

import PackageCompiler

base_sysimage = PackageCompiler.create_fresh_base_sysimage(
    String[];
    cpu_target = PackageCompiler.default_app_cpu_target(),
    sysimage_build_args = ``,
)

[DilumAluthge]

I have bisected this to JuliaLang/julia@db687ad (JuliaLang/julia#54387)

db687ad2d92b1b3dbb70dda03647284e7166dabd is the first bad commit
commit db687ad2d92b1b3dbb70dda03647284e7166dabd (HEAD)
Author: Jeff Bezanson <[email protected]>
Date:   Wed Jun 26 16:41:10 2024 -0400

    add mechanism for configuring system image builds (#54387)

    This adds the option to pass a filename of configuration settings when
    building the Core/compiler system image (from
    `base/compiler/compiler.jl`). This makes it easier to build different
    flavors of images, for example it can replace the hack that
    PackageCompiler uses to edit the list of included stdlibs, and makes it
    easy to change knobs you might want like max_methods.

 base/Base.jl              | 26 +++++++++++++++++++++++---
 base/compiler/compiler.jl | 12 ++++++++++++
 base/compiler/types.jl    |  4 ++--
 base/cpuid.jl             |  2 +-
 base/filesystem.jl        |  2 +-
 base/libc.jl              |  2 +-
 base/libuv.jl             |  2 +-
 base/pcre.jl              |  2 +-
 base/sysimg.jl            | 40 ++++++++++++++++++++++++----------------
 9 files changed, 66 insertions(+), 26 deletions(-)

[KristofferC]

Makes sense, we should use the functionality provided in that PR instead of the current text replacement for

PackageCompiler.jl/src/PackageCompiler.jl

Line 214 in 38629b0

function rewrite_sysimg_jl_only_needed_stdlibs(stdlibs::Vector{String})

.

[DilumAluthge]

We can probably split this up into two parts:

1. Pass "" as the second positional argument when we create the fresh base sysimage. This argument will be picked up by Base.jl as the value for BUILDROOT. This will fix the immediate bug we're seeing in this issue, which occurs because we currently only pass a single positional argument (namely, our sysimage_packagecompiler_$(uuid1()).jl script that we want to run), and thus Base.jl incorrectly assumes that said script is the BUILDROOT argument. This fix is in #996.
2. Teach create_fresh_base_sysimage() to use the new BuildSettings mechanism (from JuliaLang/julia#54387) in Julia 1.12+, instead of the "rewrite sysimg.jl" mechanism that we use in Julia 1.11 and earlier. This feature is implemented in #997.

I imagine that #997 will require more review and iterations than #996, and plus #996 is such a tiny change that fixes the immediate bug, so I figured I'd split them up into two different PRs.