Idea: use PackageCompiler.jl to avoid the precompilation cache nightmare?

[tkf]

• Is it possible to use PackageCompiler.jl to create a .so file with libjulia and PyCall?
  • Do we have full runtime? Can we import arbitrary Julia packages?
  • Does it ignore the standard precompilation cache of PyCall?
• Or maybe just create a custom system image with PyCall?

Though we probably still need fake-julia for the compilation step.

[stevengj]

@lucatrv, any ideas?

[tkf]

I looked at PackageCompiler a bit. I think creating the system image for "Python-mode" is possible if there is an option to pass julia_cmd (whose .exec[1] would point to fake-julia/julia in PyJulia usage). If @lucatrv et al. are OK with it I think I can make a PR to do that.

Actually, I think more problematic part is (still) the precompilation cache management on the Julia side because the precompilation cache does not change even if you use different system image:

cache_file_entry(pkg::PkgId) = joinpath(
    "compiled",
    "v$(VERSION.major).$(VERSION.minor)",
    pkg.uuid === nothing ? "$(pkg.name).ji" : joinpath(pkg.name, "$(package_slug(pkg.uuid)).ji")
)

function find_all_in_cache_path(pkg::PkgId)
    paths = String[]
    entry = cache_file_entry(pkg)
    for depot in DEPOT_PATH
        path = joinpath(depot, entry)
        isfile_casesensitive(path) && push!(paths, path)
    end
    return paths
end

The function Base.find_all_in_cache_path needs to depend on (say) the path of the system image for this plan to work. I'll suggest it in Julia's issue tracker JuliaLang/julia#27418 It sounds like useful feature to have even for pure-Julia development.

[lucatrv]

I've never attempted this... and sorry I would not have time to work on it now... but of course feel free to submit a PR, if you ask my opinion yes this would be useful!

[lucatrv]

Also have a look at PR #26 which was attempted time ago but never completed.

[tkf]

@lucatrv Thanks for the comments. Re: Python C extension, I commented here: JuliaLang/PackageCompiler.jl#25 (comment)

I guess I'll not be doing the PR for julia_cmd right now since Julia core has to be improved for this plan to work (I think). But perhaps you can join the discussion in JuliaLang/julia#27418 for how precompile files should interact with the system image? I think it would be nice to have environment-specific system image sometimes (even outside the PyJulia context).