Facilitate precompilation for juliacall calls

[ggoretkin-bdai]

I am using juliacall in Python to call a Julia function. I would like to precompile the Julia function. Here are the specializations of the function by introspecting on the Python program:

jl.MethodAnalysis.methodinstances(jl.FooPkg.bar_func)


Julia:
2-element Vector{Core.MethodInstance}:
 MethodInstance for FooPkg.bar_func(::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::FooPkg.InverseParams)
 MethodInstance for FooPkg.bar_func(::PyArray{Float64, 1, true, true, Float64}, ::PyArray{Float64, 1, true, true, Float64}, ::PyArray{Float64, 1, true, true, Float64}, ::PyArray{Float64, 1, true, true, Float64}, ::PyArray{Float64, 1, true, true, Float64}, ::PyArray{Float64, 1, true, true, Float64}, ::PyArray{Float64, 1, true, true, Float64}, ::PyArray{Float64, 1, true, true, Float64}, ::PyArray{Float64, 1, true, true, Float64}, ::PyArray{Float64, 1, true, true, Float64}, ::Float64, ::PyList{Any}, ::PyList{Any}, ::FooPkg.InverseParams)

I would like to basically insert the following into FooPkg.jl

@setup_workload begin
    ENV["JULIA_CONDAPKG_BACKEND"] = "Null" # use the system installation of Python
    using PythonCall
    np = pyimport("numpy")

    arg_1 = np.array([0.0, 0.0, 0.0])
    
    @compile_workload begin
        # all calls in this block will be precompiled, regardless of whether
        # they belong to this package or not (on Julia 1.8 and higher)
        bar_func(arg_1, ...)
    end
end

But this is not quite right, as typeof(arg_1) is Py, and not PyArray{Float64, 1, true, true, Float64}. It appears I can get a value of the right type by doing

julia> typeof(PyArray([1.0]))
PyArray{Float64, 1, true, true, Float64}

but is there a way to get it more directly from the corresponding Python code? More generally, what is a workload I can put in @compile_workload that exercises the same code paths as is exercised by juliacall?

[goretkin]

Also asked here: https://discourse.julialang.org/t/precompiling-for-pythoncall-jl-juliacall/111596

[cjdoris]

If you already have some Python code then using pyexec to execute it will be the simplest way to ensure you get the same code paths.

In your example you could do pyjl(bar_func)(arg_1, ...) might work. Or try bar_func(pyconvert(Any, arg_1), ...).

[cjdoris]

Hmm ok. I can't totally remember why I made it a submodule of Main. You could do jl = juliacall.Base.Module() to get a new anonymous module?

[ggoretkin-bdai]

I think that works in the latest PythonCall.jl v0.9.16. On the previous version (which Pkg is choosing because in our environment, we have Julia v1.10.2), I get:

ERROR: LoadError: Python: Julia: Evaluation into the closed module `anonymous` breaks incremental compilation because the side effects will not be permanent. This is likely due to some other module mutating `anonymous` with `eval` during precompilation - don't do this.

Thanks for the suggestion. Ideally Julia v1.10.3 gets released soon. Do you have a reference to the specific upstream issue?

[cjdoris]

I'm not sure what upstream issue you mean - I don't think the version of Julia/JuliaCall/PythonCall should affect the behaviour you're seeing.

[ggoretkin-bdai]

I'm not sure what upstream issue you mean

https://github.com/JuliaPy/PythonCall.jl/blob/f586f2494432f2e5366ff1e2876b8aa532630b54/docs/src/releasenotes.md#0916-2024-03-14

Removed compatibility with Julia 1.10.1 and 1.10.2 (to be fixed in 1.10.3 and 1.11.0) due to an upstream bug

[cjdoris]

Ah #464