Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing to precompile Flux and Zygote #1456

Closed
yuvalwas opened this issue Sep 14, 2023 · 6 comments
Closed

Failing to precompile Flux and Zygote #1456

yuvalwas opened this issue Sep 14, 2023 · 6 comments

Comments

@yuvalwas
Copy link

Hello, I failed to precompile Zygore on my Institute's HPC. Would appreciate any help.

Stacktrace

Precompiling project...
�[91m ✗ �[39m�[90mZygote�[39m
�[91m ✗ �[39mFlux
0 dependencies successfully precompiled in 112 seconds (337 already precompiled)

ERROR: LoadError: The following 1 direct dependency failed to precompile:

Flux [587475ba-b771-5e3f-ad9e-33799f191a9c]

Failed to precompile Flux [587475ba-b771-5e3f-ad9e-33799f191a9c] to /home/labs/tsodyks/yuvalw/.julia/compiled/v1.7/Flux/jl_6Q1dTi.
ERROR: LoadError: MethodError: no method matching iterate(::Core.CodeInfo)
Closest candidates are:
iterate(!Matched::Union{LinRange, StepRangeLen}) at /apps/easybd/easybuild/software/Julia/1.7.1-linux-x86_64/share/julia/base/range.jl:826
iterate(!Matched::Union{LinRange, StepRangeLen}, !Matched::Integer) at /apps/easybd/easybuild/software/Julia/1.7.1-linux-x86_64/share/julia/base/range.jl:826
iterate(!Matched::T) where T<:Union{Base.KeySet{<:Any, <:Dict}, Base.ValueIterator{<:Dict}} at /apps/easybd/easybuild/software/Julia/1.7.1-linux-x86_64/share/julia/base/dict.jl:695
...
Stacktrace:
[1] indexed_iterate(I::Core.CodeInfo, i::Int64)
@ Base ./tuple.jl:92
[2] pullback(::Function, ::Zygote.Context{false}, ::Int64, ::Vararg{Int64})
@ Zygote ~/.julia/packages/Zygote/4SSHS/src/compiler/interface.jl:44
[3] pullback(::Function, ::Int64, ::Int64)
@ Zygote ~/.julia/packages/Zygote/4SSHS/src/compiler/interface.jl:42
[4] gradient(::Function, ::Int64, ::Vararg{Int64})
@ Zygote ~/.julia/packages/Zygote/4SSHS/src/compiler/interface.jl:96
[5] top-level scope
@ ~/.julia/packages/Zygote/4SSHS/src/precompile.jl:17
[6] include(mod::Module, _path::String)
@ Base ./Base.jl:418
[7] include
@ ~/.julia/packages/Zygote/4SSHS/src/Zygote.jl:1 [inlined]
[8] macro expansion
@ ~/.julia/packages/Requires/Z8rfN/src/Requires.jl:40 [inlined]
[9] precompile()
@ Zygote ~/.julia/packages/Zygote/4SSHS/src/Zygote.jl:63
[10] macro expansion
@ ~/.julia/packages/PrecompileTools/kmH5L/src/workloads.jl:85 [inlined]
[11] top-level scope
@ ~/.julia/packages/Zygote/4SSHS/src/Zygote.jl:84
[12] include
@ ./Base.jl:418 [inlined]
[13] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
@ Base ./loading.jl:1318
[14] top-level scope
@ none:1
[15] eval
@ ./boot.jl:373 [inlined]
[16] eval(x::Expr)
@ Base.MainInclude ./client.jl:453
[17] top-level scope
@ none:1
in expression starting at /home/labs/tsodyks/yuvalw/.julia/packages/Zygote/4SSHS/src/precompile.jl:17
in expression starting at /home/labs/tsodyks/yuvalw/.julia/packages/Zygote/4SSHS/src/Zygote.jl:1
ERROR: LoadError: Failed to precompile Zygote [e88e6eb3-aa80-5325-afca-941959d7151f] to /home/labs/tsodyks/yuvalw/.julia/compiled/v1.7/Zygote/jl_vTqQFV.
Stacktrace:
[1] error(s::String)
@ Base ./error.jl:33
[2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, ignore_loaded_modules::Bool)
@ Base ./loading.jl:1466
[3] compilecache(pkg::Base.PkgId, path::String)
@ Base ./loading.jl:1410
[4] _require(pkg::Base.PkgId)
@ Base ./loading.jl:1120
[5] require(uuidkey::Base.PkgId)
@ Base ./loading.jl:1013
[6] require(into::Module, mod::Symbol)
@ Base ./loading.jl:997
[7] include
@ ./Base.jl:418 [inlined]
[8] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
@ Base ./loading.jl:1318
[9] top-level scope
@ none:1
[10] eval
@ ./boot.jl:373 [inlined]
[11] eval(x::Expr)
@ Base.MainInclude ./client.jl:453
[12] top-level scope
@ none:1
in expression starting at /home/labs/tsodyks/yuvalw/.julia/packages/Flux/n3cOc/src/Flux.jl:1
Stacktrace:
[1] pkgerror(msg::String)
@ Pkg.Types /apps/easybd/easybuild/software/Julia/1.7.1-linux-x86_64/share/julia/stdlib/v1.7/Pkg/src/Types.jl:68
[2] precompile(ctx::Pkg.Types.Context; internal_call::Bool, strict::Bool, warn_loaded::Bool, already_instantiated::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ Pkg.API /apps/easybd/easybuild/software/Julia/1.7.1-linux-x86_64/share/julia/stdlib/v1.7/Pkg/src/API.jl:1362
[3] precompile
@ /apps/easybd/easybuild/software/Julia/1.7.1-linux-x86_64/share/julia/stdlib/v1.7/Pkg/src/API.jl:1013 [inlined]
[4] #precompile#220
@ /apps/easybd/easybuild/software/Julia/1.7.1-linux-x86_64/share/julia/stdlib/v1.7/Pkg/src/API.jl:1011 [inlined]
[5] precompile()
@ Pkg.API /apps/easybd/easybuild/software/Julia/1.7.1-linux-x86_64/share/julia/stdlib/v1.7/Pkg/src/API.jl:1011
[6] top-level scope
@ ~/clusterless/wexac_utils/env_setup.jl:10
in expression starting at /home/labs/tsodyks/yuvalw/clusterless/wexac_utils/env_setup.jl:5


# Julia version

julia> versioninfo()
Julia Version 1.7.1
Commit ac5cc99908 (2021-12-22 19:35 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
CPU: Intel(R) Xeon(R) Gold 5320 CPU @ 2.20GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-12.0.1 (ORCJIT, icelake-server)
Environment:
EBVERSIONJULIA = 1.7.1
EBROOTJULIA = /apps/easybd/easybuild/software/Julia/1.7.1-linux-x86_64
EBDEVELJULIA = /apps/easybd/easybuild/software/Julia/1.7.1-linux-x86_64/easybuild/Julia-1.7.1-linux-x86_64-easybuild-devel


# Zygote and Flux versions
Zygote v0.6.64
Flux v0.13.17
@yuvalwas
Copy link
Author

I see that this is similar to #1449. Do you think working with the latest Julia version would solve this?

@ToucheSir
Copy link
Member

That would be ideal. It's possible some 1.7 and/or 1.8 versions may have received the same backport fix that was mentioned in #1449, but 1.7.1 likely wouldn't have because of how old it is.

@voroskoi
Copy link

voroskoi commented Sep 16, 2023 •
edited
Loading

Hi,

I have the same issue with julia-1.9.3.
If I try to precompile just Zygote that also fails without any error message, it just shows a red X.

Here is the error message:

julia> Pkg.precompile("Zygote"; strict=true)
Precompiling Zygote
  ✗ Zygote
  0 dependencies successfully precompiled in 16 seconds. 56 already precompiled.

ERROR: The following 1 dependency failed to precompile:

Zygote [e88e6eb3-aa80-5325-afca-941959d7151f]

Failed to precompile Zygote [e88e6eb3-aa80-5325-afca-941959d7151f] to "/home/voroskoi/.julia/compiled/v1.9/Zygote/jl_rZ7rSE".
Illegal inttoptr
          %magicptr = ptrtoint {} addrspace(10)* %242 to i64, !dbg !26

[82350] signal (6.-6): Aborted
in expression starting at /home/voroskoi/.julia/packages/Zygote/4SSHS/src/precompile.jl:13
unknown function (ip: 0x7fd1d868e83c)
raise at /usr/bin/../lib/libc.so.6 (unknown line)
abort at /usr/bin/../lib/libc.so.6 (unknown line)
unknown function (ip: 0x7fd1d7a7dff7)
unknown function (ip: 0x7fd1d7ba8d88)
_ZN4llvm11PassManagerINS_8FunctionENS_15AnalysisManagerIS1_JEEEJEE3runERS1_RS3_ at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
unknown function (ip: 0x7fd1d7ba7db8)
_ZN4llvm27ModuleToFunctionPassAdaptor3runERNS_6ModuleERNS_15AnalysisManagerIS1_JEEE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
unknown function (ip: 0x7fd1d7ba85f8)
_ZN4llvm11PassManagerINS_6ModuleENS_15AnalysisManagerIS1_JEEEJEE3runERS1_RS3_ at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
unknown function (ip: 0x7fd1d7ba6081)
unknown function (ip: 0x7fd1d7b13f0f)
unknown function (ip: 0x7fd1d7b148ac)
_ZN4llvm3orc16IRTransformLayer4emitESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
unknown function (ip: 0x7fd1d7b0baea)
_ZN4llvm3orc31BasicIRLayerMaterializationUnit11materializeESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
_ZN4llvm3orc19MaterializationTask3runEv at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
unknown function (ip: 0x7fd1d305ac6b)
_ZN4llvm3orc16ExecutionSession22dispatchOutstandingMUsEv at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
_ZN4llvm3orc16ExecutionSession17OL_completeLookupESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EESt10shared_ptrINS0_23AsynchronousSymbolQueryEESt8functionIFvRKNS_8DenseMapIPNS0_8JITDylibENS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISF_vEEEENSG_ISD_vEENS_6detail12DenseMapPairISD_SI_EEEEEE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
unknown function (ip: 0x7fd1d307cf7e)
_ZN4llvm3orc16ExecutionSession19OL_applyQueryPhase1ESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EENS_5ErrorE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS0_10LookupKindERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS8_EENS0_15SymbolLookupSetENS0_11SymbolStateENS_15unique_functionIFvNS_8ExpectedINS_8DenseMapINS0_15SymbolStringPtrENS_18JITEvaluatedSymbolENS_12DenseMapInfoISI_vEENS_6detail12DenseMapPairISI_SJ_EEEEEEEEESt8functionIFvRKNSH_IS6_NS_8DenseSetISI_SL_EENSK_IS6_vEENSN_IS6_SV_EEEEEE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS7_EENS0_15SymbolLookupSetENS0_10LookupKindENS0_11SymbolStateESt8functionIFvRKNS_8DenseMapIS5_NS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISI_vEEEENSJ_IS5_vEENS_6detail12DenseMapPairIS5_SL_EEEEEE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS7_EENS0_15SymbolStringPtrENS0_11SymbolStateE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS_8ArrayRefIPNS0_8JITDylibEEENS0_15SymbolStringPtrENS0_11SymbolStateE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS_8ArrayRefIPNS0_8JITDylibEEENS_9StringRefENS0_11SymbolStateE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
unknown function (ip: 0x7fd1d7b0f059)
unknown function (ip: 0x7fd1d7b10a69)
unknown function (ip: 0x7fd1d7b11882)
jl_generate_fptr_impl at /usr/bin/../lib/julia/libjulia-codegen.so.1 (unknown line)
jl_compile_method_internal at /usr/bin/../lib/julia/libjulia-internal.so.1 (unknown line)
ijl_apply_generic at /usr/bin/../lib/julia/libjulia-internal.so.1 (unknown line)
#166 at ./none:0
iterate at ./generator.jl:47 [inlined]
collect_to! at ./array.jl:840
collect_to_with_first! at ./array.jl:818
unknown function (ip: 0x7fd196f939f5)
collect at ./array.jl:792
unknown function (ip: 0x7fd196f8e716)
prune! at /home/voroskoi/.julia/packages/IRTools/jcpZw/src/passes/passes.jl:181
|> at ./operators.jl:907 [inlined]
#IR#15 at /home/voroskoi/.julia/packages/IRTools/jcpZw/src/ir/wrap.jl:151 [inlined]
IR at /home/voroskoi/.julia/packages/IRTools/jcpZw/src/ir/wrap.jl:146 [inlined]
code_ir at /home/voroskoi/.julia/packages/Zygote/4SSHS/src/compiler/interface.jl:428
unknown function (ip: 0x7fd196f652e8)
unknown function (ip: 0x7fd1d7c64e2c)
unknown function (ip: 0x7fd1d7c6460a)
unknown function (ip: 0x7fd1d7c6572b)
unknown function (ip: 0x7fd1d7c6660e)
unknown function (ip: 0x7fd1d7c82942)
unknown function (ip: 0x7fd1d7c83359)
ijl_toplevel_eval_in at /usr/bin/../lib/julia/libjulia-internal.so.1 (unknown line)
unknown function (ip: 0x7fd1bebbd6fe)
unknown function (ip: 0x7fd1be86d3b8)
unknown function (ip: 0x7fd1be86d4f2)
unknown function (ip: 0x7fd1be86d50f)
jl_f__call_latest at /usr/bin/../lib/julia/libjulia-internal.so.1 (unknown line)
include at /home/voroskoi/.julia/packages/Zygote/4SSHS/src/Zygote.jl:1 [inlined]
macro expansion at /home/voroskoi/.julia/packages/Requires/Z8rfN/src/Requires.jl:40 [inlined]
precompile at /home/voroskoi/.julia/packages/Zygote/4SSHS/src/Zygote.jl:63
macro expansion at /home/voroskoi/.julia/packages/PrecompileTools/kmH5L/src/workloads.jl:78 [inlined]
top-level scope at /home/voroskoi/.julia/packages/Zygote/4SSHS/src/Zygote.jl:84
unknown function (ip: 0x7fd1d7c82cfc)
unknown function (ip: 0x7fd1d7c83011)
unknown function (ip: 0x7fd1d7c83359)
ijl_toplevel_eval_in at /usr/bin/../lib/julia/libjulia-internal.so.1 (unknown line)
unknown function (ip: 0x7fd1bebbd6fe)
unknown function (ip: 0x7fd1be86d3b8)
unknown function (ip: 0x7fd1be6ab891)
unknown function (ip: 0x7fd1be6abab2)
unknown function (ip: 0x7fd1d7c64e2c)
unknown function (ip: 0x7fd1d7c6460a)
unknown function (ip: 0x7fd1d7c6572b)
unknown function (ip: 0x7fd1d7c6660e)
unknown function (ip: 0x7fd1d7c82942)
unknown function (ip: 0x7fd1d7c83359)
ijl_toplevel_eval_in at /usr/bin/../lib/julia/libjulia-internal.so.1 (unknown line)
unknown function (ip: 0x7fd1bebbd6fe)
unknown function (ip: 0x7fd1bec59fdc)
unknown function (ip: 0x7fd1bec5a49f)
unknown function (ip: 0x7fd1bec5a5a8)
unknown function (ip: 0x7fd1d7cb0ff0)
jl_repl_entrypoint at /usr/bin/../lib/julia/libjulia-internal.so.1 (unknown line)
main at /usr/bin/julia (unknown line)
unknown function (ip: 0x7fd1d8627ccf)
__libc_start_main at /usr/bin/../lib/libc.so.6 (unknown line)
_start at /usr/bin/julia (unknown line)
Allocations: 16079812 (Pool: 16069899; Big: 9913); GC: 24
Stacktrace:
 [1] precompile(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:strict,), Tuple{Bool}}})
   @ Pkg.API /usr/share/julia/stdlib/v1.9/Pkg/src/API.jl:156
 [2] #precompile#115
   @ /usr/share/julia/stdlib/v1.9/Pkg/src/API.jl:144 [inlined]
 [3] precompile
   @ /usr/share/julia/stdlib/v1.9/Pkg/src/API.jl:144 [inlined]
 [4] #precompile#114
   @ /usr/share/julia/stdlib/v1.9/Pkg/src/API.jl:143 [inlined]
 [5] top-level scope
   @ REPL[2]:1

julia>

@ToucheSir
Copy link
Member

That sounds more like #1455 since you're using a recent version of Julia. I'd try the steps in #1455 (comment) and open a new thread if they don't work.

@voroskoi
Copy link

I have opened #1457 after trying to build in a clean environment.

@yuvalwas
Copy link
Author

Updating to 1.9.3. more or less worked for me. (Some other packages then failed 😞 )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ViralBShah @ToucheSir @voroskoi @yuvalwas