Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

LLVM error #1693

Closed
swilliamson7 opened this issue Jul 31, 2024 · 16 comments
Closed

LLVM error #1693

swilliamson7 opened this issue Jul 31, 2024 · 16 comments

Comments

@swilliamson7
Copy link
Collaborator

In the process of creating an example of the Optim + Enzyme error, I'm now hitting an LLVM error that I haven't seen before. This was made without Checkpointing when I was trying to run my Optim experiments, and I don't know why it started. Unfortunately, to run the code there are a few data files needed, but I've put them in my Github repo, so everything needed to run and re-generate the error should be here. Specifically, running the script llvm_bug.jl should generate the error described in llvm.txt. The error itself starts with

PHI nodes not grouped at top of basic block!
  %"'ip_phi34" = phi {} addrspace(10)* , !dbg !12920
label %idxend252
; Function Attrs: mustprogress willreturn
define internal fastcc void @preprocess_julia_loop_4296({} addrspace(10)* noundef nonnull align 8 dereferenceable(2656) "enzyme_type"="{

and ends with

[ Info: Revolve: Number of checkpoints: 25
[ Info: Revolve: Number of steps: 674
[ Info: Prediction:
[ Info: Forward steps   : 1644
[ Info: Overhead factor : 2.4391691394658754
ERROR: LoadError: LLVM error: function failed verification (2)
Stacktrace:
 [1] handle_error(reason::Cstring)
   @ LLVM ~/.julia/packages/LLVM/5DlHM/src/core/context.jl:168
in expression starting at /Users/swilliamson/Documents/GitHub/ShallowWaters_work/ShallowWaters.jl/optim_plus_enzyme_bug/weird_llvm_bug.jl:304

I checked that I can run the function being differentiated separately and it did seem to work (finished running, got loss function value) so really not sure what happened
@wsmoses
Copy link
Member

wsmoses commented Jul 31, 2024

Can you paste the whole log?

@swilliamson7
Copy link
Collaborator Author

It's way too long of an error to post here, but it can be downloaded from my repo here, or should I send it some other way?

@wsmoses
Copy link
Member

wsmoses commented Aug 7, 2024

yeah this one likely requires minimization to debug

@wsmoses
Copy link
Member

wsmoses commented Aug 11, 2024

@swilliamson7 I think this should be fixed by #1726

but I can't run your original code to confirm (I don't have the "./1024_postspinup_noslip_4days_073124/eta.nc" file)

@wsmoses wsmoses closed this as completed Aug 11, 2024
@swilliamson7
Copy link
Collaborator Author

I get this now

[ Info: Size of total storage: 221.132 MiB
ERROR: Function argument passed to autodiff cannot be proven readonly.
If the the function argument cannot contain derivative data, instead call autodiff(Mode, Const(f), ...)
See https://enzyme.mit.edu/index.fcgi/julia/stable/faq/#Activity-of-temporary-storage for more information.
The potentially writing call is   %545 = call nonnull "enzyme_type"="{[-1]:Pointer}" {} addrspace(10)* ({} addrspace(10)* ({} addrspace(10)*, {} addrspace(10)**, i32)*, {} addrspace(10)*, ...) @julia.call({} addrspace(10)* ({} addrspace(10)*, {} addrspace(10)**, i32)* nonnull @ijl_apply_generic, {} addrspace(10)* addrspacecast ({}* inttoptr (i64 4750707440 to {}*) to {} addrspace(10)*), {} addrspace(10)* nonnull %435, {} addrspace(10)* nonnull %544) #107, !dbg !432, using   %435 = load atomic {} addrspace(10)*, {} addrspace(10)* addrspace(11)* %434 unordered, align 8, !dbg !432, !tbaa !117, !alias.scope !121, !noalias !124
Stacktrace:
 [1] macro expansion
   @ ~/Documents/GitHub/ShallowWaters_work/ShallowWaters.jl/optim_plus_enzyme_bug/weird_llvm_bug.jl:216
 [2] #1
   @ ~/.julia/packages/Checkpointing/CEFSa/src/Checkpointing.jl:181
 [3] #1
   @ ~/.julia/packages/Checkpointing/CEFSa/src/Checkpointing.jl:0

Stacktrace:
  [1] macro expansion
    @ ~/Documents/GitHub/ShallowWaters_work/ShallowWaters.jl/optim_plus_enzyme_bug/weird_llvm_bug.jl:216 [inlined]
  [2] #1
    @ ~/.julia/packages/Checkpointing/CEFSa/src/Checkpointing.jl:181 [inlined]
  [3] diffejulia__1_8913_inner_1wrap
    @ ~/.julia/packages/Checkpointing/CEFSa/src/Checkpointing.jl:0
  [4] macro expansion
    @ ~/.julia/packages/Enzyme/7pzUA/src/compiler.jl:7049 [inlined]
  [5] enzyme_call
    @ ~/.julia/packages/Enzyme/7pzUA/src/compiler.jl:6658 [inlined]
  [6] CombinedAdjointThunk
    @ ~/.julia/packages/Enzyme/7pzUA/src/compiler.jl:6535 [inlined]
  [7] autodiff
    @ ~/.julia/packages/Enzyme/7pzUA/src/Enzyme.jl:320 [inlined]
  [8] autodiff
    @ ~/.julia/packages/Enzyme/7pzUA/src/Enzyme.jl:348 [inlined]
  [9] autodiff
    @ ~/.julia/packages/Enzyme/7pzUA/src/Enzyme.jl:329 [inlined]
 [10] rev_checkpoint_struct_for
    @ ~/.julia/packages/Checkpointing/CEFSa/src/Schemes/Revolve.jl:471
 [11] length
    @ ./essentials.jl:10 [inlined]
 [12] copyto!
    @ ./array.jl:388 [inlined]
 [13] checkpointed_integration
    @ ~/Documents/GitHub/ShallowWaters_work/ShallowWaters.jl/optim_plus_enzyme_bug/weird_llvm_bug.jl:64 [inlined]
 [14] diffejulia_checkpointed_integration_5172wrap
    @ ~/Documents/GitHub/ShallowWaters_work/ShallowWaters.jl/optim_plus_enzyme_bug/weird_llvm_bug.jl:0
 [15] macro expansion
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/7pzUA/src/compiler.jl:7049 [inlined]
 [16] enzyme_call
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/7pzUA/src/compiler.jl:6658 [inlined]
 [17] CombinedAdjointThunk
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/7pzUA/src/compiler.jl:6535 [inlined]
 [18] autodiff
    @ Enzyme ~/.julia/packages/Enzyme/7pzUA/src/Enzyme.jl:320 [inlined]
 [19] autodiff(::ReverseMode{true, FFIABI, false, true}, ::Const{typeof(checkpointed_integration)}, ::Duplicated{Main.ShallowWaters.ModelSetup{Float32, Float32}}, ::Const{Revolve{Main.ShallowWaters.ModelSetup}})
    @ Enzyme ~/.julia/packages/Enzyme/7pzUA/src/Enzyme.jl:348
 [20] autodiff
    @ ~/.julia/packages/Enzyme/7pzUA/src/Enzyme.jl:329 [inlined]
 [21] llvm_error(Ndays::Int64)
    @ Main ~/Documents/GitHub/ShallowWaters_work/ShallowWaters.jl/optim_plus_enzyme_bug/weird_llvm_bug.jl:287
 [22] top-level scope
    @ REPL[5]:1

which previously hasn't happened? Is this a new error/maybe an easy fix?

@wsmoses
Copy link
Member

wsmoses commented Aug 12, 2024 via email

@swilliamson7
Copy link
Collaborator Author

I added the Const, so my new call to autodiff is

autodiff(Enzyme.ReverseWithPrimal, Const(checkpointed_integration), Duplicated(S, dS), Const(revolve))

but I seem to get the same error:

julia> include("weird_llvm_bug.jl")
WARNING: replacing module ShallowWaters.
[ Info: Revolve: Number of checkpoints: 25
[ Info: Revolve: Number of steps: 674
[ Info: Prediction:
[ Info: Forward steps   : 1644
[ Info: Overhead factor : 2.4391691394658754
[ Info: Size per checkpoint: 8.845 MiB
[ Info: Revolve: First Uturn
[ Info: Size of total storage: 221.132 MiB
ERROR: LoadError: Function argument passed to autodiff cannot be proven readonly.
If the the function argument cannot contain derivative data, instead call autodiff(Mode, Const(f), ...)
See https://enzyme.mit.edu/index.fcgi/julia/stable/faq/#Activity-of-temporary-storage for more information.
The potentially writing call is   %545 = call nonnull "enzyme_type"="{[-1]:Pointer}" {} addrspace(10)* ({} addrspace(10)* ({} addrspace(10)*, {} addrspace(10)**, i32)*, {} addrspace(10)*, ...) @julia.call({} addrspace(10)* ({} addrspace(10)*, {} addrspace(10)**, i32)* nonnull @ijl_apply_generic, {} addrspace(10)* addrspacecast ({}* inttoptr (i64 4782263024 to {}*) to {} addrspace(10)*), {} addrspace(10)* nonnull %435, {} addrspace(10)* nonnull %544) #107, !dbg !432, using   %435 = load atomic {} addrspace(10)*, {} addrspace(10)* addrspace(11)* %434 unordered, align 8, !dbg !432, !tbaa !117, !alias.scope !121, !noalias !124
Stacktrace:
 [1] macro expansion
   @ ~/Documents/GitHub/ShallowWaters_work/ShallowWaters.jl/optim_plus_enzyme_bug/weird_llvm_bug.jl:216
 [2] #3
   @ ~/.julia/packages/Checkpointing/CEFSa/src/Checkpointing.jl:181
 [3] #3
   @ ~/.julia/packages/Checkpointing/CEFSa/src/Checkpointing.jl:0

Stacktrace:
  [1] macro expansion
    @ ~/Documents/GitHub/ShallowWaters_work/ShallowWaters.jl/optim_plus_enzyme_bug/weird_llvm_bug.jl:216 [inlined]
  [2] #3
    @ ~/.julia/packages/Checkpointing/CEFSa/src/Checkpointing.jl:181 [inlined]
  [3] diffejulia__3_12476_inner_1wrap
    @ ~/.julia/packages/Checkpointing/CEFSa/src/Checkpointing.jl:0
  [4] macro expansion
    @ ~/.julia/packages/Enzyme/7pzUA/src/compiler.jl:7049 [inlined]
  [5] enzyme_call
    @ ~/.julia/packages/Enzyme/7pzUA/src/compiler.jl:6658 [inlined]
  [6] CombinedAdjointThunk
    @ ~/.julia/packages/Enzyme/7pzUA/src/compiler.jl:6535 [inlined]
  [7] autodiff
    @ ~/.julia/packages/Enzyme/7pzUA/src/Enzyme.jl:320 [inlined]
  [8] autodiff
    @ ~/.julia/packages/Enzyme/7pzUA/src/Enzyme.jl:348 [inlined]
  [9] autodiff
    @ ~/.julia/packages/Enzyme/7pzUA/src/Enzyme.jl:329 [inlined]
 [10] rev_checkpoint_struct_for
    @ ~/.julia/packages/Checkpointing/CEFSa/src/Schemes/Revolve.jl:471
 [11] length
    @ ./essentials.jl:10 [inlined]
 [12] copyto!
    @ ./array.jl:388 [inlined]
 [13] checkpointed_integration
    @ ~/Documents/GitHub/ShallowWaters_work/ShallowWaters.jl/optim_plus_enzyme_bug/weird_llvm_bug.jl:61 [inlined]
 [14] diffejulia_checkpointed_integration_11537wrap
    @ ~/Documents/GitHub/ShallowWaters_work/ShallowWaters.jl/optim_plus_enzyme_bug/weird_llvm_bug.jl:0
 [15] macro expansion
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/7pzUA/src/compiler.jl:7049 [inlined]
 [16] enzyme_call
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/7pzUA/src/compiler.jl:6658 [inlined]
 [17] CombinedAdjointThunk
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/7pzUA/src/compiler.jl:6535 [inlined]
 [18] autodiff
    @ Enzyme ~/.julia/packages/Enzyme/7pzUA/src/Enzyme.jl:320 [inlined]
 [19] autodiff(::ReverseMode{true, FFIABI, false, false}, ::Const{typeof(checkpointed_integration)}, ::Duplicated{Main.ShallowWaters.ModelSetup{Float32, Float32}}, ::Const{Revolve{Main.ShallowWaters.ModelSetup}})
    @ Enzyme ~/.julia/packages/Enzyme/7pzUA/src/Enzyme.jl:348
 [20] llvm_error(Ndays::Int64)
    @ Main ~/Documents/GitHub/ShallowWaters_work/ShallowWaters.jl/optim_plus_enzyme_bug/weird_llvm_bug.jl:287
 [21] top-level scope
    @ ~/Documents/GitHub/ShallowWaters_work/ShallowWaters.jl/optim_plus_enzyme_bug/weird_llvm_bug.jl:293
 [22] include(fname::String)
    @ Base.MainInclude ./client.jl:489
 [23] top-level scope
    @ REPL[1]:1
in expression starting at /Users/swilliamson/Documents/GitHub/ShallowWaters_work/ShallowWaters.jl/optim_plus_enzyme_bug/weird_llvm_bug.jl:293

@wsmoses
Copy link
Member

wsmoses commented Aug 12, 2024 via email

@swilliamson7
Copy link
Collaborator Author

Probably! It will be a sec though

@swilliamson7
Copy link
Collaborator Author

Okay, this code reproduces the error from above, and no longer needs any data files. I even somewhat minimized it :) let me know if this works for you/if you can recreate the error

@wsmoses
Copy link
Member

wsmoses commented Aug 14, 2024

What is the error message?

@wsmoses wsmoses reopened this Aug 14, 2024
@swilliamson7
Copy link
Collaborator Author 

swilliamson@CRIOS-A66253 optim_plus_enzyme_bug % julia10 --project=. readonly_bug.jl 
[ Info: Revolve: Number of checkpoints: 15
[ Info: Revolve: Number of steps: 225
[ Info: Prediction:
[ Info: Forward steps   : 522
[ Info: Overhead factor : 2.32
[ Info: Size per checkpoint: 8.595 MiB
[ Info: Revolve: First Uturn
[ Info: Size of total storage: 128.928 MiB
ERROR: LoadError: Function argument passed to autodiff cannot be proven readonly.
If the the function argument cannot contain derivative data, instead call autodiff(Mode, Const(f), ...)
See https://enzyme.mit.edu/index.fcgi/julia/stable/faq/#Activity-of-temporary-storage for more information.
The potentially writing call is   %117 = call nonnull "enzyme_type"="{[-1]:Pointer}" {} addrspace(10)* ({} addrspace(10)* ({} addrspace(10)*, {} addrspace(10)**, i32)*, {} addrspace(10)*, ...) @julia.call({} addrspace(10)* ({} addrspace(10)*, {} addrspace(10)**, i32)* nonnull @ijl_apply_generic, {} addrspace(10)* addrspacecast ({}* inttoptr (i64 4724476656 to {}*) to {} addrspace(10)*), {} addrspace(10)* nonnull %53, {} addrspace(10)* nonnull %116) #31, !dbg !77, using   %53 = load atomic {} addrspace(10)*, {} addrspace(10)* addrspace(11)* %52 unordered, align 8, !dbg !77, !tbaa !42, !alias.scope !46, !noalias !49
Stacktrace:
 [1] macro expansion
   @ ~/Documents/GitHub/ShallowWaters_work/ShallowWaters.jl/optim_plus_enzyme_bug/readonly_bug.jl:59
 [2] #1
   @ ~/.julia/packages/Checkpointing/CEFSa/src/Checkpointing.jl:181
 [3] #1
   @ ~/.julia/packages/Checkpointing/CEFSa/src/Checkpointing.jl:0

Stacktrace:
  [1] macro expansion
    @ ~/Documents/GitHub/ShallowWaters_work/ShallowWaters.jl/optim_plus_enzyme_bug/readonly_bug.jl:59 [inlined]
  [2] #1
    @ ~/.julia/packages/Checkpointing/CEFSa/src/Checkpointing.jl:181 [inlined]
  [3] diffejulia__1_6350_inner_1wrap
    @ ~/.julia/packages/Checkpointing/CEFSa/src/Checkpointing.jl:0
  [4] macro expansion
    @ ~/.julia/packages/Enzyme/7pzUA/src/compiler.jl:7049 [inlined]
  [5] enzyme_call
    @ ~/.julia/packages/Enzyme/7pzUA/src/compiler.jl:6658 [inlined]
  [6] CombinedAdjointThunk
    @ ~/.julia/packages/Enzyme/7pzUA/src/compiler.jl:6535 [inlined]
  [7] autodiff
    @ ~/.julia/packages/Enzyme/7pzUA/src/Enzyme.jl:320 [inlined]
  [8] autodiff
    @ ~/.julia/packages/Enzyme/7pzUA/src/Enzyme.jl:348 [inlined]
  [9] autodiff
    @ ~/.julia/packages/Enzyme/7pzUA/src/Enzyme.jl:329 [inlined]
 [10] rev_checkpoint_struct_for
    @ ~/.julia/packages/Checkpointing/CEFSa/src/Schemes/Revolve.jl:471
 [11] diffejulia_checkpointed_integration_4065wrap
    @ ~/Documents/GitHub/ShallowWaters_work/ShallowWaters.jl/optim_plus_enzyme_bug/readonly_bug.jl:0
 [12] macro expansion
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/7pzUA/src/compiler.jl:7049 [inlined]
 [13] enzyme_call
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/7pzUA/src/compiler.jl:6658 [inlined]
 [14] CombinedAdjointThunk
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/7pzUA/src/compiler.jl:6535 [inlined]
 [15] autodiff
    @ Enzyme ~/.julia/packages/Enzyme/7pzUA/src/Enzyme.jl:320 [inlined]
 [16] autodiff(::ReverseMode{true, FFIABI, false, false}, ::Const{typeof(checkpointed_integration)}, ::Duplicated{Main.ShallowWaters.ModelSetup{Float32, Float32}}, ::Const{Revolve{Main.ShallowWaters.ModelSetup}})
    @ Enzyme ~/.julia/packages/Enzyme/7pzUA/src/Enzyme.jl:348
 [17] llvm_error(Ndays::Int64)
    @ Main ~/Documents/GitHub/ShallowWaters_work/ShallowWaters.jl/optim_plus_enzyme_bug/readonly_bug.jl:100
 [18] top-level scope
    @ ~/Documents/GitHub/ShallowWaters_work/ShallowWaters.jl/optim_plus_enzyme_bug/readonly_bug.jl:106
in expression starting at /Users/swilliamson/Documents/GitHub/ShallowWaters_work/ShallowWaters.jl/optim_plus_enzyme_bug/readonly_bug.jl:106

@wsmoses
Copy link
Member

wsmoses commented Aug 14, 2024 via email

@swilliamson7
Copy link
Collaborator Author

Not sure I follow, but I can open an issue on Checkpointing.jl?

@wsmoses
Copy link
Member

wsmoses commented Aug 14, 2024 via email

@wsmoses
Copy link
Member

wsmoses commented Aug 21, 2024

Closing here as remaining issue looks to be in checkpointing.jl

@wsmoses wsmoses closed this as completed Aug 21, 2024
@wsmoses wsmoses mentioned this issue Aug 21, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@wsmoses @swilliamson7