Handle 4-element insert_backedges_callee

[rikhuijzer]

This PR should go a long way towards fixing SnoopCompile on nightly. I have to admit that I don't really understand most of the code, I just compared the output of invalidations in test/snoopi_deep.jl of Julia 1.8.2 with nightly and tried to match them based on dump.c after JuliaLang/julia#46920.

Note that some data which is passed by dump.c, such as callee, target_idx, and callee_idx are not used. I still annotated them with types to make it easier to use them in case that's necessary.

@timholy Feel free to modify this PR directly if you want to make changes.

Appendix

For comparison, these are the outputs of invalidations in test/snoopi_deep.jl around line 868 with some of my notes:

Julia 1.8.2

julia> invalidations
18-element Vector{Any}:
  MethodInstance for StaleA.use_stale(::Vector{Any})
 1
  MethodInstance for StaleA.build_stale(::Int64)
 2
  MethodInstance for StaleA.stale(::Any)
  "jl_method_table_insert"
  stale(x::String) in StaleC at /home/rik/git/SnoopCompile.jl/test/testmodules/Stale/StaleC/src/StaleC.jl:5
  "jl_method_table_insert"
  MethodInstance for StaleB.useA()
 1
  MethodInstance for StaleA.stale(::String)
  "jl_insert_method_instance"
  stale(x::String) in StaleC at /home/rik/git/SnoopCompile.jl/test/testmodules/Stale/StaleC/src/StaleC.jl:5
  "jl_method_table_insert"

  MethodInstance for StaleA.stale(::String) # This is pushed onto callees in SnoopCompile (note the first push already before going into the loop).
  "insert_backedges_callee"

  MethodInstance for StaleB.useA() # This is pushed into callers in SnoopCompile
  "insert_backedges"

julia> invalidation_trees(invs)[1]
callees = Any[MethodInstance for StaleA.stale(::String)]
callers = Core.MethodInstance[MethodInstance for StaleB.useA()]
inserting stale(x::String) in StaleC at /home/rik/git/SnoopCompileMaster.jl/test/testmodules/Stale/StaleC/src/StaleC.jl:5 invalidated:
   mt_backedges: 1: signature stale(x::String) in StaleC at /home/rik/git/SnoopCompileMaster.jl/test/testmodules/Stale/StaleC/src/StaleC.jl:5 (formerly stale(x) in StaleA at /home/rik/git/SnoopCompileMaster.jl/test/testmodules/Stale/StaleA/src/StaleA.jl:3) triggered MethodInstance for StaleB.useA() (0 children)
   backedges: 1: superseding stale(x) in StaleA at /home/rik/git/SnoopCompileMaster.jl/test/testmodules/Stale/StaleA/src/StaleA.jl:3 with MethodInstance for StaleA.stale(::Any) (2 children)

Julia nightly

julia> invalidations
15-element Vector{Any}:
  MethodInstance for StaleA.use_stale(::Vector{Any})
 1
  MethodInstance for StaleA.build_stale(::Int64)
 2
  MethodInstance for StaleA.stale(::Any)
  "jl_method_table_insert"
  stale(x::String) @ StaleC ~/git/SnoopCompile.jl/test/testmodules/Stale/StaleC/src/StaleC.jl:5
  "jl_method_table_insert"

  MethodInstance for StaleA.stale(::String) # `callee` belonging to "insert_backedges_callee". These 4 elements repeat for each target.
  "insert_backedges_callee"
 1 # target (of backedge) index belonging to "insert_backedges_callee"
  Any[stale(x::String) @ StaleC ~/git/SnoopCompile.jl/test/testmodules/Stale/StaleC/src/StaleC.jl:5]

  MethodInstance for StaleB.useA() # `caller` belonging to "verify_methods". These 3 elements repeat for each callee.
  "verify_methods"
 1 # callee index belonging to "verify_methods"

julia> invalidation_trees(invalidations)[1]
inserting stale(x::String) in StaleC at /home/rik/git/SnoopCompile.jl/test/testmodules/Stale/StaleC/src/StaleC.jl:5 invalidated:
   mt_backedges: 1: signature stale(x::String) in StaleC at /home/rik/git/SnoopCompile.jl/test/testmodules/Stale/StaleC/src/StaleC.jl:5 (formerly stale(x) in StaleA at /home/rik/git/SnoopCompile.jl/test/testmodules/Stale/StaleA/src/StaleA.jl:3) triggered MethodInstance for StaleB.useA() (0 children)
   backedges: 1: superseding stale(x) in StaleA at /home/rik/git/SnoopCompile.jl/test/testmodules/Stale/StaleA/src/StaleA.jl:3 with MethodInstance for StaleA.stale(::Any) (2 children)

[codecov]

Codecov Report

Base: 86.14% // Head: 85.31% // Decreases project coverage by -0.82% ⚠️

Coverage data is based on head (e2bc381) compared to base (f4d9a7b).
Patch coverage: 63.82% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #304      +/-   ##
==========================================
- Coverage   86.14%   85.31%   -0.83%     
==========================================
  Files          17       17              
  Lines        2129     2132       +3     
==========================================
- Hits         1834     1819      -15     
- Misses        295      313      +18     

Impacted Files	Coverage Δ
src/invalidations.jl	83.51% <63.82%> (-3.39%)	⬇️
SnoopPrecompile/src/SnoopPrecompile.jl	81.25% <0.00%> (-6.25%)	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[rikhuijzer]

Ugh this parsing logic is very tricky especially with all the different cases for different Julia versions.

The failure is around the following (test/snoopr.jl line 240). Before JuliaLang/julia#46920, invalidation_trees turned:

35-element Vector{Any}:
                   Tuple{typeof(PkgC.nbits), Integer}
                   "insert_backedges_callee"
                 41
 0x000000ae8df0e2d6
                   Any[nbits(::UInt8) @ Main REPL[14]:4]
                   MethodInstance for iterate(::Base.Generator{Vector{Integer}, typeof(PkgD.call_nbits)})
                  1
                   MethodInstance for Base._iterator_upper_bound(::Base.Generator{Vector{Integer}, typeof(PkgD.call_nbits)})
                  2
                   MethodInstance for Base._collect(::Vector{Integer}, ::Base.Generator{Vector{Integer}, typeof(PkgD.call_nbits)}, ::Base.EltypeUnknown, ::Base.HasShape{1})
                  3
                   MethodInstance for Base.collect_similar(::Vector{Integer}, ::Base.Generator{Vector{Integer}, typeof(PkgD.call_nbits)})
                  4
                   MethodInstance for map(::typeof(PkgD.call_nbits), ::Vector{Integer})
                  5
                   MethodInstance for PkgD.map_nbits(::Vector{Integer})
                  6
                   MethodInstance for PkgD.nbits_list()
                  7
                   MethodInstance for Base._collect(::Vector{Integer}, ::Base.Generator{Vector{Integer}, typeof(PkgD.call_nbits)}, ::Base.EltypeUnknown, ::Base.HasShape{1})
                  2
                   MethodInstance for iterate(::Base.Generator{Vector{Integer}, typeof(PkgD.call_nbits)}, ::Int64)
                  1
                   MethodInstance for Base._iterator_upper_bound(::Base.Generator{Vector{Integer}, typeof(PkgD.call_nbits)})
                  2
                   MethodInstance for Base.collect_to!(::Vector{Int64}, ::Base.Generator{Vector{Integer}, typeof(PkgD.call_nbits)}, ::Int64, ::Int64)
                  2
                   MethodInstance for Base.collect_to_with_first!(::Vector{Int64}, ::Int64, ::Base.Generator{Vector{Integer}, typeof(PkgD.call_nbits)}, ::Int64)
                  3
                   MethodInstance for Base._collect(::Vector{Integer}, ::Base.Generator{Vector{Integer}, typeof(PkgD.call_nbits)}, ::Base.EltypeUnknown, ::Base.HasShape{1})
                  4
                   MethodInstance for PkgD.call_nbits(::Integer)
                   "insert_backedges"
                 41
 0x000000ae8df0e2d6

into

1-element Vector{SnoopCompile.MethodInvalidations}:
 inserting nbits(::UInt8) @ Main REPL[14]:4 invalidated:
   mt_backedges: 1: signature Tuple{typeof(PkgC.nbits), Integer} triggered MethodInstance for PkgD.call_nbits(::Integer) (13 children)

However, after JuliaLang/julia#46920, the output is a vector of 124 elements starting with (first 20):

julia> invalidations
124-element Vector{Any}:
   Tuple{typeof(PkgC.nbits), Integer}
   "insert_backedges_callee"
 33
   Any[nbits(::UInt8) @ Main REPL[7]:4]
   MethodInstance for PkgD.call_nbits(::Integer)
   "verify_methods"
 33
   MethodInstance for iterate(::Base.Generator{Vector{Integer}, typeof(PkgD.call_nbits)}, ::Int64)
   "verify_methods"
   MethodInstance for PkgD.call_nbits(::Integer)
   MethodInstance for Base.collect_to!(::Vector{Int64}, ::Base.Generator{Vector{Integer}, typeof(PkgD.call_nbits)}, ::Int64, ::Int64)
   "verify_methods"
   MethodInstance for iterate(::Base.Generator{Vector{Integer}, typeof(PkgD.call_nbits)}, ::Int64)
   MethodInstance for iterate(::Base.Generator{Vector{Integer}, typeof(PkgD.call_nbits)}, ::Int64)
   "verify_methods"
   MethodInstance for PkgD.call_nbits(::Integer)
   MethodInstance for iterate(::Base.Generator{Vector{Integer}, typeof(PkgD.call_nbits)}, ::Int64)
   "verify_methods"
   MethodInstance for PkgD.call_nbits(::Integer)
   MethodInstance for Base.collect_to!(::Vector{Int64}, ::Base.Generator{Vector{Integer}, typeof(PkgD.call_nbits)}, ::Int64, ::Int64)
   [...]

Now all the required info is in just the first sets of elements, namely the 4 elements around "insert_backedges_callee" (invalidations[1:4]) and the 3 elements around the first "verify_methods" (invalidations[5:7]). However, I cannot figure out where to get the "13 children" part from.

EDIT: It might be the following change in JuliaLang/julia#46920 since it looks like SnoopCompile.jl was counting the elements between "insert_backedges_callee" and "insert backedges":

-            invalidate_backedges(&remove_code_instance_from_validation, caller, world, "insert_backedges");
-            if (_jl_debug_method_invalidation) {
-                targetidx = jl_box_int32((int32_t)idxbad);
-                jl_array_ptr_1d_push(_jl_debug_method_invalidation, targetidx);
-                targetidx = jl_box_uint64(jl_worklist_key(serializer_worklist));
-                jl_array_ptr_1d_push(_jl_debug_method_invalidation, targetidx);
-            }

[timholy]

Oh, sorry I failed to notice this! I tackled this in #306, I made you a co-author.