Side-effectful foreach callback mysteriously eliminated on nightly (miscompilation?)

[ToucheSir]

I discovered this while trying to determine why Flux's nightly CI was failing. Have only managed to reduce it to the following:

using ChainRulesCore

function mwe(x, v::Val{S}) where {S}
    function inner(ys)
        l = size(x, 1)
        dx = fill!(similar(x, float(eltype(x)), l * S, size(x, 2)), false)
        gs = ntuple(v) do n
            @view dx[begin+(l*(n-1)):(l*n), :]
        end
        foreach(gs, ys) do g, y
          y isa AbstractZero && return
          copy!(g, y)
        end
        # Works
        # for (g, y) in zip(gs, ys)
        #   y isa AbstractZero && continue
        #   copy!(g, y)
        # end
        return dx
    end
    inner
end

x = zeros(2, 3)
f = mwe(x, Val(3))
ys = (a=ones(2, 3), b=ZeroTangent(), c=ZeroTangent())
f(ys)

Expected output:

6×3 Matrix{Float64}:
 1.0  1.0  1.0
 1.0  1.0  1.0
 0.0  0.0  0.0
 0.0  0.0  0.0
 0.0  0.0  0.0
 0.0  0.0  0.0

Actual output:

6×3 Matrix{Float64}:
 0.0  0.0  0.0
 0.0  0.0  0.0
 0.0  0.0  0.0
 0.0  0.0  0.0
 0.0  0.0  0.0
 0.0  0.0  0.0

What makes this mysterious is what seemingly minor changes can restore the expected behaviour:

• Using the loop instead of foreach
• Defining a custom abstract type + 2 zero-sized struct subtypes mirroring ChainRulesCore's AbstractZero + NoTangent/ZeroTangent. This only seems to work with the CRC types.
• Using a Vector instead of a NamedTuple. Note that the original example used ChainRulesCore.Tangent, but I'm not sure if that indicates anything.
• Adding Core.donotdelete(false) in the foreach callback. This suggests to me that the call to said callback is being completely eliminated somehow. Unfortunately Cthulhu won't let me descend into the Core._apply_iterate(Base.iterate, f, z) call that would actually call said callback, so the best I have to go on here is the following remarks:

Call result type was widened because the return value is unused [constprop]
Disabled by argument and rettype heuristics [constprop] Disabled by entry heuristic (unimprovable result)

Versioninfo:

Julia Version 1.10.0-DEV.661
Commit 0b8e8fc216a (2023-02-25 23:01 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 8 × Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, haswell)
  Threads: 1 on 8 virtual cores

[inkydragon]

• ❌ Version 1.10.0-DEV.667 (2023-02-27) a23b29c
• ❌ Version 1.10.0-DEV.180 (2022-12-22) a074d06
• ✅ Version 1.10.0-DEV.179 (2022-12-22) 181328c
• ✅ Version 1.10.0-DEV.0 (2022-11-14) 7fe6b16
• ✅ Julia Version 1.9.0-beta4
• ✅ Version 1.8.5 (2023-01-08)

Note: No need to build it yourself, you can download pre-built artifacts in CI.

> git bisect good
a074d06e2d8d645fca6c6751325f6f31bda99a2f is the first bad commit
commit a074d06e2d8d645fca6c6751325f6f31bda99a2f
Author: Shuhei Kadowaki <[email protected]>
Date:   Thu Dec 22 20:30:28 2022 +0900

    inference: mark flag for effect-free `:call`s during abstractinterpret (#47689)

    So that they can be deleted during the first `compact!`-ion.
    This allows us to delete an inlineable and effect-free, but unused call.

    This is essentially an alternative of #47305, but doesn't introduce a
    problem like #47374.

 base/compiler/abstractinterpretation.jl | 15 +++++++++++++++
 base/compiler/inferencestate.jl         |  2 ++
 test/compiler/inline.jl                 | 23 +++++++++++++++++++++--
 3 files changed, 38 insertions(+), 2 deletions(-)

a074d06 cc: @aviatesk

[aviatesk]

Thanks so much for the bisect, @inkydragon !