callback functions see "old world" in 0.6

[stevengj]

It seems much harder to pass a proper closure as a callback function in Julia 0.6 than in previous versions due to #17057. In particular, the method described in the manual and in this blog post no longer works: passing the Function via a "thunk" void* that is called from a cfunction fails because the Function sees an "old world age".

See JuliaPy/PyCall.jl#343 and JuliaMath/Cubature.jl#22 and the mailing list, for example, but this will affect any package wrapping a C library that expects callback functions.

In PyCall, @yuyichao suggested using eval, but this is clunky and seems likely to impose a large performance penalty. Is there a better solution?

[yuyichao]

This describes a expected behavior so this is not a regression but simply a breaking change. The eval here shouldn't add that much overhead although a lighter weight one will be good.

[yuyichao]

The fix of this (adding doc or add a slightly cheaper way to call something in the latest world) won't be breaking.........

[stevengj]

We need documentation of this, and to update the blog post. Can we have eval(function, args...) or something?

Why does a call to a Function object need to use the caller's world age?

[yuyichao]

Inference cannot infer anything about a newer world so any change in world age has to be an optimization barrier. We also need to be consistent on what do we treat as optimization barrier and failure to do that is one part of #265. Since a function call can be optimized, it must never be an optimization barrier, i.e. the dispatch and the body of f() must see the same world as the caller.

[stevengj]

But a call to a Function object that is fetched from a pointer cannot be optimized, so it can be an optimization barrier, no?

[yuyichao]

The semantics shouldn't be inference dependent unless we have a clear rule of what's inferrable.

[stevengj]

Presumably unsafe_pointer_to_objref (or any ccall with a return type Any) should never result in an inferrable type?

[yuyichao]

The result can be typeasserted. The underlying implementation also allow specifying the type.

[stevengj]

Sure, but if it is not typeasserted, or is only typeasserted to an abstract type like Function, that still seems pretty clear-cut?

[yuyichao]

No. I think it's a bad idea to treat it like that. There's nothing special of it and it'll be really bad if f() and Ref{Any}(f)[]() return different result.

[yuyichao]

Adding a invokeinworld or invokelatest shouldn't be too hard. I can't do it in the next ~24hrs though.

[stevengj]

+1 for invokelatest(f, args...).

Seems like it could be built by modifying jl_apply_generic to take a world age as an argument and then passing jl_get_world_counter()? No, that wouldn't work, because it wouldn't affect functions called by the called function. Seems like you need a try -- finally that temporarily resets the world age. Looks like jl_call already does this.

[stevengj]

Looks like uv_getaddrinfocb in base/socket.jl may need this too?

[stevengj]

Regarding your comment elsewhere, maybe cfunction should not capture the world by default?

[yuyichao]

That will put a lot of overhead for almost all cfunctions.

[stevengj]

Yeah, I guess that would kill performance for calling any function from the cfunction.

[stevengj]

@yuyichao, was this closed by a commit?

[yuyichao]

#20167