RFC: ccall using clang (structs 2)

[vtjnash]

This supports all struct types for ccall. I know it needs a rebase before merging. Note that the required clang headers are not typically installed (and clang is not available at all in Travis so that fails).

@JeffBezanson

[StefanKarpinski]

Is the plan to merge this before the v0.1 deadline or not? I'm not quite comfortable with the timeframe for such a big change, but we do have two outstanding issues that are related to this (#2177, #2154).

[andreasnoack]

#2154 is not really a problem for v0.1. I've changed complex dot to use the Julia implementation and hence there is no error anymore. The error from #2177 is slightly more problematic because the complex Faddeeva functions used a C wrapper which would be annoying to reintroduce temporarily until complex return works.

[JeffBezanson]

Very impressive!! Jameson you are a beast.
This is too disruptive for 0.1. We should just disable the problematic complex versions of the Faddeeva functions for 0.1, so there isn't a segfaulting function lying around.

[StefanKarpinski]

Very impressive!! Jameson you are a beast.
This is too disruptive for 0.1. We should just disable the problematic complex versions of the Faddeeva functions for 0.1, so there isn't a segfaulting function lying around.

Yes! This is great. We'll merge it immediately after the 0.1 branch is forked.

[vtjnash]

We might want to just revert to using the Faddeeva wrapper then for 0.1 and make ccall with Structs an error (and fix the null pointer bug in alloc.c)

I wanted to get this done so that @sebastien-villemot could comment on how to package clang for this (since we need to compile against some internal clang headers)

[JeffBezanson]

Such packaging issues are exactly why we won't aim to put this in 0.1. Those kinds of things are much more problematic than a few thousand lines of code :)

If there are bug fixes here that apply to what's already in master, let's pull them over.

Reverting to using the Faddeeva wrapper seems annoying, and these functions aren't crucial.

[ViralBShah]

I would much rather revert to Faddeeva wrapper, and not ship code that we know will segfault. Alternatively, we can disable this functionality altogether for 0.1.

It can easily be got rid of after 0.1.

[andreasnoack]

The complex Faddeeva functions have been disabled for 32 bit systems. Hence no segfaults.

[ViralBShah]

I did notice WORD_SIZE==64 check in math.jl. This means that the Faddeeva functions are only available on 64-bit until ccall is fixed.

[JeffBezanson]

I did that. I think it's a good compromise.

[ViralBShah]

Yes, I think it is a good compromise.

[stevengj]

In addition to struct arguments and return values, it would be good to support accessing fields of pointers to structs (either as return values or converted from Ptr{Void}). Presumably this would use much of the same machinery.

(I'm running into this in the NumPy C API, in which a lot of functions are actually macros to access fields from a struct pointer, which makes it challenging to call from pure Julia.)

[vtjnash]

clang isn't necessarily required to support that, although it does have the potential benefit of parsing header files for you

I still need to add this to the documentation, but unsafe_ref will copy data from a Ptr{T} to a T object. I also still want to extend it to support the syntax unsafe_ref( struct_ptr, :field_name ), equivalent to unsafe_ref(struct_ptr).field_name
unsafe_assign(ptr, value, index) does the reverse operation (currently only supported for BitsKinds)

[vtjnash]

remaining known issues:

• define NDEBUG correctly (currently assumes llvm --enable-assertions)
• documentation

[vtjnash]

./usr/bin/llvm-config.exe --cflags includes NDEBUG if its needed

[Keno]

@vtjnash What's the status of this?

[JeffBezanson]

I think clang is just too large a dependency. But if we want to include it, we can get things like parsing header files, and automatically handling packages that come with C code. It's a big decision.

[Keno]

And calling C++

[vtjnash]

I'm waiting for Jeff to decide he is OK with it, then I'll rebase and update to handle immutable types. Although adding clang is a bit much for just call-by-value structs, it has some nice benefits posible if it is installed (like CIndex https://github.com/ihnorton/Clang.jl)

[stevengj]

It seems like this would hugely simplify distribution of external packages that wrap C or C++ code, which is an enormous headache right now because you can't rely on Mac or Windows systems having a compiler.

[StefanKarpinski]

You also can't rely on them having header files, so it only helps so much, but yes, it would be better.

[stevengj]

@StefanKarpinski, good point. Without libc header files a compiler is basically useless for Pkg dependencies.