ccall() with an absolute library path while outputting an .o file causes linker issues on Windows

[staticfloat]

Kristoffer and I were tracking down precompilation failures for PackageCompilerX, when we noticed a strange error coming from ld while it was trying to link together the system image:

Warning: .drectve `-export:ccalllib_C:\Users\Matthis\.julia\packages\FreeType\2dE5w\deps\usr\bin\libfreetype-6.dll,data ' unrecognized
C:/Users/Matthis/.julia/artifacts/572b61b5075459e3ed62317e674398166ca98dd4/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: Cannot export FreeType: symbol not found
C:/Users/Matthis/.julia/artifacts/572b61b5075459e3ed62317e674398166ca98dd4/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: Cannot export Users: symbol not found
C:/Users/Matthis/.julia/artifacts/572b61b5075459e3ed62317e674398166ca98dd4/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: Cannot export ccalllib_C:: symbol not found
C:/Users/Matthis/.julia/artifacts/572b61b5075459e3ed62317e674398166ca98dd4/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: Cannot export packages: symbol not found

This is being emitted as a global variable from ccall(), but appears to have some quoting issues on Windows, as the linker doesn't like the backslashes in the symbol name. We're not sure if the linker is truly to blame here, as we usually don't pass absolute library paths so we may have never actually tried to link a system image with this usage pattern before.

You can reproduce this with the following mini reproducer:

using Pkg.BinaryPlatforms

open("libfoo.c", "w") do io
    write(io, """
    #include <stdio.h>

    void foo() {
        printf("Hello, Foo!\\n");
    }
    """)
end

@info("Compiling C source...")
libfoo_fname = string("libfoo.", platform_dlext())
run(`gcc -o $libfoo_fname libfoo.c -shared`)

open("Foo.jl", "w") do io
    write(io, """
    module Foo
    const libfoo = joinpath(@__DIR__, "$libfoo_fname")
    ccall((:foo, libfoo), Cvoid, ())
    end
    """)
end

@info("Compiling Julia module...")
run(`$(Base.julia_cmd()) --startup-file=no --output-o foo.o Foo.jl`)

@info("Dumping `ccalllib` julia module symbols...")
run(pipeline(`nm foo.o`, `grep ccalllib`))

Looking at the other ccalllib symbols, they are all "soname" style:

0000000001615770 b _ccalllib_/Users/sabae/libfoo.dylib
000000000162c3d0 b _ccalllib_libcholmod
00000000016160d8 b _ccalllib_libdSFMT
0000000001615b48 b _ccalllib_libgit2
0000000001618f40 b _ccalllib_libgmp
0000000001611b50 b _ccalllib_libmpfr
000000000162ae30 b _ccalllib_libopenblas64_
0000000001623800 b _ccalllib_libopenlibm
0000000001610b60 b _ccalllib_libpcre2-8
000000000162c430 b _ccalllib_libsuitesparse_wrapper
000000000162c510 b _ccalllib_libsuitesparseconfig

[aminya]

This is the same error that I reported last year that caused the MKL building fail.
JuliaLinearAlgebra/MKL.jl#24

I posted a possible solution JuliaLinearAlgebra/MKL.jl#24 (comment), but I am not sure if it helps.

https://sourceforge.net/p/mingw-w64/mailman/message/21864960/

The problem here is the name decoration. MS doesn't extend the public 
symbols in objects by underscores, mingw-w64 does this by default. But you 
can recompile mingw-w64 crt and your app by using option 
-fno-leading-underscores. This should work.
The other solution would be, name the exported method call_me in VS 
_call_me, and reference in gcc just without the leading underscore.

[KristofferC]

This is a regression from 1.3.0 (appeared in 1.3.1) so putting on milestone. Probably #34062.

[aminya]

Can we have this backported too or not possible?

This is a regression from 1.3.0 (appeared in 1.3.1) so putting on milestone. Probably #34062.

[jwveysey]

A quick question about this issue if I could. I've been running into it while using PackageCompiler (the refactored version - dev release 1.0.0) to build a system image that includes a custom package. The process fails when gcc tries to process an import directive for a binary dependency for CodecZlib. My team was hoping to get the system image soon, so I was wondering if there's a patch I could apply to fix ccall. Would you advise trying a nightly build? Right now, I'm on Julia 1.3.1. Thanks for any pointers!

[staticfloat]

You can try this PR: #34754

[jwveysey]

You can try this PR: #34754

Thanks for the quick response. Using that PR will require building Julia myself, right? Sorry for the elementary question, but I haven't done that before. I can try following the instructions on the Julia readme, though.

[staticfloat]

That's correct, but I can help you get a binary of that version. What OS are you using?

[jwveysey]

Thank you very much! I'm on Windows 10 Professional 64-bit.

…

Sent from my mobile

________________________________ From: Elliot Saba <[email protected]> Sent: Friday, February 14, 2020 7:31:47 PM To: JuliaLang/julia <[email protected]> Cc: Jason W. Veysey <[email protected]>; Manual <[email protected]> Subject: Re: [JuliaLang/julia] `ccall()` with an absolute library path while outputting an `.o` file causes linker issues on Windows (#34680) That's correct, but I can help you get a binary of that version. What OS are you using? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#34680?email_source=notifications&email_token=AJGJHPIA5B4GB5ZLTH5CVN3RC4ZXHA5CNFSM4KQTHMO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL24FHI#issuecomment-586531485>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJGJHPPK76CV2RSVHGKYEQLRC4ZXHANCNFSM4KQTHMOQ>.

[staticfloat]

Try this link: https://julialangnightlies-s3.julialang.org/assert_pretesting/winnt/x64/1.5/julia-66ecd90bd6-win64.exe

[jwveysey]

@staticfloat, thank you very much for your guidance on this. I tried that build, and it seemed to work for my compilation. I have to do a bit more testing on the system image, but it's looking good so far! Do you have a sense when the fix will be included in an official release?

[aminya]

I tried the latest Julia 1.5 and it worked too (building MKL). Maybe the new PackageCompiler uses a workaround for that.

[KristofferC]

Didn't you use JuliaLinearAlgebra/MKL.jl#29 for that? That PR makes it so that there is no absolute path and this issue is avoided.

[KristofferC]

Do you have a sense when the fix will be included in an official release?

It will be in 1.4 which shouldn't take too long to release (hopefully).

[aminya]

Didn't you use JuliaComputing/MKL.jl#29 for that? That PR makes it so that there is no absolute path and this issue is avoided.

Yes, I used this PR. so that explains the stuff.

Do you have a sense when the fix will be included in an official release?

It will be in 1.4 which shouldn't take too long to release (hopefully).

That's awesome. Thank you!

[jwveysey]

Yes, thanks for the update on the release.