Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Julia loading Julia's libstdc++ instead of system, causes inscrutable errors #198

Closed
weech opened this issue Mar 3, 2020 · 56 comments · Fixed by JuliaLang/julia#46976
Closed

Comments

@weech
Copy link

weech commented Mar 3, 2020

Hi,
GLFW.jl is affected by the outdated libstdc++ issue. It's a problem that has been affecting several projects and has been documented in at least one Discourse thread. What happens is that Julia ships with an old version of libstdc++, and something in the linking process fails on systems that expect a newer version of libstdc++. If you run a GLFW.jl function with the LIBGL_DEBUG=verbose environment variable set, you get an error message like libGL: MESA-LOADER: failed to open /usr/lib/dri/radeonsi_dri.so: /home/alex/pkgs/julia-1.3.0/bin/../lib/julia/libstdc++.so.6: version GLIBCXX_3.4.26' not found (required by /usr/lib/libLLVM-9.so)`. The libstdc++ shipped with Julia 1.3 is version 3.4.24.

There's an open issue on the main Julia repo. Other packages like RCall have been adding notices in their docs with workarounds in the meantime. The error message GLFW.jl gives right now is especially lacking. Without the LIGBL_DEBUG=verbose, on my machine it is:

libGL error: MESA-LOADER: failed to open radeonsi (search paths /usr/lib/dri)
libGL error: failed to load driver: radeonsi
libGL error: MESA-LOADER: failed to open radeonsi (search paths /usr/lib/dri)
libGL error: failed to load driver: radeonsi
libGL error: MESA-LOADER: failed to open swrast (search paths /usr/lib/dri)
libGL error: failed to load driver: swrast
ERROR: GLFWError (VERSION_UNAVAILABLE): GLX: Failed to create context: BadValue
Stacktrace:
 [1] _ErrorCallbackWrapper(::Int32, ::Cstring) at /home/alex/.julia/packages/GLFW/g1nX6/src/callback.jl:43
 [2] CreateWindow(::Int64, ::Int64, ::String, ::GLFW.Monitor, ::GLFW.Window) at /home/alex/.julia/packages/GLFW/g1nX6/src/glfw3.jl:487 (repeats 2 times)
 [3] top-level scope at REPL[3]:1

The workaround right now is to:

  1. Delete julia-1.3.0/lib/julia/libstdc++.so.6
  2. Find the system libstdc++ with whereis libstdc++
  3. Link that location to julia-1.3.0/lib/julia-libstdc++.so.6 using ln.

I think it'd be worthwhile to put a notice up on the README that this issue is occurring. I encountered the problem while trying to use Makie, and this repo was one of the first places I went to debug it since the error comes from this library. It's probably worthwhile to put a notice up in Makie too.

@jayschwa jayschwa pinned this issue Mar 3, 2020
@giordano
Copy link
Contributor

giordano commented Mar 3, 2020

What version of GLFW.jl are you using?

@weech
Copy link
Author

weech commented Mar 3, 2020

The Manifest lists GLFW_jll version = "3.3.2+0" and GLFW version = "3.2.2". The GLFW from the repos is "glfw-x11 3.3.2-1".

@SimonDanisch
Copy link
Member

Is there any way we can fix this? Seems like quite a few people run into this!

@giordano
Copy link
Contributor

There is this issue: JuliaLang/julia#34276

@giordano
Copy link
Contributor

Is there a MWE to reproduce the error? I have GCC v9.3.0 with libstdc++.so.6.0.28 on my system and the example in the README.md works for me.

@SimonDanisch
Copy link
Member

I think you need an AMD card, since this seems to be a radeon driver issue

@weech
Copy link
Author

weech commented Apr 10, 2020

Yes, I have an AMD card, and it is the radeon driver specifically that can't be loaded while Julia's included libstdc++ is loaded.

@dawbarton
Copy link

It's not just radeon; I've just been hit by this and I've got an Intel integrated GPU with the iris driver. Thankfully now works with this fix 😄

@Kevin-Mattheus-Moerman
Copy link

I have this issue for testing Makie (Ubuntu 20.04, Julia 1.4, NVIDIA

@weech thanks for posting the work around. I'm new to Julia. Can you please elaborate on your work around.

  1. Delete julia-1.3.0/lib/julia/libstdc++.so.6
    this step is clear
  2. Find the system libstdc++ with whereis libstdc++
    This just returns: libstdc++:, am I missing something?
  3. Link that location to julia-1.3.0/lib/julia-libstdc++.so.6 using ln.
    How does this linking work can you provide me with some information?

Thanks.

@weech
Copy link
Author

weech commented May 9, 2020

Hi,
After more experimentation, it turns out steps 2 and 3 shouldn't be necessary (at least it works on my machine 😉). Just removing the libstdc++ included with Julia should be enough to stop the linker errors. (Julia will load the system libstdc++ automatically if not provided an alternative).

@Kevin-Mattheus-Moerman
Copy link

@weech thanks for the quick reply. I tried doing just step 1 but unfortunately it still does not work. When I try ] build GLFW in Julia I get nothing, not even a warning message, an error or anything, just an empty line.
I then tried ] build GLMakie and got:

Building ModernGL → `~/.julia/packages/ModernGL/rVuW2/deps/build.log`
Building GLMakie ─→ `~/.julia/packages/GLMakie/2xw8L/deps/build.log`
Error: Error building `GLMakie`: 
│ libGL error: No matching fbConfigs or visuals found
│ libGL error: failed to load driver: swrast
│ init error of GLFW
│ ERROR: LoadError: OpenGL/GLFW wasn't installed correctly. This likely means,
│ you don't have an OpenGL capable Graphic Card,
│ you don't have the newest video driver installed,
│ or the GLFW build failed. If you're on linux and `]build` GLFW failed,
│ try manually adding `sudo apt-get install libglfw3` and then `]build GLMakie`.
...
...

FYI when I run glxinfo | grep "OpenGL version" in a terminal I get:

OpenGL version string: 4.4.0 NVIDIA 340.108

Any suggestions?

Thanks. (fyi @SimonDanisch, still stuck trying to set-up Makie)

@weech
Copy link
Author

weech commented May 11, 2020

I think this might be a separate problem? I haven't seen the "No matching fbConfigs or visuals found" error before, and there's no error saying it couldn't load your driver. Can you compile and run the following program to test that the system GLFW is working? It's just a hello world app from the GLFW website that creates a window with the title "Hello World". Since you're on Ubuntu you might have to install the dev libraries and tweak where the headers are. I compiled it with clang $(pkg-config --cflags glfw3) -o glfwtest glfwtest.c $(pkg-config --libs glfw3) -lGL. Another debugging step would be to run Julia with the environment variable LIBGL_DEBUG=verbose so you get additional information.

#include <GL/gl.h>
#include <GLFW/glfw3.h>

int main(void)
{
    GLFWwindow* window;

    /* Initialize the library */
    if (!glfwInit())
        return -1;

    /* Create a windowed mode window and its OpenGL context */
    window = glfwCreateWindow(640, 480, "Hello World", NULL, NULL);
    if (!window)
    {
        glfwTerminate();
        return -1;
    }

    /* Make the window's context current */
    glfwMakeContextCurrent(window);

    /* Loop until the user closes the window */
    while (!glfwWindowShouldClose(window))
    {
        /* Render here */
        glClear(GL_COLOR_BUFFER_BIT);

        /* Swap front and back buffers */
        glfwSwapBuffers(window);

        /* Poll for and process events */
        glfwPollEvents();
    }

    glfwTerminate();
    return 0;
}

@Kevin-Mattheus-Moerman
Copy link

Kevin-Mattheus-Moerman commented May 12, 2020

@weech thanks

I think it worked, if the picture below is correct.

I followed these steps:

  1. I created that file and called it glfwtest.c
  2. I then did: sudo apt install clang
  3. I then ran clang $(pkg-config --cflags glfw3) -o glfwtest glfwtest.c $(pkg-config --libs glfw3) -lGL
  4. I then noticed the executable in the folder, if I run that I get:

Screenshot from 2020-05-12 11-54-19

How do I run julia with that environment variable LIBGL_DEBUG=verbose?
I went here: https://docs.julialang.org/en/v1/manual/environment-variables/index.html
I'll try ENV["LIBGL_DEBUG"] = "verbose" based on that link and will report back.

@Kevin-Mattheus-Moerman
Copy link

After running ENV["LIBGL_DEBUG"] = "verbose" in the REPL I tried ] build Makie again and got:

   Building ModernGL → `~/.julia/packages/ModernGL/rVuW2/deps/build.log`
   Building GLMakie ─→ `~/.julia/packages/GLMakie/S9Zib/deps/build.log`
┌┌ Error: Error building `GLMakie`: 
│ libGL: screen 0 does not appear to be DRI2 capable
│ libGL: Can't open configuration file /etc/drirc: No such file or directory.
│ libGL: Can't open configuration file /home/kevin/.drirc: No such file or directory.
│ libGL: Can't open configuration file /etc/drirc: No such file or directory.
│ libGL: Can't open configuration file /home/kevin/.drirc: No such file or directory.
│ libGL error: No matching fbConfigs or visuals found
│ libGL error: failed to load driver: swrast
│ init error of GLFW
│ ERROR: LoadError: OpenGL/GLFW wasn't installed correctly. This likely means,
│ you don't have an OpenGL capable Graphic Card,
│ you don't have the newest video driver installed,
│ or the GLFW build failed. If you're on linux and `]build` GLFW failed,
│ try manually adding `sudo apt-get install libglfw3` and then `]build GLMakie`.
│ If you're on a headless server, you still need to install x-server and
│ proper GPU drivers. You can take inspiration from this article
│ on how to get Makie running on a headless system:
│ https://nextjournal.com/sdanisch/makie-1.0
│ If you don't have a GPU, there is also a Cairo software backend
│ for Makie which you can use:
│ https://github.com/JuliaPlots/CairoMakie.jl.
│ Please check the below error and open an issue at:
│ https://github.com/JuliaPlots/GLMakie.jl.
│ After you fixed your OpenGL install, please run `]build GLMakie` again!
│ GLMakie will still load, but will be disabled as a default backend for Makie
│ 
│ Stacktrace:
│  [1] error(::String) at ./error.jl:33
│  [2] top-level scope at /home/kevin/.julia/packages/GLMakie/S9Zib/deps/build.jl:63
│  [3] include(::String) at ./client.jl:439
│  [4] top-level scope at none:5
│ in expression starting at /home/kevin/.julia/packages/GLMakie/S9Zib/deps/build.jl:31
│ caused by [exception 1]
│ GLFWError (VERSION_UNAVAILABLE): GLX: Failed to create context: BadMatch
│ Stacktrace:
│  [1] _ErrorCallbackWrapper(::Int32, ::Cstring) at /home/kevin/.julia/packages/GLFW/g1nX6/src/callback.jl:43
│  [2] CreateWindow(::Int64, ::Int64, ::String, ::GLFW.Monitor, ::GLFW.Window) at /home/kevin/.julia/packages/GLFW/g1nX6/src/glfw3.jl:487
│  [3] GLFW.Window(; name::String, resolution::Tuple{Int64,Int64}, debugging::Bool, major::Int64, minor::Int64, windowhints::Array{Tuple{UInt32,Int64},1}, contexthints::Array{Tuple{UInt32,Integer},1}, visible::Bool, focus::Bool, fullscreen::Bool, monitor::Nothing, share::GLFW.Window) at /home/kevin/.julia/packages/GLFW/g1nX6/src/glfw3.jl:338
│  [4] top-level scope at /home/kevin/.julia/packages/GLMakie/S9Zib/deps/build.jl:34
│  [5] include(::String) at ./client.jl:439
│  [6] top-level scope at none:5
└ @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/Operations.jl:892
   Building FFTW ────→ `~/.julia/packages/FFTW/5DZuu/deps/build.log`

I still get nothing after running build GLFW.

@Kevin-Mattheus-Moerman
Copy link

I upgraded to nvidia-driver-390 (was using 340) and now Makie does build without errors. So that resolved my issue. Thanks for your help @weech.

@mateuszatki
Copy link

I got

  Building FFTW ────→ `~/.julia/packages/FFTW/DMUbN/deps/build.log`
   Building GLMakie ─→ `~/.julia/packages/GLMakie/4EXKe/deps/build.log`
┌ Error: Error building `GLMakie`: 
│ libGL error: MESA-LOADER: failed to open iris (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri)
│ libGL error: failed to load driver: iris
│ libGL error: MESA-LOADER: failed to open iris (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri)
│ libGL error: failed to load driver: iris
│ libGL error: MESA-LOADER: failed to open swrast (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri)
│ libGL error: failed to load driver: swrast
│ init error of GLFW

Unfortunately proposed walk-around sounds a bit hacky, as in my opinion replacing stdc++ with other version for which Julia's binaries weren't build although might solve GLMakie build problem, it may as well cause other issues.

What worked for me instead (Nvidia users only)
I just forced julia to use Nvidia with offloading in latest drivers
http://us.download.nvidia.com/XFree86/Linux-x86_64/450.66/README/primerenderoffload.html

__NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia julia

@SimonDanisch
Copy link
Member

@jayschwa, you think we could improve the situation, by loading the systems glfw library?
I think this is a BinaryBuilder problem (well not alone, but I think the systems glfw doesn't have this problem), so something like this may just work:

try 
  using GLFW_jll
catch e
  @warn "BinaryBuilder GLFW doesn't load see issue #198, trying to load system GLFW" exception=e
  try
    const libglfw = Libdl.open("...") # I kinda forgot how to do this cleanly, but shouldn't be to hard to figure out
  catch e
    @error "Could not load system GLFW, make sure its installed (show some tips)" exception=e
  end
end

Anyone running into this issue care to try out such a solution?

@weech
Copy link
Author

weech commented Sep 18, 2020

I gave roughly that a try, and it didn't help anything. I replaced using GLFW_jll with

global libglfw_handle = Libdl.dlopen("/usr/lib/libglfw.so") 
const libglfw = "libglfw"

and ran the tests, and it gave the normal GPU driver errors.
Again, the problem is that Julia ships a version of libstdc++ that is too old for the GPU drivers and that outdated library is loaded when Julia is launched. For this problem to go away, either the GPU drivers need to link against an older libstdc++ (i.e., build them all in BinaryBuilder) or Julia needs to ship a newer one.

By the way, this issue was opened in 1.3 times, but it's still a problem with 1.5.1.

@SimonDanisch
Copy link
Member

Ok, so bumping this issue seems to be our only hope:
JuliaLang/julia#37200
JuliaLang/julia#34276

@giordano
Copy link
Contributor

An alternative is building Julia from source, that should use system libstdc++ as far as I know 🙃

@kunzaatko
Copy link

Is it possible to use the preferences capability from BinaryBuilder.jl? It seems to be the exact thing that you are doing, but within julia therefore it would overcome the issue with different environments...

@fbanning
Copy link

fbanning commented May 31, 2022

I've experienced the same problems on openSUSE Tumbleweed and Julia 1.7.2. After a bit of fiddling around, I've updated Julia to 1.7.3 and removed the .julia/compiled folder. That did the trick for me and GLMakie works again. Maybe this helps others as well.

Edit: It worked once. After that it threw the same errors as before. I seriously have no idea why that's the case. Anyways, deleting libstdc++.so and libstdc++.so.6 solved it for me (for now).

@omendezmorales
Copy link

Hi, After more experimentation, it turns out steps 2 and 3 shouldn't be necessary (at least it works on my machine wink). Just removing the libstdc++ included with Julia should be enough to stop the linker errors. (Julia will load the system libstdc++ automatically if not provided an alternative).

It helped me to remove the symbolic link indeed, as it was referencing an older version of libstdc++; If you're in Ubuntu, have a look at /usr/lib/x86_64-linux-gnu/libstdc++.so.6, to see to which version it is pointing to.

@kescobo
Copy link

kescobo commented Jul 18, 2022

I'm getting

export LD_PRELOAD=/usr/lib64/libstdc++.so.6
ERROR: ld.so: object '/usr/lib64/libstdc++.so.6' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/usr/lib64/libstdc++.so.6' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/usr/lib64/libstdc++.so.6' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.

Not sure why the triple error message,

@giordano
Copy link
Contributor

Do you have the file /usr/lib64/libstdc++.so.6?

@kescobo
Copy link

kescobo commented Jul 18, 2022

Er... no, that would have been a good thing to check.

shell> ls /usr/lib64/
ld-linux-x86-64.so.2@

shell> ls -l /usr/lib64/
total 0
lrwxrwxrwx 1 root root 42 Mar  3 21:54 ld-linux-x86-64.so.2 -> /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2*

This is on pop!_os (ubunty jammy)

@kescobo
Copy link

kescobo commented Jul 18, 2022

@giordano so should I delete that link instead?

@kescobo
Copy link

kescobo commented Jul 19, 2022

So, following this SO answer, I get

find / -name libstdc++.so.6 2>/dev/null
/usr/lib/i386-linux-gnu/libstdc++.so.6
/usr/lib/x86_64-linux-gnu/libstdc++.so.6
#... plus a bunch of others from flatpack, etc

So I did set LD_PRELOAD /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (I'm using fish, I thought this is the equivalent of export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libstdc++.so.6), then started julia. Still got same problem trying to load GLMakie.

But I noticed that the variable was not set in ENV, so I did ENV["LD_PRELOAD"]="/usr/lib/x86_64-linux-gnu/libstdc++.so.6", the using GLMakie, and that worked! At least it finally precompiled... but now:

julia> plot(rand(10), rand(10))
libGL error: MESA-LOADER: failed to open iris: /usr/lib/dri/iris_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri, suffix _dri)

That file is there in /usr/lib/x86_64-linux-gnu/dri - any ideas?

@j-fu
Copy link

j-fu commented Jul 19, 2022

Careful with set ! The equivalent of export in fish is set -x.
I also doubt that the ENV thing really works as it comes too late in the loading process (see above).

As for the error message: what is your graphics card ?
Which graphics drivers have been installed ?
Do you work on a remote system ?
Here it seems that opengl branches into software rendering (which is done by the MESA library).

BTW there are others with a similar problem...

@kescobo
Copy link

kescobo commented Jul 19, 2022

Careful with set ! The equivalent of export in fish is set -x.

Ah, set does it permanently? 👍 Good to know


I actually have this same problem on my laptop and the remote server I use. This makes me think it might be an OS thing, rather than graphics thing, since one is nvidia and one is AMD. Both are running pop!_os (ubuntu jammy)

As for the error message: what is your graphics card ?

  • Latptop: nvidia GTX 1650
  • Server: Radeon RX 5700 XT

Which graphics drivers have been installed

  • Laptop: nvidia driver 515
  • Server: not totally sure how to check

Do you work on a remote system ?

  • Laptop: no
  • Server: yes

@jean-pierreBoth
Copy link

Same problem with Debian testing with
glxinfo | grep "OpenGL version"
OpenGL version string: 4.6 (Compatibility Profile) Mesa 22.0.5

GLMakie parabola test gives:
fig = lines(x, x.^2; label = "Parabola") gives

libGL error: MESA-LOADER: failed to open iris: /usr/lib/dri/iris_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:$${ORIGIN}/dri:/usr/lib/dri, suffix _dri)
libGL error: failed to load driver: iris
libGL error: MESA-LOADER: failed to open swrast: /usr/lib/dri/swrast_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:$${ORIGIN}/dri:/usr/lib/dri, suffix _dri)
libGL error: failed to load driver: swrast
┌ Warning: GLFW couldn't create an OpenGL window.
│ This likely means, you don't have an OpenGL capable Graphic Card,
│ or you don't have an OpenGL 3.3 capable video driver installed.
│ Have a look at the troubleshooting section in the GLMakie readme:
https://github.com/JuliaPlots/Makie.jl/tree/master/GLMakie#troubleshooting-opengl.
└ @ GLMakie ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:339
Error showing value of type Makie.FigureAxisPlot:
ERROR: GLFWError (VERSION_UNAVAILABLE): GLX: Failed to create context: GLXBadFBConfig

with iris_dri.so and and swrast_dri.so present in /usr/lib/x86_64-linux-gnu/dri.

glwgears runs without problems.

export LD_PRELOAD=/usr/lib64/libstdc++.so.6 before calling julia
solves the problem

@jean-pierreBoth
Copy link

a precision : julia 1.7.3
GLMakie "0.5.5"
GLFW "3.4.1"

@lmshk
Copy link

lmshk commented Aug 3, 2022

On Arch, this happens for me everytime a Julia version goes stale, i.e. when I update gcc-libs to a version newer than Julia. Most recently, this happened for gcc-libs-12.1.0 which contains libstdc++.so.6.0.30 (up to GLIBCXX_3.4.30).

Unfortunately, even recompiling Julia does not fix the issue because it will not pick up the newer libstdc++.so installed on the system. This looks at first glance as if the Julia build is supposed to use the system version if it is newer than whatever it knows about, but at least when I make clean and make that does not happen.

Setting LD_PRELOAD or replacing the .so files as described above works for me.

@frankier
Copy link

A fix is planned in JuliaCI/julia-buildkite#205, where the newest libstdc++ out of (bundled, system) is chosen at startup time.

@prittjam
Copy link

I've run into this problem as well. Pop OS (Ubuntu variant).

@vchuravy
Copy link

Hopefully this will be fixed by JuliaLang/julia#46976

@SimonDanisch
Copy link
Member

Fixed forever?? ❤️ Jeez, 1.9 looks like it will be an amazing release for Makie :)

@giordano
Copy link
Contributor

Yes, until gcc will completely break the ABI

@Krastanov
Copy link

Maybe this can be marked as closed now that Julia 1.9-beta2 is out?

@giordano
Copy link
Contributor

JuliaLang/julia#46976 is also in Julia v1.8.4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.