Using the LD LIBRARY PATH to load local libraries does not work on Linux

[wli3]

From @sunliusi on Sunday, March 22, 2020 4:21:18 AM

I need to use the local library for cross-platform. Local libraries have different implementations and Put them in different folders.

it work fine in windows just like this:
Environment.SetEnvironmentVariable("PATH", path + ";" + Environment.GetEnvironmentVariable("PATH"));

it's not working in linux just like this:
Environment.SetEnvironmentVariable("LD_LIBRARY_PATH", dir + ":" + Environment.GetEnvironmentVariable("LD_LIBRARY_PATH"));

Copied from original issue: dotnet/sdk#10957

[wli3]

From @sunliusi on Sunday, March 22, 2020 4:28:55 AM

NativeLibrary doesn't work either. There are two local libraries, one dependent on the other. This works only if the local library has no dependencies.

var path1 = Path.Combine(assemblyDirectory, "unix", "testaa.so");
               var p1 = NativeLibrary.Load(path1);
               if (p1 == IntPtr.Zero)
               {
                   throw new ApplicationException("load assembly fail:" + path1);
               }


               var path = Path.Combine(assemblyDirectory, "unix", "test.so");
               var p = NativeLibrary.Load(path);
               if (p == IntPtr.Zero)
               {
                   throw new ApplicationException("load assembly fail:" + path);
               }

---> System.DllNotFoundException: Unable to load shared library 'test.so' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: ./testaa.so: cannot open shared object file: No such file or directory
at System.Runtime.InteropServices.NativeLibrary.LoadFromPath(String libraryName, Boolean throwOnError)
at System.Runtime.InteropServices.NativeLibrary.Load(String libraryPath)

[wli3]

From @SourceSkyBoxer on Tuesday, March 24, 2020 9:48:41 AM

Hello,
I will tell you example with NativeLibrary()

      /*
        *  Valid library name for Linux of X11, Extensions, OpenGL and Vulkan
        */
       private const string libX11 = "libX11.so.6";
       private const string libXrender = "libXrender.so.0";
       private static IntPtr handle;

       static XLib()
       {
           // Important instance of TryGetExport()
           handle = NativeLibrary.Load(libX11, typeof(XLib).Assembly, 0);
       }

       /*
        *  Display *XOpenDisplay(char *displayname);
        */
       [UnmanagedFunctionPointer(CallingConvention.Cdecl)]
       private delegate IntPtr _XOpenDisplay(string displayname);
       public static Display XOpenDisplay(string displayname)
       {
           _XOpenDisplay opendisplay_delegate;
           if (NativeLibrary.TryGetExport(handle, "XOpenDisplay", out IntPtr opendisplay))
           {
               opendisplay_delegate = Marshal.GetDelegateForFunctionPointer<_XOpenDisplay>(opendisplay);
           }
           else
           {
               opendisplay_delegate = (string name) => { throw new NotImplementedException("Error: XOpenDisplay not found."); };
           }

           return new Display(opendisplay_delegate(displayname));
       }

If you want FullPath like this.
You should use "AppDomain.CurrentDomain.BaseDirectory"
like this:

            ...
            string rootdir = AppDomain.CurrentDomain.BaseDirectory;
            handle = NativeLibrary.Load(Path.Combine(rootdir, "unix", "sofile.so"), typeof(YourClass).Assembly, 0);
            ...

You should create delegate with UnmanagedFunctionPointer

        [UnmanagedFunctionPointer(CallingConvention.Cdecl)]
        private delegate void _Method(string sayhello);

Than you can try as public method if you try out.

        public static void Method(string sayhello)
        {
            _Method method_delegate;
            if (NativeLibrary.TryGetExport(handle, "Method", out IntPtr method_ptr))
            {
                method_delegate = Marshal.GetDelegateForFunctionPointer<_Method>(method_ptr);
            }
            else
            {
                method_delegate = (string name) => { throw new NotImplementedException("Error: Method not found."); };
            }

            method_delegate(sayhello);
        }

Just it is small example of my made.
I hope my solution helps you as well.

// EDIT: You mean multiple so files?
No problem for me

            ...
            // multiple so files
            string[] sofiles =
            {
                "file1.so", "file2.so"
            };
            string rootdir = AppDomain.CurrentDomain.BaseDirectory;
            foreach (string sofile in sofiles)
            {
                handle = NativeLibrary.Load(Path.Combine(rootdir, "unix", sofile));
            }
            ...

It is example:
For old version of NetFX 4.5 picture.

It is very easy to understand like I made it.

It is simple for you. I hope that.

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. Please help me learn by adding exactly one area label.

[wli3]

From @sunliusi on Wednesday, March 25, 2020 4:29:01 AM

Thanks @SourceSkyBoxer for such a detailed explanation.

Follow the example above:
// multiple so files string[] sofiles = { "file1.so", "file2.so" }; string rootdir = AppDomain.CurrentDomain.BaseDirectory; foreach (string sofile in sofiles) { handle = NativeLibrary.Load(Path.Combine(rootdir, "unix", sofile)); }

The problem is that NativeLibrary.Load will fail when file2.so references file1.so.

If you put file2.so and file1.so in the root directory, there is no problem

[wli3]

From @sunliusi on Wednesday, March 25, 2020 4:30:22 AM

System.DllNotFoundException: Unable to load shared library '.../unix/file2.so' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: ./file1.so: cannot open shared object file: No such file or directory

[jkoritzinsky]

I believe that LD_LIBRARY_PATH has to be set before the program starts (either exported into the environment or when it is started). We don't propagate environment variable changes back out to the system when they're changed in .NET on Unix.

[jkoritzinsky]

Related to/possible duplicate of #9529

[janvorli]

I believe that LD_LIBRARY_PATH has to be set before the program starts (either exported into the environment or when it is started). We don't propagate environment variable changes back out to the system when they're changed in .NET on Unix.

That's correct. And it is not a .NET problem and it is unrelated to propagation env variables settings into the system, setting LD_LIBRARY_PATH has effect only before a process launch for any application written in any language. That's how the dynamic linker works.

[ssa3512]

I am running into a similar issue with interop on linux. I have a nuget package that is distributing some native libraries that I am calling with P/Invoke. The libraries are in the bin folder as such:

bin/Debug/netcoreapp3.1/runtimes
├── linux-x64
│   └── native
│       ├── libfile1.so
│       ├── libfile2.so
│       └── libfile3.so
├── linux-x86
│   └── native
│       ├── libfile1.so
│       ├── libfile2.so
│       └── libfile3.so
├── win-x64
│   └── native
│       ├── file1.dll
│       ├── file2.dll
│       └── file3.dll
└── win-x86
    └── native
        ├── file1.dll
        ├── file2.dll
        └── file3.dll

file1 has a dependency on file2 and file3. The code runs fine on Windows x86 and x64, but fails on linux with this error:

The active test run was aborted. Reason: Test host process crashed : Unhandled exception.
System.DllNotFoundException: Unable to load shared library 'file1' or one of its dependencies. In
order to help diagnose loading problems, consider setting the LD_DEBUG environment variable:
libfile1: cannot open shared object file: No such file or directory

The code is calling into this library as such:

[DllImport("file1")
public static extern void NativeMethod();

If I copy the files libfile1.so, libfile2.so and libfile3.so to /usr/lib the code executes fine. Is this related to the above issue? I would expect .NET core to include the appropriate runtimes folder (in this case runtimes/linux-x64/native in the search path for interop.

ldd output without runtimes folder in LD_LIBRARY_PATH

projectdir> ldd bin/Debug/netcoreapp3.1/runtimes/linux-x64/native/libfile1.so
        linux-vdso.so.1 (0x00007ffd95b94000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa4abdbd000)
        libfile2.so => not found
        libfile3.so => not found
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fa4aba1f000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa4ab62e000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fa4ac34e000)

Including runtimes folder in LD_LIBRARY_PATH

projectdir> LD_LIBRARY_PATH=$LD_LIBRARY_PATH:bin/Debug/netcoreapp3.1/runtimes/linux-x64/native ldd bin/Debug/netcoreapp3.1/runtimes/linux-x64/native/libfile1.so
        linux-vdso.so.1 (0x00007fff7c4d4000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2a2163e000)
        libfile2.so => bin/Debug/netcoreapp3.1/runtimes/linux-x64/native/libfile2.so (0x00007f2a2142c000)
        libfile3.so => bin/Debug/netcoreapp3.1/runtimes/linux-x64/native/libfile3.so (0x00007f2a21183000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f2a20de5000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2a207ed000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f2a21bcf000)

[sunliusi]

Put so in the root directory on linux, and put dll in custom directory on windows, it also works.

Set LD_LIBRARY_PATH before the program start is not a good idea. It's best to control it using code, just like Windows

[ssa3512]

It is my understanding that the /runtimes/[platform]/native folder is supposed to be supported as a delivery mechanism for native libraries https://docs.microsoft.com/en-us/nuget/create-packages/supporting-multiple-target-frameworks#architecture-specific-folders

The challenge seems to come in when those libraries have other dependencies also being distributed this way.

[emmenlau]

I'm also affected by this problem, and I've spent the past day trying to figure out what is going wrong. Reading this issue I still do not see the best solution (or workaround).

Why does .NET not resolve libraries in architecture specific folders (i.e. in runtimes/linux-x64/native/)? I've tried to see with strace where my DllImport library is searched, and it seems that dotnet is not even checking the runtimes/linux-x64/native/ folder for the DllImport (let alone its dependencies).

[sweemer]

I have encountered this issue as well and agree with @ssa3512 and @emmenlau that the .NET runtime should be responsible for somehow properly adding all nupkg native libs and their dependencies to the runtime search path.

In the meantime, I did find a workaround that I would like to share. You can use the compiler option -Wl,-rpath,"\$ORIGIN" when compiling (linking) the library that you specify in DllImport, which will add the executable directory to the search path. Since dotnet publish puts the executable and all the native libs in the publish directory together, the libraries are all available on the search path; i.e. the publish directory.

If you are using CMake then you can add set(CMAKE_BUILD_RPATH "$\{ORIGIN\}") to your CMakeLists.txt file to accomplish the same thing.