dlfaker does not intercept dlopen(, RTLD_DEEPBIND) flag

[obilaniu]

If an application dlopen()'s a library that makes use of OpenGL with the RTLD_DEEPBIND flag set, it will eventually crash due to a BadRequest.

In my case the application in question is proprietary and thus cannot be modified to not pass RTLD_DEEPBIND. I could write myself a small library that also interposes dlopen() and disables that flag, but it seems like this should be something that dlfaker does, because if it does not do this, there exists a way for applications to "escape" their GL-virtualized sandbox (and promptly die).

[dcommander]

It is apparently such a rare occurrence that this is the first time I've ever heard of that flag. I'll investigate. I agree that if using RTLD_DEEPBIND with OpenGL symbols breaks VirtualGL, dlfaker should intercept and counteract that flag.

[obilaniu]

@dcommander Thanks for having a look. I've done some more research:

• A similar issue was reported against ASAN (Address Sanitizer fails to intercept function in shared library opened with RTLD_DEEPBIND google/sanitizers#611)
• Applications using jemalloc crash in the presence of deep-bound libraries (because the deep-bound libraries are made to use libc's malloc()/free() instead of jemalloc's)
• The docs for the memory debugger IBM Rational Purify explicitly state that it will not work correctly if a deep-bound DSO calls malloc()/free().
• Ulrich Drepper's How to Write Shared Libraries describes RTLD_DEEPBIND and points out that this feature is really a last resort.

In my estimation, an application will crash if it deep-binds a DSO that makes calls to X. Some calls will be intercepted and aimed at X display :8, some won't and will be aimed at :0. But an application may still work when RTLD_DEEPBIND is disabled (as is the case for me), and besides this is a flag that should be avoided.

Perhaps it would be worthwhile to add a simple +-deepbind flag, defaulting to -, for this usecase. You can choose to set +deepbind if you're sure that 1) Your application needs it and 2) The DSOs concerned don't make calls into X.

[dcommander]

Just so I'm completely clear on what you're proposing:

Would it be sufficient to ignore RTLD_DEEPBIND only if dlopen() is being redirected to the VirtualGL interposer? That is, could the flag be ignored in _vgl_dlopen() instead of the interposed dlopen() function?

[dcommander]

In other words, does the following simple patch fix the problem? If so, I can easily allow it to be enabled using an environment variable.

--- a/server/dlfaker.c
+++ b/server/dlfaker.c
@@ -63,6 +63,7 @@ void *dlopen(const char *filename, int flag)
                        fprintf(stderr,
                                "[VGL] NOTICE: Replacing dlopen(\"%s\") with dlopen(\"%s\")\n",
                                filename? filename:"NULL", env? env:"NULL");
+               flag&=(~RTLD_DEEPBIND);
                retval=_vgl_dlopen(env, flag);
        }
        else if(filename && (!strncmp(filename, "libdl.", 6)

[obilaniu]

@dcommander I'm not sure; How could I test this? I thought dlfaker was LD_PRELOAD'ed and intercepted all dlopen() calls?

I interpose dlopen() as follows:

#include <stdlib.h>
#include <string.h>
#include <stdio.h>
#include <dlfcn.h>


extern void *__libc_dlopen_mode (const char *__name, int __mode);
extern void *__libc_dlsym (void *__map, const char *__name);


typedef void* (*DLOPENFN)(const char*, int);

void *dlopen(const char *filename, int flag){
        static void*       DLHANDLE           = NULL;
        static DLOPENFN    DLOPENHANDLE       = NULL;
        static const char* SUBSTITUTEFILENAME = NULL;


        if(!DLHANDLE){
                DLHANDLE     = __libc_dlopen_mode("libdl.so", 0x00002);
                DLOPENHANDLE = __libc_dlsym(DLHANDLE, "dlopen");
        }

        if(filename && strstr(filename, getenv("TARGET"))){
                flag &= ~RTLD_DEEPBIND;
        }

        DLOPENHANDLE(filename, flag);
}

[dcommander]

You would have to apply the patch to the VirtualGL source, build it, and verify that the binary works with your application. I think that is the best approach, since I have no access to your application.

[dcommander]

libdlfaker does intercept all dlopen() calls, but if I understand correctly, the problem is that your application is using the RTLD_DEEPBIND flag to load libGL or libX11 (both?), and when VirtualGL rewrites that dlopen() call to load libvglfaker.so instead, it causes problems. Most likely this is because the RTLD_DEEPBIND flag puts the interposed functions in libvglfaker.so at the top of the preference list, and thus when libvglfaker tries to use dlsym() to load the actual OpenGL or X11 functions from libGL or libX11, it receives pointers to its own interposed versions of those functions. Correct me if I'm wrong. If this is what's happening, then I think it would be sufficient to invert the RTLD_DEEPBIND flag only when dlopen() is rewritten to load libvglfaker.so, and that's what the one-line patch I posted above does.

[obilaniu]

The "application" is, from my understanding, more of a wrapper that loads the main logic from a DSO as if it were a plugin, albeight a big one. This is done by dlopen() with flags RTLD_NOW | RTLD_DEEPBIND.

If I got this right, libdlfaker.so contains only server/dlfaker.c, which implements a custom dlopen() that interposes glibc's. This custom dlopen() calls out elsewhere to _vgl_dlopen(), and _vgl_dlopen() implements in a slightly cleaner way my code above.

If that's correct, what I think happens is that libdlfaker.so's dlopen() correctly intercepts all calls to dlopen(), but because it passes through the mode bits with the offending flag RTLD_DEEPBIND unchanged all the way through _vgl_dlopen() and down to glibc dlopen(), Glibc works its magic and all references to X and OpenGL functions that VirtualGL had hoped to interpose coming from that DSO are instead routed directly to the originals.

My additional dlopen() interposer blocks the RTLD_DEEPBIND flag from reaching Glibc dlopen(). Because it acts only on the flags, it's orthogonal to libdlfaker.so's dlopen(), and they managed to cooperate on my system. There would have been no difference the call stack had been app -> mylib.so(dlopen) -> libdlfaker.so(dlopen) -> libvglfaker.so(_vgl_dlopen) -> libc.so(dlopen), or instead app -> libdlfaker.so(dlopen) -> libvglfaker.so(_vgl_dlopen) -> mylib.so(dlopen) -> libc.so(dlopen).

I think your proposed patch is essentially exactly what my interposer currently performs, and it will work.

[dcommander]

OK, so hypothetically it should work, but I need to know for sure whether my patch fixes the problem before I can include it in VirtualGL.

[obilaniu]

It turns out it's not enough to put flag &= (~RTLD_DEEPBIND); where you put it. When dlopen("lib_not_GL_or_X11.so", RTLD_NOW | RTLD_DEEPBIND) is called, and said library depends on GL or X11, dlopen() does not call itself recursively for libGL.so and libX11.so. Instead GL & X11 libraries are loaded as part of dlopen("lib_not_GL_or_X11.so", ...) and any GL/X11 symbols are deep-bound directly to their original versions, not the VirtualGL versions.

flag &= (~RTLD_DEEPBIND); belongs in

• server/dlfaker.c:40 or
• server/faker.cpp:190.

RTLD_DEEPBIND simply can't be allowed to reach Glibc dlopen(). Any library loaded with RTLD_DEEPBIND will not have the correct bindings for GL and X11 symbols, as well as dlopen() itself (which would be deep-bound to Glibc's and not libdlfaker.so's version of dlopen()).

[dcommander]

Was finally able to figure out how to build a reproducible test case for this, which I added to dlfakerut, along with the proposed fix to libdlfaker. Fixed in 87ad605. CI build should be available shortly.