Private loading not working on Android 9

[summershrimp]

The Bionic library for Android 9 have changes from Android 6 that fails dynamorio.

Below is a small trick to get it working, but not a perfectly way.

In order to running dynamorio on android9-arm64, we change privload_call_lib_func

diff --git a/core/unix/loader.c b/core/unix/loader.c
index 89d5d31c..00c3519e 100644
--- a/core/unix/loader.c
+++ b/core/unix/loader.c
@@ -911,7 +911,7 @@ privload_call_lib_func(fp_t func)
      */
     dummy_argv[0] = dummy_str;
     dummy_argv[1] = NULL;
-    func(1, dummy_argv, our_environ);
+    //func(1, dummy_argv, our_environ);
 }

 bool

Firstly, we thought programs with init or init_array would behave abnormal, but a few testcases shows that It's working perfectly. Then we thought the module load callback would behave abnormal, but it also works.

The question is, what's really happening when call init functions from dynamorio privload? What would failed if we don't call init functions from dynamorio privload?

PS. Testing with libinscount.so with patch below.

+++ b/api/samples/inscount.cpp
@@ -85,6 +85,14 @@ static dr_emit_flags_t
 event_app_instruction(void *drcontext, void *tag, instrlist_t *bb, instr_t *inst,
                       bool for_trace, bool translating, void *user_data);

+static void
+event_module_load(void *drcontext, const module_data_t *info, bool loaded)
+{
+    const char *module_name = dr_module_preferred_name(info);
+    dr_fprintf(STDOUT, "Module Name: %s\n", module_name);
+
+}
+
 DR_EXPORT void
 dr_client_main(client_id_t id, int argc, const char *argv[])
 {
@@ -108,6 +116,7 @@ dr_client_main(client_id_t id, int argc, const char *argv[])
     dr_register_exit_event(event_exit);
     drmgr_register_bb_instrumentation_event(event_bb_analysis, event_app_instruction,
                                             NULL);
+    drmgr_register_module_load_event(event_module_load);

     /* make it easy to tell, by looking at log file, which client executed */
     dr_log(NULL, DR_LOG_ALL, 1, "Client 'inscount' initializing\n");

Testing program:

bad_case.cpp:

// g++ bad_case.cpp -o bad_case.so -fPIC -shared 
#include <cassert>
#include <cstdio>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>

unsigned long global_magic;

__attribute__((constructor)) static void initall(){
    global_magic = 0xdead0000;
    printf("global_magic: %lx\n", global_magic);
}

__attribute__((destructor))  static void deinitall(){
    global_magic = 0;
    printf("global_magic: %lx\n", global_magic);
}


class BadClass{
public:
    BadClass() {
        this->magic_number = global_magic;
        this->add_number = 0xbeef;
        this->magic_number = this->magic_number + this->add_number;
    }

    bool isOK(){
        printf("%lx\n", this->magic_number);
        return this->magic_number == 0xdeadbeef;
    }

    ~BadClass() {
        printf("deconstruct\n");
        this->magic_number = 0;
    }

private:
    unsigned long magic_number;
    unsigned long add_number;
};

BadClass globalbad;
extern "C"{
int bad_case(void) {
    char tmp[4096];
    int cnt;
    assert(globalbad.isOK());
    int fd = open("/proc/self/maps", O_RDONLY);
    while((cnt = read(fd, tmp, 4096)) != 0)
        write(1, tmp, cnt);
    return 0;
}
}

main.c:

// gcc main.c -o main -fPIE -pie -ldl
#include <dlfcn.h>
#include <stdlib.h>
int (*bad_case)(void);

int main(void){
    void* handle = dlopen("./bad_case.so", RTLD_LAZY);
    if(!handle) {
        perror("dlopen");
        exit(-1);
    }
    bad_case = dlsym(handle, "bad_case");
    if(!bad_case) {
        perror("dlsym");
        exit(-1);
    }

    bad_case();
}

[summershrimp]

It seems the privload loads modules for clients, not for target executable. Will this modification affect thread init?

[AssadHashmi]

Warning: everything below is based on my limited understanding (@derekbruening and others can correct me).

The question is, what's really happening when call init functions from dynamorio privload?

It's the same as would happen as when Linux loads shared objects, i.e. .init and .fini get called on load and unload. DynamoRIO takes control of the shared object loading process on behalf of Linux and maintains runtime transparency by calling .init and .fini as Linux would do.

What would failed if we don't call init functions from dynamorio privload?

What happens depends on your client. If init doesn't get called and nothing in your client relies on it then things should be fine.

Will this modification affect thread init?

I don't think thread initialisation in client shared objects is affected by whether init is called. The threads library should take care of that.

[derekbruening]

That sounds right. I would just elaborate by saying:

If a privately-loaded library's initialization function is failing, it is probably due to some problem or limitation with the isolation (separation of private libraries from application libraries) provided by DR, and something in that library is not going to work properly. If your client's use of that library does not depend on whatever is broken, then it may work fine in practice for you. However, I would not recommend disabling the calling of every init function: only the one that is failing.

Ideally you would debug the problem and find a solution. That's not always easy, as we have seen on Windows where libraries are doing complex things that interact with csrss and other low-level Windows services. On Android it should be easier, especially if you have the source code to the library in question. The procedure would be to step through the init function in the debugger and see where it fails.

You never said what the symptoms were of calling the init function that caused you to disable it in the first place, nor what library we're talking about, nor what client, nor what DR version. If this is supposed to be a bug in our issue tracker all of that needs to be supplied. If this is a general question, please use the users list https://groups.google.com/forum/#!forum/DynamoRIO-Users rather than the issue tracker, as the latter is for specific detailed bugs, and the former will reach the other users who may find the information beneficial.

[summershrimp]

@derekbruening @AssadHashmi
Thank for replying. The latest Android Bionic loader have so many global states that if you do not start from /system/bin/loader, It's hardly to run init functions for libc.so, libm.so and so on. We've tried to fix the broken state, but it needs many works.
We will try to make privload working on Android in a more beautiful way, and I'd to change the issue tittle to 'Not working on Android 9'.

[derekbruening]

I see, thank you for the information and the effort to make it work.

[derekbruening]

Another user hit this with the outward visible symptom a fatal SIGILL: https://groups.google.com/forum/#!topic/DynamoRIO-Users/diIYsP0TK_8

[petrochenkov]

More detailed explanation of why the first entry in .init_array of libc.so fails if run by a DR's private loader:

• The first entry is __libc_preinit which quickly calls __loader_shared_globals which is defined outside of libc.so, so the call goes through PLT.
• To call __loader_shared_globals through PLT we are resolving it by name going through DT_NEEDED dependencies of libc.so. libc.so has dependencies on ld-android.so and libdl.so and __loader_shared_globals is found in ld-android.so during the name search (DR's module_lookup_symbol).
• However ld-android.so is just a stub (https://android.googlesource.com/platform/bionic/+/refs/heads/master/linker/ld_android.cpp) all functions in which result in __builtin_trap, these functions are never supposed to be found by real Android linker.
• The real home for __loader_shared_globals and other symbols from the link above is the /usr/bin/linker(64) binary (the dynamic linker) (https://android.googlesource.com/platform/bionic/+/refs/heads/master/linker/dlfcn.cpp). During real Android dynamic linking /usr/bin/linker(64) is always first in the list of modules when __loader_shared_globals is looked up by name, so the lookup completes successfully and we never reach ld-android.so.
• So the main solution that I see here (besides removing the init_array calls and hoping for the better) is to load a private copy of /usr/bin/linker(64) and put it at the start of the private loader module list. /usr/bin/linker(64) is a static PIE, it is able to relocate itself at an arbitrary address after start, so DR will probably be able to load and relocate it correctly as well.

(Note that if a DR client is linked with -static and absorbs libc.a and friends into itself, then it gets a different set of init functions and never refers to dynamic linker, so none of the issues described above appear, and things just work.)

[petrochenkov]

so DR will probably be able to load and relocate it correctly as well.

UPD: DR can load /usr/bin/linker(64) as a generic binary fixing the issue with __loader_shared_globals, but it'll miss some initialization steps unique to that binary resulting in some global state staying uninitialized (e.g. libc_shared_globals::auxv staying NULL) and causing further crashes in __libc_preinit.

UPD2: Also, to relocate the linker DR needs to support DT_RELR/DT_ANDROID_RELR relocations.

[petrochenkov]

DR can load /usr/bin/linker(64) as a generic binary ... but it'll miss some initialization steps unique to that binary resulting in some global state staying uninitialized

One more idea is to to load the bionic linker by just putting it into memory, and then finding the __linker_init function in it and transferring control to it so the linker initializes itself, including all the custom steps.
The main problem in this case is setting up some kind of "sandboxed environment" before calling __linker_init, so it only does self-initialization and doesn't make changes to code outside of it, like attempting to relocate libdynamorio.so again or something like that.

[petrochenkov]

putting it into memory, and then finding the __linker_init function in it and transferring control to it so the linker initializes itself, including all the custom steps

Somewhat similar example - #5134.

[derekbruening]

putting it into memory, and then finding the __linker_init function in it and transferring control to it so the linker initializes itself, including all the custom steps

Somewhat similar example - #5134.

Right, it did remind me of Android, with hardcoded dependencies between the loader and libc (and libpthread), though a relatively simple case of calling an extra function with a specific name if it exists.