libc independence on Linux

[derekbruening]

From [email protected] on February 24, 2009 11:22:37

I did not yet remove these libc routines:

worried about, but for now only at init:
U dlclose@@GLIBC_2.0
U dlopen@@GLIBC_2.1
used only at init:
U fclose@@GLIBC_2.1
U feof@@GLIBC_2.0
U fgets@@GLIBC_2.0
U fopen@@GLIBC_2.1
U get_nprocs_conf@@GLIBC_2.0
used only for debugging but will likely be problematic:
U getchar@@GLIBC_2.0
U abort@@GLIBC_2.0
U execlp@@GLIBC_2.0
U remove@@GLIBC_2.0
do not use system calls (I did not verify though, just assuming):
U __environ@@GLIBC_2.0
U __udivdi3@@GLIBC_2.0
U __umoddi3@@GLIBC_2.0
U backtrace@@GLIBC_2.1
U backtrace_symbols_fd@@GLIBC_2.1
U dlerror@@GLIBC_2.0
U dlsym@@GLIBC_2.0
U getenv@@GLIBC_2.0
U memcpy@@GLIBC_2.0
U memmove@@GLIBC_2.0
U memset@@GLIBC_2.0
U rindex@@GLIBC_2.0
U setenv@@GLIBC_2.0
U sqrt@@GLIBC_2.0
U sscanf@@GLIBC_2.0
U stderr@@GLIBC_2.0
U stdout@@GLIBC_2.0
U strcasecmp@@GLIBC_2.0
U strchr@@GLIBC_2.0
U strcmp@@GLIBC_2.0
U strncasecmp@@GLIBC_2.0
U strncat@@GLIBC_2.0
U strncmp@@GLIBC_2.0
U strncpy@@GLIBC_2.0
U strstr@@GLIBC_2.0
U strtoul@@GLIBC_2.0
U tolower@@GLIBC_2.0

Note that we actually call dlclose at process exit which is potentially a
more fragile point then during init. Xref PR 308654 for a crash when we
call dlclose.

xref PR 204554: early injection on Linux, which relies on this case, and
for which we need to access env vars directly

xref glibc issues with binary compatibility: issue #36

Original issue: http://code.google.com/p/dynamorio/issues/detail?id=46

[derekbruening]

From [email protected] on February 24, 2009 08:28:54

this was PR 206369

[derekbruening]

From [email protected] on April 02, 2009 08:30:00

sscanf is problematic for portability: issue #36 and also for transparency: issue #105

[derekbruening]

From [email protected] on April 02, 2009 11:10:13

Is it possible to load two libc into the memory, one for application and one for DR?
It is something similar to static link DR with libc.
This should solve some of the problems.

[derekbruening]

From [email protected] on April 02, 2009 12:39:42

since we need so little of libc we've always assumed through the years that having
our own "mini libc" would be the eventual solution, where any implementation of the
string routines would do whether written ourselves or using some BSD-licensed libc
code. but we should also think about leveraging a solution here to also help with issue #30 where we want to provide libc routines to clients and we need them to have
the same names. on Windows we forward a number of common libc routines to ntdll.dll
(PR 219380) and it would be nice to have something similar on Linux but backed by
private libc routines (we would probably shift Windows to use the same routines).

[derekbruening]

From [email protected] on April 02, 2009 13:28:45

DR itself might only need a minimal libc, but what I am worry about the client, which
might want much more.
We should have a solution that client can easily use all the user libraries without
conflicting with applications.
Using wrap only works if DR has the support functions.

[derekbruening]

From [email protected] on April 07, 2009 13:55:01

yes, we're in agreement, you're echoing comment 4 from "but ..."

[derekbruening]

From [email protected] on December 07, 2009 12:12:18

xref errno issues b/c we can no longer change libc's errno address: issue #238 xref private loader: issue #157

[derekbruening]

From [email protected] on October 19, 2010 07:30:05

On my system, the ld-linux.so did not create heap before initializing DynamoRIO.
get_nprocs_conf@libc, which is called by get_num_processor from util_init(), creates the heap region for the precess (calling brk system call).

[derekbruening]

From [email protected] on April 10, 2012 19:11:23

I hit this issue while attempting to run an ASan-ified binary under DR.

ASan intercepts memcpy and others by exporting its version of memcpy, which gets picked up by libdynamorio.so before we get to libc. ASan's memcpy fails a CHECK:
==10475== CHECK failed: AddrIsInMem(p) at /usr/local/google/chrome/src/third_party/llvm/projects/compiler-rt/lib/asan/asan_mapping.h:82

AddrIsInMem() looks like:
static inline bool AddrIsInMem(uintptr_t a) {
return AddrIsInLowMem(a) || AddrIsInHighMem(a);
}

gdb gives this trace:
Breakpoint 3, 0x00000000142e2670 in __asan::AsanDie() ()
(gdb) bt
#0 0x00000000142e2670 in __asan::AsanDie() ()
#1 0x00000000142e266b in __asan::ShowStatsAndAbort() ()
#2 0x00000000142e291d in asan::CheckFailed(char const, char const, int) ()
#3 0x00000000142de0fb in memcpy ()
#4 0x00000000710ab898 in copy_and_re_relativize_raw_instr (dcontext=0x76236a40, instr=0x764aecb8, dst_pc=0x7682a350 "") at /usr/include/bits/string3.h:52
#5 0x00000000710ae99e in instr_encode_common (dcontext=0x76236a40, instr=0x764aecb8, pc=0x7682a350 "", check_reachable=true) at /home/rnk/dynamorio/core/x86/encode.c:2235
#6 0x000000007106e50c in set_linkstub_fields (num_indirect_stubs=, num_direct_stubs=, ilist=, f=, dcontext=, emit=)
at /home/rnk/dynamorio/core/emit.c:390
#7 emit_fragment_common (dcontext=0x76236a40, tag=, ilist=, flags=, vmlist=, link_fragment=, add_to_htable=true, replace_fragment=0x0)
at /home/rnk/dynamorio/core/emit.c:680
#8 0x000000007106f1ce in emit_fragment_ex (dcontext=0x1d29bcc0, tag=0x0, ilist=0x1d29bcc0, flags=41270857, vmlist=0xff, link=, visible=false) at /home/rnk/dynamorio/core/emit.c:1011
#9 0x00000000710cdd5c in build_basic_block_fragment (dcontext=0x76236a40, start=0xffffffffff600140 "UH\211\345ff\220\017\256\350\017\061\211\301ff\220\017\256\350H\211\320H\213\025J", initial_flags=0, link=,
visible=true, for_trace=false, unmangled_ilist=0x0) at /home/rnk/dynamorio/core/x86/interp.c:4234
#10 0x000000007106c7a6 in dispatch (dcontext=0x76236a40) at /home/rnk/dynamorio/core/dispatch.c:186
#11 0x000000007624a2cd in ?? ()
#12 0x0000000000000000 in ?? ()

Can you think of any good ways to avoid the interception, or do we have to define our own hidden memcpy, str*, etc symbols for libc independence?

Cc: [email protected] [email protected]

[derekbruening]

From [email protected] on April 10, 2012 19:15:17

What if we try env var ASAN_OPTIONS=replace_intrin=0:replace_str=0
It may not be enough, but it should be easy to fix

[derekbruening]

From [email protected] on April 10, 2012 19:23:03

Thanks, that seems to get past this particular issue. Long term we'll still need to solve this.

[derekbruening]

From [email protected] on April 12, 2012 07:35:17

Issue 728 has been merged into this issue.

[derekbruening]

From [email protected] on July 11, 2012 12:28:05

Our list of libc imports is getting smaller:

[rnk@wittenberg build]$ nm -D -u lib64/debug/libdynamorio.so
U __environ
v _environ
U abort
U backtrace
U backtrace_symbols_fd
U dlclose
U dlerror
U dlopen
U dlsym
v environ
U execlp
U getenv
U memmove
U remove
U setenv
U sqrt
U stderr
U stdin
U stdout

Most of these we never actually use. dlopen&co are behind the -no_private_loader flag. abort, backtrace, execlp, remove, and some others are all for debugging, and I don't have good ways to test any changes I make to them.

The environment and stdio related imports are actually used, and we'll need to cut them for early injection.

At this point I can run dr_asan without any special ASAN_OPTIONS, but memmove might still be a problem. I missed that in my string CL ( r1458 ) because I just grepped for str. =P

Owner: [email protected]

[derekbruening]

From [email protected] on August 04, 2012 08:01:56

Here's the current list of imports:
$ nm -D -u ./lib64/debug/libdynamorio.so
U abort
U backtrace
U backtrace_symbols_fd
U dlclose
U dlerror
U dlopen
U dlsym
U execlp
U remove
U sqrt

My plan is to make the dlfcn imports weak, so we can support -no_private_loader when using LD_PRELOAD. This still provides somewhat useful functionality, so I'd rather not break it completely.

sqrt is for computing stddev. I'm not sure what to do here. I try to do a simple approximation.

The rest are from stackdump.c. I have a patch to eliminate the abort, execlp, and remove imports.

How do we feel about backtrace? Can we kill it off, along with KEEP_SYMBOLS_FOR_GLIBC backtrace? I've never been able to get it to work, and once we've moved off the loaded modules list, it probably won't work at all.

We also have another problem in release builds:
$ nm -D -u ./lib64/release/libdynamorio.so
U __ctype_tolower_loc
U __memcpy_chk
U __memmove_chk
U __strncpy_chk
U abort
U backtrace
U backtrace_symbols_fd
U dlclose
U dlerror
U dlopen
U dlsym
U execlp
U remove

These __*_chk guys are slipping in because of transformations gcc makes. For example, when doing memcpy with an unknown length into a buffer of a known size, gcc will pass the known size to __memcpy_chk. Otherwise, the prototypes match, so I think we can just alias them to the un-checked versions. This is C and the caller cleans up args, so we should be OK.

Not sure what's up with __ctype_tolower_loc.

[derekbruening]

From [email protected] on November 15, 2012 12:59:40

This is basically fixed, except for -no_private_loader, which will never work, and special stats builds.

Status: Verified