hard_rss_limit_mb broken on Android O

[eugenis]

tl;dr we spawn a background thread before the global stack cookie is initialized.

Libc has at least two constructors there, and it initializes the stack protector cookie in the second one.

First happens this:
#12 0xf6ff8ed8 in AsanInitInternal () at /code/llvm-project/compiler-rt/lib/asan/asan_rtl.cc:454
#13 0xf6fb5124 in pthread_mutex_lock ()
at /code/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:3801
#14 0xf6f034ca in jemalloc_constructor () from target:/system/lib/libc.so
#15 0xf772959a in _dl__ZL13call_functionPKcPFviPPcS2_ES0 () from target:/system/bin/linker

AsanInitInternal spawns the background thread.

Then this:
#0 0xf6f0e550 in __libc_preinit() () from target:/system/lib/libc.so
#1 0xf772959a in _dl__ZL13call_functionPKcPFviPPcS2_ES0 () from target:/system/bin/linker

That's where the stackprotector cookie is set. Our background thread, meanwhile, is waiting in
#0 0xf75c14f8 in usleep () from target:/system/lib/libc.so
#1 0xf6dec05c in BackgroundThread () at /code/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_common_libcdep.cc:100

and fails the stack cookie check when leaving usleep().

Apparently, we can not create new threads in pthread_mutex_lock interceptor, or they would race with the initialization of the cookie.

I don't think this is a bug in bionic, because user code, normally, is not supposed to run that early.
Perhaps we could delay thread creation somehow?

[eugenis]

This does not reproduce on the current master branch of Android.

[eugenis]

And the reason is: at some pointer stack protector cookie initialization has been moved to an earlier stage in __linker_init_post_relocation.