-replace_malloc: /MD apps fail during init w/ OOM

[derekbruening]

From [email protected] on August 06, 2012 23:18:54

mallocMD, cs2bugMD, and cfrac print this app error:

runtime error R6016 - not enough space for thread data

% cd ~/extsw/heaplayers-351/benchmarks/cfrac/
% /usr/bin/time ~/drmemory/git/build_x86_dbg/bin/drmemory.exe -replace_malloc -dr d:/derek/dr/git/exports -batch -- ./cfrac.exe 41757646344123832613190542166099121
runtime error R6016 - not enough space for thread data

comparing the bbs, this is where -replace_malloc diverges:
new basic block @0x6d5f0715 == MSVCR100.dll!_getptd_noexit
new basic block @0x6d6175b0 == MSVCR100.dll!_getptd_noexit
new basic block @0x6d5f1e1c == MSVCR100.dll!_calloc_crt
new basic block @0x6d5f09a9 == MSVCR100.dll!_calloc_impl
new basic block @0x6d5f09b5 == MSVCR100.dll!_calloc_impl
new basic block @0x6d5f09c5 == MSVCR100.dll!_calloc_impl
new basic block @0x6d5f09d4 == MSVCR100.dll!_calloc_impl
new basic block @0x6d5f09df == MSVCR100.dll!_calloc_impl
new basic block @0x6d5f09ee == MSVCR100.dll!_calloc_impl
new basic block @0x6d61f3ec == MSVCR100.dll!_calloc_impl
new basic block @0x6d61f418 == MSVCR100.dll!_calloc_impl
new basic block @0x6d5f09f6 == MSVCR100.dll!_calloc_impl
new basic block @0x6d5f1e32 == MSVCR100.dll!_calloc_crt
new basic block @0x6d61f1c4 == MSVCR100.dll!_calloc_crt
new basic block @0x6d5f1e3f == MSVCR100.dll!_calloc_crt
new basic block @0x6d6175bc == MSVCR100.dll!_getptd_noexit
new basic block @0x6d5f071f == MSVCR100.dll!_getptd_noexit
new basic block @0x75b211a9 == KERNEL32.dll!SetLastError
new basic block @0x6d5f0726 == MSVCR100.dll!_getptd_noexit
new basic block @0x6d5f0733 == MSVCR100.dll!_getptd
new basic block @0x6d61760a == MSVCR100.dll!_getptd
new basic block @0x6d63bf5b == MSVCR100.dll!_amsg_exit

from -no_replace_malloc:
new basic block @0x6d5f0715 == MSVCR100.dll!_getptd_noexit
new basic block @0x6d6175b0 == MSVCR100.dll!_getptd_noexit
new basic block @0x6d5f1e1c == MSVCR100.dll!_calloc_crt
new basic block @0x6d5f09a9 == MSVCR100.dll!_calloc_impl
new basic block @0x6d5f09b5 == MSVCR100.dll!_calloc_impl
new basic block @0x6d5f09c5 == MSVCR100.dll!_calloc_impl
new basic block @0x6d5f09d4 == MSVCR100.dll!_calloc_impl
new basic block @0x6d5f09df == MSVCR100.dll!_calloc_impl
new basic block @0x77ae2d37 == ntdll.dll!RtlpAllocateHeap
new basic block @0x77ae3a5d == ntdll.dll!RtlpAllocateHeap
new basic block @0x77ae2c37 == ntdll.dll!RtlpAllocateHeap
new basic block @0x77ae31a6 == ntdll.dll!RtlpAllocateHeap
new basic block @0x77ae3c2c == ntdll.dll!RtlpAllocateHeap
new basic block @0x77ae31c1 == ntdll.dll!RtlpCreateSplitBlock
new basic block @0x77ae31d4 == ntdll.dll!RtlpCreateSplitBlock
new basic block @0x77ae3e14 == ntdll.dll!RtlpCreateSplitBlock
new basic block @0x77ae2c97 == ntdll.dll!RtlpAllocateHeap
new basic block @0x77ae2c87 == ntdll.dll!RtlpAllocateHeap
new basic block @0x77addf40 == ntdll.dll!memset
new basic block @0x7380180c == drmemorylib.dll!ILT+2055(_replace_memset)
new basic block @0x73a88000 == drmemorylib.dll!replace_memset
new basic block @0x73a8805e == drmemorylib.dll!replace_memset
new basic block @0x73a88064 == drmemorylib.dll!replace_memset
new basic block @0x73a88080 == drmemorylib.dll!replace_memset
new basic block @0x73a880a2 == drmemorylib.dll!replace_memset
new basic block @0x77ae2c8f == ntdll.dll!RtlpAllocateHeap
new basic block @0x6d5f09ee == MSVCR100.dll!_calloc_impl
new basic block @0x6d5f09f6 == MSVCR100.dll!_calloc_impl
new basic block @0x6d5f1e32 == MSVCR100.dll!_calloc_crt
new basic block @0x6d5f1e3f == MSVCR100.dll!_calloc_crt
new basic block @0x6d6175bc == MSVCR100.dll!_getptd_noexit

0:000> U 0x6d5f09df
MSVCR100!_calloc_impl+0x3a:
6d5f09df 56 push esi
6d5f09e0 6a08 push 0x8
6d5f09e2 ff35b048696d push dword ptr [MSVCR100!_crtheap (6d6948b0)]
6d5f09e8 ff15f8115e6d call dword ptr [MSVCR100!_imp__HeapAlloc (6d5e11f8)]

w/o syms this is "0x6d5f09df == MSVCR100.dll!unlock"

replace_RtlAllocateHeap heap=0x000a0000 (=> 0x00000000) flags=0x8 size=0x214

this is the issue #606 internal _calloc_impl, bypassing libc Heap from issue #939 this is the additional heap w/ /MD (b/c we come in after
dynamic crt init, as opposed to exe's static crt init) => hits failure of
-replace_malloc to handle pre-us heaps other than ProcessHeap.

walking 3 heaps
walking heap 0 0x01070000
adding heap region 0x01070000-0x01170000 arena
walking heap 1 0x000a0000
adding heap region 0x000a0000-0x000b0000 arena
walking heap 2 0x00cc0000
skipping private heap 0x00cc0000

solutions:

• early injection: or just earlier injection.
  could impl that in drinject.
  except we'll hit issue APP CRASH and HANG (ipc_tests IPCChannelTest.ChannelTest on xp32) #699 which is why currently using -no_early_inject!
  need a custom entry point after kernel32 but before msvcrt.
• handle multiple pre-us heaps: issue -replace_malloc: support multiple pre-us Heaps #959 .

sure, should handle multiple pre-us heaps: but in this case can we use
issue #939 arena_for_libc_alloc()? unfortunately here we don't
intercept MSVCR100.dll!_calloc_impl (added to list for issue #940 )
and we see this at Rtl level only.
so really if we had issue #606 fixed this would just work?

trying to see if having symbols solves it I hit some problems:

• bug in get_libc_base() on windows
• _calloc_impl only shows up in private sym search:
  % bin/symquery.exe -e C:/Windows/system32/MSVCR100.dll --search -s "__impl"
  % bin/symquery.exe -e C:/Windows/system32/MSVCR100.dll --searchall -s "__impl"
  _calloc_impl +0x109a9
  and lookup_symbol_common() turns a "full" search into a singleton lookup
  but after fixing those => it works!
  presumably full search perf hit won't be too bad when only on msvcrt (and
  not on say chrome.dll)

so once I put in a solution for issue #606 part A _calloc_impl (for issue #607 ) this
particular instance will be solved.

Original issue: http://code.google.com/p/drmemory/issues/detail?id=960

[derekbruening]

From [email protected] on February 21, 2013 18:33:02

once fixed, re-enable the replace_operatorsMDd for issue #1083

[derekbruening]

From [email protected] on April 30, 2013 19:40:34

This issue was closed by revision r1306 .

Status: Fixed