at Windows libc level, the right Heap to use is unknown

[derekbruening]

From [email protected] on July 12, 2012 15:22:03

splitting from main -replace_malloc case issue #794 xref issue #893 _msize in libc calls "HeapSize(_crtheap, 0, pblock)"
malloc in libc calls "HeapAlloc(_crtheap, 0, size ? size : 1)"
we intercept before we see the param _crtheap!
so how do we know which Heap to use?
while for _msize we can ignore the Heap if we don't mind not reporting an
invalid arg error, we need to know the Heap to properly allocate and free.

solutions:

1. move interception one layer in, and thus miss out on the
   (rare) cases of libc doing its own parceling.
   small-block heap is rarely used (off by default for win2k+), but there are
   other problems:

• dbg crt will add its 40 extra bytes, wasting some memory, but worse
  setting up its own redzone and extra info such that DrMem won't catch
  overflows ( issue runsuite return 0 when tests fail #284 ).
• dbg crt also has features like _CRTDBG_DELAY_FREE_MEM_DF => delays frees,
  which will cause DrMem to not see the free and thus not catch
  use-after-free errors.
• xref PR 595801 where not intercepting _dbg crt resulted in unaddrs,
  though re-visiting I'm not sure why: _dbg crt was reading the Rtl header?
• need to intercept operators, and they will have the same problem!
  can't easily not intercept them: on windows they violate abstractions and
  directly read headers ( issue APP ASSERT _BLOCK_TYPE_IS_VALID(pHead->nBlockUse) on operator delete with /MTd #26 ) (also xref issue enable redzones in Debug CRT #832 )

certainly seems best to intercept crt layer.

2. do a lookup ptr_to_arena() on every single free/size call.
   such a lookup needs to walk heap_region tree which is too expensive.

• can do quick checks up front for being in cur_arena, and perhaps some
  checks of 64K back-aligns (though windows arena allocs reserve 4MB by
  default)
• could add backpointer in every single chunk to its arena but that seems
  overkill
• could have an arena cache per heap routine set (== dll): but don't have
  mechanism to pass extra arg to replacement routines. do have retaddr so
  could look up the module it's in: but that's not overhead-free, even
  counting on the callstack module cache.

but what do we do on malloc()?
break Rtl compat below libc layer and ignore libc's Heap?
can we detect its HeapCreate call, or honor it but it will just sit empty?
what if it calls some Heap stats/etc. routines?
worse, what if it's in a library that's unloaded and thus calls
HeapDestroy trying to free all its allocs? we'll have leaks.

could honor the libc Heaps, and cache the last few retaddr-to-arena
mappings, and do full heap region walk on a miss. this wouldn't support
multiple Heaps from the same libc routine set, which is unlikely for crt
but may happen in the future w/ custom app routines ( issue #107 )

Original issue: http://code.google.com/p/drmemory/issues/detail?id=939

[derekbruening]

From [email protected] on July 12, 2012 14:59:08

proposal: create custom Heap per libc routine set, and all of those
routines use that Heap. honor any HeapCreate and HeapDestroy calls from a
lib but never use those Heaps unless passed to Rtl routine. on lib unload,
destroy its libc routine Heap. this requires augmenting
drwrap_replace_native() to pass a custom arg via SPILL_SLOT_2 (or else
messy gencode wrappers).

[derekbruening]

From [email protected] on July 16, 2012 07:59:19

This issue was closed by revision r920 .

Status: Fixed