Persistent Memory leaks reported when using ASan

[mgaudet]

When running the Tril tests compiled with Asan, comptest reports a large amount of leaked memory all passing through jitPersistentAlloc.

One answer is that these leaks may be in some sense false positives because of the persistent alloc story, but it would be good to figure if it's possible to remove them.

Direct leak of 27720 byte(s) in 385 object(s) allocated from:
    #0 0x7fce4306d602 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98602)
    #1 0xc66faf in OMR::RawAllocator::allocate(unsigned long, std::nothrow_t, void*) (/home/magaudet/open/omr/build/fvtest/compilertriltest/comptest+0xc66faf)
    #2 0xd155e4 in OMR::PersistentAllocator::allocate(unsigned long, std::nothrow_t, void*) (/home/magaudet/open/omr/build/fvtest/compilertriltest/comptest+0xd155e4)
    #3 0xc77111 in TR_PersistentMemory::allocatePersistentMemory(unsigned long, TR_MemoryBase::ObjectType) (/home/magaudet/open/omr/build/fvtest/compilertriltest/comptest+0xc77111)
    #4 0xd1296f in TR_MemoryBase::jitPersistentAlloc(unsigned long, TR_MemoryBase::ObjectType) (/home/magaudet/open/omr/build/fvtest/compilertriltest/comptest+0xd1296f)
    #5 0x10817f2 in OMR::PointerType::operator new(unsigned long, TR::Internal::PersistentNewType) (/home/magaudet/open/omr/build/fvtest/compilertriltest/comptest+0x10817f2)
    #6 0x107ee78 in OMR::TypeDictionary::TypeDictionary() (/home/magaudet/open/omr/build/fvtest/compilertriltest/comptest+0x107ee78)
    #7 0xbf9e97 in TR::TypeDictionary::TypeDictionary() (/home/magaudet/open/omr/build/fvtest/compilertriltest/comptest+0xbf9e97)
    #8 0xbf9f78 in Tril::TRLangBuilder::TRLangBuilder(ASTNode const*, TR::TypeDictionary*) (/home/magaudet/open/omr/build/fvtest/compilertriltest/comptest+0xbf9f78)
    #9 0xbf95f3 in Tril::SimpleCompiler::compileWithVerifier(TR::IlVerifier*) (/home/magaudet/open/omr/build/fvtest/compilertriltest/comptest+0xbf95f3)
    #10 0xbf9410 in Tril::SimpleCompiler::compile() (/home/magaudet/open/omr/build/fvtest/compilertriltest/comptest+0xbf9410)
    #11 0xab53f5 in Int32Arithmetic_UsingConst_Test::TestBody() (/home/magaudet/open/omr/build/fvtest/compilertriltest/comptest+0xab53f5)
    #12 0xc5204e in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/home/magaudet/open/omr/build/fvtest/compilertriltest/comptest+0xc5204e)
    #13 0xc47957 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/home/magaudet/open/omr/build/fvtest/compilertriltest/comptest+0xc47957)
    #14 0xc1a023 in testing::Test::Run() (/home/magaudet/open/omr/build/fvtest/compilertriltest/comptest+0xc1a023)
    #15 0xc1afbb in testing::TestInfo::Run() (/home/magaudet/open/omr/build/fvtest/compilertriltest/comptest+0xc1afbb)
    #16 0xc1b93b in testing::TestCase::Run() (/home/magaudet/open/omr/build/fvtest/compilertriltest/comptest+0xc1b93b)
    #17 0xc277b3 in testing::internal::UnitTestImpl::RunAllTests() (/home/magaudet/open/omr/build/fvtest/compilertriltest/comptest+0xc277b3)
    #18 0xc542cd in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/home/magaudet/open/omr/build/fvtest/compilertriltest/comptest+0xc542cd)
    #19 0xc492ea in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/home/magaudet/open/omr/build/fvtest/compilertriltest/comptest+0xc492ea)
    #20 0xc24fb0 in testing::UnitTest::Run() (/home/magaudet/open/omr/build/fvtest/compilertriltest/comptest+0xc24fb0)
    #21 0xa7a375 in RUN_ALL_TESTS() (/home/magaudet/open/omr/build/fvtest/compilertriltest/comptest+0xa7a375)
    #22 0xa7a1f9 in testMain (/home/magaudet/open/omr/build/fvtest/compilertriltest/comptest+0xa7a1f9)
    #23 0xbf180c in main (/home/magaudet/open/omr/build/fvtest/compilertriltest/comptest+0xbf180c)
    #24 0x7fce4216d82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

[dsouzai]

I think anything allocated via TR_PersistentMemory or TR::PersistentAllocator is going to result in external tools thinking there's a memory leak. One potential way of verifying that there aren't any leaks, would be to maybe have a special build where we replace the bodies of jitPersistentAlloc and jitPersistentFree with calls to malloc and free.

EDIT: Better would be to replace the bodies of TR::PersistentAllocator::allocate and TR::PersistentAllocator::deallocate with calls to malloc and free.

[mstoodle]

Is this pointing out a problem with persistent allocator, per se, or a problem in TypeDictionary and its "managed" types? I suppose the example pasted is just one; but are the others all coming in via TypeDictionary types or through a variety of persistent allocations?

[mgaudet]

All seem to be TypeDictionary related. - Matt

…

On Nov 24, 2017, at 12:36 PM, Mark Stoodley ***@***.***> wrote: Is this pointing out a problem with persistent allocator, per se, or a problem in TypeDictionary and its "managed" types? I suppose the example pasted is just one; but are the others all coming in via TypeDictionary types or through a variety of persistent allocations? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[dsouzai]

The problem might be due to the fact though we always call malloc through TR::PersistentAllocator::allocate, that memory never gets freed via free; TR:PersistentAllocator::deallocate keeps that memory lying around in internal free lists.

[janvrany]

I just realized the same as @mgaudet while trying to reduce test memory footprint in context of #6905.
With path below I managed to get significantly lower memory consumption when running jit tests
(from ~1GB RAM down to ~350MB RAM).

diff --git a/compiler/ilgen/OMRTypeDictionary.cpp b/compiler/ilgen/OMRTypeDictionary.cpp
index 3fae65fdd..214a42668 100644
--- a/compiler/ilgen/OMRTypeDictionary.cpp
+++ b/compiler/ilgen/OMRTypeDictionary.cpp
@@ -463,36 +463,36 @@ OMR::TypeDictionary::TypeDictionary() :
    TR::DataTypes Vector128Double = OMR::DataType::createVectorType(TR::Double, TR::VectorLength128);
 
    // primitive types
-   NoType       = _primitiveType[TR::NoType]                = new (PERSISTENT_NEW) OMR::PrimitiveType("NoType", TR::NoType);
-   Int8         = _primitiveType[TR::Int8]                  = new (PERSISTENT_NEW) OMR::PrimitiveType("Int8", TR::Int8);
-   Int16        = _primitiveType[TR::Int16]                 = new (PERSISTENT_NEW) OMR::PrimitiveType("Int16", TR::Int16);
-   Int32        = _primitiveType[TR::Int32]                 = new (PERSISTENT_NEW) OMR::PrimitiveType("Int32", TR::Int32);
-   Int64        = _primitiveType[TR::Int64]                 = new (PERSISTENT_NEW) OMR::PrimitiveType("Int64", TR::Int64);
-   Float        = _primitiveType[TR::Float]                 = new (PERSISTENT_NEW) OMR::PrimitiveType("Float", TR::Float);
-   Double       = _primitiveType[TR::Double]                = new (PERSISTENT_NEW) OMR::PrimitiveType("Double", TR::Double);
-   Address      = _primitiveType[TR::Address]               = new (PERSISTENT_NEW) OMR::PrimitiveType("Address", TR::Address);
-   VectorInt8   = _primitiveType[Vector128Int8]             = new (PERSISTENT_NEW) OMR::PrimitiveType("VectorInt8", Vector128Int8);
-   VectorInt16  = _primitiveType[Vector128Int16]            = new (PERSISTENT_NEW) OMR::PrimitiveType("VectorInt16", Vector128Int16);
-   VectorInt32  = _primitiveType[Vector128Int32]            = new (PERSISTENT_NEW) OMR::PrimitiveType("VectorInt32", Vector128Int32);
-   VectorInt64  = _primitiveType[Vector128Int64]            = new (PERSISTENT_NEW) OMR::PrimitiveType("VectorInt64", Vector128Int64);
-   VectorFloat  = _primitiveType[Vector128Float]            = new (PERSISTENT_NEW) OMR::PrimitiveType("VectorFloat", Vector128Float);
-   VectorDouble = _primitiveType[Vector128Double]           = new (PERSISTENT_NEW) OMR::PrimitiveType("VectorDouble", Vector128Double);
+   NoType       = _primitiveType[TR::NoType]                = new (trMemory()->heapMemoryRegion()) OMR::PrimitiveType("NoType", TR::NoType);
+   Int8         = _primitiveType[TR::Int8]                  = new (trMemory()->heapMemoryRegion()) OMR::PrimitiveType("Int8", TR::Int8);
+   Int16        = _primitiveType[TR::Int16]                 = new (trMemory()->heapMemoryRegion()) OMR::PrimitiveType("Int16", TR::Int16);
+   Int32        = _primitiveType[TR::Int32]                 = new (trMemory()->heapMemoryRegion()) OMR::PrimitiveType("Int32", TR::Int32);
+   Int64        = _primitiveType[TR::Int64]                 = new (trMemory()->heapMemoryRegion()) OMR::PrimitiveType("Int64", TR::Int64);
+   Float        = _primitiveType[TR::Float]                 = new (trMemory()->heapMemoryRegion()) OMR::PrimitiveType("Float", TR::Float);
+   Double       = _primitiveType[TR::Double]                = new (trMemory()->heapMemoryRegion()) OMR::PrimitiveType("Double", TR::Double);
+   Address      = _primitiveType[TR::Address]               = new (trMemory()->heapMemoryRegion()) OMR::PrimitiveType("Address", TR::Address);
+   VectorInt8   = _primitiveType[Vector128Int8]             = new (trMemory()->heapMemoryRegion()) OMR::PrimitiveType("VectorInt8", Vector128Int8);
+   VectorInt16  = _primitiveType[Vector128Int16]            = new (trMemory()->heapMemoryRegion()) OMR::PrimitiveType("VectorInt16", Vector128Int16);
+   VectorInt32  = _primitiveType[Vector128Int32]            = new (trMemory()->heapMemoryRegion()) OMR::PrimitiveType("VectorInt32", Vector128Int32);
+   VectorInt64  = _primitiveType[Vector128Int64]            = new (trMemory()->heapMemoryRegion()) OMR::PrimitiveType("VectorInt64", Vector128Int64);
+   VectorFloat  = _primitiveType[Vector128Float]            = new (trMemory()->heapMemoryRegion()) OMR::PrimitiveType("VectorFloat", Vector128Float);
+   VectorDouble = _primitiveType[Vector128Double]           = new (trMemory()->heapMemoryRegion()) OMR::PrimitiveType("VectorDouble", Vector128Double);
 
    // pointer to primitive types
-   pNoType       = _pointerToPrimitiveType[TR::NoType]       = new (PERSISTENT_NEW) OMR::PointerType(NoType);
-   pInt8         = _pointerToPrimitiveType[TR::Int8]         = new (PERSISTENT_NEW) OMR::PointerType(Int8);
-   pInt16        = _pointerToPrimitiveType[TR::Int16]        = new (PERSISTENT_NEW) OMR::PointerType(Int16);
-   pInt32        = _pointerToPrimitiveType[TR::Int32]        = new (PERSISTENT_NEW) OMR::PointerType(Int32);
-   pInt64        = _pointerToPrimitiveType[TR::Int64]        = new (PERSISTENT_NEW) OMR::PointerType(Int64);
-   pFloat        = _pointerToPrimitiveType[TR::Float]        = new (PERSISTENT_NEW) OMR::PointerType(Float);
-   pDouble       = _pointerToPrimitiveType[TR::Double]       = new (PERSISTENT_NEW) OMR::PointerType(Double);
-   pAddress      = _pointerToPrimitiveType[TR::Address]      = new (PERSISTENT_NEW) OMR::PointerType(Address);
-   pVectorInt8   = _pointerToPrimitiveType[Vector128Int8]   = new (PERSISTENT_NEW) OMR::PointerType(VectorInt8);
-   pVectorInt16  = _pointerToPrimitiveType[Vector128Int16]  = new (PERSISTENT_NEW) OMR::PointerType(VectorInt16);
-   pVectorInt32  = _pointerToPrimitiveType[Vector128Int32]  = new (PERSISTENT_NEW) OMR::PointerType(VectorInt32);
-   pVectorInt64  = _pointerToPrimitiveType[Vector128Int64]  = new (PERSISTENT_NEW) OMR::PointerType(VectorInt64);
-   pVectorFloat  = _pointerToPrimitiveType[Vector128Float]  = new (PERSISTENT_NEW) OMR::PointerType(VectorFloat);
-   pVectorDouble = _pointerToPrimitiveType[Vector128Double] = new (PERSISTENT_NEW) OMR::PointerType(VectorDouble);
+   pNoType       = _pointerToPrimitiveType[TR::NoType]       = new (trMemory()->heapMemoryRegion()) OMR::PointerType(NoType);
+   pInt8         = _pointerToPrimitiveType[TR::Int8]         = new (trMemory()->heapMemoryRegion()) OMR::PointerType(Int8);
+   pInt16        = _pointerToPrimitiveType[TR::Int16]        = new (trMemory()->heapMemoryRegion()) OMR::PointerType(Int16);
+   pInt32        = _pointerToPrimitiveType[TR::Int32]        = new (trMemory()->heapMemoryRegion()) OMR::PointerType(Int32);
+   pInt64        = _pointerToPrimitiveType[TR::Int64]        = new (trMemory()->heapMemoryRegion()) OMR::PointerType(Int64);
+   pFloat        = _pointerToPrimitiveType[TR::Float]        = new (trMemory()->heapMemoryRegion()) OMR::PointerType(Float);
+   pDouble       = _pointerToPrimitiveType[TR::Double]       = new (trMemory()->heapMemoryRegion()) OMR::PointerType(Double);
+   pAddress      = _pointerToPrimitiveType[TR::Address]      = new (trMemory()->heapMemoryRegion()) OMR::PointerType(Address);
+   pVectorInt8   = _pointerToPrimitiveType[Vector128Int8]   = new (trMemory()->heapMemoryRegion()) OMR::PointerType(VectorInt8);
+   pVectorInt16  = _pointerToPrimitiveType[Vector128Int16]  = new (trMemory()->heapMemoryRegion()) OMR::PointerType(VectorInt16);
+   pVectorInt32  = _pointerToPrimitiveType[Vector128Int32]  = new (trMemory()->heapMemoryRegion()) OMR::PointerType(VectorInt32);
+   pVectorInt64  = _pointerToPrimitiveType[Vector128Int64]  = new (trMemory()->heapMemoryRegion()) OMR::PointerType(VectorInt64);
+   pVectorFloat  = _pointerToPrimitiveType[Vector128Float]  = new (trMemory()->heapMemoryRegion()) OMR::PointerType(VectorFloat);
+   pVectorDouble = _pointerToPrimitiveType[Vector128Double] = new (trMemory()->heapMemoryRegion()) OMR::PointerType(VectorDouble);
 
    if (TR::Compiler->target.is64Bit())
       {

[janvrany]

The above patch also helps a lot when running tests under QEMU - with patch above applied, (RISC-V) comptest needs with ~450MB max instead of exhausting 12GB (on my machine).
I also modified JitTest to avoid initializing / shutting down JIT for each test as this also leaks some memory
(and doing it every 10000 tests or so). But major savings come from TypeDictionary

@mstoodle: Obviously, I do not know what I'm doing so the change might be completely bogus but I think is well worth exploring. What's the lifecycle of TypeDictionary? Maybe tests should reuse it?