Using jemalloc instead of tcmalloc.

[alexey-milovidov]

In ClickHouse we do many medium-sized allocations of arrays (tens of kilobytes to few megabytes).
Not every allocator is good at this scenario. If the allocator fails to cache these allocations, performance will be terrible due to page faults.

We have used tcmalloc, but we had to tune it:

/** Without this option, tcmalloc returns memory to OS too frequently for medium-sized memory allocations
      *  (like IO buffers, column vectors, hash tables, etc.),
      *  that lead to page faults and significantly hurts performance.
      */
    MallocExtension::instance()->SetNumericProperty("tcmalloc.aggressive_memory_decommit", false);

But in some scenarios it still looks to be working suboptimal:

  13.24%  clickhouse                      [.] tcmalloc::CentralFreeList::RemoveRange
   6.67%  clickhouse                      [.] tcmalloc::ThreadCache::ReleaseToCentralCache
   5.20%  clickhouse                      [.] tcmalloc::CentralFreeList::InsertRange

When using jemalloc, there are no such issues even without any tuning.
Some queries will run faster up to 20%.
Memory usage (RSS) is about 10% lower and is more stable.

In general point of view, jemalloc seems to be better supported and better engineered than tcmalloc.