feat: support jemalloc

[empiredan]

It's well known that jemalloc is an excellent memory allocator, especially scalable for multi-threaded applications. It's adopted or supported by many famous projects, such as redis, clickhouse, rocksdb, etc.

By this PR, developer can choose to use jemalloc as the allocator of rDSN (default allocator/profiler is still tcmalloc/gperftools). Once chosen, rDSN can be compiled and linked with jemalloc dynamically or statically.

apache/incubator-pegasus#913

[levy5307]

Thanks for your pull request. And I think you can split it into serveral pull requests, because this patch is too big

[empiredan]

Thanks for you pull request. And I think you can split it into serveral pull requests, because this patch is too big

Ok, I'll first commit build-related code (e.g. cmake files and build scripts), and the remote_command tools for jemalloc will be provided later.

[Smityz]

It's a good idea to support jemalloc! Could you show the performance test between tcmalloc(default choice) and jemalloc?

[empiredan]

It's a good idea to support jemalloc! Could you show the performance test between tcmalloc(default choice) and jemalloc?

Sure, I'll give the performance comparison as we've showed at the meetup.

[empiredan]

Performance test between jemalloc and tcmalloc

Write and read performance test have been made between jemalloc and tcmalloc.

First, all the tests are based the requirements that all rocksdb's memory are managed by block cache, include both memtables and index & fitler blocks.

And block cache is 12GB, where at most 8GB will be used as the max size of all memtables totally. The necessary configurations are as below:

rocksdb_block_cache_capacity = 12884901888
rocksdb_enable_write_buffer_manager = true
rocksdb_total_size_across_write_buffer = 8589934592
rocksdb_index_type = two_level_index_search
rocksdb_partition_filters = true
rocksdb_cache_index_and_filter_blocks = true
rocksdb_pin_l0_filter_and_index_blocks_in_cache = true

1. Hardware configurations

cpu: AMD EPYC Processor 2900 MHz (4 cores)
memory: 32GB
network: 25000Mb/s
disk: HDD 400GB

2. Write test

There are 10 sort keys for each hash key, with each value 320B. A single client with 48 threads is used to put 80530636 records (about 29GB totally) into a table.

3. Read test

A single client with 96 threads is used to get 80530636 records (about 29GB totally) from a table that generated by the write test.

4. Scan test

A full scan over the whole table generated by the write test, all the 80530636 records.

5. jemalloc/tcmalloc configurations

5.1 jemalloc config 1

background_thread=false
dirty_decay_ms=10000
muzzy_decay_ms=0

5.2 jemalloc config 2

background_thread=true
dirty_decay_ms=10000
muzzy_decay_ms=0

5.3 tcmalloc config 1

tcmalloc_release_rate=10.0
mem_release_enabled=true
mem_release_check_interval_ms=10000
mem_release_max_reserved_mem_percentage=0

5.4 tcmalloc config 2

tcmalloc_release_rate=1.0
mem_release_enabled=false

6. Test results

6.1 QPS/Latency between jemalloc and tcmalloc

We can conclude from the table that jemalloc can increase QPS about 10% ~ 20% for write. Write qps trend is described from Prometheus as below (from left to right: jemalloc config 1 and 2, tcmalloc config 1 and 2, ignoring the second narrow bar graph):

6.2 Memory usage between jemalloc and tcmalloc

First load data by the write test, and then run the read test over the generated data. The physical memory usage trend is described from Prometheus as below (from left to right: jemalloc config 1 and 2, tcmalloc config 1 and 2):

First run scan test over the data that are generated by the write test, and then run another identical write test into another table.

We can conclude that jemalloc config 2 and tcmalloc config 1 have made the better restriction on the memory, and tcmalloc config 2 has made the loosest restriction on the memory as expected.

[Smityz]

Good job! Thank you empiredan!

[levy5307]

Thanks for your pull request again! Looking forward to your next work