[RFC/PROPOSAL] Benchmarking with jemalloc

[dblock]

What/Why

What are you proposing?

In opensearch-project/OpenSearch#9496 we are discussing whether to package or not package jemalloc in the distribution. Either way we want to know the performance benefits of jemalloc with features such as k-nn that use native libraries. The ask is to add jemalloc supports + runs.

What users have asked for this feature?

See opensearch-project/OpenSearch#9496.

What problems are you trying to solve?

Reliably measure the performance improvements of jemalloc so that we can/cannot recommend it.

What is the developer experience going to be?

1. Documentation of how to enable jemalloc in opensearch-benchmark.
2. Daily runs with performance metrics.

[gkamat]

Use of jemalloc as the preferred allocator is not controlled by OSB, since it is simply a client. This is entirely determined on the OpenSearch side, and the mechanism is to set the LD_PRELOAD environment variable appropriately.

That being said, the capability can be added to the nightly performance runs to configure this in the cluster CDK. This task will be tracked separately.

[rishabh6788]

@dblock As long as we get a tar distribution created (-min or bundle) with jemalloc it should be straight forward to start nightly benchmark on them.

From the issue description it seems you want to enable it using OSB, can you please elaborate on this?

One quick way to test this out is to handle this in opensearch-cluster-cdk, add an option to use jemalloc and then handle installation (if not bundled with OpenSearch) and setting env variable.
Thoughts?