pack: enhance memory allocation strategy for large JSON payloads

When converting msgpack buffers with large JSON text content, the buffering
strategy is not quite optimal since the outgoing buffer always set an
output buffer of size 1.5x, then if more space is needed the code reallocate
by adding 256 bytes more.

If the large payload needs more than 1.5x, the 256 bytes allocation generates
a slow path and may lead to hundreds of memory reallocations, adding latency
to the pipeline.

This patch improves the strategy by allocating an initial buffer size of 2x
and then if more space is needed, increase in rounds of 10% of the original
payload size.

Signed-off-by: Eduardo Silva <[email protected]>

src/flb_pack.c

@@ -690,12 +690,17 @@ flb_sds_t flb_msgpack_raw_to_json_sds(const void *in_buf, size_t in_size)
     int ret;
     size_t off = 0;
     size_t out_size;
+    size_t realloc_size;
+
     msgpack_unpacked result;
     msgpack_object *root;
     flb_sds_t out_buf;
     flb_sds_t tmp_buf;
 
-    out_size = in_size * 3 / 2;
+    /* buffer size strategy */
+    out_size = in_size * 2;
+    realloc_size = in_size * 0.10;
+
     out_buf = flb_sds_create_size(out_size);
     if (!out_buf) {
         flb_errno();
@@ -714,10 +719,10 @@ flb_sds_t flb_msgpack_raw_to_json_sds(const void *in_buf, size_t in_size)
     while (1) {
         ret = flb_msgpack_to_json(out_buf, out_size, root);
         if (ret <= 0) {
-            tmp_buf = flb_sds_increase(out_buf, 256);
+            tmp_buf = flb_sds_increase(out_buf, realloc_size);
             if (tmp_buf) {
                 out_buf = tmp_buf;
-                out_size += 256;
+                out_size += realloc_size;
             }
             else {
                 flb_errno();