Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segfault in cb_modifier_filter #4013

Closed
andreas-schroeder opened this issue Aug 27, 2021 · 6 comments
Closed

Segfault in cb_modifier_filter #4013

andreas-schroeder opened this issue Aug 27, 2021 · 6 comments

Comments

@andreas-schroeder
Copy link

Bug Report

Fluentbit segfaults on some JSON inputs. I'm opening this bug report as followup to this issue: aws/aws-for-fluent-bit#224

To Reproduce

I'm not entirely sure about what is causing the problem, and I don't have a local setup available to try to reproduce it. However, the segfaults stopped once we changed the field names in the logs from a dotted notation to proper nested JSON objects. I.e., we had log messages including (among other fields) fields with names of the form

{ "error.name1.errors":"error","error.name1.value": "value",
  "error.name2.errors":"error","error.name2.value": "value",
  ...
}

with up to around 10 names.

Expected behavior
Not crashing ;)

Your Environment

  • Version used: 1.8.3 (aws-for-fluentbit distro 2.19.0)
  • Configuration:
[SERVICE]
    Parsers_File /fluent-bit-conf/parser.conf
    Streams_File /fluent-bit-conf/stream-processing.conf

[FILTER]
    Name          parser
    Match         *
    Key_Name      log
    Parser        json
    Reserve_Data  True

[OUTPUT]
    Name              cloudwatch
    Match             logs.default
    region            eu-central-1
    log_group_name    <default-log-group>
    log_stream_name   <default-log-stream>

[OUTPUT]
    Name              cloudwatch
    Match             logs.security
    region            eu-central-1
    log_group_name    <security-log-group>
    log_stream_name   <security-log-stream>

[OUTPUT]
    Name              cloudwatch
    Match             logs.trace
    region            eu-central-1
    log_group_name    <trace-log-group>
    log_stream_name   <trace-log-stream>

Parsers File

[PARSER]
    Name   json
    Format json
Streams File

[STREAM_TASK]
    Name   default_logs
    Exec   CREATE STREAM default WITH (tag='logs.default') AS SELECT * from TAG:'app*' WHERE trace != true;

[STREAM_TASK]
    Name   security_logs
    Exec   CREATE STREAM security WITH (tag='logs.security') AS SELECT * from TAG:'app*' WHERE security = true;

[STREAM_TASK]
    Name   trace_logs
    Exec   CREATE STREAM trace WITH (tag='logs.trace') AS SELECT * from TAG:'app*' WHERE trace = true;
  • Environment name and version: ECS Fargate 1.4.0
  • Filters and plugins: see config above (?)

Additional context

This setup works fine for three workloads, and crashes for a fourth workload when enabling trace logs (i.e. logs that are larger in size, and have a richer structure). The logs are all generated with logback logstash encoder, so I think it's valid to assume they are proper JSON.

I ran fluentbit with valgrind, and this is the stacktrace/issues it found.

------------------------------------------------------------------------------------------------------
|   timestamp   |                                      message                                       |
|---------------|------------------------------------------------------------------------------------|
| 1629383594445 | ==1==                                                                              |
| 1629383594445 | ==1== Invalid read of size 8                                                       |
| 1629383594445 | ==1==    at 0x594191: cb_modifier_filter (filter_modifier.c:318)                   |
| 1629383594445 | ==1==    by 0x4CE497: flb_filter_do (flb_filter.c:118)                             |
| 1629383594445 | ==1==    by 0x4FB4A1: flb_input_chunk_append_raw (flb_input_chunk.c:911)           |
| 1629383594445 | ==1==    by 0x5288D0: cb_chunks_append (sp.c:75)                                   |
| 1629383594445 | ==1==    by 0x4CDF58: flb_input_collector_fd (flb_input.c:1005)                    |
| 1629383594445 | ==1==    by 0x4DE5BF: flb_engine_handle_event (flb_engine.c:363)                   |
| 1629383594445 | ==1==    by 0x4DE5BF: flb_engine_start (flb_engine.c:627)                          |
| 1629383594445 | ==1==    by 0x4C3B5A: flb_lib_worker (flb_lib.c:493)                               |
| 1629383594446 | ==1==    by 0x4E3F40A: start_thread (in /usr/lib64/libpthread-2.26.so)             |
| 1629383594446 | ==1==    by 0x68660BE: clone (in /usr/lib64/libc-2.26.so)                          |
| 1629383594446 | ==1==  Address 0x2208d258 is 1,528 bytes inside a block of size 8,192 free'd       |
| 1629383594446 | ==1==    at 0x4C2CD28: free (vg_replace_malloc.c:530)                              |
| 1629383594446 | ==1==    by 0x4F4213: flb_free (flb_mem.h:122)                                     |
| 1629383594446 | ==1==    by 0x4F4B4B: collect_outputs (flb_metrics_exporter.c:143)                 |
| 1629383594446 | ==1==    by 0x4F4C20: collect_metrics (flb_metrics_exporter.c:167)                 |
| 1629383594446 | ==1==    by 0x4F4D81: flb_me_fd_event (flb_metrics_exporter.c:220)                 |
| 1629383594446 | ==1==    by 0x4DE5E5: flb_engine_handle_event (flb_engine.c:370)                   |
| 1629383594446 | ==1==    by 0x4DE5E5: flb_engine_start (flb_engine.c:627)                          |
| 1629383594446 | ==1==    by 0x4C3B5A: flb_lib_worker (flb_lib.c:493)                               |
| 1629383594446 | ==1==    by 0x4E3F40A: start_thread (in /usr/lib64/libpthread-2.26.so)             |
| 1629383594446 | ==1==    by 0x68660BE: clone (in /usr/lib64/libc-2.26.so)                          |
| 1629383594446 | ==1==  Block was alloc'd at                                                        |
| 1629383594446 | ==1==    at 0x4C2BABC: malloc (vg_replace_malloc.c:298)                            |
| 1629383594446 | ==1==    by 0x4C2DC6F: realloc (vg_replace_malloc.c:785)                           |
| 1629383594446 | ==1==    by 0x4F3AA1: msgpack_sbuffer_write (sbuffer.h:81)                         |
| 1629383594446 | ==1==    by 0x4F37B1: msgpack_pack_map (pack_template.h:745)                       |
| 1629383594446 | ==1==    by 0x4F407B: flb_metrics_dump_values (flb_metrics.c:219)                  |
| 1629383594446 | ==1==    by 0x4F4ADB: collect_outputs (flb_metrics_exporter.c:139)                 |
| 1629383594446 | ==1==    by 0x4F4C20: collect_metrics (flb_metrics_exporter.c:167)                 |
| 1629383594446 | ==1==    by 0x4F4D81: flb_me_fd_event (flb_metrics_exporter.c:220)                 |
| 1629383594446 | ==1==    by 0x4DE5E5: flb_engine_handle_event (flb_engine.c:370)                   |
| 1629383594446 | ==1==    by 0x4DE5E5: flb_engine_start (flb_engine.c:627)                          |
| 1629383594446 | ==1==    by 0x4C3B5A: flb_lib_worker (flb_lib.c:493)                               |
| 1629383594446 | ==1==    by 0x4E3F40A: start_thread (in /usr/lib64/libpthread-2.26.so)             |
| 1629383594446 | ==1==    by 0x68660BE: clone (in /usr/lib64/libc-2.26.so)                          |
| 1629383594446 | ==1==                                                                              |
| 1629383594447 | ==1== Invalid read of size 8                                                       |
| 1629383594447 | ==1==    at 0x5941C7: cb_modifier_filter (filter_modifier.c:319)                   |
| 1629383594447 | ==1==    by 0x4CE497: flb_filter_do (flb_filter.c:118)                             |
| 1629383594447 | ==1==    by 0x4FB4A1: flb_input_chunk_append_raw (flb_input_chunk.c:911)           |
| 1629383594447 | ==1==    by 0x5288D0: cb_chunks_append (sp.c:75)                                   |
| 1629383594447 | ==1==    by 0x4CDF58: flb_input_collector_fd (flb_input.c:1005)                    |
| 1629383594447 | ==1==    by 0x4DE5BF: flb_engine_handle_event (flb_engine.c:363)                   |
| 1629383594447 | ==1==    by 0x4DE5BF: flb_engine_start (flb_engine.c:627)                          |
| 1629383594447 | ==1==    by 0x4C3B5A: flb_lib_worker (flb_lib.c:493)                               |
| 1629383594447 | ==1==    by 0x4E3F40A: start_thread (in /usr/lib64/libpthread-2.26.so)             |
| 1629383594447 | ==1==    by 0x68660BE: clone (in /usr/lib64/libc-2.26.so)                          |
| 1629383594447 | ==1==  Address 0x2208d280 is 1,568 bytes inside a block of size 8,192 free'd       |
| 1629383594447 | ==1==    at 0x4C2CD28: free (vg_replace_malloc.c:530)                              |
| 1629383594447 | ==1==    by 0x4F4213: flb_free (flb_mem.h:122)                                     |
| 1629383594447 | ==1==    by 0x4F4B4B: collect_outputs (flb_metrics_exporter.c:143)                 |
| 1629383594447 | ==1==    by 0x4F4C20: collect_metrics (flb_metrics_exporter.c:167)                 |
| 1629383594447 | ==1==    by 0x4F4D81: flb_me_fd_event (flb_metrics_exporter.c:220)                 |
| 1629383594447 | ==1==    by 0x4DE5E5: flb_engine_handle_event (flb_engine.c:370)                   |
| 1629383594447 | ==1==    by 0x4DE5E5: flb_engine_start (flb_engine.c:627)                          |
| 1629383594447 | ==1==    by 0x4C3B5A: flb_lib_worker (flb_lib.c:493)                               |
| 1629383594447 | ==1==    by 0x4E3F40A: start_thread (in /usr/lib64/libpthread-2.26.so)             |
| 1629383594447 | ==1==    by 0x68660BE: clone (in /usr/lib64/libc-2.26.so)                          |
| 1629383594447 | ==1==  Block was alloc'd at                                                        |
| 1629383594447 | ==1==    at 0x4C2BABC: malloc (vg_replace_malloc.c:298)                            |
| 1629383594447 | ==1==    by 0x4C2DC6F: realloc (vg_replace_malloc.c:785)                           |
| 1629383594447 | ==1==    by 0x4F3AA1: msgpack_sbuffer_write (sbuffer.h:81)                         |
| 1629383594447 | ==1==    by 0x4F37B1: msgpack_pack_map (pack_template.h:745)                       |
| 1629383594447 | ==1==    by 0x4F407B: flb_metrics_dump_values (flb_metrics.c:219)                  |
| 1629383594448 | ==1==    by 0x4F4ADB: collect_outputs (flb_metrics_exporter.c:139)                 |
| 1629383594448 | ==1==    by 0x4F4C20: collect_metrics (flb_metrics_exporter.c:167)                 |
| 1629383594448 | ==1==    by 0x4F4D81: flb_me_fd_event (flb_metrics_exporter.c:220)                 |
| 1629383594448 | ==1==    by 0x4DE5E5: flb_engine_handle_event (flb_engine.c:370)                   |
| 1629383594448 | ==1==    by 0x4DE5E5: flb_engine_start (flb_engine.c:627)                          |
| 1629383594448 | ==1==    by 0x4C3B5A: flb_lib_worker (flb_lib.c:493)                               |
| 1629383594448 | ==1==    by 0x4E3F40A: start_thread (in /usr/lib64/libpthread-2.26.so)             |
| 1629383594448 | ==1==    by 0x68660BE: clone (in /usr/lib64/libc-2.26.so)                          |
| 1629383594448 | ==1==                                                                              |
| 1629383594448 | ==1== Invalid read of size 8                                                       |
| 1629383594448 | ==1==    at 0x5941CA: cb_modifier_filter (filter_modifier.c:319)                   |
| 1629383594448 | ==1==    by 0x4CE497: flb_filter_do (flb_filter.c:118)                             |
| 1629383594448 | ==1==    by 0x4FB4A1: flb_input_chunk_append_raw (flb_input_chunk.c:911)           |
| 1629383594448 | ==1==    by 0x5288D0: cb_chunks_append (sp.c:75)                                   |
| 1629383594448 | ==1==    by 0x4CDF58: flb_input_collector_fd (flb_input.c:1005)                    |
| 1629383594448 | ==1==    by 0x4DE5BF: flb_engine_handle_event (flb_engine.c:363)                   |
| 1629383594448 | ==1==    by 0x4DE5BF: flb_engine_start (flb_engine.c:627)                          |
| 1629383594448 | ==1==    by 0x4C3B5A: flb_lib_worker (flb_lib.c:493)                               |
| 1629383594448 | ==1==    by 0x4E3F40A: start_thread (in /usr/lib64/libpthread-2.26.so)             |
| 1629383594448 | ==1==    by 0x68660BE: clone (in /usr/lib64/libc-2.26.so)                          |
| 1629383594448 | ==1==  Address 0x2208d278 is 1,560 bytes inside a block of size 8,192 free'd       |
| 1629383594448 | ==1==    at 0x4C2CD28: free (vg_replace_malloc.c:530)                              |
| 1629383594448 | ==1==    by 0x4F4213: flb_free (flb_mem.h:122)                                     |
| 1629383594448 | ==1==    by 0x4F4B4B: collect_outputs (flb_metrics_exporter.c:143)                 |
| 1629383594448 | ==1==    by 0x4F4C20: collect_metrics (flb_metrics_exporter.c:167)                 |
| 1629383594448 | ==1==    by 0x4F4D81: flb_me_fd_event (flb_metrics_exporter.c:220)                 |
| 1629383594449 | ==1==    by 0x4DE5E5: flb_engine_handle_event (flb_engine.c:370)                   |
| 1629383594449 | ==1==    by 0x4DE5E5: flb_engine_start (flb_engine.c:627)                          |
| 1629383594449 | ==1==    by 0x4C3B5A: flb_lib_worker (flb_lib.c:493)                               |
| 1629383594450 | ==1==    by 0x4E3F40A: start_thread (in /usr/lib64/libpthread-2.26.so)             |
| 1629383594450 | ==1==    by 0x68660BE: clone (in /usr/lib64/libc-2.26.so)                          |
| 1629383594450 | ==1==  Block was alloc'd at                                                        |
| 1629383594450 | ==1==    at 0x4C2BABC: malloc (vg_replace_malloc.c:298)                            |
| 1629383594450 | ==1==    by 0x4C2DC6F: realloc (vg_replace_malloc.c:785)                           |
| 1629383594450 | ==1==    by 0x4F3AA1: msgpack_sbuffer_write (sbuffer.h:81)                         |
| 1629383594450 | ==1==    by 0x4F37B1: msgpack_pack_map (pack_template.h:745)                       |
| 1629383594450 | ==1==    by 0x4F407B: flb_metrics_dump_values (flb_metrics.c:219)                  |
| 1629383594451 | ==1==    by 0x4F4ADB: collect_outputs (flb_metrics_exporter.c:139)                 |
| 1629383594451 | ==1==    by 0x4F4C20: collect_metrics (flb_metrics_exporter.c:167)                 |
| 1629383594451 | ==1==    by 0x4F4D81: flb_me_fd_event (flb_metrics_exporter.c:220)                 |
| 1629383594451 | ==1==    by 0x4DE5E5: flb_engine_handle_event (flb_engine.c:370)                   |
| 1629383594451 | ==1==    by 0x4DE5E5: flb_engine_start (flb_engine.c:627)                          |
| 1629383594451 | ==1==    by 0x4C3B5A: flb_lib_worker (flb_lib.c:493)                               |
| 1629383594451 | ==1==    by 0x4E3F40A: start_thread (in /usr/lib64/libpthread-2.26.so)             |
| 1629383594451 | ==1==    by 0x68660BE: clone (in /usr/lib64/libc-2.26.so)                          |
| 1629383594451 | ==1==                                                                              |
| 1629383594451 | ==1== Invalid read of size 8                                                       |
| 1629383594451 | ==1==    at 0x5941CD: cb_modifier_filter (filter_modifier.c:319)                   |
| 1629383594451 | ==1==    by 0x4CE497: flb_filter_do (flb_filter.c:118)                             |
| 1629383594451 | ==1==    by 0x4FB4A1: flb_input_chunk_append_raw (flb_input_chunk.c:911)           |
| 1629383594451 | ==1==    by 0x5288D0: cb_chunks_append (sp.c:75)                                   |
| 1629383594451 | ==1==    by 0x4CDF58: flb_input_collector_fd (flb_input.c:1005)                    |
| 1629383594451 | ==1==    by 0x4DE5BF: flb_engine_handle_event (flb_engine.c:363)                   |
| 1629383594451 | ==1==    by 0x4DE5BF: flb_engine_start (flb_engine.c:627)                          |
| 1629383594451 | ==1==    by 0x4C3B5A: flb_lib_worker (flb_lib.c:493)                               |
| 1629383594451 | ==1==    by 0x4E3F40A: start_thread (in /usr/lib64/libpthread-2.26.so)             |
| 1629383594451 | ==1==    by 0x68660BE: clone (in /usr/lib64/libc-2.26.so)                          |
| 1629383594451 | ==1==  Address 0x2208d270 is 1,552 bytes inside a block of size 8,192 free'd       |
| 1629383594451 | ==1==    at 0x4C2CD28: free (vg_replace_malloc.c:530)                              |
| 1629383594451 | ==1==    by 0x4F4213: flb_free (flb_mem.h:122)                                     |
| 1629383594452 | ==1==    by 0x4F4B4B: collect_outputs (flb_metrics_exporter.c:143)                 |
| 1629383594452 | ==1==    by 0x4F4C20: collect_metrics (flb_metrics_exporter.c:167)                 |
| 1629383594452 | ==1==    by 0x4F4D81: flb_me_fd_event (flb_metrics_exporter.c:220)                 |
| 1629383594452 | ==1==    by 0x4DE5E5: flb_engine_handle_event (flb_engine.c:370)                   |
| 1629383594452 | ==1==    by 0x4DE5E5: flb_engine_start (flb_engine.c:627)                          |
| 1629383594452 | ==1==    by 0x4C3B5A: flb_lib_worker (flb_lib.c:493)                               |
| 1629383594452 | ==1==    by 0x4E3F40A: start_thread (in /usr/lib64/libpthread-2.26.so)             |
| 1629383594452 | ==1==    by 0x68660BE: clone (in /usr/lib64/libc-2.26.so)                          |
| 1629383594452 | ==1==  Block was alloc'd at                                                        |
| 1629383594452 | ==1==    at 0x4C2BABC: malloc (vg_replace_malloc.c:298)                            |
| 1629383594452 | ==1==    by 0x4C2DC6F: realloc (vg_replace_malloc.c:785)                           |
| 1629383594452 | ==1==    by 0x4F3AA1: msgpack_sbuffer_write (sbuffer.h:81)                         |
| 1629383594452 | ==1==    by 0x4F37B1: msgpack_pack_map (pack_template.h:745)                       |
| 1629383594452 | ==1==    by 0x4F407B: flb_metrics_dump_values (flb_metrics.c:219)                  |
| 1629383594452 | ==1==    by 0x4F4ADB: collect_outputs (flb_metrics_exporter.c:139)                 |
| 1629383594452 | ==1==    by 0x4F4C20: collect_metrics (flb_metrics_exporter.c:167)                 |
| 1629383594452 | ==1==    by 0x4F4D81: flb_me_fd_event (flb_metrics_exporter.c:220)                 |
| 1629383594452 | ==1==    by 0x4DE5E5: flb_engine_handle_event (flb_engine.c:370)                   |
| 1629383594452 | ==1==    by 0x4DE5E5: flb_engine_start (flb_engine.c:627)                          |
| 1629383594452 | ==1==    by 0x4C3B5A: flb_lib_worker (flb_lib.c:493)                               |
| 1629383594452 | ==1==    by 0x4E3F40A: start_thread (in /usr/lib64/libpthread-2.26.so)             |
| 1629383594452 | ==1==    by 0x68660BE: clone (in /usr/lib64/libc-2.26.so)                          |
| 1629383594452 | ==1==                                                                              |
| 1629383594453 | ==1== Invalid read of size 8                                                       |
| 1629383594456 | ==1==    at 0x960755: msgpack_pack_object (objectc.c:113)                          |
| 1629383594456 | ==1==    by 0x5941D7: cb_modifier_filter (filter_modifier.c:319)                   |
| 1629383594456 | ==1==    by 0x4CE497: flb_filter_do (flb_filter.c:118)                             |
| 1629383594456 | ==1==    by 0x4FB4A1: flb_input_chunk_append_raw (flb_input_chunk.c:911)           |
| 1629383594456 | ==1==    by 0x5288D0: cb_chunks_append (sp.c:75)                                   |
| 1629383594456 | ==1==    by 0x4CDF58: flb_input_collector_fd (flb_input.c:1005)                    |
| 1629383594456 | ==1==    by 0x4DE5BF: flb_engine_handle_event (flb_engine.c:363)                   |
| 1629383594456 | ==1==    by 0x4DE5BF: flb_engine_start (flb_engine.c:627)                          |
| 1629383594456 | ==1==    by 0x4C3B5A: flb_lib_worker (flb_lib.c:493)                               |
| 1629383594456 | ==1==    by 0x4E3F40A: start_thread (in /usr/lib64/libpthread-2.26.so)             |
| 1629383594456 | ==1==    by 0x68660BE: clone (in /usr/lib64/libc-2.26.so)                          |
| 1629383594456 | ==1==  Address 0x15 is not stack'd, malloc'd or (recently) free'd                  |
| 1629383594456 | ==1==                                                                              |
| 1629383594462 | [2021/08/19 14:33:14] [engine] caught signal (SIGSEGV)                             |
| 1629383594494 | ==1==                                                                              |
| 1629383594494 | ==1== Process terminating with default action of signal 6 (SIGABRT): dumping core  |
| 1629383594494 | ==1==    at 0x67ACC20: raise (in /usr/lib64/libc-2.26.so)                          |
| 1629383603398 | ==1==                                                                              |
| 1629383603398 | ==1== HEAP SUMMARY:                                                                |
| 1629383603398 | ==1==     in use at exit: 1,798,158 bytes in 532 blocks                            |
| 1629383603398 | ==1==   total heap usage: 54,166 allocs, 53,634 frees, 205,565,269 bytes allocated |
| 1629383603398 | ==1==                                                                              |
| 1629383605029 | ==1== LEAK SUMMARY:                                                                |
| 1629383605029 | ==1==    definitely lost: 96 bytes in 3 blocks                                     |
| 1629383605029 | ==1==    indirectly lost: 157 bytes in 6 blocks                                    |
| 1629383605029 | ==1==      possibly lost: 1,413,399 bytes in 433 blocks                            |
| 1629383605029 | ==1==    still reachable: 384,506 bytes in 90 blocks                               |
| 1629383605029 | ==1==                       of which reachable via heuristic:                      |
| 1629383605029 | ==1==                         newarray           : 144 bytes in 3 blocks           |
| 1629383605029 | ==1==         suppressed: 0 bytes in 0 blocks                                      |
| 1629383605029 | ==1== Rerun with --leak-check=full to see details of leaked memory                 |
| 1629383605029 | ==1==                                                                              |
| 1629383605030 | ==1== For counts of detected and suppressed errors, rerun with: -v                 |
| 1629383605030 | ==1== Use --track-origins=yes to see where uninitialised values come from          |
| 1629383605030 | ==1== ERROR SUMMARY: 5467031 errors from 8174 contexts (suppressed: 0 from 0)      |
| 1629383605466 | valgrind: the 'impossible' happened:                                               |
| 1629383605466 |    main(): signal was supposed to be fatal                                         |
| 1629383605466 | host stacktrace:                                                                   |
| 1629383605466 | ==1==    at 0x58041058: ??? (in /usr/lib64/valgrind/memcheck-amd64-linux)          |
| 1629383605466 | sched status:                                                                      |
| 1629383605466 |   running_tid=21                                                                   |
| 1629383605466 | Note: see also the FAQ in the source distribution.                                 |
| 1629383605466 | It contains workarounds to several common problems.                                |
| 1629383605466 | In particular, if Valgrind aborted or crashed after                                |
| 1629383605466 | identifying problems in your program, there's a good chance                        |
| 1629383605466 | that fixing those problems will prevent Valgrind aborting or                       |
| 1629383605466 | crashing, especially if it happened in m_mallocfree.c.                             |
| 1629383605466 | If that doesn't help, please report this bug to: www.valgrind.org                  |
| 1629383605466 | In the bug report, send all the above text, the valgrind                           |
| 1629383605466 | version, and what OS and version you are using.  Thanks.                           |
------------------------------------------------------------------------------------------------------
@nokute78
Copy link
Collaborator

How many number of elements do you send ?
If the elements of map is greater than 128, filter_record_modifier will read invalid memory.

Note : Similar issue #3968. I sent a patch #3982 (is not merged yet)

@nokute78 nokute78 mentioned this issue Aug 28, 2021
Merged
3 tasks
@andreas-schroeder
Copy link
Author

This might indeed explain why I did hit this issue and why it disappeared. The log statement definitely surpassed the 128 top level fields on occasion. With the change they always stay below 128 fields.

Good to know that this limit currently exists, I have workloads that run at risk to hit that limit in the future.

@andreas-schroeder andreas-schroeder mentioned this issue Sep 2, 2021
Closed
@nokute78 nokute78 mentioned this issue Sep 3, 2021
Closed
@nokute78
Copy link
Collaborator

nokute78 commented Sep 10, 2021 •
edited
Loading

The patch #3982 is merged and it will be released as v1.8.7.
The limit of elements will be 65535.

@nokute78
Copy link
Collaborator

Can we close this issue ?
v1.8.7 already released.

@agup006
Copy link
Member

agup006 commented Oct 11, 2021

Closing @andreas-schroeder could you see if the new release helps?

@agup006 agup006 closed this as completed Oct 11, 2021
@andreas-schroeder
Copy link
Author

It probably does, but I took care for now that the workload respects the original field limit. Still good to know an upgrade option exists in case my workload runs into this issue again. Thanks for reaching out!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@agup006 @andreas-schroeder @nokute78