config: yaml: add support for cfl_arrays when a property is declared as a list.

[pwhelan]

Summary

When a property is declared as a list with variant values it will be parsed using cfl_array with variants inside.

An example is this:

pipeline:
  inputs:
    - name: test
      hosts:
        - foo
        - bar

This should also work with arrays that have key/vales, ie:

pipeline:
  inputs:
    - name: test
      hosts:
        - ip: 127.0.0.1
          port: 4444

This also works with processors and will be key for an upcoming feature for the content_modifier processor:

pipeline:
  inputs:
    - name: dummy
      dummy: '{"message": "foobar", "foobar": "barfoo"}'
      processors:
        logs:
          - name: content_modifier
            actions:
              - action: insert
                key: foo
                value: bar
              - action: delete
                key: foo
              - action: delete
                key: foobar
              - action: insert
                key: bar
                value: foo

This pull request does not attempt to actually allow passing these properties to a plugin. Changes to the config and config_map layer are required for these values to actually be passed to a plugin instance.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• [N/A] Example configuration file for the change
• [N/A] Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

Documentation

• [N/A] Documentation required for this feature

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[pwhelan]

Here is a valgrind log:

2024/04/01 18:14:14] [ info] [fluent bit] version=3.0.1, commit=cf3e53ce03, pid=275574
[2024/04/01 18:14:14] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/04/01 18:14:14] [ info] [output:stdout:stdout.0] worker #0 started
[2024/04/01 18:14:14] [ info] [cmetrics] version=0.7.1
[2024/04/01 18:14:14] [ info] [ctraces ] version=0.4.0
[2024/04/01 18:14:14] [ info] [input:dummy:dummy.0] initializing
[2024/04/01 18:14:14] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2024/04/01 18:14:14] [ info] [sp] stream processor started
[0] dummy.0: [[1712006055.007001399, {}], {"message"=>"foobar", "foobar"=>"barfoo"}]
[0] dummy.0: [[1712006056.003066368, {}], {"message"=>"foobar", "foobar"=>"barfoo"}]
^C[2024/04/01 18:14:17] [engine] caught signal (SIGINT)
[2024/04/01 18:14:17] [ warn] [engine] service will shutdown in max 5 seconds
[0] dummy.0: [[1712006056.991977440, {}], {"message"=>"foobar", "foobar"=>"barfoo"}]
[2024/04/01 18:14:17] [ info] [input] pausing dummy.0
[2024/04/01 18:14:17] [ info] [engine] service has stopped (0 pending tasks)
[2024/04/01 18:14:17] [ info] [input] pausing dummy.0
[2024/04/01 18:14:17] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2024/04/01 18:14:17] [ info] [output:stdout:stdout.0] thread worker #0 stopped
==275574==
==275574== HEAP SUMMARY:
==275574==     in use at exit: 0 bytes in 0 blocks
==275574==   total heap usage: 1,992 allocs, 1,992 frees, 1,450,877 bytes allocated
==275574==
==275574== All heap blocks were freed -- no leaks are possible
==275574==
==275574== For lists of detected and suppressed errors, rerun with: -s
==275574== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

[cosmo0920]

Basically, it looks good to me.
Is there any chance to add an internal unit test case for this PR?
Or, this will be covered by another derived works?

I'm also interested in defining to the multiple server destinations on out_forward plugin like Fluentd does.

[pwhelan]

Is there any chance to add an internal unit test case for this PR?
Or, this will be covered by another derived works?

I will be adding tests as soon as the work for config and config_format is done which will allow me to parse yaml files with the new syntax.

I could also push those changes to this branch if it is better to try and get the entire feature merged at once but I felt the config_format changes will be disruptive enough to warrant their own PR.

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[pwhelan]

Superseded elsewhere.