[Feature Request] Dynamic Auto-Map

[adplotzk]

  {
      "src_key_0.1": "val_0.1",
      "src_key_0.2": "val_0.2",
      "src_key_0.3": "val_0.3",
      "src_key_0.4": {
	      "src_key_0.5": {
		      "src_key_0.6": "val_0.4"
	      }
      },
      "some_source": [
	      {
		      "nested_array": [
			      {
				      "src_key_1": "val_1",
				      "src_key_2": "val_2",
				      "src_key_3": "val_3"
			      },
			      {
				      "src_key_1": "val_5",
				      "src_key_2": "val_6",
				      "src_key_3": "val_7"
			      }
		      ]
	      }
      ]
  }

#TO

  {
      "somekey": [
			[
				"src_key_0.1",
				""val_0.1"
			],
			[
				"src_key_0.2",
				""val_0.2"
			],
			[
				"src_key_0.3",
				""val_0.3"
			],
			[
				"src_key_0.4.src_key0.5.src_key0.6",
				"val_0.4"
			],
			[
				"some_source[].nested_array[].src_key_1",
				"val_3, val5"
			],
			[
				"some_source[].nested_array[].src_key_2",
				"val_3, val_6"
			],
			[
				"some_source[].nested_array[].src_key_3",
				"val_3, val_7"
			],
		]
  }

#MAPPING

  - set_default_map_to: "somekey"

[oeyh]

This looks to me is to collapse/flatten nested json objects and use dot (or some other char) as delimiter in keys.

If the source keys are known, this can be done with existing copy_values processor and with each from_key and to_key specified:

  copy_values:
    entries:
      - from_key: "src_key_0.1"
        to_key: "somekey/src_key_0.1"
      - from_key: "src_key_0.4/src_key_0.5/src_key_0.6"
        to_key: "somekey/src_key_0.4.src_key_0.5.src_key_0.6"
      ...

but it can be a tedious process to set this up if there are many fields in the json.

To flatten all fields in json and put under a destination key, we can have a new processor flatten_json with config like this:

flatten_json:
  source: "source-key"
  target: "target-key"
  delimiter: "."

source: flatten all fields under source; defaults to root if not specified
target: put flattened fields under target; defaults to root if not specified
delimiter: the delimiter to use in keys for flattened fields

[oeyh]

Discussed offline. The description has been updated. The transformation involves these steps:

• Flatten the json object. We will end up with bunch of fields on the same level with keys like "key1.key2.key3" or "key4[].key5" (when there are arrays). If multiple values corresponds to the same key, put them in an array
• In the flattened json, if the value is an array, convert it to a string, eg ["value1", "value2"] --> "value1, value2"
• Convert the flattened fields (key-value pairs) to an array, eg {"key1": "value1", "key2": "value2"} --> [["key1", "value1"], ["key2", "value2"]]. This can be done with an additional boolean option called convert_object_to_list to map_to_list processor. If that’s set to true, the processor further transforms objects to a list.
  The configuration would be:

  processor:
    - map_to_list:
        source: "my-map"
        target: "my-list"
        convert_object_to_list: true

[oeyh]

Resolved via #4033, #4075 and #4128