[Fleet] - Support dynamic templates for "date" types and array of objects containing same type of attributes

[ShourieG]

Describe the feature: Support dynamic templates for date types and array of objects containing same type of attributes.

Describe a specific use case for the feature: In many integrations we need to dynamically map groups of date types and object arrays that have the same type of attribute value. Eg - Object where all attributes are strings/keywords or integers etc. Having this support will help us build more flexible integrations moving forward. According to the code here "date" types and object arrays are not supported.

cc: @chrisberkhout @kcreddy @narph

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[kpollich]

Hi @ShourieG thanks for opening this issue. I'm having a hard time understanding what the ask is here for Fleet. Could you provide a specific example of integration mappings before and after this proposed change?

Is this something we'd need to add support for in https://github.com/elastic/package-spec as well, or could this be handled entirely within the Fleet plugin in Kibana?

[ShourieG]

Hi @kpollich, so currently when we want to use dynamic templates in integration pipelines, it has certain limitations.

1. Dynamic templates are not supported for templates specifying a date type.

Example:

Object: 
  fieldA_time: 'val_a'
  fieldB_time: 'val_b'
  fieldC: 'val_c'
  fieldD: 'val_d'

Here, I want to dynamically template fieldA_time and fieldB_time while having explicit mappings for C and D. Both A and B are date types. The template would look like as follows:

name: some_object
type: group
fields:
 - name: '*_time'
   type: date
   format: epoch_second
 - name: fieldC
   type: keyword
 - name: fieldD
   type: integer

But this doesn't work and while running system tests with elastic-package we encountered the following error:

error: failed to setup system runner: can't install the package: there was an apply error: installation failed: can't install the package: could not zip-install package; API status code = 400; response body = {"statusCode":400,"error":"Bad Request","message":"No dynamic mapping generated for field ocsf.actor.user.ldap_person.*_time of type date"}

On further investigation of the code block here, we saw that date types are not handled.

2. In a similar fashion, array of objects are not supported. Now I understand the main reason for this as objects could have mixed types within them and dynamically templating them would become tricky and expensive, but for use cases, where we know all the attributes of the object we want to dynamically template and we know that they have a singular type, then we should be allowed to do so as it gives a lot of flexibility for building integrations.

Example:

Suppose I have the following object:

  {
    "group": [
      {
        "desc": "Example group description",
        "name": "ExampleGroupName",
        "privileges": "Example privileges",
        "type": "Example type",
        "uid": "ExampleUID"
      },
       {
        "desc": "Example 2",
        "name": "Example2Name",
        "privileges": "Example2 privileges",
        "type": "Example2 type",
        "uid": "Example2UID"
      }
    ]
  }

And I want to map as follows:

 - name: group.*
   type: object
   object_type: keyword
   object_type_mapping_type: "*"

Here we know that group contains keyword attributes only even though it's an array of objects. So the ability to have dynamic template definitions for such cases would solve a lot of complexity for building simpler integration pipelines.

[kpollich]

Thanks for the concrete examples - this is very helpful! I've put this on our backlog for now and we'll work on prioritizing this.