Support validation ECS style dataset format

[jonathan-buttner]

Once the dataset fields are included in ecs the yaml file will look like this:

---
- name: dataset
  title: dataset
  group: 2
  short: Fields describing the new indexing strategy
  description: >
    Fields describing the new indexing strategy <type>-<name>-<namespace>
  type: group
  fields:
    - name: type
      level: custom
      type: constant_keyword
      description: >
        Dataset type.

    - name: name
      level: custom
      type: constant_keyword
      description: >
        Dataset name.

    - name: namespace
      level: custom
      type: constant_keyword
      description: >
        Dataset namespace.

This results in the following fields.yml:

- name: dataset
  title: dataset
  group: 2
  description: Fields describing the new indexing strategy <type>-<name>-<namespace>
  type: group
  fields:
  - name: name
    level: custom
    type: constant_keyword
    description: Dataset name.
    default_field: false
  - name: namespace
    level: custom
    type: constant_keyword
    description: Dataset namespace.
    default_field: false
  - name: type
    level: custom
    type: constant_keyword
    description: Dataset type.
    default_field: false
- name: destination
  title: Destination
  group: 2
  description: 'Destination fields describe details about the destination of a packet/event.

    Destination fields are usually populated in conjunction with source fields.'
  type: group
  fields:
  - name: address
<more fields>

This will fail validation because we're searching for dataset.*.

This PR adds support for splitting the dataset.* string and looking inside the map. It finds name: dataset first and then pulls out the array fields and looks for the last part of the string (e.g. namespace or type).

This implementation is suupeeeer slow. So if you have better ideas that'd be great.

Basically I'd like to define the dataset.* fields in ecs style schema instead of using:

- name: dataset.type
  type: constant_keyword
  description: >
    Dataset type.
- name: dataset.name
  type: constant_keyword
  description: >
    Dataset name.
- name: dataset.namespace
  type: constant_keyword
  description: >
    Dataset namespace.

It doesn't look like this line: https://github.com/elastic/package-registry/blob/master/util/dataset.go#L282

Is flattening it like the way I hoped. It still results in:

{
  name: dataset
  fields: [
    {name: name, ...}, {name: type, ...}, {name: namespace, ...}
  ]
}

[elasticmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [Pull request #520 updated]

• Start Time: 2020-06-18T14:18:36.777+0000

• Duration: 9 min 25 sec

Test stats 🧪

Test	Results
Failed	0
Passed	105
Skipped	0
Total	105

[mtojek]

Thank you for your contribution. I think I managed to simplify your code. Indeed, the issue is around flattening data.

Please take a look and share your thoughts on this: #521

[mtojek]

Resolving in favor of #521

[mtojek]

The PR #521 doesn't properly support nested arrays while flattenning, hence reverting: #527

[mtojek]

I couldn't figure out a solution that's really shorter than this. Feel free to merge this change if the CI goes green.

It would be good to add an entry to the CHANGELOG ;)

[ruflin]

To test your changes, I assume you could update one of the testdata packages with the new format and if it still works, all should be good: https://github.com/elastic/package-registry/blob/master/testdata/package/default_pipeline/0.0.2/dataset/foo/fields/base-fields.yml

[jonathan-buttner]

To test your changes, I assume you could update one of the testdata packages with the new format and if it still works, all should be good: https://github.com/elastic/package-registry/blob/master/testdata/package/default_pipeline/0.0.2/dataset/foo/fields/base-fields.yml

Oh sorry just saw this after I merged haha. I'll open up another PR to update the tests.