Make ECS tooling friendly to generating custom templates based on ECS

[webmat]

A frequent request about ECS is: "How can I generate Elasticsearch templates that include ECS fields and my custom fields?"

There's many ways we could provide tooling for that: a Kibana app would probably be the most user-friendly. However the tooling in the ECS repository is very close to being able to offer this already. As a short term solution, we will be tweaking the generator in this repo to make it easier for users to manage their templates with it. As a bonus, the generator will continue generating all of the artifacts that includes people's fields (asciidoc, csv), not only the ES templates. Some users may find that interesting.

Here's a list of tasks that will gradually make it easier to achieve this goal:

• Add ability to include external directory of schemas when running generator #494 Add the ability to include external directory of schemas
• Adding --out flag #748 Add the ability to specify an output directory
• Document the usage of the ECS generator #746 Document how to use the ECS tooling to generate custom templates
• Let users select which artifacts they want to generate
• Allow generation of a subset of ECS and custom schema fields #737 Let users specify which ECS schemas to include, to skip the fields they will not use
• Forcefully set level: custom for any field loaded from an external directory
• Make the index template settings configurable
• Allow for adding fields within existing ECS field sets from external files

These will help address #95 and #324

[Randy-312]

We were going to do this with ordering for elasticsearch templates.
Have our base fields in a 001, and all of the various version specific fields in 002.
I've not thought of all the problems in that approach.. what are they? i.e, if we change OUR fields and dont' want to use those everywhere.

[webmat]

Please move your question to discuss.elastic.co, and I'll answer it there. This issue is meant to discuss the plan I've outlined above.

[pmuellr]

I'm starting to look into adopting ECS into my life, and this sort of tooling is of interest to me. Just starting, haven't written any code for this yet, so ... can't speak with any real credence. We'd either be taking base ECS and adding more bits to it, or starting with our own ECS-compatible-ish base, and adding relevant ECS bits on top of that.

My initial thoughts were to write some of my own tooling in node, that would just read one of the generated schema files, and then operate on that and my custom definitions. I'm actually quite comfortable doing that, vs using an existing tool - it's just JSON algebra in the end, and it wouldn't surprise me to find I need a bit more than some generic tooling might provide. And I guess I've done this so many other times over the years :-)

Given that, my biggest problem is getting access to the repo. This is for some new code in Kibana, and so the perfect vehicle to ingest this repo would be via a node npm package. Not sure that really makes sense, and would require quite a bit more book-keeping for an ecs release, and so I think I'll probably be happy enough to just curl the appropriate files in some stand-alone script I'd run when a new ECS version is released.

If it WAS available as an npm package tho, I could just pre-req it, and make that script part of our build, vs making it a manual process per ECS release.

There are some options for making a GH repo like this available without publishing to npm (a node package can pre-req a package directly from a git repo, either master or with a specific tag), which might be an interesting start tho.

[webmat]

@pmuellr Other initiatives such as the ECS logging libraries also depend on ECS artifacts and aren't in this repo. I'll be open to helping make this repo more amenable to consuming programmatically from other projects, such as yours or the ECS logging libraries. Let me know what challenges you encounter.

Here's a few pointers to help you get started:

• By default, have your scripts operate on the release tags, not on the master branch. Until something gets released (and therefore tagged), it's still open to change, or even be removed completely.
• When leveraging the ECS schema from your script, don't try to parse what's in schemas/*.yml, these files are for humans to read and edit. The generator does a pass on the YAML files, fills in defaults, cleans everything up, and produces two other YAML files: ecs_flat.yml and ecs_nested.yml. Same content, different layout. Both will be maintained, as the various ECS generators use them both. Flat is easier to run through for a simple file such as the csv, whereas nested keeps the details of the fieldsets & so on. Nested is a bit trickier to navigate, but more complete.

With that said, this repo cannot adopt a deployment to NPM, because by the same token, people may then request it to Rubygems, Maven, or to any other arbitrary package manager :-) But as I said already, I'm happy to help people consume this programmatically otherwise.

[webmat]

Another note: Elastic SIEM embeds ECS and Beats field definitions already. Ask the SIEM team where the files are, perhaps there's something you can leverage from that, elsewhere in Kibana?

[webmat]

For those following this issue, there's good progress happening on helping users leverage the ECS tooling to manage their custom fields. See the issue body for detail 🙂

[ebeahan]

While there are always improvements to be made, we've come a long way with the USAGE.md guide and the usage example. Any future improvements or ideas can be captured in additional issues.