Make the ECS tooling work for customizations on top of branch 1.5

[webmat]

Many improvements have been added to the ECS tooling, in order to help curate an index template that contains custom fields or that selects only a subset of ECS fields.

The unofficial guidance (#746 (comment)) suggests using a tag or branch rather than master, in order for the result to actually be based on a stable ECS release.

Many of the tooling improvements have however been done after the ECS 1.5 release, which means the guidance doesn't actually work 😬 The goal of the work here is to backport some of the tooling improvements to the current ECS release only (1.5), not older versions.

MVP backports to make customizations possible:

• Merge reusable location lists when merging schemas #751 Make it possible to add reusable locations via custom fields files.
  • backport: Backport #751 to 1.5: Merge reusable location lists when merging a custom schema with an ECS schema (#751) #836
• Handle nestings better and refactor asciidoc generation #803 Handle nestings better and adjust asciidoc
  • Starting with this PR, using either subset or include no longer generates the asciidoc (which is was a cause for problems, when removing categorization fields)
  • This changes the listing of nestings visible in the docs, and may therefore warrant cutting ECS 1.5.1?

Additional backports that could be considered:

• Fix bug with subsets and reusable fields #753 Fix a bug with subsets and reusable fields
• Shorthand subset notation #805 Support shorthand subset notation
• Refactor schema_reader.py #811 Refactor schema_reader.py
  • removes the unused order attribute from the ECS yaml files, which may also warrant a patch bump
• Fix bug #495 #813 Remove fields at the top level that should never have been there in some artifacts (only expected in "reused" locations)
  • this bug fix absolutely warrants a version bump

[webmat]

cc @joshdevins

[webmat]

@marshallmain @jonathan-buttner Are you guys running the ECS tooling on top of master, or on branch 1.5?

If the latter, how much of this would you need backported?

[marshallmain]

I was actually just thinking about this as I'm trying to use the tooling to make a combined schema to put in the SIEM detection engine. I don't immediately need the subsets for that, but I do need #751 to be able to add reusable locations, and would probably want #813 as well.

My initial thought was it would be awesome to be able to run the tooling in master against any cut version of the ECS schema by passing in a target ECS version as a cmdline parameter. This way in the future when we make tooling changes in master they can immediately be used without having to backport. I haven't thought through how to implement that though. Thoughts?

[webmat]

Yeah perhaps we could add an option to pass a git ref. Although the last thing I want is for the tooling to depend on a full fledged Git library, or doing HTTP calls to the main ECS repo for this.

We should look into doing this via git show. E.g. looking at the version file as of branch 1.4: git show 1.4:version.

I'm not going to start work on this right now, however. I'm already knee deep in working on the generator for self-nesting.

So I still think I need to backport some of the above to branch 1.5. I'll start with the first two as planned, as this is what will fix an important bug when trying to select a subset of fields that doesn't include categorization fields.

[webmat]

@marshallmain For now if you really need #813, I'd like to know if you can temporarily do it by filtering out the fields via --subset instead. I'd like to avoid cutting 1.5.1 if we can. And #813 clearly needs cutting a new version.

[jonathan-buttner]

@webmat I've been running the ecs tooling on top of master. Should I only be using the latest release?

[webmat]

Well, master is a development branch. So if something hasn't been cut into a release yet, it may still change in a breaking manner. This doesn't happen often, as we usually try to resolve these things in the PRs, but unreleased things in master are fair game for breaking changes.

Also, from one of your script runs to another, you may suddenly inherit whole new field sets, which may come as a surprise :-)

So for predictability's, sake my recommendation is to do this based on a tag, or at least a version branch. But I don't mind if you like to live dangerously ;-)

[jonathan-buttner]

Haha ah that makes sense. I think I'll just have to stick with living dangerously haha, I think I need most of the changes that Marshall and I put into master. When do you think we'll release 1.6?

[webmat]

Should be about a month before 7.9 FF. But the exact date hasn't been determined yet.

[webmat]

Rather than backporting all of these changes and potentially even creating a patch release because of that, I think we should modify the tooling to be able to load the ECS schema files from a git ref instead.

Closing this in favour of #837