ECS tooling rewrite brought about by the need to reuse field sets as a different name

CHANGELOG.next.md

@@ -43,6 +43,8 @@ Thanks, you're awesome :-) -->
   had `reusable.top_level:false`. This PR affects `ecs_flat.yml`, the csv file
   and the sample Elasticsearch templates. #495, #813
 * Removed the `order` attribute from the `ecs_nested.yml` and `ecs_flat.yml` files. #811
+* In `ecs_nested.yml`, the array of strings that used to be in `reusable.expected`
+  has been replaced by an array of objects with 3 keys: 'as', 'at' and 'full'. #864
 
 #### Bugfixes
 
@@ -63,6 +65,20 @@ Thanks, you're awesome :-) -->
 * Allow shorthand notation for including all subfields in subsets. #805
 * Add `ref` option to generator allowing schemas to be built for a specific ECS version. #851
 * Add `template-settings` and `mapping-settings` options to allow override of defaults in generated ES templates. #856
+* When overriding ECS field sets via the `--include` flag, it's no longer necessary
+  to duplicate the field set's mandatory attributes. The customizations are merged
+  before validation. #864
+* Add ability to nest field sets as another name. #864
+* Add ability to nest field sets within themselves (e.g. `process` => `process.parent`). #864
+* New attribute `reused_here` is added in `ecs_nested.yml`. It obsoletes the
+  previous attribute `nestings`, and is able to fully capture details of other
+  field sets reused under this one. #864
+* When chained reuses are needed (e.g. `group` => `user`, then `user` => many places),
+  it's now necessary to force the order with new attribute `reusable.order`. This
+  attribute is otherwise optional. It's currently only needed for `group`. #864
+* There's a new representation of ECS at `generated/ecs/ecs.yml`, which is a deeply nested
+  representation of the fields. This file is not in git, as it's only meant for
+  developers working on the ECS tools. #864
 
 #### Deprecated
 

Makefile

@@ -102,7 +102,7 @@ setup: ve
 # Run the ECS tests
 .PHONY: test
 test: ve
-	$(PYTHON) -m unittest discover --start-directory scripts/tests
+	$(PYTHON) -m unittest discover -v --start-directory scripts/tests
 
 # Create a virtualenv to run Python.
 .PHONY: ve

docs/field-details.asciidoc

@@ -453,12 +453,6 @@ example: `co.uk`
 // ===============================================================
 
 
-| <<ecs-group,client.user.group.*>>
-| User's group relevant to the event.
-
-// ===============================================================
-
-
 |=====
 
 [[ecs-cloud]]
@@ -1012,12 +1006,6 @@ example: `co.uk`
 // ===============================================================
 
 
-| <<ecs-group,destination.user.group.*>>
-| User's group relevant to the event.
-
-// ===============================================================
-
-
 |=====
 
 [[ecs-dll]]
@@ -2755,12 +2743,6 @@ example: `1325`
 // ===============================================================
 
 
-| <<ecs-group,host.user.group.*>>
-| User's group relevant to the event.
-
-// ===============================================================
-
-
 |=====
 
 [[ecs-http]]
@@ -5269,12 +5251,6 @@ example: `co.uk`
 // ===============================================================
 
 
-| <<ecs-group,server.user.group.*>>
-| User's group relevant to the event.
-
-// ===============================================================
-
-
 |=====
 
 [[ecs-service]]
@@ -5610,12 +5586,6 @@ example: `co.uk`
 // ===============================================================
 
 
-| <<ecs-group,source.user.group.*>>
-| User's group relevant to the event.
-
-// ===============================================================
-
-
 |=====
 
 [[ecs-threat]]
@@ -6728,7 +6698,7 @@ example: `outside`
 
 ==== Field Reuse
 
-The `vlan` fields are expected to be nested at: `network.vlan`, `network.inner.vlan`, `observer.egress.vlan`, `observer.ingress.vlan`.
+The `vlan` fields are expected to be nested at: `network.inner.vlan`, `network.vlan`, `observer.egress.vlan`, `observer.ingress.vlan`.
 
 Note also that the `vlan` fields are not expected to be used directly at the top level.
 

generated/ecs/.gitignore

@@ -0,0 +1 @@
+ecs.yml