Beta1 related but not blocker tasks

[ruflin]

Misc

• Domain breakdown field (adding the field for domain without subdomain) Use domain and registered_domain for the domain breakdown. #163
• Reflect object reuse in template.json
• Reflect object reuse in schema.csv
• Fix script issue with multi_fields, introduced when adding for Core/Extended
• Discuss whether we nest protocol field sets under network.
  • Add field set for dns
  • Re-introduce http (see removal Temporarily remove the http object... #171)
• Re-introduce user_agent (see removal Temporarily remove the user_agent field set... #172)

Documentation

• Non-ECS fields (Document various options to adding non-ECS fields to a stream that wants to conform to ECS #114, WIP Documentation on how to add non-ECS fields in one's schema #130, Reserve some top level prefixes for custom non-ecs fields #101)
  • Document how to work around ECS (Document various options to adding non-ECS fields to a stream that wants to conform to ECS #114, WIP Documentation on how to add non-ECS fields in one's schema #130)
  • Decide on specific approach(es) (Reserve some top level prefixes for custom non-ecs fields #101)
  • sharing schema examples with non-ECS data (doc) (How to share schemas used with fields not in ECS #52)
    • this issue contains discussion about naming outside ECS
• list upcoming top level fields as prevention measure vs customization clashes
• Readme should suggest strategy of getting to ECS compatibility in two phases:
  1. Map your stream's existing fields to ECS
  2. Consider "filling" more ECS fields than you had initially, if it can be done easily from that system's vantage point
• Cleanup ECS use cases v new names (Cleanup ECS use cases #110, let's do close to v1.0, when things are closer to GA)
• development pointers (Add a DEVELOPMENT.md #92)

Related Projects

• Share template with Beats
• Open PRs
  • Beats
    • Document breaking changes (see Append ECS fields to fields.yml beats#8313)
    • Packetbeat [Packetbeat] - Change field names to follow ECS beats#7968
    • More issues for default fields & popular modules
  • Logstash plugins
    • user_agent (closes Logstash User Agent Filter vs. ECS Fields #112)
    • GeoIP
    • Create issues for all other Elastic-supported plugins
    • Update the CEF codec? Add an option to convert?
  • ES Ingest Node processors
    • user_agent
    • GeoIP
    • Any other needed?

Automation

• Script to generate an ECS index template (related: ECS Schema Management #95)
  • Option to generate separate fields.yml files (core only, core+ext)
• Offer a script that can look at existing ES index templates and evaluate for ECS compliance: breaking changes, unknown fields, etc
• Script and/or procedures to help migrate existing indices to ECS
  • Consider the following simplification/approach: implement one or a few pipelines that can automatically take known streams (Beats 6.x) and change them to ECS

Close

Not necessarily related to 1.0, but during my review, these issues seemed to have been addressed or sound like we're not going to go that direction. So consider closing.

• pr Revised Network Metrics #2? network stats directionality
• pr Proposal: Introduce connection prefix, move source / destination #51? conn prefix not closing it
• pr Rename hostname to host.name fields to be consistent #74? host.name stays hostname
• Reserve some top level prefixes for custom non-ecs fields #101

[MikePaquette]

@webmat @ruflin should we look at any of these items in light of beta2?

I know I'd still like to see us reflect object reuse in schema.csv . :-)

[ruflin]

@MikePaquette I see all the above issues as tooling / docs around ECS that can also be backported from master to the 1.0 branch after we cut off beta2. Some of the above definitively are need for the GA release.

[webmat]

Yes, the end of this week will be a bit weird. But starting next week, I'd like to get back to working on the tooling around ECS. Fixing schema.csv and template.json are definitely at the top of my list.

Looking at some of the other cool code contributions (from both Andrews) is also right in there :-)

[ruflin]

@webmat @MikePaquette I suggest we also close this issue and instead create issue for the things we don't have an issue yet and assign a 1.0.0-GA label if we think it's need for GA. This will make it also very easy to adjust things.

[webmat]

Most of the tasks I see here have been done over the years 🎉 . Closing as stale