From 18e5e945fcfc050717ba167240e8fb6b41c31bc2 Mon Sep 17 00:00:00 2001 From: Court Ewing Date: Thu, 30 Apr 2020 16:26:13 -0400 Subject: [PATCH 01/13] Document ECS RFC process --- PROPOSING_CHANGES.md | 50 ++++++++++++++ rfcs/0000-rfc-template.md | 95 ++++++++++++++++++++++++++ rfcs/README.md | 39 +++++++++++ stages.html | 138 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 322 insertions(+) create mode 100644 PROPOSING_CHANGES.md create mode 100644 rfcs/0000-rfc-template.md create mode 100644 rfcs/README.md create mode 100644 stages.html diff --git a/PROPOSING_CHANGES.md b/PROPOSING_CHANGES.md new file mode 100644 index 0000000000..a9c2378728 --- /dev/null +++ b/PROPOSING_CHANGES.md @@ -0,0 +1,50 @@ +# Proposing material changes to ECS + +Changes to ECS are proposed as Requests for Comments (RFC) in [rfcs/](./rfcs/) and iterated on through a series of [stages](https://elastic.github.io/ecs/stages.html). To advance to a specific stage, an RFC must meet the documented requirements for that stage. After being accepted into a given stage, there are stage-specific expectations and goals to be met. The overall goal of this process is to thoroughly evaluate and verify the assumptions being made about a change before formally committing it to the schema. + +Each RFC is represented as a markdown document following a prescribed template that gets committed to the repo. Each stage of the RFC is represented as a pull request against that document. + +Generally speaking, the ECS team will help steward the process, but the work of researching and iterating on aspects of an RFC will be owned by that RFC's contributor. If an RFC is being contributed by a community member, then someone at Elastic will need to act as a sponsor of the change to act as a long term owner after completion of the process. + +## Key questions we seek to answer through RFC process + +* Is this change appropriate for ECS? +* Does this change provide enough utility for its intended use cases? +* Does this change strike sufficient balance between introducing new fields and reusing existing common fields? +* Is ownership for the ongoing maintenance of this change clearly defined and accepted? +* Is the scope of impact of this change to ingestion, existing applications, and the ECS project itself understood? +* Are the technical details of the change defined clearly enough to implement in the schema? +* Are we confident these changes can be stable upon release without requiring revisions or breaking changes? +* Have our assumptions about the shape and utility of these changes been verified by real-world, production-ready usage? + +## Goals with this contributing process + +* Allow contributors to quickly iterate and receive feedback on their fields in a transparent way without the high bar set for general availability in the schema +* Clarify the level of stability to expect from a change in ECS while still allowing early adopters to try it out and provide feedback +* Offer assurance that once an RFC reaches stage 4, we're able to guarantee backwards compatibility + +## Responsibilities in this process + +Member(s) of the **ECS committee**: +* evaluates whether the changes are appropriate in terms of the goals of the ECS project +* provides recommendations on the which common fields would be best suited for reuse versus adding new fields +* determines whether each RFC is accepted into the next target stage by merging the RFC PR + +The **ECS team**: +* provides procedural guidance for moving an RFC through stages +* curates the overall RFC process, including closing stalled or abandoned RFCs +* reports on the status of open RFCs +* acts on behalf of the committee for some but not all PRs + +The **contributor**: +* takes responsibility for doing all necessary legwork to move their RFC forward including but not limited to responding to feedback, identifying and bringing in subject matter experts, and researching scope of impact +* demonstrates how the fields in the RFC are expected to be used: from the data source, all the way to its consumption +* commits to iterating on the RFCs through to stage 4 if necessary +* creates and iterate on RFC PRs +* implements all necessary changes to their RFC PRs + +The **sponsor** at Elastic: +* can be the same person as the contributor if they're an appropriate engineer at Elastic +* is involved at least from stage 1 onward if a different person than the contributor +* signs off on each stage if a different person than the contributor +* takes or coordinates ownership of the addition in terms of support and maintenance after the RFC process is completed diff --git a/rfcs/0000-rfc-template.md b/rfcs/0000-rfc-template.md new file mode 100644 index 0000000000..d527fcf203 --- /dev/null +++ b/rfcs/0000-rfc-template.md @@ -0,0 +1,95 @@ +# 0000: Name of RFC + + +- Stage: **1 (strawperson)** +- Date: **TBD** + + + +## Fields + + + + + + + +## Usage + + + +## Source data + + + + + + + +## Scope of impact + + + +## Concerns + + + + + + + + + +## Real-world implementations + + + +## People + +The following are the people that consulted on the contents of this RFC. + +* TBD | author + + diff --git a/rfcs/README.md b/rfcs/README.md new file mode 100644 index 0000000000..ca874483a9 --- /dev/null +++ b/rfcs/README.md @@ -0,0 +1,39 @@ +# Elastic Common Schema RFCs + +While smaller and less controversial changes can still be made directly through pull requests, more substantial changes follow a multi-stage RFC process to ensure they are sufficiently thought out and vetted before being released for general availability in the schema. + +The types of changes that warrant this RFC process include but are not limited to: + +* Breaking changes targeting the next major version +* New top level fieldsets +* New fields to accommodate an unaddressed use case +* Changes that would alter the scope of ECS as a whole + +Check out [Proposing Changes](../PROPOSING_CHANGES.md) for high level information about the RFC process. + +## How this works + +It's important to understand the overall goals and intentions of the RFC process, so it is recommended that you read this and the document listed above. When you're ready to dive in, + +1. Create a new RFC document from the RFC template (described below) +2. Fill in the details for your strawperson +3. Open a PR to commit your RFC to [rfcs/](./) +4. ECS committee and/or team members review +5. ECS committee and/or team member merges RFC, formally accepting the strawperson +6. Expand existing RFC document with additional details +7. Open a PR to commit your proposed changes to the RFC and advance to stage 1 +8. ECS committee and/or team members review +9. ECS committee and/or team member merges RFC, formally accepting the proposal +10. Repeat steps 6-9 for stage 2, 3, and 4 + +## Using the RFC template + +All new RFCs should be started by copying [0000-rfc-template.md](./0000-rfc-template.md) with a name format of `.md`. Take care to select an RFC number that is not already being used, and especially one that has not already been committed. + +Throughout the RFC template are comments that provide guidance on what type of content to include in each stage. It's ideal if you remove comments from your RFC as you fill out the content and they become obsolete. A pristine, finished RFC will have no explanatory comments remaining. + +For the most part, content is additive as you move through the stages. Occasionally, advancing a stage will require modifying existing content. This is OK! This process should be iterative, and the RFC document is considered living until it is finished (i.e. accepted as stage 4). + +## Skipping stages + +While advancing through multiple stages at a time is possible if all the necessary information is provided and thoroughly vetted, moving changes through stage by stage provides the greatest opportunity to iterate efficiently on changes and a much lower change of an author wasting a ton of their time. diff --git a/stages.html b/stages.html new file mode 100644 index 0000000000..c7480666b5 --- /dev/null +++ b/stages.html @@ -0,0 +1,138 @@ + + + + + +ECS Proposal Stages + +

ECS Proposal Stages

+ +

These are the stages that an individual RFC advances through before being released for general availability in the Elastic Common Schema (ECS). See the Contributing Guide for broader details about contributing changes to ECS through the RFC process. + + + + + + + + + + + + + + + + + + + + + + +
+ Stage + Goals during this stage + Criteria for consideration for this stage + Acceptance into this stage signifies + Acceptable changes to schema in this stage + Breaking changes expected after acceptance into this stage + Recommended types of usage implementation +
0 + Strawperson + +
    +
  • Discuss with domain or subject matter experts the utility of these changes +
  • Discuss with ECS team whether these changes seem appropriate for ECS +
+ 		Opened RFC pull request for this strawperson at elastic/ecs + The premise of these changes is not obviously useless or inappropriate for ECS + None + Major + N/A +
1 + Proposal + +
    +
  • Make the case for these changes +
  • Describe the field definitions at a high level +
  • Identify potential challenges and areas that need more clarity +
+ 		+
    +
  • Opened pull request for this proposal revising the existing strawperson +
  • Identified "sponsor" at Elastic who will participate in RFC process and take ownership of the change after the process completes +
  • Types of fields or changes outlined +
  • High-level description of examples of usage +
  • High-level description of example sources of data +
  • Identified potential concerns and implementation challenges/complexity +
  • Subject matter experts identified and weighed in on the high level utility of these changes in the pull request +
  • ECS team weighed in on appropriateness of these changes in the pull request +
+ 		ECS team accepts the premise of the addition and commits to considering this proposal as it advances. + None + Major + Proof of concepts, demos +
2 + Draft + Identify a comprehensive set of field definitions that could be appropriate for real-world usage + +
    +
  • Opened pull request for this draft revising the existing proposal +
  • Outlined initial field definitions +
  • Included a real world example source document +
  • Identifies scope of impact of changes to ingestion mechanisms (e.g. beats/logstash), usage mechanisms (e.g. Kibana applications, detections), and the ECS project (e.g. docs, tooling) +
  • Subject matter experts weighed in on technical utility of field definitions in the pull request +
+ 		The initial field definitions are a comprehensive, though not necessarily complete, model of the addition to the schema. Fundamental questions and concerns are resolved, though some less significant questions remain open. + Draft field definitions can be committed to the ECS schema as "experimental" fields + Iterative + Experimental and beta features +
3 + Candidate + Indicate that direct experience from implementations and users is necessary to validate the additions + +
    +
  • Opened pull request for this candidate revising the existing draft +
  • Completed field definitions +
  • Included multiple real world example source documents +
  • Existing or newly raised questions and concerns are addressed +
+ 		There are no further open questions or unaddressed concerns, and the field definitions are complete based on the information and usage experience we have. + Candidate field definitions can be committed to the ECS schema as "beta" fields + Minimal: only those determined to be critical based on usage experience + GA features +
4 + Finished + Indicate that the addition is ready for GA release in ECS + +
    +
  • Opened pull request for this candidate revising the existing candidate +
  • At least one real-world, production-ready usage implementation exists for the field definitions +
  • Existing or newly raised questions and concerns are addressed +
  • No objections from sponsor, ECS team, or subject matter experts +
+ 		The field definitions are in use as defined in real-world, production-ready software without raising additional questions or concerns that need to be resolved. The changes are ready to be released as GA in ECS. + Field definitions can be committed to the ECS schema as "GA" fields + None outside major versions + Any +
From fee4ba549232769b6cdf4684198d253e957c8829 Mon Sep 17 00:00:00 2001 From: Court Ewing Date: Wed, 3 Jun 2020 13:57:55 -0400 Subject: [PATCH 02/13] Remove erroneous "the" Co-authored-by: Eric Beahan --- PROPOSING_CHANGES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/PROPOSING_CHANGES.md b/PROPOSING_CHANGES.md index a9c2378728..46a1c4e6c8 100644 --- a/PROPOSING_CHANGES.md +++ b/PROPOSING_CHANGES.md @@ -27,7 +27,7 @@ Generally speaking, the ECS team will help steward the process, but the work of Member(s) of the **ECS committee**: * evaluates whether the changes are appropriate in terms of the goals of the ECS project -* provides recommendations on the which common fields would be best suited for reuse versus adding new fields +* provides recommendations on which common fields would be best suited for reuse versus adding new fields * determines whether each RFC is accepted into the next target stage by merging the RFC PR The **ECS team**: From 1ea3c8fdacab22f1bf90e6d38978b4b0e92b6696 Mon Sep 17 00:00:00 2001 From: Court Ewing Date: Wed, 3 Jun 2020 13:58:24 -0400 Subject: [PATCH 03/13] Default to stage 0 in RFC template Co-authored-by: Eric Beahan --- rfcs/0000-rfc-template.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0000-rfc-template.md b/rfcs/0000-rfc-template.md index d527fcf203..f7b326ad8f 100644 --- a/rfcs/0000-rfc-template.md +++ b/rfcs/0000-rfc-template.md @@ -1,7 +1,7 @@ # 0000: Name of RFC -- Stage: **1 (strawperson)** +- Stage: **0 (strawperson)** - Date: **TBD** + - Stage: **0 (strawperson)** -- Date: **TBD** +- Date: **TBD**