Document ECS RFC process

CONTRIBUTING.md

@@ -20,9 +20,14 @@ ECS is an open source project and we love to receive contributions from our comm
 
 ## How to Contribute
 
+There are two primary ways in which you can contribute to ECS.
+
+1. The [RFC process](./rfcs/README.md) is used for significant additions or breaking changes to the schema itself.
+2. For bug fixes or incremental, non-controversial additions to ECS, changes can be made directly to the ECS project and submitted as pull request.
+
 ### Dev Tools
 
-You need these tools to contribute to ECS:
+You need these tools to contribute to the ECS repo:
 
 * [Git](https://git-scm.com/)
 * [Python 3.6+](https://www.python.org/)

README.md

@@ -15,7 +15,8 @@ Visit [the official ECS Reference Documentation](https://www.elastic.co/guide/en
 ## Contributing
 
 If you're looking to contribute to ECS, you're invited to look at our
-[contribution guide](CONTRIBUTING.md)
+[contribution guide](CONTRIBUTING.md). Substantial changes to ECS are completed
+through our [RFC process](./rfcs/README.md).
 
 ## Generated artifacts
 

rfcs/0000-rfc-template.md

@@ -0,0 +1,95 @@
+# 0000: Name of RFC
+<!--^ The ECS team will assign a unique, contiguous RFC number upon merging the initial stage of this RFC, taking care not to conflict with other RFCs.-->
+
+- Stage: **0 (strawperson)** <!-- Update to reflect target stage -->
+- Date: **TBD** <!-- Update to reflect date of most recent stage advancement -->
+
+<!--
+Stage 0: Provide a high level summary of the premise of these changes. Briefly describe the nature, purpose, and impact of the changes. ~2-5 sentences.
+-->
+
+## Fields
+
+<!--
+Stage: 1: Describe at a high level how this change affects fields. Which fieldsets will be impacted? How many fields overall? Are we primarily adding fields, removing fields, or changing existing fields? The goal here is to understand the fundamental technical implications and likely extent of these changes. ~2-5 sentences.
+-->
+
+<!--
+Stage 2: Include new or updated yml field definitions for all of the essential fields in this draft. While not exhaustive, the fields documented here should be comprehensive enough to deeply evaluate the technical considerations of this change. The goal here is to validate the technical details for all essential fields and to provide a basis for adding experimental field definitions to the schema. Use GitHub code blocks with yml syntax formatting.
+-->
+
+<!--
+Stage 3: Add or update all remaining field definitions. The list should now be exhaustive. The goal here is to validate the technical details of all remaining fields and to provide a basis for releasing these field definitions as beta in the schema. Use GitHub code blocks with yml syntax formatting.
+-->
+
+## Usage
+
+<!--
+Stage 1: Describe at a high-level how these field changes will be used in practice. Real world examples are encouraged. The goal here is to understand how people would leverage these fields to gain insights or solve problems. ~1-3 paragraphs.
+-->
+
+## Source data
+
+<!--
+Stage 1: Provide a high-level description of example sources of data. This does not yet need to be a concrete example of a source document, but instead can simply describe a potential source (e.g. nginx access log). This will ultimately be fleshed out to include literal source examples in a future stage. The goal here is to identify practical sources for these fields in the real world. ~1-3 sentences or unordered list.
+-->
+
+<!--
+Stage 2: Included a real world example source document. Ideally this example comes from the source(s) identified in stage 1. If not, it should replace them. The goal here is to validate the utility of these field changes in the context of a real world example. Format with the source name as a ### header and the example document in a GitHub code block with json formatting.
+-->
+
+<!--
+Stage 3: Add more real world example source documents so we have at least 2 total, but ideally 3. Format as described in stage 2.
+-->
+
+## Scope of impact
+
+<!--
+Stage 2: Identifies scope of impact of changes. Are breaking changes required? Should deprecation strategies be adopted? Will significant refactoring be involved? Break the impact down into:
+ * Ingestion mechanisms (e.g. beats/logstash)
+ * Usage mechanisms (e.g. Kibana applications, detections)
+ * ECS project (e.g. docs, tooling)
+The goal here is to research and understand the impact of these changes on users in the community and development teams across Elastic. 2-5 sentences each.
+-->
+
+## Concerns
+
+<!--
+Stage 1: Identify potential concerns, implementation challenges, or complexity. Spend some time on this. Play devil's advocate. Try to identify the sort of non-obvious challenges that tend to surface later. The goal here is to surface risks early, allow everyone the time to work through them, and ultimately document resolution for posterity's sake.
+-->
+
+<!--
+Stage 2: Document new concerns or resolutions to previously listed concerns. It's not critical that all concerns have resolutions at this point, but it would be helpful if resolutions were taking shape for the most significant concerns.
+-->
+
+<!--
+Stage 3: Document resolutions for all existing concerns. Any new concerns should be documented along with their resolution. The goal here is to eliminate the risk of churn and instability by resolving outstanding concerns.
+-->
+
+<!--
+Stage 4: Document any new concerns and their resolution. The goal here is to eliminate risk of churn and instability by ensuring all concerns have been addressed.
+-->
+
+## Real-world implementations
+
+<!--
+Stage 4: Identify at least one real-world, production-ready implementation that uses these updated field definitions. An example of this might be a GA feature in an Elastic application in Kibana.
+-->
+
+## People
+
+The following are the people that consulted on the contents of this RFC.
+
+* TBD | author
+
+<!--
+Who will be or has consulted on the contents of this RFC? Identify authorship and sponsorship, and optionally identify the nature of involvement of others. Link to GitHub aliases where possible. This list will likely change or grow stage after stage.
+
+e.g.:
+
+* @Yasmina | author
+* @Monique | sponsor
+* @EunJung | subject matter expert
+* @JaneDoe | grammar, spelling, prose
+* @Mariana
+-->

rfcs/PROCESS.md

@@ -0,0 +1,50 @@
+# Proposing material changes to ECS
+
+Changes to ECS are proposed as Requests for Comments (RFC) in [rfcs/](./rfcs/) and iterated on through a series of [stages](https://elastic.github.io/ecs/stages.html). To advance to a specific stage, an RFC must meet the documented requirements for that stage. After being accepted into a given stage, there are stage-specific expectations and goals to be met. The overall goal of this process is to thoroughly evaluate and verify the assumptions being made about a change before formally committing it to the schema.
+
+Each RFC is represented as a markdown document following a prescribed template that gets committed to the repo. Each stage of the RFC is represented as a pull request against that document.
+
+Generally speaking, the ECS team will help steward the process, but the work of researching and iterating on aspects of an RFC will be owned by that RFC's contributor. If an RFC is being contributed by a community member, then someone at Elastic will need to act as a sponsor of the change to act as a long term owner after completion of the process. If it's not obvious who such a sponsor might be, then the ECS committee will assign a sponsor.
+
+## Key questions we seek to answer through RFC process
+
+* Is this change appropriate for ECS?
+* Does this change provide enough utility for its intended use cases?
+* Does this change strike a sufficient balance between introducing new fields and reusing existing common fields?
+* Is ownership for the ongoing maintenance of this change clearly defined and accepted?
+* Is the scope of impact of this change to ingestion, existing applications, and the ECS project itself understood?
+* Are the technical details of the change defined clearly enough to implement in the schema?
+* Are we confident these changes can be stable upon release without requiring revisions or breaking changes?
+* Have our assumptions about the shape and utility of these changes been verified by real-world, production-ready usage?
+
+## Goals with this contributing process
+
+* Allow contributors to quickly iterate and receive feedback on their fields in a transparent way without the high bar set for general availability in the schema
+* Clarify the level of stability to expect from a change in ECS while still allowing early adopters to try it out and provide feedback
+* Offer assurance that once an RFC reaches stage 4, we're able to guarantee backward compatibility
+
+## Responsibilities in this process
+
+Member(s) of the **ECS committee**:
+* evaluates whether the changes are appropriate in terms of the goals of the ECS project
+* provides recommendations on which common fields would be best suited for reuse versus adding new fields
+* determines whether each RFC is accepted into the next target stage by merging the RFC PR
+
+The **ECS team**:
+* provides procedural guidance for moving an RFC through stages
+* curates the overall RFC process, including closing stalled or abandoned RFCs
+* reports on the status of open RFCs
+* acts on behalf of the committee for some but not all PRs
+
+The **contributor**:
+* takes responsibility for doing all necessary legwork to move their RFC forward including but not limited to responding to feedback, identifying and bringing in subject matter experts, and researching the scope of impact
+* demonstrates how the fields in the RFC are expected to be used: from the data source, all the way to its consumption
+* commits to iterating on the RFCs through to stage 4 if necessary
+* creates and iterate on RFC PRs
+* implements all necessary changes to their RFC PRs
+
+The **sponsor** at Elastic:
+* can be the same person as the contributor if they're someone at Elastic that can take ownership of this change through membership on the ECS committee
+* is involved at least from stage 1 onward if a different person than the contributor
+* signs off on each stage if a different person than the contributor
+* takes or coordinates ownership of the addition in terms of support and maintenance after the RFC process is completed

rfcs/README.md

@@ -0,0 +1,39 @@
+# Elastic Common Schema RFCs
+
+While smaller and less controversial changes can still be made directly through pull requests, more substantial changes follow a multi-stage RFC process to ensure they are sufficiently thought out and vetted before being released for general availability in the schema.
+
+The types of changes that warrant this RFC process include but are not limited to:
+
+* Breaking changes targeting the next major version
+* New top level fieldsets
+* New fields to accommodate an unaddressed use case
+* Changes that would alter the scope of ECS as a whole
+
+Check out [Proposing Changes](./PROCESS.md) for high level information about the RFC process.
+
+## How this works
+
+It's important to understand the overall goals and intentions of the RFC process, so it is recommended that you read this and the document listed above. When you're ready to dive in,
+
+1. Create a new RFC document from the RFC template (described below)
+2. Fill in the details for your strawperson
+3. Open a PR to commit your RFC to [rfcs/](./)
+4. ECS committee and/or team members review
+5. ECS committee and/or team member merges RFC, formally accepting the strawperson
+6. Expand existing RFC document with additional details
+7. Open a PR to commit your proposed changes to the RFC and advance to stage 1
+8. ECS committee and/or team members review
+9. ECS committee and/or team member merges RFC, formally accepting the proposal
+10. Repeat steps 6-9 for stage 2, 3, and 4
+
+## Using the RFC template
+
+All new RFCs should be started by copying [0000-rfc-template.md](./0000-rfc-template.md) with a name format of `0000-<dash-separated-name>.md`. When the first RFC stage is accepted, the ECS team will assign a unique RFC number, which will identify this RFC throughout all stages of the process.
+
+Throughout the RFC template are comments that provide guidance on what type of content to include in each stage. It's ideal if you remove comments from your RFC as you fill out the content and they become obsolete. A pristine, finished RFC will have no explanatory comments remaining.
+
+For the most part, content is additive as you move through the stages. Occasionally, advancing a stage will require modifying existing content. This is OK! This process should be iterative, and the RFC document is considered living until it is finished (i.e. accepted as stage 4).
+
+## Skipping stages
+
+While advancing through multiple stages at a time is possible if all the necessary information is provided and thoroughly vetted, moving changes through stage by stage provides the greatest opportunity to iterate efficiently on changes and a much lower chance of an author wasting a ton of their time.

stages.html

@@ -0,0 +1,138 @@
+<!DOCTYPE html>
+<meta charset="utf-8">
+
+<style>
+  html {
+    font-family: sans-serif;
+  }
+
+  table {
+    border-collapse: collapse;
+  }
+
+  th, td {
+    vertical-align: top;
+    border: 1px solid black;
+    padding: 0.3em;
+  }
+
+  ul {
+    padding: 0 0 0 1.2em;
+    margin: 0;
+  }
+</style>
+
+<title>ECS Proposal Stages</title>
+
+<h1>ECS Proposal Stages</h1>
+
+<p>These are the stages that an individual RFC advances through before being released for general availability in the Elastic Common Schema (ECS). See the <a href="https://github.com/elastic/ecs/blob/master/rfcs/PROCESS.md">Contributing Guide</a> for broader details about contributing changes to ECS through the RFC process.
+
+<table>
+  <thead>
+    <tr>
+      <th>
+      <th>Stage
+      <th>Goals during this stage
+      <th>Criteria for consideration for this stage
+      <th>Acceptance into this stage signifies
+      <th>Acceptable changes to schema in this stage
+      <th>Breaking changes expected after acceptance into this stage
+      <th>Recommended types of usage implementation
+    </tr>
+  </thead>
+  <tr>
+    <td>0
+    <td>Strawperson
+    <td>
+      <ul>
+        <li>Discuss with domain or subject matter experts the utility of these changes
+        <li>Discuss with ECS team whether these changes seem appropriate for ECS
+      </ul>
+    </td>
+    <td>Opened RFC pull request for this strawperson at <a href="https://github.com/elastic/ecs">elastic/ecs
+    <td>The premise of these changes is not obviously useless or inappropriate for ECS
+    <td>None
+    <td>Major
+    <td>N/A
+  </tr>
+  <tr>
+    <td>1
+    <td>Proposal
+    <td>
+      <ul>
+        <li>Make the case for these changes
+        <li>Describe the field definitions at a high level
+        <li>Identify potential challenges and areas that need more clarity
+      </ul>
+    </td>
+    <td>
+      <ul>
+        <li>Opened pull request for this proposal revising the existing strawperson
+        <li>Identified "sponsor" at Elastic who will participate in RFC process and take ownership of the change after the process completes
+        <li>Types of fields or changes outlined
+        <li>High-level description of examples of usage
+        <li>High-level description of example sources of data
+        <li>Identified potential concerns and implementation challenges/complexity
+        <li>Subject matter experts identified and weighed in on the high level utility of these changes in the pull request
+        <li>ECS team weighed in on appropriateness of these changes in the pull request
+      </ul>
+    </td>
+    <td>ECS team accepts the premise of the addition and commits to considering this proposal as it advances.
+    <td>None
+    <td>Major
+    <td>Proof of concepts, demos
+  </tr>
+  <tr>
+    <td>2
+    <td>Draft
+    <td>Identify a comprehensive set of field definitions that could be appropriate for real-world usage
+    <td>
+      <ul>
+        <li>Opened pull request for this draft revising the existing proposal
+        <li>Outlined initial field definitions
+        <li>Included a real world example source document
+        <li>Identifies scope of impact of changes to ingestion mechanisms (e.g. beats/logstash), usage mechanisms (e.g. Kibana applications, detections), and the ECS project (e.g. docs, tooling)
+        <li>Subject matter experts weighed in on technical utility of field definitions in the pull request
+      </ul>
+    </td>
+    <td>The initial field definitions are a comprehensive, though not necessarily complete, model of the addition to the schema. Fundamental questions and concerns are resolved, though some less significant questions remain open.
+    <td>Draft field definitions can be committed to the ECS schema as "experimental" fields
+    <td>Iterative
+    <td>Experimental and beta features
+  </tr>
+  <tr>
+    <td>3
+    <td>Candidate
+    <td>Indicate that direct experience from implementations and users is necessary to validate the additions
+    <td>
+      <ul>
+        <li>Opened pull request for this candidate revising the existing draft
+        <li>Completed field definitions
+        <li>Included multiple real world example source documents
+        <li>Existing or newly raised questions and concerns are addressed
+      </ul>
+    </td>
+    <td>There are no further open questions or unaddressed concerns, and the field definitions are complete based on the information and usage experience we have.
+    <td>Candidate field definitions can be committed to the ECS schema as "beta" fields
+    <td>Minimal: only those determined to be critical based on usage experience
+    <td>GA features
+  </tr>
+  <tr>
+    <td>4
+    <td>Finished
+    <td>Indicate that the addition is ready for GA release in ECS
+    <td>
+      <ul>
+        <li>Opened pull request for this candidate revising the existing candidate
+        <li>At least one real-world, production-ready usage implementation exists for the field definitions
+        <li>Existing or newly raised questions and concerns are addressed
+        <li>No objections from sponsor, ECS team, or subject matter experts
+      </ul>
+    </td>
+    <td>The field definitions are in use as defined in real-world, production-ready software without raising additional questions or concerns that need to be resolved. The changes are ready to be released as GA in ECS.
+    <td>Field definitions can be committed to the ECS schema as "GA" fields
+    <td>None outside major versions
+    <td>Any
+  </tr>
+</table>