Stable release management

[mattklein123]

I'm opening this issue to discuss a few different things and receive community comment:

1. What are the community's expectations around stable releases, release candidates, etc. Right now our policy is to snap from master every 3 months and only release dot releases for security issues on the last released version (< 3 months previous). As the community continues to grow I suspect there is interest in release candidates, stable dot releases with bug fixes, etc. but I'm not sure.
2. If the community desires more releases and rigor around releases, we will need additional resources. This might come in the form of stable branch managers, release managers, etc.

I'm going to leave this issue open for discussion right now. I would like to hear what people would like to see in this area. Additionally, if you would like to contribute to Envoy in the form of release management, stable branch management, etc. please let us know.

[Mythra]

I currently work at Instructure, and would like to share our thoughts on this topic. Since it's something we've been looking at ourselves internally for awhile now.

• For us the "stable release" once every 3 months is: "fine". We feel comfortable at this point cutting our own releases if need be. However, that brings up the question: "Why do we need to cut our own releases?" For us this boils down to usually one of two scenarios:

  1. A "fix" gets added that isn't necessarily security related, but is something causing a lot of pain/issues. A Prime example I can point to is the first issue we ran into:
     properly handle 'proxy-connection: close' header #6749
     Here all current versions of envoy that were supported had this issue. Since a release wasn't available, we had to roll our own. Having more frequent releases would have helped us in this case since it's much easier to have confidence in something others have gained first.
  2. Something breaks a build in newer versions, so we have to update our build scripts. (For example switching to bazel 0.28, we had to use a new build container). Since we build our own code into extensions at the native level, that meant any changes to that code also need to be bundled with a new Envoy release (using the newer bazel), since we personally only want to roll forward and not cherry pick "back"/support multiple build setups. So whenever a build "update" comes out like this. We generally try to upgrade as soon as we can to ensure theirs no issues. Potentially holding up something later.

In both of these senses, "running the same as everyone else" (i.e. a more frequent point release) would be better for troubleshooting purposes. Because instead of saying: "At this Git Hash we see the problem" to: "is anyone else running version v1.12.2 seeing this problem"? This would be ideal.

As for point releases, for us that isn't as important. We have our own series of contracts/integration/unit/fuzz/perf/chaos testing to: "Validate" a release. We'd be happy to contribute to validating an RC, but aren't itching for one, because we'll always validate a release the same way, and if we have to wait a bit until it can pass all those tests. That's okay.

• We agree that more resources would be necessary to do it "well", and think that we shouldn't rush a release process that can't be effectively run compared to now.

[mattklein123]

cc @antoniovicente. Here is the issue we can use to describe the policy updates we have been discussing.

[mattklein123]

Assigning this over to @PiotrSikora who is going to convert the in flight proposal into an in repo MD doc which will resolve this issue.