KubeVirt requires a clear policy on how features are introduced, evaluated and finally graduated or removed.
This proposal defines the steps and policies to follow in order to manage a feature and its lifecycle in KubeVirt.
The proposal is focusing on introducing features in
a stable API (CRD) version, e.g. kubevirt.io/v1.
KubeVirt has grown into a matured virtualization management solution with a large set of features.
New features are being proposed and added regularly to its portfolio.
With time, the challenge of supporting and maintaining such a large set of features raised the need to re-examine their relevance. It also raised the need to examine with more care features graduation.
The KubeVirt community has tried to control the flow of features informally through feature-gates, similar to Kubernetes. However, as time passed, several challenges presented themselves:
- Evaluated features rarely got graduated to GA or removed, causing
feature consumption to be risky for users and a maintenance burden
for the project contributors.
Note: As of this writing (pre v1.2), there are 37 FGs, out of which 5 GA-ed and 2 marked for deprecation. For more information, explore the source.
- We do not have agreed-upon procedures on how and when features graduate or discontinue. This causes each feature to take different approaches, possibly surprising users.
We conclude that the current use of FGs is insufficient due to the lack of well-defined processes and policy on how a feature should progress in its lifecycle.
Eventually we would like to see features being evaluated carefully before they are introduced, while they are experimented with and proven to be actually in use (and useful) before graduating.
Note: Once a feature graduates, it is included in a General Availability (GA) release with its functionality available to all users. GA features need to comply with semver which add constraints on their ability to change (including deprecation).
- Define the process a feature needs to pass in order to be Generally Available.
- Define the process a feature needs to pass in order to be removed.
- Provide policies and rules on how to manage a feature during its lifetime.
- Implement enforcement tooling to keep features in sync with the lifecycle rules.
- Development contributors.
- Cluster operators.
- As a KubeVirt contributor, I would like to introduce a new useful feature and follow it to graduation (GA).
- As a KubeVirt contributor, I would like to remove a feature that has not yet reached its formal graduation.
- As a KubeVirt contributor, I would like to remove a feature
that has already graduated.
Note: Removal of a GA feature is considered an exception. Strong arguments and a wide agreement is required for such an action.
- As a KubeVirt cluster operator, I would like to experiment with a newly-proposed ("Alpha") feature in a controlled environment, to see if it makes sense to me.
- As a KubeVirt cluster operator, I would like to evaluate an undergraduate ("Beta") feature in a real-life environment with actual users.
- As a KubeVirt cluster operator, I would like to keep using a feature that got graduated after I used it during the "Beta" evaluation period.
- As a KubeVirt cluster operator, I would like to know that an experimental ("Alpha") feature is planned to be removed.
- As a KubeVirt cluster operator, I would like to know that an undergraduate ("Beta") feature is planned to be removed.
- As a KubeVirt cluster operator, I would like to stop using a feature that got removed.
This is a cross repo project policy under the kubevirt organization.
The proposal on how to define a feature lifecycle is influenced by processes and policies from the Kubernetes project. These sources are scattered around, each focusing on different aspects of a feature:
The proposal takes the top-down approach, starting with the high level flow that a common feature will traverse through. Continuing with actions that need to be taken and timeline suggestions.
Both feature graduation and discontinuation flows are covered. Including the implications on users.
Depending on individual topics, follow-up proposals may extend the basic points raised in this proposal.
Feature Gates and feature configuration are often used interchangeable or not differentiated.
- Feature Gate: A flag that controls the presence or availability of a feature in the cluster.
- Feature configuration: Cluster or workload level configuration that allows an admin or user (depending on the feature) to control aspects of a feature operation. A common usage is to determine if features are opt-in or opt-out by default.
A feature is expected to pass in the following order through the following stages:
- Enhancement proposal.
- Implementation.
- Release as Alpha (experimental).
- Release as Beta (pre-release for evaluation).
- Release as General Availability (graduation).
- Removal.
Starting from the Alpha release, it can be removed with restrictions that depend on the release stage (Alpha, Beta, GA).
Removal of features is widely discussed later in this proposal.
As the first step for introducing a new feature, a formal proposal is expected to be shared for public review via mailinglist and a design proposal.
This is the first opportunity to evaluate a new feature. The proposal needs to include motivation, goals, implementation details and phases. Review the proposal template for more information.
The development work on the feature is expected to include coding, testing, integration and documentation.
-
Alpha: An initial release of the feature for experimental purposes. Recommended for non-production usages, evaluation or testing.
The API is considered unstable and may change significantly. There are no backward compatability considerations and it can be removed at any time.
The period in which a feature can remain in Alpha is limited, assuring features are not piling up without control. See release stage transition table for more information.
The feature presence is controlled using a Feature-Gate (FG) during runtime. It must be specified for the feature to be active.
-
Beta: The first release that can be evaluated with care in production. Acting as a pre-release, its main objective is to collect feedback from users to assure its usefulness and readiness for graduation. If there is no confidence of usage or usefulness, it may remain in this stage for some time.
However, the period in which a feature can remain in Beta is limited, assuring features are not piling up without control. See release stage transition table for more information.
The API is considered stable with care not to break backward compatibility with previous beta releases. This implies that fields may only be added during this stage, not removed or renamed.
The feature presence is controlled using a Feature-Gate (FG) during runtime. It must be specified for the feature to be active.
-
GA: The feature graduated to general-availability (GA) and is now part of the core features.
The API is considered stable with care not to break backward compatibility with the previous releases.
The feature functionality is no longer controlled by a FG.
Warning: A Feature Gate flag is solely intended to control feature lifecycle. It should not be confused and used as a cluster configurable enablement of the functionality. In cases where the cluster admin should control a functionality, regardless to the feature stage, dedicated configuration field/s should be included.
If a feature is targeted for deprecation and retirement, it needs to pass a deprecation process, depending on its current release stage (Alpha, Beta, GA).
For more details, see here.
The following table summarized the different release stages with their transition requirements and restrictions.
| Stage | Period range | F.Gate | Removal Availability |
|---|---|---|---|
| Alpha | 1 to 2 releases | YES | Between minor releases |
| Beta | 1 to 3 releases | YES | Between minor releases |
| GA | - | NO | Between major releases |
Through Alpha and Beta feature releases, a FG must be set in order for the feature to function. By default, no FG is specified, therefore the feature is disabled.
If a feature is not able to transition to the next stage in the defined period, it should be removed automatically.
Note: Exceptions to the period range may apply if 2/3 of active maintainers come to agreement to prolong a specific feature.
One reason for features to go through the Alpha and Beta stages, is the opportunity to examine their usefulness and adoption. Same goes with major releases that intentionally allow breaking backward compatibility (as specified by semver).
Therefore, it is only natural that some features will not graduate between the stages, or will be found irrelevant after some time and be removed when transitioning between major releases.
KubeVirt follows semver versioning, in which major versions may break API compatibility. Therefore, discontinuation of features is somehow simpler when incrementing the major version.
However, this is not without a cost. When a new major release is introduced, the previous one is still maintained and supported, something that does not exist with minor releases.
Only Alpha and Beta features can be removed during a minor release.
These are the steps needed to deprecate & remove a feature:
- Proposal: Prepare a proposal to remove a feature with proper reasoning, phases, exact timelines and functional alternatives (if any). The proposal should be reviewed and approved.
- Notification: Notify the project community about the feature discontinuation based on the approved proposal. All details of the plan should be provided to allow users and possibly down-stream projects to adjust. Use all community media options to distribute this information (e.g. mailing list, slack channel, community meetings).
- Deprecation warnings: Add deprecation warnings at runtime to warn users
that the feature is planned to be removed.
Warnings should be raised when:
- Feature API fields are accessed.
- Feature FG is activated.
- Behavior related to the feature is detected (optional).
- Removal: Feature removal involves removing the core functionality
of a feature and its exposed API.
-
The core implementation can be removed in two steps:
- The FG is removed by assuring it is never reported as set (i.e. even if it is left by the operator configured, internally it is ignored). At this stage, the core implementation will follow the FG conditions and therefore from the outside the feature is inactive.
- In case there are no side effects, the core implementation code can be removed.
-
The API types are not to be removed, as it may have implications with the underlying storage which has already persisted them. Kubernetes has not removed fields, it just kept them around with the warning that they have been deprecated and no longer available.
While keeping fields around for a period of a release or two makes sense, beyond a limited period it adds a burden on dragging leftover fields around to eternity.
-
Note: The only reference seen on why fields should not be removed was mentioned here. But it is unclear if this is relevant for Alpha stage features. Starting with a strict policy, similar to Kubernetes is recommended, i.e. once fields are introduced, they should not be removed no matter the feature release stage. Per need, the topic can be revisited in follow-up adjustments.
While the project strives to maintain a stable contract with its users, there may be scenarios where the policy described here will not be a fit.
Therefore, it should be acceptable to have exceptions from time to time given a very good reasoning and an agreement from 2/3 of the project maintainers (also known as "approvers").
- Add a section in the design proposal template that describes the planned timelines for the feature stages.
- Add a reference to the feature-lifecycle documentation to assure contributors know the process and policy.
- Prepare a user-facing document that describes the usability implications of this feature lifecycle.
-
New features are to be introduced to major and minor release versions only. For clarification, this implies that new features are not to be backport.
-
CI:
- Alpha stage features should not be gating on CI.
- Beta and GA features should be gating on CI.
-
API fields may be marked with the following information:
- Description
- Release stage (alpha/beta/ga) and release version.
It is left to a follow-up implementation proposal to define the exact format and required/optional information.