Skip to content

Latest commit

 

History

History
227 lines (177 loc) · 8.91 KB

20200220-cluster-resource-set.md

File metadata and controls

227 lines (177 loc) · 8.91 KB
title authors reviewers creation-date last-updated status
ClusterResourceSet
@sedefsavas
@vincepri
@detiber
@ncdc
@fabriziopandini
2020-02-20
2020-05-11
experimental

ClusterResourceSet

Table of Contents

Glossary

Refer to the Cluster API Book Glossary.

Summary

Provide a mechanism for applying resources in a cluster once it is created.

Motivation

Clusters created by Cluster API are minimally functional. For instance,they do not have a container networking interface (CNI), which is required for pod-to-pod networking, or any StorageClasses, which are required for dynamic persistent volume provisioning. Users today must manually add these components to every cluster they create.

Having a mechanism to apply an initial set of default resources after clusters are created makes clusters created with Cluster API functional and ready for workloads from the beginning, without requiring additional user intervention.

To achieve this, ClusterResourceSet CRD is introduced that will be responsible for applying a set resources defined by users to the matching clusters.

Goals

  • Provide a means to specify a set of resources to apply automatically to newly-created and existing Clusters. Resources will be applied only once.
  • Support additions to the resource list by applying the new added resources to both new and existing matching clusters.
  • Support both json and yaml resources.

Non-Goals/Future Work

  • Replace or compete with the Cluster Addons subproject.
  • Support deletion of resources from clusters. Deleting a resource from a ClusterResourceSet or deleting a ClusterResourceSet does not result in deletion of those resources from clusters.
  • Lifecycle management of the installed resources (such as CNI).
  • Support reconciliation of resources on resource hash change and/or periodically. This can be a future enhancement work.

Proposal

User Stories

Story 1

As someone creating multiple clusters, I want all my clusters to have a CNI provider of my choosing installed automatically, so I don’t have to manually repeat the installation for each new cluster.

Story 2

As someone creating multiple clusters, I want all my clusters to have a StorageClass installed automatically, so I don't have to manually repeat the installation for each new cluster.

Story 3

As someone creating multiple clusters, I want to be able to provide different values for some fields in the resources for different clusters. For example, CNIs podCIDRs may be required to be distinct for each cluster, hence some templating mechanism for variable substitution in the resources is needed.

Implementation Details/Notes/Constraints

Data model changes to existing API types

None. We are planning to implement this feature without modifying any of the existing structure to minimize the footprint of ClusterResourceSet Controller. This enhancement will follow Kubernetes’s feature-gate structure and will be under the experimental package with its APIs, and enabled/disabled with a feature gate.

ClusterResourceSet Object Definition

This is the CRD that has a set of components to be applied to clusters that match the label selector in it.

The resources field is a list of secrets/configmaps in the same namespace. The clusterSelector field is a Kubernetes label selector that matches against labels on clusters (only the clusters in the same namespace with the ClusterResourceSet resource). ClusterResourceSet is namespace-scoped, all resources and clusters needs to be in the same namespace as the ClusterResourceSet. Sample ClusterResourceSet YAML

---
apiVersion: cluster.x-k8s.io/v1alpha3
kind: ClusterResourceSet
metadata:
 name: crs-conf
 namespace: default
spec:
 mode: "ApplyOnce"
 clusterSelector:
   matchLabels:
     postcreatelabelcni: calico
 resources:
   - name: calico-addon
     kind: Secret
   - name: network-policy-addon
     kind: ConfigMap
status:
 LastUpdated:  "2020-05-05T08:24:17Z"

Initially, the only supported mode will be ApplyOnce and it will be the default mode if no mode is provided. In the future, we may consider adding a Reconcile mode that reapplies the resources on resource hash change and/or periodically. If ClusterResourceSet resources will be managed by an operator after they are applied by ClusterResourceSet controller, "ApplyOnce" mode must be used so that reconciliation on those resources can be delegated to the operator.

Each item in the resources specifies a kind (must be either ConfigMap or Secret) and a name. Each referenced ConfigMap/Secret contains yaml/json content as value.

Sample Secret Format

apiVersion: v1
kind: Secret
metadata:
  name: db-secret
type: clusterresourceset
stringData:
  value: |-
    kind: Secret
    apiVersion: v1
    metadata:
     name: mysql-secret

For preventing all secrets to be reached by all clusters in a namespace, only secrets with type "clusterresourceset" can be accessed by ClusterResourceSet controller.

Sample ConfigMap Format

apiVersion: v1
kind: ConfigMap
metadata:
  name: calico-addon
data:
  value: |-
    kind: ConfigMap
    apiVersion: v1
    metadata:
     name: calico-conf

The resources in ClusterResourceSet will be applied to matching clusters. There is many-to-many mapping between Clusters and ClusterResourceSets: Multiple ClusterResourceSets can match with a cluster; and multiple clusters can match with a single ClusterResourceSet. To keep information on which resources applied to which clusters, a new CRD is used, ClusterResourceSetBinding will be created in the management cluster. There will be one ClusterResourceSetBinding per workload cluster.

Example:

apiVersion: v1
kind: ClusterResourceBinding
metadata:
  name: <cluster-name>
  namespace: <cluster-namespace>
clusterresourcesets:
  <ClusterResourceSet-name1>:   
    <secret-name1>:
      kind: Secret
      hash: <>
      status: success
      error: ""
      lastAppliedTime: "2020-04-05T08:24:17Z"
    <configmap-name1>:
      kind: ConfigMap
      hash: <>
      status: failed
      error: "some error"
      lastAppliedTime: "2020-05-05T08:24:17Z"
  <ClusterResourceSet-name2>:  
    <secret-name2>:
      kind: Secret
      hash: <>
      status: success
      error: ""
      lastAppliedTime: "2020-04-05T08:24:17Z"
Status: InProgress/Completed

Status will be Completed when all matching ClusterResourceSet reconciles are completed for that cluster. In case of new resource addition to a matching ClusterResourceSet, Status becomes InProgress Also, the errors / overall progress will be tracked in the ClusterResourceSet’s status.

Risks and Mitigations

Alternatives

The Alternatives section is used to highlight and record other possible approaches to delivering the value proposed by a proposal.

Upgrade Strategy

This is an experimental feature supported by a new CRD and controller so there is no need to handle upgrades for existing clusters.

Additional Details

Test Plan [optional]

Extensive unit testing for all the cases supported when applying ClusterResourceSet resources. e2e testing as part of the cluster-api e2e test suite.

Graduation Criteria [optional]

This proposal will follow all maturity stages (alpha, beta, GA) and then may be merged with cluster-api apis and controllers.

Implementation History