Skip to content

Latest commit

 

History

History
218 lines (166 loc) · 14 KB

operatorgroups.md

File metadata and controls

218 lines (166 loc) · 14 KB
Raw

Operator Multitenancy with OperatorGroups

An OperatorGroup is an OLM resource that provides rudimentary multitenant configuration to OLM installed operators.

OperatorGroup Overview

OperatorGroup Membership

An operator defined by CSV csv-a is said to be a member of OperatorGroup op-a in namespace ns-a if both of the following hold:

  • op-a is the only OperatorGroup in ns-a
  • csv-a's InstallModes support op-a's target namespace set

TooManyOperatorGroups

If there exists more than one OperatorGroup in a single namespace, any CSV created in that namespace will transition to a failure state with reason TooManyOperatorGroups. CSVs in a failed state for this reason will transition to pending once the number of OperatorGroups in their namespaces reaches one.

InstallModes and Supported OperatorGroups

An InstallMode consists of an InstallModeType field and a boolean Supported field. A CSV's spec can contain a set of InstallModes of four distinct InstallModeTypes:

  • OwnNamespace: If supported, the operator can be a member of an OperatorGroup that selects its own namespace
  • SingleNamespace: If supported, the operator can be a member of an OperatorGroup that selects one namespace
  • MultiNamespace: If supported, the operator can be a member of an OperatorGroup that selects more than one namespace
  • AllNamespaces: If supported, the operator can be a member of an OperatorGroup that selects all namespaces (target namespace set is the empty string "")

Note: If a CSV's spec omits an entry of InstallModeType, that type is considered unsupported unless support can be inferred by an existing entry that implicitly supports it.

UnsupportedOperatorGroup

If a CSV's InstallModes do not support the target namespace selection of the OperatorGroup in its namespace, the CSV will transition to a failure state with reason UnsupportedOperatorGroup. CSVs in a failed state for this reason will transition to pending once either the OperatorGroups's target namespace selection changes to a supported configuration, or the CSV's InstallModes are modified to support the OperatorGroup's target namespace selection.

Target Namespace Selection

Select the set of namespaces by specifying a label selector with the spec.selector field:

apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
  name: my-group
  namespace: my-namespace
spec:
  selector:
    matchLabels:
      cool.io/prod: "true"

or by explicitly naming target namespaces with the spec.targetNamespaces field:

apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
  name: my-group
  namespace: my-namespace
spec:
  targetNamespaces:
  - my-namespace
  - my-other-namespace
  - my-other-other-namespace

Note: If both spec.targetNamespaces and spec.selector are defined, spec.selector is ignored.

Additionally, a global OperatorGroup (which selects all namespaces) is specified by omitting both spec.selector and spec.targetNamespaces:

apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
  name: my-group
  namespace: my-namespace

The resolved set of selected namespaces is surfaced in an OperatorGroup's status.namespaces field. A global OperatorGroup's status.namespace is of length 1 and contains the empty string, "", which signals a consuming operator that it should watch all namespaces.

Note: The consuming operator must know to treat "" as an all namespace configuration.

OperatorGroup CSV Annotations

Member CSVs of an OperatorGroup get the following annotations:

  • olm.operatorGroup=<group-name>
    • Contains the name of the OperatorGroup
  • olm.operatorGroupNamespace=<group-namespace>
    • Contains the namespace of the OperatorGroup
  • olm.targetNamespaces=<target-namespaces>
    • Contains a comma-delimited string listing the OperatorGroup's target namespace selection. This annotation is projected onto the pod template of a CSV's deployments and can be consumed by a pod instance via The Downward API

Note: All annotations except olm.targetNamespaces are included with copied CSVs. Omitting the olm.targetNamespaces annotation on copied CSVs prevents the names of target namespaces from being leaked between tenants.

Provided APIs Annotation

Information about what GroupVersionKindss (GVK) are provided by an OperatorGroup are surfaced in an olm.providedAPIs annotation. The annotation's value is a string consisting of a set of <Kind>.<version>.<group>s delimited with commas. The GVKs of CRDs and APIServices provided by all active member CSVs of an OperatorGroup are included.

Here's an example of an OperatorGroup with a single active member CSV providing the PackageManifests resource:

apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
  annotations:
    olm.providedAPIs: PackageManifest.v1alpha1.packages.apps.redhat.com
  name: olm-operators
  namespace: local
  ...
spec:
  selector: {}
  serviceAccount:
    metadata:
      creationTimestamp: null
  targetNamespaces:
  - local
status:
  lastUpdated: 2019-02-19T16:18:28Z
  namespaces:
  - local

RBAC

When an OperatorGroup is created, 3 ClusterRoles each containing a single AggregationRule are generated:

  • <operatorgroup-name>-admin

    • ClusterRoleSelector set to match the olm.opgroup.permissions/aggregate-to-admin: <operatorgroup-name> label

  • <operatorgroup-name>-edit

    • ClusterRoleSelector set to match the olm.opgroup.permissions/aggregate-to-edit: <operatorgroup-name> label

  • <operatorgroup-name>-view

    • ClusterRoleSelector set to match the olm.opgroup.permissions/aggregate-to-view: <operatorgroup-name> label

When a CSV becomes an active member of an OperatorGroup and is not in a failed state with reason InterOperatorGroupOwnerConflict, the following RBAC resources are generated:

  • For each provided API resource from a CRD:

    • A <kind.group-version-admin> ClusterRole is generated with the * verb on <group> <kind> with aggregation labels rbac.authorization.k8s.io/aggregate-to-admin: true and olm.opgroup.permissions/aggregate-to-admin: <operatorgroup-name>
    • A <kind.group-version-edit> ClusterRole is generated with the create, update, patch, delete verbs on <group> <kind> with aggregation labels rbac.authorization.k8s.io/aggregate-to-edit: true and olm.opgroup.permissions/aggregate-to-edit: <operatorgroup-name>
    • A <kind.group-version-view> ClusterRole is generated with the get, list, watch verbs on <group> <kind> with aggregation labels rbac.authorization.k8s.io/aggregate-to-view: true and olm.opgroup.permissions/aggregate-to-view: <operatorgroup-name>
    • A <kind.group-version-view-crd> ClusterRole is generated with the get verb on apiextensions.k8s.io customresourcedefinitions <crd-name> with aggregation labels rbac.authorization.k8s.io/aggregate-to-view: true and olm.opgroup.permissions/aggregate-to-view: <operatorgroup-name>

  • For each provided API resource from an APIService:

    • A <kind.group-version-admin> ClusterRole is generated with the * verb on <group> <kind> with aggregation labels rbac.authorization.k8s.io/aggregate-to-admin: true and olm.opgroup.permissions/aggregate-to-admin: <operatorgroup-name>
    • A <kind.group-version-edit> ClusterRole is generated with the create, update, patch, delete verbs on <group> <kind> with aggregation labels rbac.authorization.k8s.io/aggregate-to-edit: true and olm.opgroup.permissions/aggregate-to-edit: <operatorgroup-name>
    • A <kind.group-version-view> ClusterRole is generated with the get, list, watch verbs on <group> <kind> with aggregation labels rbac.authorization.k8s.io/aggregate-to-view: true and olm.opgroup.permissions/aggregate-to-view: <operatorgroup-name>

  • For CSV in the global OperatorGroup:

    • A ClusterRole and corresponding ClusterRoleBinding are generated for each permission defined in the CSV's permissions field. All resources generated are given the olm.owner: <csv-name> and olm.owner.namespace: <csv-namespace> labels

  • Else for each target namespace:

    • All Roles and RoleBindings in the operator namespace with the olm.owner: <csv-name> and olm.owner.namespace: <csv-namespace> labels are copied into the target namespace.

Copied CSVs

OLM will create copies of all active member CSVs of an OperatorGroup in each of that OperatorGroup's target namespaces. The purpose of a Copied CSV is to tell users of a target namespace that a specific operator is configured to watch resources created there. Copied CSVs have a status reason Copied and are updated to match the status of their source CSV. The olm.targetNamespaces annotation is stripped from copied CSVs before they are created on the cluster. Omitting the target namespace selection avoids an unnecessary information leak. Copied CSVs are deleted when their source CSV no longer exists or the operator group their source CSV belongs to no longer targets the copied CSV's namespace.

Static OperatorGroups

An OperatorGroup is static if it's spec.staticProvidedAPIs field is set to true. As a result, OLM does not modify the OperatorGroups's olm.providedAPIs annotation, which means that it can be set in advance. This is useful when a user wishes to use an OperatorGroup to prevent resource contention in a set of namespaces, but does not have active member CSVs that provide the APIs for those resources.

Here's an example of an OperatorGroup that "protects" prometheus resources in all namespaces with the something.cool.io/cluster-monitoring: "true" annotation:

apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
  name: cluster-monitoring
  namespace: cluster-monitoring
  annotations:
    olm.providedAPIs: Alertmanager.v1.monitoring.coreos.com,Prometheus.v1.monitoring.coreos.com,PrometheusRule.v1.monitoring.coreos.com,ServiceMonitor.v1.monitoring.coreos.com
spec:
  staticProvidedAPIs: true
  selector:
    matchLabels:
      something.cool.io/cluster-monitoring: "true"

OperatorGroup Intersection

OperatorGroup Intersection Terminology

  • Two OperatorGroups are said to be intersecting if the intersection of their target namespace sets is not the empty set
  • Two OperatorGroups are said to have intersecting provided APIs if they are intersecting and the intersection of their provided API sets (defined by olm.providedAPIs annotations) is not the empty set

What Can Go Wrong?

OperatorGroups with intersecting provided APIs can compete for the same resources in the set of intersecting namespaces.

Rules for Intersection

Each time an active member CSV syncs, OLM queries the cluster for the set of intersecting provided APIs between the CSV's OperatorGroup and all other OperatorGroups. OLM then checks if that set is the empty set:

  • If true and the CSV's provided APIs are a subset of the OperatorGroup's:
    • Continue transitioning
  • If true and the CSV's provided APIs are not a subset of the OperatorGroup's:
    • If the OperatorGroup is static:
      • Clean up any deployments that belong to the CSV
      • Transition the CSV to a failed state with status reason CannotModifyStaticOperatorGroupProvidedAPIs
    • Else:
      • Replace the OperatorGroup's olm.providedAPIs annotation with the union of itself and the CSV's provided APIs
  • If false and the CSV's provided APIs are not a subset of the OperatorGroup's:
    • Clean up any deployments that belong to the CSV
    • Transition the CSV to a failed state with status reason InterOperatorGroupOwnerConflict
  • If false and the CSV's provided APIs are a subset of the OperatorGroup's:
    • If the OperatorGroup is static:
      • Clean up any deployments that belong to the CSV
      • Transition the CSV to a failed state with status reason CannotModifyStaticOperatorGroupProvidedAPIs
    • Else:
      • Replace the OperatorGroup's olm.providedAPIs annotation with the difference between itself and the CSV's provided APIs

Note: Failure states caused by OperatorGroups are non-terminal.

Note: When checking intersection rules, an OperatorGroup's namespace is always included as part of its selected target namespaces.

Each time an OperatorGroup syncs:

  • The set of provided APIs from active member CSV's is calculated from the cluster (ignoring copied CSVs)
  • The cluster set is compared to olm.providedAPIs:
    • If olm.providedAPIs contains any extraneous provided APIs:
      • olm.providedAPIs is pruned of any extraneous provided APIs (not provided on cluster)
  • All CSVs that provide the same APIs across all namespaces (including those removed) are requeued. This notifies conflicting CSVs in intersecting groups that their conflict has possibly been resolved, either through resizing or through deletion of the conflicting CSV.