Forbidden node labels and annotations

[MaxFedotov]

Currently, we can allow tenant admins to modify nodes using capsule-proxy if appropriate permission is set in tenant CRD.

While this is a great option for them, which adds additional flexibility, there are some types of labels or annotations, which must be protected from modifications (for example, which are set by cloud-providers or autoscalers)

My proposal is to add an ability for cluster administrators to protect some labels or annotations from modification by tenant owners (the same, as it is done with namespaces)

If the cluster administrator wants to disallow tenant owner to add some labels or annotations on nodes, he had to add one of the following annotations on a tenant:

• capsule.clastix.io/forbidden-nodes-labels
• capsule.clastix.io/forbidden-nodes-labels-regexp
• capsule.clastix.io/forbidden-nodes-annotations
• capsule.clastix.io/forbidden-nodes-annotations-regexp

When new capsule api will be release, these annotations will be moved to the new NodeOptions struct (together with nodeSelector:

type NodeOptions struct {
        // Specifies the label to control the placement of pods on a given pool of worker nodes. All namespaces created within the Tenant will have the node selector annotation. This annotation tells the Kubernetes scheduler to place pods on the nodes having the selector label. Optional.
	NodeSelector map[string]string `json:"nodeSelector,omitempty"`
	// Specifies forbidden a labels the Tenant owner can't place on Nodes resources in the Tenant. Optional.
	ForbiddenLabels *ForbiddenListSpec `json:"forbiddenLabels,omitempty"`
	// Specifies forbidden annotations the Tenant owner can't place on Nodes resources in the Tenant. Optional.
	ForbiddenAnnotations *ForbiddenListSpec `json:"forbiddenAnnotations,omitempty"`
}

@bsctl @prometherion WDYT?

[prometherion]

I think this is a good improvement, nice catch.

Do you think we could address this for v0.1.1 release?

[MaxFedotov]

@prometherion yes, i think will be able to make a PR next week :)

[MaxFedotov]

After discussion with @prometherion decided to move these annotations to CapsuleConfiguration and later add them to the CapsuleConfigurationSpec:

type CapsuleConfigurationSpec struct {
	// Names of the groups for Capsule users.
	// +kubebuilder:default={capsule.clastix.io}
	UserGroups []string `json:"userGroups,omitempty"`
	// Enforces the Tenant owner, during Namespace creation, to name it using the selected Tenant name as prefix,
	// separated by a dash. This is useful to avoid Namespace name collision in a public CaaS environment.
	// +kubebuilder:default=false
	ForceTenantPrefix bool `json:"forceTenantPrefix,omitempty"`
	// Disallow creation of namespaces, whose name matches this regexp
	ProtectedNamespaceRegexpString string `json:"protectedNamespaceRegex,omitempty"`
        // Specifies options for the Node resources, such as forbidden labels and annotations. Optional.
        NodeOptions *NodeOptions `json:"nodeOptions,omitempty"`
}

type NodeOptions struct {
	// Specifies forbidden labels the Tenant owners can't place or modify on Nodes resources. Optional.
	UserLabels *ForbiddenListSpec `json:"userLabels,omitempty"`
	// Specifies forbidden annotations the Tenant owners can't place or modify on Nodes resources Optional.
	UserAnnotations *ForbiddenListSpec `json:"userAnnotations,omitempty"`
}

The reason was that each user can have multiple tenants, and in the previous scenario, each of these tenants can have a different set of forbidden annotations. And as nodes can be shared between tenants, it will be impossible to find out which labels\annotations are set as forbidden for a node.