KEP: Namespace Population #2177

easeway · 2018-05-22T18:10:36Z

Summary

Namespace Population is an automated mechanism to make sure the predefined policy objects
(e.g. NetworkPolicy, Role, RoleBinding) are present in selected namespaces.

Motivation

Kubernetes users create namespaces (self-service namespace creation) and want to
ensure security/isolation policies are enforced between them and other users.
Just like addon manager creates cluster-scope policy objects like PodSecurityPolicy,
ClusterRole, ClusterRoleBinding at cluster creation time.
It creates namespace-scope policy objects during namespace creation time.

This is part of Security Profile proposal

/sig auth
/sig api-machinery
/sig cluster-lifecycle

/cc @davidopp
/cc @tallclair
/cc @ericchiang
/cc @liggitt
/cc @bgrant0607
/cc @roberthbailey
/cc @lavalamp

davidopp · 2018-05-23T01:34:28Z

/cc @bgrant0607

smarterclayton · 2018-05-23T01:42:41Z

keps/draft-20180426-namespace-template.md

+
+Using external admission webhook increases the complexity, and also latency.
+As the webhook must be reliable all the time, it must be deployed in HA.
+It's recommended to create a built-in admission controller for namespace initialization.


I don't think this statement is true. I certainly don't think we should put this admission controller in kube/kube if it can be built with an external admission controller, which I believe it can.

I'm on the fence about whether this feels like a part of Kube or not, but if this is a controller and a CRD and is plugging in to extension, I don't see anything that requires built in web hook admission yet, certainly not until it graduates from beta.

I'm OK with building an external webhook. I will remove this statement.

smarterclayton · 2018-05-23T01:51:18Z

keps/draft-20180426-namespace-template.md

+individually.
+No dynamic objects (generating the object definition during population,
+e.g. string substitution) is supported - all objects are defined statically, 
+except `$(CREATOR)` and `$(NAMESPACE)`.


Please reference prior proposals here on templating. This is still somewhat problematic - you'll need to define what the rules are here, what fields it can be applied to, what happens when it is applied incorrectly (like to an array or to a string key).

smarterclayton · 2018-05-23T01:52:23Z

keps/draft-20180426-namespace-template.md

+    labels: {}
+    # annotations to be injected into all objects defined in templates
+    annotations: {}
+  templates:


This object has some scaling problems - include at minimum a discussion on what would happen at the max limit for templates. Also you'll need to define ordering (is ordering in this significant) in case you didn't already and I just missed it.

@bgrant0607 had suggested that we could put the namespace templates in kube-system namespace. In that case we could use ResourceQuota to protect against creating too many. (His suggestion wasn't related to quota, but I think it would allow quota.) AIUI intention is that only cluster admin can create namespace templates, since they are part of the security profile.

Crazy idea: what if the template was a namespace? I.e. have some way of indicating that a namespace is a template and shouldn't actually be instantiated, and then clone all its objects into new namespaces when the template applies.

EDIT: oops, I see this was already proposed below.

Details in Apply the template section. Will update.

smarterclayton · 2018-05-23T01:52:40Z

keps/draft-20180426-namespace-template.md

+    # excludes lists names of namespaces which must be excluded from being applied 
+    # with current namespace template.
+    excludes: []
+  common:


I'd rather not do this - it's too clever. Users can easily duplicate labels and be precise.

The intention was for other components to specify labels they are interested in when integrating namespace population mechanism, not for users. I will remove it, as it's not necessary for now.

smarterclayton · 2018-05-23T01:52:59Z

keps/draft-20180426-namespace-template.md

+    # labelSelector selects namespaces by labels to populate defined objects
+    labelSelector: {}
+    # excludes lists names of namespaces which must be excluded from being applied 
+    # with current namespace template.


This doesn't scale at all. I don't know that this is worth it.

Maybe this should also be a label selector, so the semantics are "take all namespaces that match labelSelector and then remove any than match excludes"? We have complicated label selection rules in features like node affinity (in, not in, gt, lt, present, absent) and this would be even simpler than that.

Remove the excludes. This is a general object selecting problem which should not belong to here. I will call out in Non-Goals.

smarterclayton · 2018-05-23T01:54:54Z

keps/draft-20180426-namespace-template.md

+namespace from `NamespaceTemplate`, it has no idea who created the namespace.
+To be able to substitute `$(CREATOR)`, a `MutatingAdmissionWebhook` is involved in namespace
+creation request. 
+It adds an annotation `policy/namespace-creator=name` to the namespace being created.


This should be a proper kubernetes namespace, probably "kubernetes.io". I don't see why namespace creation is special here. And I don't know that this is the correct naming "creator" - it probably needs to match the syntax and semantics of the user object.

@smarterclayton Would calling this something like authentication.alpha.kubernetes.io/creator and putting the username, uid, groups, and extra (assuming all of those are available an admission controller) as the value, satisfy what you're asking for?

The resource types in the proposal was originally targeted under policy (which doesn't have k8s.io suffix). Anyway, I can change it to authorization.k8s.io/creator. This can be changed anyway when people agree on something.

smarterclayton · 2018-05-23T01:56:12Z

keps/draft-20180426-namespace-template.md

+To be able to substitute `$(CREATOR)`, a `MutatingAdmissionWebhook` is involved in namespace
+creation request. 
+It adds an annotation `policy/namespace-creator=name` to the namespace being created.
+This annotation is immutable from further update/patch requests.


This is a pretty big deal, it means that any naive code that changes the annotation on namespaces will break when this is introduced. I'm not sure we should be doing a one off "this annotation is special" approach without a bit more discussion on next steps.

The protection is necessary. I will update the details with compatibility into consideration.

smarterclayton · 2018-05-23T01:56:51Z

keps/draft-20180426-namespace-template.md

+A newly created namespace is not ready before all objects in `NamespaceTemplate`
+are fully populated.
+To determine the readiness of a namespace, simply inspect `phase` in `status`
+subresource of the namespace. It's ready when the `phase` is `Active`.


It's not clear to me whether you are proposing adding a new phase.

derekwaynecarr · 2018-05-23T01:52:48Z

keps/draft-20180426-namespace-template.md

+A newly created namespace is not ready before all objects in `NamespaceTemplate`
+are fully populated.
+To determine the readiness of a namespace, simply inspect `phase` in `status`
+subresource of the namespace. It's ready when the `phase` is `Active`.


There are only two phases: active and terminating. Are you saying there will be a third phase?

derekwaynecarr · 2018-05-23T01:54:22Z

keps/draft-20180426-namespace-template.md

+To determine the readiness of a namespace, simply inspect `phase` in `status`
+subresource of the namespace. It's ready when the `phase` is `Active`.
+Alternatively, the `NamespaceTemplate` controller is registered as an _Initialization Controller_
+and uses `Initializer` admission control to hide the namespace before it's fully populated.


Please don’t do this (hide ns)

I agree. Mentioning this as an alternative. We will not go this path.

Unfortunately we can't change phase (it's an enum, so clients won't expect a new phase). See my comments at the bottom - I think we want to adapt the spirit of initializers (which were designed around this use case) without requiring the existing initializer concept (of which this is the primary use case we have, and we can do it as first class on namespace rather than implicit).

derekwaynecarr · 2018-05-23T01:55:34Z

keps/draft-20180426-namespace-template.md

+
+To avoid the situation, a few mechanisms can be used:
+
+##### Creating namespace through service


We have a similar concept in openshift via a feature called projectrequest

After talking with jordan I'm almost convinced we can kill project request off in favor of this, with the security caveats.

derekwaynecarr · 2018-05-23T01:57:05Z

keps/draft-20180426-namespace-template.md

+users/groups the privilege to create namespaces.
+Beyond that, it is the simplest and recommended approach.
+
+##### Use of Initializers


I feel like just creating namespace through delegated service avoids the need for this complexity

With help of a delegated service, it makes the problem simpler. However that requires the delegated service, with concerns in some use cases:

The user experience is inconsistent using delegated service, as in different environment/teams/corps, the delegated services are different

The cluster admin will either spend effort developing their own delegated service or use an existing solution, and they have to educate the users of the cluster about how to create a namespace.

In a self-service namespace creation use case, the users of the cluster want to simply use "kubectl" to create namespaces.

smarterclayton · 2018-05-23T02:00:00Z

keps/draft-20180426-namespace-template.md

+
+#### Opt-out
+
+Cluster administrator is able to opt-out some namespaces from being populated by the controller.


I don't think the mechanism described here is going to fit well in existing clusters.

Since this mechanism does nothing to rate limit or control namespace creation under quota, your proposed solution doesn't really solve "defend against malicious creators", so I don't think namespace template should default to covering all namespaces or attempt to solve that problem. Instead, this proposal is about "automated defaulting", and so it needs to be opt in by the user specifying a label. If the administrator sets a namespace template label that intersects with a system namespace, that's the administrator's fault.

Agree with opt-in

I may be misunderstanding, but I believe that making this opt-in runs counter to the goal here, which is to allow the cluster admin to force policies on new namespaces.

Of course obviously it's opt-in in the sense that if the cluster admin decides not to use the feature, then nothing happens, i.e. no change from today.

It seems that your main objection to this proposal is that it does not address quota for namespace creation. Am I understanding correctly?

smarterclayton · 2018-05-23T02:01:43Z

keps/draft-20180426-namespace-template.md

+The mechanism is agnostic to object types as long as they are namespace scoped.
+The mechanism works in a single cluster.
+
+### Non-Goals


I think you need to add as a non-goal (because this solution can't provide it)

Defend against namespace exhaustion or malicious namespace creators.

and any elements of the proposed solution that overlap should be dropped. Instead it is a defaulting solution that could be composed with a future change that defends namespace creation itself via some pattern or hook.

Self-service namespace creation is a much harder problem than this proposal.

@smarterclayton IIUC when you say "Self-service namespace creation is a much harder problem than this proposal" you are meaning "Self-service namespace creation with certain properties that this proposal does not provide is a much harder problem than this proposal."

Assuming that's what you meant, could you be specific what other properties you had in mind, other than namespace exhaustion? What did you mean by "malicious namespace creators" -- are those malicious namespace creators trying to do something other than namespace exhaustion (for example do you see a way they would be able to create namespaces that don't get populated with the requisite policy objects, or that would allow them to later remove/modify those policy objects)?

smarterclayton · 2018-05-23T02:03:04Z

keps/draft-20180426-namespace-template.md

+
+#### Self-serviced namespace creation
+
+Self-serviced namespace creation allows certain users/groups to create namespaces


I don't think this proposal as written can correctly protect against this, so it either needs to be dropped, split, or a lot more work put into it.

@smarterclayton What is the "this" you're referring to where you say "protect against this"? I think you are referring to something that is not in this sentence, but I wasn't sure exactly what.

Certainly people are doing self-service namespace creation today on stock upstream Kubernetes clusters, so I assume you are referring to doing that with some additional guarantees. Can you specify which guarantees you think this proposal does not cover? The one you have mentioned elsewhere is quota on namespace creation. Are there others?

smarterclayton · 2018-05-23T02:03:29Z

keps/draft-20180426-namespace-template.md

+
+## Graduation Criteria
+
+Namespaces can be securely populated.


This is something this proposal doesn't provide, so this needs to be rethought.

@smarterclayton Is there an aspect of security that is missing other than quota on namespace creation?

easeway · 2018-06-21T18:16:10Z

@smarterclayton could you please clarify:

Is metadata.initializers mechanism (https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/registry/generic/registry/store.go) alpha, same as Initializers admission controller?
Are you proposing adding Initializers into NamespaceSpec, because metadata.initializers is alpha and going to be deprecated?

If yes, then the core API change is required because of adding Initializers field in NamespaceSpec.

smarterclayton · 2018-06-21T18:21:42Z

For clarity, we have generally agreed to remove alpha generic initializers because they don’t offer enough value over webhooks. This is a more targeted namespace-specific lifecycle which is consistent with the special nature of namespaces as a “ACL bucket” that doesn’t apply to other objects. I suppose we could preserve the generic field, but I might argue that each object has its own lifecycle (pods vs ns vs nodes) and so init containers vs namespace initialization are both domain specific. Also, the things that use objects are inextricably linked to how they are initialized - a Kube proxy behaves one way with services as opposed to how an ingress behaves with secrets.

smarterclayton · 2018-06-29T02:03:56Z

I'll open a PR to the initializers proposal to describe the current state and that we desire to deprecate and remove the feature as not having enough value relative to web hooks to justify the additional complexity. I think this proposal covers the core use case we do care about (namespace initialization). In the short term, we could use the alpha initializers field in metadata to test this, and then if we reach beta on it we can move the field out of metadata and into namespace (metadata.initializers -> spec.initializers) as a beta field and then drop metadata.initializers.

Or we could leave the initializers field present, and remove the infrastructure that supports hiding the objects from end users. We would still have a namespace finalizers field, but others could use metadata.initializers as they like with no semantics imposed by the apiserver.

jbeda · 2018-07-03T16:58:51Z

In any case, I think it makes a ton of sense to make this KEP explicit in what the changes to "core" kubernetes are and what is being built outside. I'd suggest perhaps even splitting it into 2 KEPs to ensure that the "core" features can stand on their own.

idealhack · 2018-07-04T08:08:16Z

/ok-to-test

k8s-ci-robot · 2018-07-09T18:55:07Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: jbeda

If they are not already assigned, you can assign the PR to them by writing /assign @jbeda in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

keps/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

easeway · 2018-07-09T18:56:25Z

Updated to explicitly call out this proposal doesn't include anything related to changes in core Kubernetes which should be discussed in a separate proposal.

easeway · 2018-07-20T18:16:06Z

/wg policy

ericavonb · 2018-07-20T21:17:32Z

Can we be explicit about:

What are the exact user problems this proposal aims to solve?
- Can you give some user stories, specific examples of users encountering these issues?
Why can't they be solved with the current set of features in Kubernetes?
- What user requirements are left unmet or are compromised to other needs with past attempts to address this issue?
What external solutions have been tried?
- Why are these attempts insufficient?
- What blocks users from adopting the successful strategies out there?
- Why should we have a standardized solution upstream for this problem in particular?
What are some other ways these problems could be solved within Kubernetes?
- What are the pros and cons to each?
- Why do you recommend this particular solution?

I am especially unclear about no. 2 and 4. It's difficult to evaluate this from a policy management perspective without a better understanding of the exact problem.

re @easeway

easeway · 2018-07-25T19:23:39Z

Requires FeatureGate Initializers to be on:
https://github.com/kubernetes/kubernetes/blob/64bb688ecec332ee4e1ecdad1f3582fb61f82bae/staging/src/k8s.io/apiserver/pkg/registry/rest/create.go#L97

easeway · 2018-08-15T19:00:07Z

@ericavonb Updated motivation section

easeway · 2018-08-15T19:01:25Z

@smarterclayton @jbeda Separate KEP filed containing core Kubernetes changes: #2543 and this KEP doesn't contain anything requiring changes in core Kubernetes.

justaugustus · 2018-10-13T03:41:32Z

/kind kep

mattfarina · 2018-11-05T21:27:11Z

Now that #2543 is in flight, what does that mean for this KEP?

easeway · 2018-11-07T17:58:45Z

@mattfarina #2543 is a dependency of this KEP. #2543 is for core Kubernetes changes required by this feature and this KEP focuses on non-core Kubernetes changes. This can't land without #2543.

justaugustus · 2018-11-20T04:46:34Z

REMINDER: KEPs are moving to k/enhancements on November 30. Please attempt to merge this KEP before then to signal consensus.
For more details on this change, review this thread.

Any questions regarding this move should be directed to that thread and not asked on GitHub.

justaugustus · 2018-12-01T08:11:31Z

KEPs have moved to k/enhancements.
This PR will be closed and any additional changes to this KEP should be submitted to k/enhancements.
For more details on this change, review this thread.

Any questions regarding this move should be directed to that thread and not asked on GitHub.
/close

k8s-ci-robot · 2018-12-01T08:11:32Z

@justaugustus: Closed this PR.

In response to this:

KEPs have moved to k/enhancements.
This PR will be closed and any additional changes to this KEP should be submitted to k/enhancements.
For more details on this change, review this thread.

Any questions regarding this move should be directed to that thread and not asked on GitHub.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot requested review from bgrant0607, davidopp, ericchiang, lavalamp, liggitt, roberthbailey and tallclair May 22, 2018 18:10

easeway mentioned this pull request May 22, 2018

KEP: Security Profile Design Proposal #2052

Closed

easeway force-pushed the kep-namespace-template branch from 14c552f to d3fb413 Compare May 22, 2018 20:48

smarterclayton reviewed May 23, 2018

View reviewed changes

derekwaynecarr reviewed May 23, 2018

View reviewed changes

smarterclayton reviewed May 23, 2018

View reviewed changes

k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 4, 2018

easeway force-pushed the kep-namespace-template branch from aa13f64 to dc7e429 Compare July 9, 2018 18:54

jdumars mentioned this pull request Jul 12, 2018

KEP #TBD: Namespace Population kubernetes-retired/architecture-tracking#22

Closed

k8s-ci-robot added the wg/policy Categorizes an issue or PR as relevant to WG Policy. label Jul 20, 2018

easeway mentioned this pull request Aug 15, 2018

KEP: Namespace Initializer #2543

Closed

KEP: Namespace Population

4b8eab5

easeway force-pushed the kep-namespace-template branch from dc7e429 to 4b8eab5 Compare August 15, 2018 18:59

k8s-ci-robot added the kind/kep label Oct 13, 2018

kkasravi mentioned this pull request Oct 28, 2018

create a self-serve component for data-scientists kubeflow/kubeflow#1872

Merged

k8s-ci-robot closed this Dec 1, 2018

easeway mentioned this pull request Dec 5, 2018

KEP: Namespace Initializer kubernetes/enhancements#645

Closed


		To avoid the situation, a few mechanisms can be used:

		##### Creating namespace through service


		#### Opt-out

		Cluster administrator is able to opt-out some namespaces from being populated by the controller.


		#### Self-serviced namespace creation

		Self-serviced namespace creation allows certain users/groups to create namespaces

KEP: Namespace Population #2177

KEP: Namespace Population #2177

Conversation

easeway commented May 22, 2018

Summary

Motivation

davidopp commented May 23, 2018

smarterclayton May 23, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tallclair May 24, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

easeway May 30, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

easeway May 30, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

easeway commented Jun 21, 2018 • edited Loading

smarterclayton commented Jun 21, 2018 via email • edited Loading

smarterclayton commented Jun 29, 2018

jbeda commented Jul 3, 2018

idealhack commented Jul 4, 2018

k8s-ci-robot commented Jul 9, 2018

easeway commented Jul 9, 2018

easeway commented Jul 20, 2018

ericavonb commented Jul 20, 2018

easeway commented Jul 25, 2018

easeway commented Aug 15, 2018

easeway commented Aug 15, 2018

justaugustus commented Oct 13, 2018

mattfarina commented Nov 5, 2018

easeway commented Nov 7, 2018

justaugustus commented Nov 20, 2018

justaugustus commented Dec 1, 2018

k8s-ci-robot commented Dec 1, 2018

smarterclayton May 23, 2018 •

edited

Loading

tallclair May 24, 2018 •

edited

Loading

easeway May 30, 2018 •

edited

Loading

easeway May 30, 2018 •

edited

Loading

easeway commented Jun 21, 2018 •

edited

Loading

smarterclayton commented Jun 21, 2018 via email •

edited

Loading