RFD - Start moving services into their own Kubernetes Namespaces

[Adam-D-Lewis]

Status	Open for comments 💬
Author(s)	@Adam-D-Lewis
Date Created	29-08-2024
Date Last updated	29-08-2024
Decision deadline	dd-MM-YYY

Start moving services into their own Kubernetes Namespaces

Summary

Services integrated into Nebari often suggest putting the service in it's own namespace when used with Kubernetes (e.g. Rook Ceph, Argo Workflows). Jupyterhub also has the option to put users in their own namespace. So far we've kept everything in a single namespace. The idea being that if someone wants to deploy Nebari into an existing Kubernetes cluster they can be assured that by default everything is deployed in a single namespace.

Benefit

Putting services and potentially users in their own namespaces brings some advantages around controlling resource limits on users. It also allows us to follow the best practices recommended by the services themselves.

It can also help with some aspects of RBAC permissions. E.g. All pods in a namespace have all the the permissions granted to the default service account (assuming automountServiceToken: true). This is a smaller benefit in my mind. We currently don't use the default service account.

Design Proposal

I propose we move Argo Workflows, Keycloak, Rook-Ceph, Monitoring (Prometheus/Grafana), potentially Jupyterhub each into their own namespaces. We should ensure the namespace is configurable so users could put everything in a single namespace if they wish, but by default we separate these things out.

Alternatives or approaches considered (if any)

Best practices

Services recommend being put in their own namespace. It also gives better resource controls on various services and potentially Nebari users.

User impact

Ideally none, there may be some special instructions/manual steps required around upgrading Nebari b/c of these changes.

Unresolved questions

• What changes will need to be made to support services in their own namespace? I think it will be minimal for most services with the possible exception of Juyterhub.

[viniciusdc]

I do like this approach, and I would prefer we opted it as default behavior. We already have control over the services endpoint injection in each helm chart, as well as, the infra resources itself. So, we could have the existing deployment option default to single namespaces.

[viniciusdc]

Services accounts such as the jupyterhub one would possibly require access to the user's namespaces, so that would need to use some level of tagging system or similar if that exists.

[Adam-D-Lewis]

Copying over some internal discussion here.

Dharhas Pothina
The earlier issue about doing that was the idea that some orgs wouldn't be able to do a wildcard dns entry.
so it would need be optional.

Vinicius Cerutti
I think the main goal is to separate the resource at the kubernetes level to separate namespaces, the endpoints entries should, in theory, be kept the same. (edited)
my single concern is with deployments on existing infra, where the number of namespaces and their access might be limited. (edited)

Adam Lewis
Yes, that's what I'm proposing Vinicius Cerutti. My understanding is the single namespace design originally was intended for orgs wanting to do a deployment into an existing k8s cluster. At least they could have some peace of mind that everything is going into a single namespace. However, it doesn't seem like that is very common at all. Do we know of anyone doing this currently? It seems like not the best to ignore best practices for a scenario which isn't really happening.

Adam Lewis
We could continue to support single namespace deployments, but just make the default multi namespace. (edited)

Adam Lewis
Most of the CRDs, cluster roles are for services like Prometheus, Argo, etc. I think making it easier for orgs doing existing k8s Nebari deployments to use their own e.g. Prometheus deployment, Argo Deployment, etc. with Nebari if they have one would be a better way. (edited)

Chuck McAndrew
I am in favor, but we should expose ways to set the namespaces for each service so that if an org wants everything in one, we can do that

Adam Lewis
Yeah, I think keeping namespace configurable is totally reasonable.

[krassowski]

On meeting today there was also a mention of whether this has an interaction with:

• enabling per-user subdomains in JupyterHub https://jupyterhub.readthedocs.io/en/stable/explanation/websecurity.html#enable-user-subdomains
• c.KubeSpawner.enable_user_namespaces https://discourse.jupyter.org/t/how-to-create-their-own-namespace-for-each-user/8039