New DevWorkspace CRD

[l0rd]

This epic is about using a Kubernetes Custom Resource to provision Che Workspaces and transferring some of Che Master responsibilities to a Kubernetes controller.

Current architecture

Che Workspaces are currently provisioned by Che Master: a Java application that exposes a secured REST API to manage Che workspaces.

To create/start/stop a workspace a client sends an HTTP request to wsmaster. Some examples of the wsmaster clients are the Che User Dashboard (Che landing page), chectl (che command line tool) or the OpenShift Dev Console.

Note that the Workspace API is not the only API exposed by the Che Master API. Other APIs include auth, factory, ssh, metrics, user.

Che Master is stateful and data are persisted in a Postgresql database. Persisted data includes workspaces metadata and users preferences and data.

New architecture

We want to move the part of the Che Master that is responsible for scheduling workspace containers to a Kubernetes CRD and controller. This has a few benefits:

1. Lightweight and Simpler: delegating to Kubernetes auth (through RBAC) and persistence (using etcd key-value store)
2. Part of the Kubernetes API: makes it easy for developers that already know the Kubernetes API to provision a Che workspace
3. Adhere to the operator pattern: that will allow for a more advanced workspace control like configuring workspaces based on users geographic location or automatically disable plugins that are deteriorate workspaces performances.

Authentication and Authorization

In multi-user mode, requests to the Che Master API need to be authenticated and authorized as before. Nothing should change for the Che Master API auth.

Authentication to the Workspaces services should continue to work as it is currently. With the JWT Proxy securing communications at Workspaces level.

What changes with the introduction of the DevWorkspace CRD is that the provisioning of the JWT Proxy Pod and Workspaces ingresses/routes will be the responsibility of a Kubernetes controller rather then the Che Master.

The PoC of the DevWorkspace already comes with one example of such a controller (for OpenShift OAuth). The controller is associated with a secondary CRD called DevWorkspaceRouting (managed by the DevWorkspace operator as the primary CRD). A new controller named DevWorkspaceRoutingController should be implemented to support the current Che Workspaces auth model with the JWT Proxy.

Deployment

The deployment will be done through a new DevWorkspace operator. The current Che operator should be updated to programmatically subscribe to the DevWorkspace operator when a user subscribe to it. That will guarantee that the creation of a CheCluster CRD will be always associated to the creation of a DevWorkspaces CRD.

FAQ

How do we see the CRD workspace controller

The controller has the responsibility for reconciling a DevWorkspace spec and status with one or more pods implementing the workspace.

What are the logical relation and technical connections between CRD controller and workspace master

The CRD controller will be a pre-requirement of the wsmaster. The wsmaster will need the controller to create the workspace abstraction. The CRD controller instead can be deployed and be operational even without a wsmaster.

What is the scope of the responsibilities of the Che Master?

The scope of the Che Master would still be the same but instead of low level Kubernetes API (like pods, deployment, etc..) it will manage higher level custom resources (DevWorkspace CR but also other CRs as detailed in the following answers about users, orgs, etc ...):

• Create the CRs and let the CR controller reconcile them
• React to CR status change (asynchronously as much as possible)
• Follow the events of CRs to complete the logs, etc ..., feed monitoring and tracing, etc ...
• Implement WSMaster Rest APIs by Managing CRs under-the-cover (Create, list, patch, etc ...)

Will the wsmaster or API server continue exposes a REST API?

No reason why the API would change. But the implementation of the API would change to use the DevWorkspace CRs under-the-cover.

What is the scope of the responsibilities of CRD controller?

Reconcile the DevWorkspace Custom Resources and create the corresponding Deployment and associated Kubernetes objects. It mainly replaces low-level parts of the Kubernetes or Openshift infrastructure implementation.

How do we see authentication/authorization in a CRD controller/workspace master future?

The main idea would be to use:

• A secondary DevWorkspaceRouting CRD with a customizable secondary controller that enables customizing the creation of ingresses / routes + the related authentication method
  This was already implemented in the POC with an example using OpenShift authentication through oauth-proxy
• RBAC rules for authorization

How do we see Che organization/groups/resource management?

Che user would mainly become a cluster-wide Custom Resource (DevWorkspaceUser), which would also allow storing preferences.
The same could be true for Organization (DevWorkspaceUsersOrganisation) if there are specific settings to store on them.
These CRs should be linked to the underlying K8S users / groups so that RBAC rules can then be used to manage and check permissions.

What are the steps of integration CRD controller into Che?

1st Phase:

• Replace the current Kubernetes infrastructure by creating a DevWorkspace CR and reacting to the reconciliation by the CRD controller
• Create a DevWorkspaceRouting controller that exposes the workspace with the jwt-proxy-based authentication provided by Keycloak

2nd phase:
Replace Users and profiles (and possibly other data) stored in the Postgres database by cluster-wide CRs, and manage them accordingly in the Che Master. We should be able to get rid of Postgres at this point.

3rd phase:

• Add alternative types of authentication (especially use the authentication provided by the K8S platform) in the wsmaster (and provide alternative implementations of DevWorkspaceRouting controllers)
• Modify the Dashboard, and other client applications (IDE loader, etc...), in order to remove Keycloak-based authentication, and use the authentication currently configured in the wsmaster.
  We should be able to remove Keycloak or replace it with other alternate authentication method.

Are we going to get rid of Keycloak?

Keycloak should at least become an option among others. But the priority, in terms of scalability, and management burden, is more about removing Postgres and have everything stored as CRs in ETCD.

Are we going to get rid of PostgreSQL?

That's the idea yes. Currently we don't even support safe rolling-update of the postgres database. It doesn't seem to be very scalable.

Are we going to get rid of something else?

No. But the REST API could become a Quarkus application, and thus even benefit from serverless scale-to-0 features. Not getting rid of it, but at least some interesting refactoring work.

Can a Che workspace be provisioned and run without the Che Master? In other words, once we get rid of Keycloak and Postgres, can we envision a lighter Che installation with Kubernetes CRDs and controllers only?

Yes, it’s already the case in the existing POC. However, in order to correctly support the Che-Theia component, we still need to have access to a minimal implementation of the https://github.com/eclipse/che-workspace-client client API. However it isn’t a problem and is already provided by an optional extra-simple sidecar that mainly provides this required endpoints by just reading the devfile and other fields on the DevWorkspace Custom resource.
Of course this is only useful in order to transparently support che-theia in the current state of the overall Che architecture.

In the perspective to use Che to implement a Kubernetes Cloud shell: in what namespace is the DevWorkspace CR been created? What if that namespace doesn’t exist? What if it already exists with resources already running there? If the user opens 2 tabs will we have 2 workspaces or only one?

See the CloudShell epic

[l0rd]

@skabashnyuk @sleshchenko please review
cc @davidfestal @gorkem

[alexeykazakov]

I would like to get the authZ side clarified...
Currently Che uses the user's openshift token (in case of OpenShift auth) to create kube resources such as deployments, routes, etc when starting a workspace. It requires users to have permissions in the workspace namespace to create all these resources.

But when Workspace CRD & Controller are introduced it will work in a traditional operator/controller way:

• User has to have edit permissions (*) for Workspace CRDs in the target namespace. So user can create a Workspace CR in the target namespace. But user doesn't have to have any permissions for deployments, routes, etc in that namespace.
• Workspace Controller will reconcile the created Workspace CR and will create/update the deployments, routes, etc using its own Service Account which will have all needed permissions in the namespaces where the operator is installed (one more reason to support all-namespaces installation mode ;)

It means it will be possible to grand users permissions to manipulate with Workspace CRs only. So, they can't create their own deployments. Such users still be able to start workspaces.

Is this the plan?

[kfox1111]

Do keep in mind too that k8s does not have a mechanism the same as openshift tokens.

[alexeykazakov]

Do keep in mind too that k8s does not have a mechanism the same as openshift tokens.

OpenShift uses the same RBAC for controllers and CRs as Kube. So, the flow I described for Workspace Controller is the same for OpenShift and Kube.
So, not sure what you mean by openshift specific token mechanism in this context.

[davidfestal]

I would like to get the authZ side clarified...
Currently Che uses the user's openshift token (in case of OpenShift auth) to create kube resources such as deployments, routes, etc when starting a workspace. It requires users to have permissions in the workspace namespace to create all these resources.

But when Workspace CRD & Controller are introduced it will work in a traditional operator/controller way:

• User has to have edit permissions (*) for Workspace CRDs in the target namespace. So user can create a Workspace CR in the target namespace. But user doesn't have to have any permissions for deployments, routes, etc in that namespace.
• Workspace Controller will reconcile the created Workspace CR and will create/update the deployments, routes, etc using its own Service Account which will have all needed permissions in the namespaces where the operator is installed (one more reason to support all-namespaces installation mode ;)

It means it will be possible to grand users permissions to manipulate with Workspace CRs only. So, they can't create their own deployments. Such users still be able to start workspaces.

Is this the plan?

That's totally the plan yes.
And in fact the workspace CRD controller POC already had support for OperatorHub deployment (demonstrated at the last EclipseCon), and the related CSV already use the all-namespaces installation mode.

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.

[gattytto]

The controller has the responsibility for reconciling a DevWorkspace spec and status with one or more pods implementing the workspace.

I am very interested in this part, where a devWorkspace is consisted of more than one pod, for example to handle different architectures. but will it be possible to make it invisible to the containers in those pods and still have them act as if they were all part of the same pod?. Like, having a pipe socket connection between 2 containers in separate pods