From d385032e3e7d3fd8958b5b3ac11f2fca197d7ab1 Mon Sep 17 00:00:00 2001 From: duanmengkk Date: Thu, 8 Feb 2024 13:47:25 +0800 Subject: [PATCH] Proposal: add proposal for implement k8s in k8s by kosmos Signed-off-by: duanmengkk --- .../k8sink8s/img/Kubernetes_in_Kubernetes.svg | 1 + .../k8sink8s/kubernetes_inkubernetes.md | 77 +++++++++++++++++++ 2 files changed, 78 insertions(+) create mode 100644 docs/proposals/k8sink8s/img/Kubernetes_in_Kubernetes.svg create mode 100644 docs/proposals/k8sink8s/kubernetes_inkubernetes.md diff --git a/docs/proposals/k8sink8s/img/Kubernetes_in_Kubernetes.svg b/docs/proposals/k8sink8s/img/Kubernetes_in_Kubernetes.svg new file mode 100644 index 000000000..571dab63d --- /dev/null +++ b/docs/proposals/k8sink8s/img/Kubernetes_in_Kubernetes.svg @@ -0,0 +1 @@ +k8s-apiserver 6444k8s-controller-managerk8s-scheduleretcdkosmosk8s-apiserver 6445k8s-controller-managerk8s-scheduleretcdkosmosk8s-apiserver 6446k8s-controller-managerk8s-scheduleretcdkosmosUser A（2* 2C 16G 1GPU）PODNodeIP： 192.168.0.1规格： 48C256G10GPUMasterMasterMasterPODNodeIP： 192.168.0.2规格： 48C256G10GPUUser B（4* 2C 16G 1GPU）User C（2* 2C 16G 4GPU） \ No newline at end of file diff --git a/docs/proposals/k8sink8s/kubernetes_inkubernetes.md b/docs/proposals/k8sink8s/kubernetes_inkubernetes.md new file mode 100644 index 000000000..ece36d0b5 --- /dev/null +++ b/docs/proposals/k8sink8s/kubernetes_inkubernetes.md @@ -0,0 +1,77 @@ +# Summary + +Kosmos now supports the management of different Kubernetes (k8s) clusters, offering a unified control plane view for resource management and scheduling. This proposal introduces a method to achieve Kubernetes within Kubernetes (k8s in k8s) using Kosmos. + +The approach involves creating different Kubernetes control plane services within a single k8s cluster and deploying Kosmos' ClusterTree service. The ClusterTree listens to the kube-apiserver address of the created k8s control plane and incorporates the current cluster as a node into the dynamically created control plane. Each control plane has its exclusive etcd, ensuring the isolation of permissions and data for different tenants. + +# Motivation + +In some cases, cloud providers offer a large k8s cluster as infrastructure to different tenants, providing computational resources like CPU and GPU. Therefore, a k8s in k8s solution is required to isolate permissions and data for different tenants. + +## Goals + +Divide the computational power of a single k8s cluster into multiple distinct k8s clusters, achieving permission and data isolation for different tenants. + +## Non-Goals + +# Proposal + +Implement a k8s in k8s solution with Kosmos, allocating a single cluster's computational resources to different tenants. + +## Function Overview + +1. Enable splitting a single cluster into different smaller clusters through Kosmos. +2. Provide tenants with exclusive etcd and k8s control planes to ensure complete isolation of permissions and resources. +3. Ensure tenants' control planes offer the same functionalities as a single k8s control plane, without changing the user experience. + +## User Stories (Optional) + +### Story 1 + +Imagine a supercluster with tens of thousands of nodes, hosting many different tenants. These tenants do not require all the computational resources of the entire k8s cluster, and for security reasons, it's crucial to ensure complete permission isolation between tenants. The native RBAC permission isolation solution in k8s might be insufficient, and k8s's reliance on namespaces for resource isolation could alter the user experience, restricting operations to the namespace level, which could be user-unfriendly. + +# Design Details + +## Overall Architecture + +![Kubernetes_in_Kubernetes.svg](img/Kubernetes_in_Kubernetes.svg) + +As shown in the figure, each tenant has a separate k8s control plane and etcd. Each tenant's k8s control plane uses Kosmos to manage the nodes of the current cluster as virtual nodes for the control plane. The CRD configuration determines the number of nodes from the current cluster each control plane should manage. + +## CRD Design + +```yaml +apiVersion: kosmos.io/v1alpha1 +kind: VirtualCluster +metadata: + name: tenant1-cluster +spec: + kubeconfig: XXXXXX + promoteResources: + nodes: + - node1 + - node2 + resources: + cpu: 10 + memory: 100Gi +status: + phase: Initialized +``` + +## Overall Process + +1. A cluster administrator creates a VirtualCluster CR resource. +2. The Kosmos-operator listens for the creation of the VirtualCluster, starts the creation of the Kubernetes control plane services including the kube-apiserver, kube-controller-manager, kube-scheduler, and etcd. Once the Kubernetes control plane services are setup, the VirtualCluster's status.phase will be set to Initialized, and the spec.kubeconfig field will be populated. +3. After the control plane is set up, the Kosmos-operator creates the Kosmos clusterTree service and configures the clusterTree's apiserver address to the current control plane using the kubeconfig value from the VirtualCluster. +4. Once the clusterTree is ready, the Kosmos-operator manages the tenant's required node resources or computational resources like CPU and memory into the tenant's control plane based on the VirtualCluster's promoteResource configuration. Tenants can then manage resources in the control plane using the spec.kubeconfig. + +# Test Plan + +## Unit Test + +## E2E Test + +| Test Case | Case Description | +| ----------------------------- | ------------------------------------------------------------- | +| Virtual Control Plane Creation | Test the creation of a virtual control plane through the VirtualCluster CRD | +| Service Creation in Virtual Control Plane | Test the creation of services in the virtual control plane and their external functionality |