Skip to content

Commit

Permalink
Address frisso comments
Browse files Browse the repository at this point in the history
  • Loading branch information
@claudiolor
claudiolor committed Nov 15, 2024
1 parent acdd4bd commit f4b8613
Showing 1 changed file with 74 additions and 51 deletions.
125 changes: 74 additions & 51 deletions docs/advanced/peering/peering-via-cr.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
Declarative peerings are supported starting from Liqo 1.0, which means that you can create a set of CRs describing the peering with a remote cluster, which is automatically set up once the CRs are applied on both sides.
This simplifies automation, GitOps and continuous delivery: for example, you might have your Git repository with the manifests describing the peerings, and an instance of [ArgoCD](https://argo-cd.readthedocs.io) which synchronizes the changes on the clusters, creating and destroying the peerings.

The following lines analizes how to describe the configuration of each of the Liqo modules:
This documentation page analizes how to declaratively configure each of the Liqo modules:

- Networking
- Authentication
Expand All @@ -17,13 +17,31 @@ This tenant namespace must **refer to the peering with a specific cluster**, hen
A tenant namespace **can have an arbitrary name**, but **it must have the following labels**:

```text
liqo.io/remote-cluster-id: <HERE_THE_CLUSTER_ID_OF_PEER_CLUSTER>
liqo.io/remote-cluster-id: <PEER_CLUSTER_ID>
liqo.io/tenant-namespace: "true"
```

### Configuring the tenant namespace on cluster consumer
Where `PEER_CLUSTER_ID` is the cluster id of the peer cluster defined at installation time, it can be obtained by launching the following command on the **remote cluster**:

`````{tab-set}
````{tab-item} liqoctl
```bash
liqoctl info --get clusterid
```
````
````{tab-item} kubectl
```bash
kubectl get configmaps -n liqo liqo-clusterid-configmap \
--template {{.data.CLUSTER_ID}}
```
````
`````

### Configuring the tenant namespace on consumer cluster

The following is an example of tenant namespace referring to the peering with a cluster with id `cl-provider`:
The following is an example of tenant namespace named `liqo-tenant-cl-provider`, which refers to the peering with a cluster with id `cl-provider`:

```yaml
apiVersion: v1
Expand All @@ -36,10 +54,9 @@ metadata:
spec: {}
```
### Configuring the tenant namespace on the cluster provider
### Configuring the tenant namespace on the provider cluster
**You will need a similar namespace on the provider cluster** pointing to the cluster ID of your local cluster (cluster consumer).
For example, if the cluster consumer has cluster ID `cl-consumer`, its tenant namespace might look like the following:
The following is an example of tenant namespace on the provider cluster, named `liqo-tenant-cl-consumer`, which refers to the peering with the consumer cluster with id `cl-consumer`:

```yaml
apiVersion: v1
Expand All @@ -52,15 +69,18 @@ metadata:
spec: {}
```

```{admonition} Note
When peering is configured, the names of the tenant namespaces of the provider and consumer clusters do not have to be equal or follow any pattern.
```

## Declarative network configuration

By default, the network connection between clusters is established using a secure channel created via [Wireguard](https://www.wireguard.com/).
To allow the creation of the tunnel, one cluster, typically the provider, needs to host a server gateway that exposes a port for the client gateway (usually running on the consumer cluster) to connect to.

Therefore, it is required that the server gateway is reachable from the peer cluster, to enable the creation of the tunnel between the two clusters.
In this case, one cluster (usually the provider) needs to host a server gateway that exposes a UDP port that must be reachable from the client gateway (usually running on the consumer cluster).

In this guide, **we will configure the client gateway on the consumer cluster and the server gateway on the provider cluster**, which is the most common setup.
However, depending on your requirements, you may choose to reverse these roles.

However, given that the setup of the network peering is independent from the offloading role of the cluster (i.e., consumer vs. provider), you may choose to invert the client/server roles in case this is more convenient for your setup.

### Creating and exchanging the network configurations (both clusters)

Expand Down Expand Up @@ -97,7 +117,7 @@ echo "Private key:"; cat private.der | tail -c 32 | base64
echo "Public key:"; cat public.der | tail -c 32 | base64
```

At this point, **in each cluster**, we will need to create a secret containing this pair of keys:
At this point, you need to create **in each cluster** a secret containing this pair of keys:

```{code} yaml
apiVersion: v1
Expand Down Expand Up @@ -137,19 +157,19 @@ liqo.io/remote-cluster-id: <HERE_THE_CLUSTER_ID_OF_PEER_CLUSTER>
networking.liqo.io/gateway-resource: "true"
```

### Configuring the server gateway (cluster provider)
### Configuring the server gateway (provider cluster)

As mentioned before, a Wireguard tunnel connects the clusters peered with Liqo.
In this section we are going to configure the gateway server, where the client will connect to.
By default, a Wireguard tunnel connects the clusters peered with Liqo.
This section shows how to configure the gateway server, where the client will connect to.

The `GatewayServer` resource describes the configuration of the gateway server, and should be applied on the cluster acting as server for the tunnel creation.
You can check [here](./inter-cluster-network.md#creation-of-a-gateway-server) an example of the `GatewayServer` CR.

When you create the `GatewayServer` resource, **make sure to specify the `secretRef`** pointing to the key pairs we created before.

Note that under `.spec.endpoint` of the `GatewayServer` resource you can configure a fixed `nodePort` or `loadBalancerIP` (if supported by your provider) to have a deterministic port or ip address for the gateway, so that the configuration of the client that connects to it can be defined in advance.
Note that under `.spec.endpoint` of the `GatewayServer` resource you can configure a fixed `nodePort` or `loadBalancerIP` (if supported by your provider) to have a precise UDP port or IP address for the gateway, so that the configuration of the client that connects to it can be defined in advance.

The following is an example of the `GatewayServer` resource, configured in the cluster provider, exposing the gateway using `NodePort` on port `30742`:
The following is an example of the `GatewayServer` resource, configured in the provider cluster, exposing the gateway using `NodePort` on port `30742`:

```{code} yaml
apiVersion: networking.liqo.io/v1beta1
Expand Down Expand Up @@ -177,12 +197,12 @@ spec:
Where:

- `CONSUMER_CLUSTER_ID` is the cluster ID of the consumer, where the gateway client runs;
- `PROVIDER_TENANT_NAMESPACE` is the tenant namespace on the cluster provider, where, in thie case, we are configuring the gateway server;
- `WIREGUARD_KEYS_SECRET_NAME` is the name of the secret with the Wireguard key paris we created before.
- `PROVIDER_TENANT_NAMESPACE` is the tenant namespace on the provider cluster, where, in this case, we are configuring the gateway server;
- `WIREGUARD_KEYS_SECRET_NAME` is the name of the secret with the Wireguard key pairs we created before.

### Configuring the client gateway (cluster consumer)
### Configuring the client gateway (consumer cluster)

The other cluster, in this case the consumer, needs to run the client gateway connecting to the service exposed by the cluster provider.
The other cluster, in this case the consumer, needs to run the client gateway connecting to the service exposed by the provider cluster.

The `GatewayClient` resource describes the configuration of the gateway client, **it should contain the parameters** to connect to service exposed by the gateway server.

Expand Down Expand Up @@ -216,9 +236,9 @@ spec:
Where:

- `PROVIDER_CLUSTER_ID` is the cluster ID of the provider, where the gateway server is running;
- `CONSUMER_TENANT_NAMESPACE` is the tenant namespace on the cluster consumer, where, in thie case, we are configuring the gateway client;
- `WIREGUARD_KEYS_SECRET_NAME` is the name of the secret with the Wireguard key paris we created before;
- `REMOTE_IP`: is the IP address of one of the nodes of the cluster provider, as we configured a `NodePort` service. If the service was a `LoadBalancer` the ip would be the one of the load balancer.
- `CONSUMER_TENANT_NAMESPACE` is the tenant namespace on the consumer cluster, where, in this case, we are configuring the gateway client;
- `WIREGUARD_KEYS_SECRET_NAME` is the name of the secret with the Wireguard key pairs we created before;
- `REMOTE_IP`: is the IP address of one of the nodes of the provider cluster, as we configured a `NodePort` service. If the service was a `LoadBalancer` the IP would be the one of the load balancer.

### Summary of network configuration

Expand All @@ -228,19 +248,20 @@ To sum up, to set up the network, **both clusters need**:
- a `Secret` containing the Wireguard public and private keys
- a `PublicKey` with the Wireguard public key **of the peer cluster**

The **cluster provider will have a `GatewayServer`** resource
In addition:

The **cluster consumer a `GatewayClient` resource** connecting to the peer Gateway server.
- the **provider cluster will have a `GatewayServer`** resource
- the **consumer cluster a `GatewayClient` resource** connecting to the peer Gateway server.

Once you applied all the required resources, the client should be able to connect to the server and create the tunnel.

You can get the `Connection` resource to check the status of the tunnel, as shown [here](./inter-cluster-network.md#connection-crds).

## Declarative configuration of clusters authentication

In this section we are going to configure the authentication between the clusters, allowing the cluster consumer to ask for resources.
This section shows how to configure the authentication between the clusters, allowing the consumer cluster to ask for resources.

When authentication is manually configured, **the user is in charge of providing the credentials with the permission required** by the cluster consumer to operate.
When authentication is manually configured, **the user is in charge of providing the credentials with the permission required** by the consumer cluster to operate.

If your are not familiar with how authentication works in Kubernetes you can check [this documentation page](https://kubernetes.io/docs/reference/access-authn-authz/authentication/).
You can also check [here to know how to issue a certificate for a user](https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/#normal-user).
Expand All @@ -250,13 +271,13 @@ Note that with EKS authentication via client certificate [is not directly suppor
You can check [here](https://docs.aws.amazon.com/eks/latest/userguide/cluster-auth.html) how access control works in eks.
```

### Cluster consumer role binding (cluster provider)
### consumer cluster role binding (provider cluster)

Once we created the credentials the consumer can work with, we will need to provide the minimum permission required by the consumer to operate.
Note that **the cluster consumer will never directly create workloads on the remote cluster** and, at this stage, it should have only the permissions to create the liqo resources to ask for the approval of a `ResourceSlice`.
Once we created (in the provider cluster) the credentials the consumer can work with, we need to provide the minimum permission required by the consumer to operate.
Note that **the consumer cluster will never directly create workloads on the remote cluster** and, at this stage, it should have only the permissions to create the liqo resources to ask for the approval of a `ResourceSlice`.

To do so, we will need to bind the newly created user to the `liqo-remote-controlpane` role.
This can be done **by creating the following `RoleBinding` resource in the tenant namespace of the cluster provider**:
To do so, you need to bind the newly created user to the `liqo-remote-controlpane` role.
This can be done **by creating the following `RoleBinding` resource in the tenant namespace of the provider cluster**:

```{code} yaml
apiVersion: rbac.authorization.k8s.io/v1
Expand All @@ -276,14 +297,14 @@ subjects:
name: <USER_COMMON_NAME>
```

Where, when the user authenticate via certificate signed by the cluster CA, `USER_COMMON_NAME` is the CN field of the certificate.
where, when the user authenticates via a certificate signed by the cluster CA, `USER_COMMON_NAME` is the `CN` field of the certificate.

### Creation of a tenant for the cluster consumer (cluster provider)
### Creation of a tenant for the consumer cluster (provider cluster)

On the provider side, to allow the authentication of a consumer, we will need to create a `Tenant` resource for it.
This resource is useful to control the remote consumer (e.g. if we the provider wants to prevent a remote consumer to negotiate more resources, it can set the tenant condition to `Cordoned`, stopping any other resources negotiation).
This resource is useful to control the remote consumer (e.g. if the provider would like to prevent a remote consumer to negotiate more resources, it can set the tenant condition to `Cordoned`, stopping any other resources negotiation).

Note that, in the case of declarative configuration, there won't be any handshake between the clusters, so we will need to configure the tenant so that it accepts the `ResourceSlice` of the given consumer, even though no handshake occurred.
Note that, in the case of declarative configuration, there will not be any handshake between the clusters, so we will need to configure the tenant so that it accepts the `ResourceSlice` of the given consumer, even though no handshake occurred.
This can be done setting `TolerateNoHandshake` as `authzPolicy` like in the following example:

```{code} yaml
Expand All @@ -300,10 +321,10 @@ spec:
tenantCondition: Active
```

### Add the credentials on the cluster consumer (cluster consumer)
### Add the credentials on the consumer cluster (consumer cluster)

The previously created credentials on the cluster provider, should be provided to the cluster consumer.
To do so, you should create a `Secret` containing the kubeconfig with the credentials to operate on the cluster provider, having the following labels:
The previously created credentials on the provider cluster should be given to the consumer cluster.
To do so, you should create a `Secret` containing the kubeconfig with the credentials to operate on the provider cluster, having the following labels:

```{code} yaml
liqo.io/identity-type: ControlPlane
Expand All @@ -316,7 +337,7 @@ and annotation:
liqo.io/remote-tenant-namespace: <PROVIDER_TENANT_NAMESPACE>
```

Where the `PROVIDER_TENANT_NAMESPACE` is the tenant namespace that we created on the cluster provider for the peering with this consumer.
Where the `PROVIDER_TENANT_NAMESPACE` is the tenant namespace that we created on the provider cluster for the peering with this consumer.

The following is an example of identity secret:

Expand Down Expand Up @@ -351,23 +372,25 @@ I1107 10:05:29.355741 1 reflector.go:163] [cl02] Reflection of authenticat

### Summary of authentication configuration

To sum up, to set up the authentication, **on the cluster provider** you will need to:
To sum up, to set up the authentication, **on the provider cluster** you will need to:

- Create the credentials to be used by the consumer
- Bind the credentials to the `liqo-remote-controlplane` role
- Create a `Tenant` resource for the consumer

While, **on the consumer side**:

- Create a new secret containing the kubeconfig with the credentials to access to the cluster provider
- Create a new secret containing the kubeconfig with the credentials to access to the provider cluster

## Declarative configuration of namespace offloading

While offloading is independent from the network, which means that it is possible to negotiate resources and configure a namespace offloading without the inter-cluster network enabled, **a [working authentication configuration](#declarative-configuration-of-clusters-authentication) is a pre-requisite to enable offloading**.

### Ask for resources: configure a ResourceSlice
### Ask for resources: configure a ResourceSlice (consumer cluster)

The `ResourceSlice` resource is the CR that defines the computational resources requested by the consumer to the provider cluster.
It should be created on the tenant namespace of the consumer cluster, and it is automatically forwarded to the provider cluster, which can accept or reject it.

The `ResourceSlice` resource is the one to be created to ask a cluster provider for resources.
The following is an example of `ResourceSlice`:

```{code} yaml
Expand All @@ -381,7 +404,7 @@ metadata:
liqo.io/remote-cluster-id: <PROVIDER_CLUSTER_ID>
liqo.io/remoteID: <PROVIDER_CLUSTER_ID>
name: test
namespace: liqo-tenant-cl02
namespace: <CONSUMER_TENANT_NAMESPACE>
spec:
class: default
providerClusterID: <PROVIDER_CLUSTER_ID>
Expand All @@ -390,7 +413,7 @@ spec:
ram 128Gi
```

With the configuration above, once the resources are accepted from the provider side, a new (virtual) node, impersonating the provider cluster, will make available the requested resorces on the consumer cluster.
If the request above is successfully accepted by the provider, a new (virtual) node, impersonating the provider cluster, will make available the requested resorces on the consumer cluster.

To know more about `ResourceSlice` and `VirtualNode` check [this section of the documentation](./offloading-in-depth.md#create-resourceslice).

Expand All @@ -412,17 +435,17 @@ spec:
podOffloadingStrategy: LocalAndRemote
```

The `NamespaceOffloading` resource should be created in the namespace that we want to extend on the remote clusters.
The `NamespaceOffloading` resource should be created in the namespace that we would like to extend on the remote clusters.

For example, the resource above, extends the `demo` namespace on all the configured provider clusters.
As the policy is `LocalAndRemote` the pod will be executed locally or remotely and it depends on the choice made by the the Kubernetes scheduler.
Since the `podOffloadingStrategy` policy is `LocalAndRemote`, the a new pod could be executed either locally or remotely depending on the choice made by the vanilla Kubernetes scheduler (e.g., if the remote virtual node is plenty of free resources, it may be preferred against local nodes that are already used by other pods).

[Check here](../../usage/namespace-offloading.md#namespace-mapping-strategy) to know more about namespace offloading.

```{warning}
Currently, **there is a caveat with the namespace offloading preventing the pods created before the creation of the `NamespaceOffloading` resource to be scheduled on a remote cluster**.
Currently, the `NamespaceOffloading` resource **must be created before scheduling a pod on a remote cluster**.

For example, if we configure a pod to run on the remote cluster, but pod is created before the `NamespaceOffloading`, that pod remains in a `Pending` state, also after the namespace is actually offloaded.
For example, if we configure a pod to run on the remote cluster, but the pod is created before setting the `NamespaceOffloading` resource, that pod will remain in a `Pending` state forever, also after the namespace is actually offloaded.

Therefore, **make sure to offload a namespace** before starting scheduling pods on it.
```

0 comments on commit f4b8613

Please sign in to comment.