Add support for Rancher desktop

[99hops]

What is the new feature about?

I want to use Gefyra with Rancher desktop (containerd+k3s) but they don't work nicely together. Basically gefyra up can't connect in any way I've tried to the running k8s. I've tried many different combinations of --endpoint and without endpoint with no luck.
Here is a sample:

$ gefyra up --endpoint 192.168.5.15:31820 --kubeconfig=~/.kube/config
[INFO] Installing Gefyra Operator
[INFO] Created network 'gefyra' (b28c846440)
[INFO] Container image "quay.io/gefyra/operator:0.13.1" already present on machine
[INFO] Pulling image "quay.io/gefyra/stowaway:0.13.1"
[INFO] Successfully pulled image "quay.io/gefyra/stowaway:0.13.1" in 613.015103ms
[INFO] Operator became ready in 3.4344 seconds
[INFO] Deploying Cargo (network sidecar) with IP 192.168.96.149
[ERROR] Gefyra could not successfully confirm the Wireguard connection working. Please make sure you are using the --endpoint argument for remote clusters and that 192.168.5.15:31820 can reach Kubernetes node port 31820 from this machine. Please check your firewall settings, too. If you are running a local Minikube cluster, please use the 'gefyra up --minikube' flag.
[INFO] Removing running bridges
[INFO] Uninstalling Operator
^[[A[INFO] Removing Cargo
[INFO] Removing Docker network gefyra

Why would such a feature be important to you?

Gefyra + Rancher desktop will be a great stack

Anything else we need to know?

No response

[SteinRobert]

Hey @nforced ! Thank you for sharing your experience and this particular issue you ran into! This is super valuable to improve the developer experience of more and more developers working with Kubernetes by using Gefyra.

I suspect the port 31820 of your cluster nodes (or more specific I believe rancher-desktop uses lima) is not reachable from your host. However, I haven't worked with rancher-desktop yet. I guess it could just be a setting missing in rancher-desktop. Do you see any setting to open this port for the VM your cluster runs in? Maybe this helps. If it does we could just document it in the Gefyra documentation.

[99hops]

Hi @SteinRobert, yes it's using lima. There are no settings in rancher-desktop that can make a difference. What I see is upon gefyra up in Port forwarding section it create two new Gefyra entries one of which is gefira-wireguard which is assigned a random port and when I try to change it to 31820 it displays port 31820 already in use. This is visible only while gefyra up and lasts till if fail. Please give it a try as you know better how to debug this (*: I am looking for a way to ditch docker-desktop and migrate to rancher-desktop and this is a blocker.

[Schille]

I could run Gefyra with Rancher-Desktop, although it was not apparent to me. I am not an expert in RD nor in Lima. Some of my findings might be obvious to you.

First, I noticed that my docker CLI was not compatible with the containerd socket provided by RD:

> docker context use rancher-desktop
rancher-desktop
Current context is now "rancher-desktop"

> docker ps
error during connect: Get "http://%2Fhome%2Fmschilonka%2F.rd%2Fdocker.sock/v1.24/containers/json": read unix @->/home/mschilonka/.rd/docker.sock: read: connection reset by peer

When using nerdctl shipped by RD I could actually talk (nerdctl ps) to the container runtime.

Indeed, RD starts a qemu VM (I am on Linux):

> ps -ef | grep qemu-system-x86_64
mschilo+   91367   91342 57 18:09 ?        00:00:58 /usr/bin/qemu-system-x86_64 -m 5120 -cpu host -machine q35,accel=kvm -smp 2,sockets=1,cores=2,threads=1 -boot [...]

I switched in RD the container runtime to dockerd/moby and RD restarted the Kubernetes cluster (potentially K8s gets reset during this operation).
Now I switched back to the default docker context:

> docker context use default
default
Current context is now "default"

(docker ps is working again)

Now the following worked out:

> gefyra up --endpoint=192.168.5.15:31820
[INFO] Installing Gefyra Operator
[INFO] Created network 'gefyra' (02d5f5e0d5)
[INFO] Pulling image "quay.io/gefyra/operator:0.13.1"
[INFO] Successfully pulled image "quay.io/gefyra/operator:0.13.1" in 9.607257114s
[INFO] Pulling image "quay.io/gefyra/stowaway:0.13.1"
[INFO] Successfully pulled image "quay.io/gefyra/stowaway:0.13.1" in 13.927913411s
[INFO] Operator became ready in 29.3126 seconds
[INFO] Deploying Cargo (network sidecar) with IP 172.18.0.149

> gefyra status
{
  "summary": "Gefyra is up and connected",
  "cluster": {
    "connected": true,
    "operator": true,
    "operator_image": "quay.io/gefyra/operator:0.13.1",
    "stowaway": true,
    "stowaway_image": "quay.io/gefyra/stowaway:0.13.1",
    "namespace": true
  },
  "client": {
    "version": "0.13.1",
    "cargo": true,
    "cargo_image": "gefyra-cargo:20221107182146",
    "network": true,
    "connection": true,
    "containers": 0,
    "bridges": 0,
    "kubeconfig": "~/.kube/config",
    "context": "rancher-desktop",
    "cargo_endpoint": "192.168.5.15:31820"
  }
}

It seems Rancher-Desktop does not assign an external IP address to the Kubernetes nodes:

> kubectl get node -o wide
NAME                   STATUS   ROLES                  AGE   VERSION        INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION   CONTAINER-RUNTIME
lima-rancher-desktop   Ready    control-plane,master   27m   v1.24.7+k3s1   192.168.5.15   <none>        Alpine Linux v3.16   5.15.64-0-virt   containerd://1.6.8

On the one hand, the qemu VM does not install a routable network interface on my host, so it seems difficult to reach these containers. On the other hand, it's a pitty that Gefyra does not work properly with the containerd runtime coming with RD.

That's all I have for the moment. 🤔

[99hops]

Interesting, this is what I think as well it's just communication problem between the two parts. And yes, containerd is what I am looking to move over with RD and remove docker forever (:

[SteinRobert]

Also took a look into this.
Basically it looks like we could potentially make use of this package:
https://github.com/gabrieldemarmiesse/python-on-whales
It wraps the docker client, podman and nerdctl into a common api (more or less).

However - I am a bit hesitant doing this without further thought. It seems docker-py cannot simply talk to the containerd socket on one's machine (which makes sense I guess)....

[99hops]

As I think about it maybe Wireguard is an overkill for local development where everything is setup on local machine and adds complexity in cases like this one.

[SteinRobert]

We talked about this in the team. We're currently not able to create a client, talking to containerd directly - right now we're using the Docker client Python implementation docker-py which itself cannot talk to the containerd socket directly.

Furthermore - as you stated - it seems it is not possible to expose the Rancher Desktop VM to the host system as needed for Gefyra. Maybe Rancher could extend this. However, we cannot add Rancher VM (lima) manipulation stuff into Gefyra - that would be out of scope.

I don't think the issue should end here - it could be very interesting in the future. If there is enough interest or even a feature sponsor we would be willing to put this on our roadmap again. This issue will be moved to discussion.

Please upvote - in case this is a relevant use case for you.