Heath Check Deployment is not Compatible with KIND

[abhisek]

KIND nodes by default uses containerd as the CRI directly instead of Docker

kubelet command line:

/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=remote --container-runtime-endpoint=/run/containerd/containerd.sock --fail-swap-on=false --node-ip= --fail-swap-on=false

The same scenario can expose containerd.sock but exploitation will be a bit different. Either talk to it directly or start a dockerd on containerd.sock and then use docker client.

Error from Pod events:

Events:
  Type     Reason       Age                  From                   Message
  ----     ------       ----                 ----                   -------
  Normal   Scheduled    3m38s                default-scheduler      Successfully assigned default/health-check-deployment-54fd854474-gvhmq to kind-worker3
  Warning  FailedMount  95s                  kubelet, kind-worker3  Unable to attach or mount volumes: unmounted volumes=[docker-sock-volume], unattached volumes=[docker-sock-volume default-token-kvh25]: timed out waiting for the condition
  Warning  FailedMount  90s (x9 over 3m38s)  kubelet, kind-worker3  MountVolume.SetUp failed for volume "docker-sock-volume" : hostPath type check failed: /var/run/docker.sock is not a file

[madhuakula]

Yup. this is one of the limitation with KIND for this scenario. Will try to solve with KIND, for now https://katacoda.com/madhuakula/scenarios/kubernetes-goat

If you have a PR, I am happy to review & merge. As this is know issue and part of the ROADMAP to fix

[mihir-shah99]

Will work on this issue, please assign this to me

[macagr]

This is not a fix, but an extremely hacky way to enable the same attack vector using KIND: create a KIND node using extraMounts, mounting the docker.sock of the host (assuming you're using docker as your host container runtime). The YAML description would be something like:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes: 
  - role: control-plane
    extraMounts:
      - hostPath: /var/run/docker.sock
        containerPath: /var/run/docker.sock

Then, comment out the type: File from the deployment.yaml file in /scenarios/health-check/ (not entirely sure why this is needed, but I'm guessing the type of a mounted hostname volume is not a file?). Then, using the same attack technique you should be able to see all the containers running in your own host, not on the node unfortunately, but as I mentioned, it works for educational purposes.

If I have more time later I may play around a bit more to see if it is possible to do something a bit less hacky: I am thinking of using, e.g., ansible, to configure nodes to install Docker whenever you're running local instances like KIND or Minikube.

[madhuakula]

Hey @macagr

Thank you so much for the suggestion. I will try this out and see if it works.

Please feel free to try it out and let me know. If you think there is a potentially better approach please feel free to make a pull request.

Cheers!

[macagr]

Hey @madhuakula,

Thanks! I'll keep you informed if I find a way to make the implementation as agnostic as possible to the node container runtime.

Edit: Just added a PR enabling Kind support using this idea.

Cheers!

[madhuakula]

Thank you @macagr 👍

Closing this issue #20