kublet metrics not available with k8s 1.24

[a-thaler]

Description
It seems that something changed with the kubelet metrics on k8s 1.24.
The prow job post-main-k3d-k8s-preview got updated to 1.24 and the monitoring tests are failing as metric container_memory_usage_bytes returns an empty result.

Please check what is going on and assure that the metrics are available on old and new k8s version.

See all failing jobs https://status.build.kyma-project.io/?job=post-main-k3d-k8s-preview, a specific one https://status.build.kyma-project.io/view/gs/kyma-prow-logs/logs/post-main-k3d-k8s-preview/1547222455657959424 with the actual message:

  1) Executing Standard Testsuite:
       Prometheus Tests:
         Metrics used by Kyma Dashboard should exist:
     AssertionError: Following queries return no results: topk(10,container_memory_usage_bytes{container=~"..*"})metric from service monitor: monitoring-kubelet: expected [ 'topk(10,container_memory_usage_bytes{container=~"..*"})metric from service monitor: monitoring-kubelet' ] to be empty
      at assertTimeSeriesExist (monitoring/prometheus.js:280:10)
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (internal/process/task_queues.js:95:5)
      at async Object.assertMetricsExist (monitoring/prometheus.js:141:9)
      at async Context.<anonymous> (monitoring/index.js:56:7)

Expected result

Actual result

Steps to reproduce

Troubleshooting

[a-thaler]

The problem is K3S specific, on a gardener/gcp cluster the metric is returning results

[a-thaler]

Looks like the default containerd socket is not matching anymore. Setting --kubelet-arg containerd=/run/k3s/containerd/containerd.sock on k3s seem to fix the issue, also read k3s-io/k3s#473 and k3s-io/k3s#2831

So running the provision command with kyma provision k3d -k 1.24.2 -s --kubelet-arg=containerd=/run/k3s/containerd/containerd.sock@"all:*" fixed the problem local.

[a-thaler]

The problem must be related to the switch from k8s 1.23 to k8s 1.24. Here the most prominent change is the removal of the dockershim. When using k3s (even with k3d), containerd is used and it should not have an impact. Also the default setting of the containerd socket seem to be the same as the recommended arg.
As the setting is fixing the problem and it is anyway the default, it seems safe to just configure it now as a default param to circumvent the problem. I assume it is a bug in upstream as it was already the case twice (see the linked tickets).

I attached a PR for setting the param by default