Use socket with unix:/// in CAPD.

[sbueringer]

We are currently using /var/run/containerd/containerd.sock in our CAPD configurations.

We should actually be using unix:///var/run/containerd/containerd.sock. Goal of this issue is to go over test/ and replace all occurrences.

P.S. there will be a warning with kubeadm v1.24 for paths which are not using a URL scheme. (xref: kubernetes/kubeadm#2426 (comment))

/kind cleanup
/area provider-docker
/good-first-issue

[One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels]

[k8s-ci-robot]

@sbueringer: The label(s) area/provider-docker cannot be applied, because the repository doesn't have them.

In response to this:

We are currently using /var/run/containerd/containerd.sock in our CAPD configurations.

We should actually be using unix:///var/run/containerd/containerd.sock. Goal of this issue is to go over test/ and replace all occurrences.

P.S. there will be a warning with kubeadm v1.24 for paths without a scheme.

/kind cleanup
/area provider-docker
/good-first-issue

[One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sbueringer]

@fabriziopandini I assume we don't have to do anything for upgrades as we don't do in-place upgrades:

during kubeadm upgrade, we should iterate all nodes in the cluster and patch "kubeadm.alpha.kubernetes.io/cri-socket"
kubernetes/kubeadm#2426

[stmcginnis]

I don't see anywhere that we are referencing this directly in the CAPD code:

$ rg socket test/*
test/go.sum
520:github.com/gorilla/websocket v0.0.0-20170926233335-4201258b820c/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ=
521:github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ=
522:github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
883:github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
884:github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
885:github.com/tmc/grpc-websocket-proxy v0.0.0-20201229170055-e5319fda7802/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=

test/infrastructure/docker/Dockerfile
60:# NOTE: CAPD can't use non-root because docker requires access to the docker socket

I think with our use of the docker API, any interaction with the socket is all encapsulated inside that library.

[sbueringer]

That is correct. I was referring to the places where we configure the criSocket in YAML: https://grep.app/search?q=/var/run/containerd/containerd.sock&filter[repo.pattern][0]=kubernetes-sigs/cluster-api&filter[lang][0]=YAML

P.S. This config is used by kubeadm "inside" the nodes, afaik to configure Kubelet.

[fabriziopandini]

@sbueringer we should tell the user to update their kubeconfig, but I'm -1 to upgrade settings automatically because this will trigger rollouts
@vincepri @CecileRobertMichon @enxebre opinions ^^

[sbueringer]

@sbueringer we should tell the user to update their kubeconfig, but I'm -1 to upgrade settings automatically because this will trigger rollouts @vincepri @CecileRobertMichon @enxebre opinions ^^

+1. I think we should have a section about 1.24 in our version.md and link that in the next releases which add support for 1.24. Maybe it's a good/sustainable pattern to document the really really breaking changes for CAPI users in Kubernetes releases there. (I'm referring to those parts which are kind of abstracted away by CAPI like kubeadm, not generic Kubernetes things like dropping support for old apiGroups in upstream Kubernetes)

[CecileRobertMichon]

Just to confirm, the scope here is only CAPD? If so, considering CAPD is not meant for production and only for development, I don't think we need to be as strict with not causing rollouts. IMO the term "user" doesn't really apply here.

[sbueringer]

Just to confirm, the scope here is only CAPD? If so, considering CAPD is not meant for production and only for development, I don't think we need to be as strict with not causing rollouts. IMO the term "user" doesn't really apply here.

The scope of what I want to fix is CAPD.

Fabrizio was referring to that we don't want to automatically migrate in KCP or CABPK to not trigger rollouts everywhere.

So essentially we should communicate to CAPI users that they should use a path with a URL scheme going forward, given that "the kubelet has long deprecated paths without unix:// and it might stop supporting them in the future" (kubernetes/kubeadm#2426 (comment))

[fabriziopandini]

Thanks, Stefan for clarifying!
WRT to CAPD what about dropping set cri setting entirely? (kubeadm should auto-detect containerd, right?)

[sbueringer]

Thanks, Stefan for clarifying! WRT to CAPD what about dropping set cri setting entirely? (kubeadm should auto-detect containerd, right?)

Yes. Let's try if that works!

[vincepri]

Agree with @CecileRobertMichon's message above, if the scope is only for CAPD we should probably not bother and just do the breaking change. For general KCP consumption, we could consider to only migrate the configuration when we detect an upgrade to 1.24?

[sbueringer]

For general KCP consumption, we could consider to only migrate the configuration when we detect an upgrade to 1.24?

But I think we can't do it for KubeadmConfig(Template) which means we would produce different configs in KCP / worker nodes. What do you think about trying to make some progress on webhook warnings in controller-runtime (not sure what the current state is there) and then adding a warning for it?

[sbueringer]

@ good first issue readers :)

Concrete task that can be already done:

• Drop the criSocket: /var/run/containerd/containerd.sock configurations in all YAMLs below test/*
  • If the tests are still green, that should be it.
  • Otherwise let's take a look at the error together, we might have to add the prefix then instead.

Meanwhile we're further discussing on this issue what we can/should do in KCP/CABPK.

[shubham14bajpai]

/assign

[sbueringer]

Thx @shubham14bajpai
We'll continue here to figure out what/if we have to do in KCP/CABPK
/reopen

[k8s-ci-robot]

@sbueringer: Reopened this issue.

In response to this:

Thx @shubham14bajpai
We'll continue here to figure out what/if we have to do in KCP/CABPK
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sbueringer]

/remove-good-first-issue

CAPD YAMLs have been changed accordingly.

Let's pivot back to what we want do for KCP / CABPK. Trying to summarize:

The old criSocket format (without URL scheme) is deprecated on multiple levels now (kubeadm (with 1.24) / kubelet)

Our options are (listing all for completeness):

1. Document that folks should migrate to the new format in a Kubernetes 1.24 specific section in version.md. We can link to that section when we announce Kubernetes 1.24 support
2. Extend our validating webhooks to return a warning when the old format is found
   • This requires some prior work in controller-runtime to enable warnings in webhooks (I think it's this issue: Support warnings in webhook utils controller-runtime#1788)
   • I think getting this in controller-runtime would enable a lot of other use cases where we want to return warnings.
3. Try to automatically migrate criSocket fields:
   • This is not an option as it would trigger rollouts and probably lead to problems with GitOps tools and potentially ClusterClass too

My take is, let's implement 1. and open a new focused issue for 2.

Opinions? @CecileRobertMichon @enxebre @fabriziopandini @vincepri

[fabriziopandini]

I'm +1 to option 1 because AFAIK most of the users rely on kubeadm CRI autodetection (and it is just a warning now)
WRT to support warning in webhooks I'm +100 but I would consider a separated work, not an alternative to 1

[fabriziopandini]

/milestone v1.2

[sbueringer]

I'm tracking this as part of #5968, so we can document that folks should migrate when we have support for Kubernetes 1.24. (currently versions.md doesn't include v1.24 which makes sense as it hasn't been released yet)