Stop using "k8s.gcr.io" registry, use "registry.k8s.io" for new

[afbjorklund]

Background

https://github.com/kubernetes/k8s.io/wiki/New-Registry-url-for-Kubernetes-(registry.k8s.io)

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.25.md#changelog-since-v1240

There is a new OCI image registry (registry.k8s.io) that can be used to pull kubernetes images. The old registry (k8s.gcr.io) will continue to be supported for the foreseeable future, but the new name should perform better because it frontends equivalent mirrors in other clouds. Please point your clusters to the new registry going forward.

---

All the images from kubeadm, needs to be moved: (they have already changed, in v1.25.0-alpha.1)

• The DefaultKubernetesRepo changed for 1.25.0 #14768

Probably would be a good idea to put the 5 year old (!) k8s.gcr.io/echoserver:1.4 out of its misery, too ?

• Replace references to echoserver with supported image from all documentation #11107

---

Unfortunately, "k8s.gcr.io" seems to be hardcoded in a lot of add-ons. Not sure if they are affected ?

pkg/addons/addons.go:				"IngressController": "k8s.gcr.io",
pkg/minikube/assets/addons.go:		"Elasticsearch":        "k8s.gcr.io",
pkg/minikube/assets/addons.go:		"FluentdElasticsearch": "k8s.gcr.io",
pkg/minikube/assets/addons.go:		"KubeWebhookCertgenCreate": "k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1@sha256:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660",
pkg/minikube/assets/addons.go:		"KubeWebhookCertgenPatch": "k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1@sha256:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660",
pkg/minikube/assets/addons.go:		"IngressController": "k8s.gcr.io",
pkg/minikube/assets/addons.go:		"MetricsServer": "k8s.gcr.io",
pkg/minikube/assets/addons.go:		"NvidiaDriverInstaller": "k8s.gcr.io",
pkg/minikube/assets/addons.go:		"Pause":                 "k8s.gcr.io",
pkg/minikube/assets/addons.go:		"NvidiaDevicePlugin": "k8s.gcr.io",
pkg/minikube/assets/addons.go:		"KubeWebhookCertgen": "k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0@sha256:f3b6b39a6062328c095337b4cadcefd1612348fdd5190b1dcbcb9b9e90bd8068",
pkg/minikube/assets/addons.go:		"SnapshotController": "k8s.gcr.io",
pkg/minikube/assets/addons.go:		"Attacher":              "k8s.gcr.io",
pkg/minikube/assets/addons.go:		"HostMonitorAgent":      "k8s.gcr.io",
pkg/minikube/assets/addons.go:		"HostMonitorController": "k8s.gcr.io",
pkg/minikube/assets/addons.go:		"NodeDriverRegistrar":   "k8s.gcr.io",
pkg/minikube/assets/addons.go:		"HostPathPlugin":        "k8s.gcr.io",
pkg/minikube/assets/addons.go:		"LivenessProbe":         "k8s.gcr.io",
pkg/minikube/assets/addons.go:		"Resizer":               "k8s.gcr.io",
pkg/minikube/assets/addons.go:		"Snapshotter":           "k8s.gcr.io",
pkg/minikube/assets/addons.go:		"Provisioner":           "k8s.gcr.io",

The "pause" image should be overridden at runtime (in the container runtimes), so should be fine...

Updating old preloads and cache images will be an interesting research project, sometime "later".

[afbjorklund]

Needs to be updated also for 1.22 - 1.24, and there are still places using the old echoserver.

[afbjorklund]

• Replace k8s.gcr.io with registry.k8s.io #15777

[mrbobbytables]

Thanks for creating the tracking issue @afbjorklund I should have started there instead of just opening up a PR 😅

For further context, while gcr won't be shut down - we need to shift as much traffic to use registry.k8s.io which will load balance across multiple providers and acts as a CDN (e.g. if a person is running on aws, it will pull images from the aws mirror).

The big driver for being proactive on this is that we're currently tracking to exhaust our 3M in GCP credits (to the tune of an additional 1M 😬 ), well before end of year and are looking to reduce those costs as quickly as possible and (ideally) as little interruption to prod environments as possible.

We are looking at other potential disruptive actions, like setting a retention policy to age out old images after say 3 years (9 releases)

[afbjorklund]

The idea behind not changing legacy releases, except for no longer matching what kubeadm is saying, is that it would invalidate all local caches and all preloads - thus causing those to have to be re-downloaded, generating more traffic

There are similar issues to use dl.k8s.io, or apt.k8s.io or yum.k8s.io, but those are more about staying hosting agnostic. So instead of hardcoding https://storage.googleapis.com/ or https://packages.cloud.google.com/, it uses the mirrors...

• Stop using gs://kubernetes-release bucket, use dl.k8s.io for downloads #14828

But I don't think they make any practical difference - yet. Possibly if OBS gets used more.

[xmudrii]

@afbjorklund How are we doing with this? I see there are some PRs that got merged, is there anything else that can be done for this issue?

[spowelljr]

This is completed as of #16029 and #16200