Replace k8s.gcr.io with registry.k8s.io #15777

mrbobbytables · 2023-02-02T23:20:38Z

k8s.gcr.io is in the process of being deprecated with future images no longer being served from that endpoint.

This is a general "find & replace" I ignored the changelog, but updated everything else.

For more information - see this blog post: https://kubernetes.io/blog/2022/11/28/registry-k8s-io-faster-cheaper-ga/

k8s-ci-robot · 2023-02-02T23:20:43Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mrbobbytables
Once this PR has been reviewed and has the lgtm label, please assign prezha for approval by writing /assign @prezha in a comment. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

minikube-bot · 2023-02-02T23:21:05Z

Can one of the admins verify this patch?

afbjorklund · 2023-02-03T07:04:48Z

@mrbobbytables : the goal with the code changes, was to have the minikube cache/preload match kubeadm.

This is why there was a cut-off, so that we don't have to invalidate old caches and old preloads needlessly ?

So it is good that the examples and tests are updated, but not changing e.g. 1.20 was done on purpose...

Versions -1.21 use old, versions 1.25+ use new. The others have their cutoff points, 1.22.18, 1.23.15, 1.24.9

afbjorklund · 2023-02-03T07:06:21Z

site/content/en/docs/handbook/offline.md

-cache/images/k8s.gcr.io/kube-addon-manager_v9.0
-cache/images/k8s.gcr.io/k8s-dns-kube-dns-amd64_1.14.13
-cache/images/k8s.gcr.io/kube-proxy_v1.14.0
+cache/images/registry.k8s.io/k8s-dns-sidecar-amd64_1.14.13


the kubernetes version needs to be updated, in this example (not changing anything for 1.14)

afbjorklund · 2023-02-03T07:07:09Z

site/content/en/docs/contrib/tests.en.md

@@ -294,7 +294,7 @@ Steps:
 asserts basic "service" command functionality

 Steps:
- Create a new `k8s.gcr.io/echoserver` deployment
+- Create a new `registry.k8s.io/echoserver` deployment


The echoserver no longer lives in the main registry, it has been forked to multiple locations.

For minikube, I think k8s.gcr.io/echoserver:1.4 was replaced with kicbase/echo-server:1.0

afbjorklund · 2023-02-03T07:07:38Z

pkg/minikube/config/types.go

@@ -76,7 +76,7 @@ type ClusterConfig struct {
 	KubernetesConfig        KubernetesConfig
 	Nodes                   []Node
 	Addons                  map[string]bool
-	CustomAddonImages       map[string]string // Maps image names to the image to use for addons. e.g. Dashboard -> k8s.gcr.io/echoserver:1.4 makes dashboard addon use echoserver for its Dashboard deployment.
+	CustomAddonImages       map[string]string // Maps image names to the image to use for addons. e.g. Dashboard -> registry.k8s.io/echoserver:1.4 makes dashboard addon use echoserver for its Dashboard deployment.


the echoserver image needs to be updated to the new one

afbjorklund · 2023-02-03T07:08:06Z

pkg/minikube/bootstrapper/images/repo.go

@@ -21,7 +21,7 @@ import (
 )

 // OldDefaultKubernetesRepo is the old default Kubernetes repository
-const OldDefaultKubernetesRepo = "k8s.gcr.io"
+const OldDefaultKubernetesRepo = "registry.k8s.io"


Changing the old registry to be the same as the new, makes this rather meaningless.

ah yeah - sorry, this was a more generic find & replace. Will update

afbjorklund · 2023-02-03T07:08:19Z

pkg/minikube/bootstrapper/images/kubeadm_test.go

-			"k8s.gcr.io/coredns:1.6.2",
-			"k8s.gcr.io/etcd:3.3.15-0",
-			"k8s.gcr.io/pause:3.1",
+			"registry.k8s.io/kube-proxy:v1.16.0",


not changing anything for 1.16 ?

I must have accidentally just replaced that 1 line vs all in the file, will update

afbjorklund · 2023-02-03T07:11:02Z

The old echoserver needs to be updated to the new location,

Replace references to echoserver with supported image from all documentation #11107

@spowelljr I think this will still use the "kicbase" echoserver ?

Different echoserver images used in different places #15650

mrbobbytables · 2023-02-06T15:31:23Z

@afbjorklund should I just close this? It looks like things are being tackled / tracked separately in #14769

FWIW - theres a policy discussion going on about agining out old images, we're sadly still on track to exceed our 3M in GCP credits (to the tune of an additional 1M 😬 ) and are looking for ways to reduce costs.

Outside of policy decisions on removing old images, one of the big things we can do is get as much user facing things to switch over to the registry.k8s.io which will spread the load across multiple providers. Updating k/website and this repo looked like an easy win because they are so frequently used / referenced by end users.

afbjorklund · 2023-02-06T15:33:05Z

A little reminder is a good thing, it missed the minikube 1.29.0 release for instance.

I think the best approach for minikube, would be to make a "preload" version also for the "none" driver.
Instead of tarring up the entire container storage (which doesn't work), it could include all the images ?

428M	preloaded-images-k8s-v18-v1.26.1-containerd-overlay2-amd64.tar.lz4
429M	preloaded-images-k8s-v18-v1.26.1-cri-o-overlay-amd64.tar.lz4
398M	preloaded-images-k8s-v18-v1.26.1-docker-overlay2-amd64.tar.lz4

Currently it is pulling them one by one from the registry, and storing them in cache (without preload)

But when there is a combined tarball, that could be used instead - and also for the "none" driver.
The only thing was that Podman was buggy with multiple images in one archive, but I think it is fixed...

Basically: kubeadm config images list | xargs docker save

mrbobbytables · 2023-02-06T15:38:51Z

I'll close this for now and move things over to separate PRs, and track in #14769

/close

k8s-ci-robot · 2023-02-06T15:39:22Z

@mrbobbytables: Closed this PR.

In response to this:

I'll close this for now and move things over to separate PRs, and track in #14769

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

afbjorklund · 2023-02-06T15:44:52Z

@mrbobbytables if there is any other discussion on how to make the downloads smaller, please invite

I don't know if the preload bucket and release bucket are also in the same spreadsheet, somewhere...

https://storage.googleapis.com/minikube-preloaded-volume-tarballs/

https://storage.googleapis.com/kubernetes-release/

mrbobbytables · 2023-02-06T16:02:03Z

I don't know of any discussion on making the images smaller at this time, most of the discussion is around shutting traffic to other providers or aging out old image. I don't know if image size would impact much of the traffic right now - from various sources it looks like most users are running older images, and thats what drove the backporting discussion / push to get everyone to use registry.k8s.io.

afbjorklund · 2023-02-06T20:22:44Z

Ok, for minikube it is mostly those two.

Downloading all the images into one file, instead of one-by-one with crane like now.
When not using a daemon, there is nobody to notice that the images are sharing layers.

111M	images/amd64/registry.k8s.io/etcd_3.5.6-0
17M	images/amd64/registry.k8s.io/kube-scheduler_v1.26.1
17M	images/amd64/registry.k8s.io/coredns/coredns_v1.9.3
34M	images/amd64/registry.k8s.io/kube-apiserver_v1.26.1
21M	images/amd64/registry.k8s.io/kube-proxy_v1.26.1
344K	images/amd64/registry.k8s.io/pause_3.9
31M	images/amd64/registry.k8s.io/kube-controller-manager_v1.26.1
230M	total

145M images/amd64/v1.26.1.tar.xz

Downloading a compressed version of the binaries, instead of getting the executables.
Currently compression (xz) is only offered for the packages (deb, rpm) but not the exe.

116M	linux/amd64/v1.26.1/kubelet
46M	linux/amd64/v1.26.1/kubectl
45M	linux/amd64/v1.26.1/kubeadm
207M	total

39M linux/amd64/v1.26.1.tar.xz

But the URLs should be changed, as well.

BenTheElder · 2023-03-28T05:03:42Z

Versions -1.21 use old, versions 1.25+ use new. The others have their cutoff points, 1.22.18, 1.23.15, 1.24.9

But it would only invalidate old caches? How long will minikube continue to support these old versions?
The reason it only goes back so far in Kubernetes is we can no longer release older versions.

https://storage.googleapis.com/minikube-preloaded-volume-tarballs/

I have no idea where this is and who pays for it ... 😬

https://storage.googleapis.com/kubernetes-release/

This is billed to google internally / different bill that SIG K8s Infra. There's other work ongoing to migrate that out to fully community control but it's a lot so we're seeking out other funding before doing so ....

@mrbobbytables if there is any other discussion on how to make the downloads smaller, please invite

We've had pretty active work targeting images like kube-proxy but nothing at this time. distroless kube-proxy shipped a pretty major reduction in 1.25 kubernetes/kubernetes#109406

I think out of the top images, for current tags only ingress-nginx/controller jumps out as something that appears to be particularly large.

afbjorklund · 2023-03-28T11:15:56Z

Versions -1.21 use old, versions 1.25+ use new. The others have their cutoff points, 1.22.18, 1.23.15, 1.24.9

But it would only invalidate old caches? How long will minikube continue to support these old versions? The reason it only goes back so far in Kubernetes is we can no longer release older versions.

This was just about the default in kubeadm, the plan is to use registry.k8s.io for everything

I don't see why the version support should be different in minikube, from the rest of the project...
But as long as the tutorials* are running 1.20, I guess some leeway can be "allowed" and tolerated ?

* the old tutorials will soon be deleted anyway

https://storage.googleapis.com/minikube-preloaded-volume-tarballs/

I have no idea where this is and who pays for it ... grimacing

The preload is an optional pre-installed archive, so I guess it is the same as the kind "node" image.

It would be possible to not provide this feature anymore, and use the regular container images.
Maybe generate the preload on the client, but then it would only speed up the second installation ?

@mrbobbytables if there is any other discussion on how to make the downloads smaller, please invite

We've had pretty active work targeting images like kube-proxy but nothing at this time. distroless kube-proxy shipped a pretty major reduction in 1.25 kubernetes/kubernetes#109406

I think out of the top images, for current tags only ingress-nginx/controller jumps out as something that appears to be particularly large.

The preload is all about making the install go faster, so it is actually bigger than the usual images. (it uses lz4)

But it would be possible to make a "batteries included" version and optimize for a small download... (using xz)

Otherwise I think we are hoping for improvements in the container runtimes and registries will "fix" the problem.
Like when allowing for peer-to-peer pulling, and so on ? Having local mirrors of the official registry also helps a lot.

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Feb 2, 2023

k8s-ci-robot requested review from afbjorklund and prezha February 2, 2023 23:20

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Feb 2, 2023

Replace k8s.gcr.io with registry.k8s.io

38a6a6b

mrbobbytables force-pushed the reg branch from eb0cbbb to 38a6a6b Compare February 3, 2023 00:30

afbjorklund requested changes Feb 3, 2023

View reviewed changes

afbjorklund mentioned this pull request Feb 3, 2023

Stop using "k8s.gcr.io" registry, use "registry.k8s.io" for new #14769

Closed

k8s-ci-robot closed this Feb 6, 2023

afbjorklund mentioned this pull request Feb 8, 2023

Update k8s.gcr.io refs to registry.k8s.io #15819

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace k8s.gcr.io with registry.k8s.io #15777

Replace k8s.gcr.io with registry.k8s.io #15777

mrbobbytables commented Feb 2, 2023

k8s-ci-robot commented Feb 2, 2023

minikube-bot commented Feb 2, 2023

afbjorklund commented Feb 3, 2023 •

edited

Loading

afbjorklund Feb 3, 2023

afbjorklund Feb 3, 2023

afbjorklund Feb 3, 2023

afbjorklund Feb 3, 2023

afbjorklund Feb 3, 2023

mrbobbytables Feb 6, 2023

afbjorklund Feb 3, 2023

mrbobbytables Feb 6, 2023

afbjorklund commented Feb 3, 2023

mrbobbytables commented Feb 6, 2023

afbjorklund commented Feb 6, 2023 •

edited

Loading

mrbobbytables commented Feb 6, 2023

k8s-ci-robot commented Feb 6, 2023

afbjorklund commented Feb 6, 2023 •

edited

Loading

mrbobbytables commented Feb 6, 2023

afbjorklund commented Feb 6, 2023

BenTheElder commented Mar 28, 2023

afbjorklund commented Mar 28, 2023 •

edited

Loading

Replace k8s.gcr.io with registry.k8s.io #15777

Replace k8s.gcr.io with registry.k8s.io #15777

Conversation

mrbobbytables commented Feb 2, 2023

k8s-ci-robot commented Feb 2, 2023

minikube-bot commented Feb 2, 2023

afbjorklund commented Feb 3, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

afbjorklund commented Feb 3, 2023

mrbobbytables commented Feb 6, 2023

afbjorklund commented Feb 6, 2023 • edited Loading

mrbobbytables commented Feb 6, 2023

k8s-ci-robot commented Feb 6, 2023

afbjorklund commented Feb 6, 2023 • edited Loading

mrbobbytables commented Feb 6, 2023

afbjorklund commented Feb 6, 2023

BenTheElder commented Mar 28, 2023

afbjorklund commented Mar 28, 2023 • edited Loading

afbjorklund commented Feb 3, 2023 •

edited

Loading

afbjorklund commented Feb 6, 2023 •

edited

Loading

afbjorklund commented Feb 6, 2023 •

edited

Loading

afbjorklund commented Mar 28, 2023 •

edited

Loading