From 1919dc5534898221e6e0ee42054f0f1b9caa4575 Mon Sep 17 00:00:00 2001 From: Hippie Hacker Date: Tue, 7 Dec 2021 23:46:16 +1300 Subject: [PATCH 01/17] KEP 3000: Artifact Distribution Policy Co-authored-by: Stephen Augustus --- .../3000-artifact-distribution/README.md | 132 ++++++++++++++++++ .../3000-artifact-distribution/kep.yaml | 19 +++ 2 files changed, 151 insertions(+) create mode 100644 keps/sig-release/3000-artifact-distribution/README.md create mode 100644 keps/sig-release/3000-artifact-distribution/kep.yaml diff --git a/keps/sig-release/3000-artifact-distribution/README.md b/keps/sig-release/3000-artifact-distribution/README.md new file mode 100644 index 00000000000..65488917c6c --- /dev/null +++ b/keps/sig-release/3000-artifact-distribution/README.md @@ -0,0 +1,132 @@ +# KEP/MST-3000: Artifact Promotion and Distribution Policy + + +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Proposal](#proposal) + - [User Stories](#user-stories) + - [SIG Release - Image Promotion](#sig-release---image-promotion) + - [Cloud Customer - Installing K8s via kubeadm](#cloud-customer---installing-k8s-via-kubeadm) + - [Notes/Constraints/Caveats](#notesconstraintscaveats) + - [Risks and Mitigations](#risks-and-mitigations) +- [Design Details](#design-details) + - [Release Promotion](#release-promotion) + - [Policy](#policy) + - [Process](#process) + - [Artifact Distribution](#artifact-distribution) + - [Policy](#policy-1) + - [Process](#process-1) +- [Alternatives / Background](#alternatives--background) + - [How much is this going to save us?](#how-much-is-this-going-to-save-us) +- [Infrastructure Needed](#infrastructure-needed) + + +## Summary + +The container images and release binaries produced by our community need a clear path to be hosted by multiple service/cloud providers. + +The global community should be routed to the appropriate mirror for their country or cloud provider to ensure cost effective worldwide access. + +This KEP should cover the policy and distribution mechanisms we will put in place to allow creating a globally distributed, multi-cloud and country solution. + +## Motivation + +Currently we push to a single provider, and distributing to the rest of community comes at great cost nearing $150k/month (mostly egress) in donations. + +Additonally some of our community members are unable to access the official release artifacts due to country level firewalls that do not them connect to Google services. + +Ideally we can dramatically reduce cost and allow everyone in the world to download the artifacts released by our community. + +### Goals + +A policy and procedure for use by SIG Release to promote container images and release binaries to multiple registries and mirrors. + +A solution to allow redirection to appropriate mirrors to lower cost and allow access from any cloud or country globally. + +### Non-Goals + +Anything related to creation of artifacts, bom, digital signatures. + +## Proposal + +There are two intertwined concepts that are part of this proposal. + +First, the policy and procedures to promote/upload our artifacts to multiple providers. Our existing processes upload only to GCS buckets. Ideally we extend the existing software/promotion process to push directly to multiple providers. Alternatively we use a second process to syncronize artifacts from our existing production buckets to similar contructs at other providers. + +Additionally we require a registry and artifact url-redirection solution to the local cloud provider or country. + +### User Stories + +#### SIG Release - Image Promotion + +```feature +As a SIG Release volunteer +I want to promote our binaries/images to multiple clouds + +Given a promotion / mainifest +When my PR is merged +Then the promotion process occurs +``` + +#### Cloud Customer - Installing K8s via kubeadm + +```feature +As a CLOUD end-user +I want to install kubernetes + +Given some compute resources at CLOUD +When I use kubeadm to deploy Kubernetes +Then I will be redirected to a local CLOUD registry +``` + +### Notes/Constraints/Caveats + +The primary purpose of the KEP is getting consensus on the agreed policy and procedure to unblock our community and move forward together. + +There has been a lot of activity around the technology and tooling for both goals, but we need shared agreement on policy and procedure first. + +### Risks and Mitigations + +This is the primary pipeline for delivering Kubernetes worldwide. Ensuring the appropriate SLAs and support as well as artifact integrite are crucial. + +## Design Details + +### Release Promotion + +#### Policy + +(more details needed, #sig-release-eng?) + +#### Process + +Currently the promotion process is primarily driven by the CIP/[promo-tool#kpromo](https://github.com/kubernetes-sigs/promo-tools#kpromo)? + +### Artifact Distribution + +#### Policy + +#### Process + +## Alternatives / Background +- Apache has a widespread mirror network + - @dims has experince here + - http://ws.apache.org/mirrors.cgi + - https://infra.apache.org/mirrors.html +- [Umbrella issue: k8s.gcr.io => registry.k8s.io solution k/k8s.io#1834 +](https://github.com/kubernetes/k8s.io/issues/1834) +- [ii/registry.k8s.io Implementation proposals](https://github.com/ii/registry.k8s.io#registryk8sio) +- [ii.nz/blog :: Building a data pipline for displaying Kubernetes public artifact traffic +](https://ii.nz/post/building-a-data-pipline-for-displaying-kubernetes-public-artifact-traffic/) +### How much is this going to save us? +Cost of K8s Artifact hosting - Data Studio Graphs + +![](https://i.imgur.com/LAn4UIE.png) + + +## Infrastructure Needed + +It would be good to request some donations for some larger providers, including one in China, via cncf.io/credits + +[![hackmd-github-sync-badge](https://hackmd.io/KjHufZssQR654ShkZFUzyA/badge)](https://hackmd.io/KjHufZssQR654ShkZFUzyA) diff --git a/keps/sig-release/3000-artifact-distribution/kep.yaml b/keps/sig-release/3000-artifact-distribution/kep.yaml new file mode 100644 index 00000000000..4d1af565638 --- /dev/null +++ b/keps/sig-release/3000-artifact-distribution/kep.yaml @@ -0,0 +1,19 @@ +title: Artifact Promotion and Distribution Policy and Procedure +kep-number: NNNN +authors: + - "@hh" +owning-sig: sig-architecture +participating-sigs: + - sig-release +status: provisional +creation-date: 2021-11-26 +reviewers: + - TBD + approvers: + - TBD +stage: alpha +latest-milestone: "v1.24" +milestone: + alpha: "NA" + beta: "NA" + stable: "v1.24" From 467d84a4389687a5ea1226dc36b3e9b7469c2e14 Mon Sep 17 00:00:00 2001 From: Stephen Augustus Date: Tue, 7 Dec 2021 12:13:20 -0500 Subject: [PATCH 02/17] sig-release/3000-artifact-distribution: Update kep.yaml - update title - update KEP number - set owner to SIG Release - set participating to SIG K8s Infra - set reviewers to TLs for SIGs - set approvers to Chairs for SIGs (one apiece) - set alpha milestone to v1.24 Signed-off-by: Stephen Augustus --- .../3000-artifact-distribution/kep.yaml | 24 +++++++++++-------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/keps/sig-release/3000-artifact-distribution/kep.yaml b/keps/sig-release/3000-artifact-distribution/kep.yaml index 4d1af565638..35ed2731829 100644 --- a/keps/sig-release/3000-artifact-distribution/kep.yaml +++ b/keps/sig-release/3000-artifact-distribution/kep.yaml @@ -1,19 +1,23 @@ -title: Artifact Promotion and Distribution Policy and Procedure -kep-number: NNNN +title: Artifact Distribution Policy +kep-number: 3000 authors: - "@hh" -owning-sig: sig-architecture +owning-sig: sig-release participating-sigs: - - sig-release + - sig-k8s-infra status: provisional creation-date: 2021-11-26 reviewers: - - TBD - approvers: - - TBD + - "@cpanato" + - "@puerco" + - "@spiffxp" + - "@thockin" +approvers: + - "@ameukam" + - "@justaugustus" stage: alpha latest-milestone: "v1.24" milestone: - alpha: "NA" - beta: "NA" - stable: "v1.24" + alpha: "v1.24" + beta: "v1.25" + stable: "v1.26" From bfdee8734b25b58177dee0b99a56e9be168015e0 Mon Sep 17 00:00:00 2001 From: Stephen Augustus Date: Tue, 7 Dec 2021 12:26:30 -0500 Subject: [PATCH 03/17] sig-release/3000-artifact-distribution: Typo/markdown lint fixes Signed-off-by: Stephen Augustus --- .../3000-artifact-distribution/README.md | 31 ++++++++++--------- 1 file changed, 17 insertions(+), 14 deletions(-) diff --git a/keps/sig-release/3000-artifact-distribution/README.md b/keps/sig-release/3000-artifact-distribution/README.md index 65488917c6c..e4356d384af 100644 --- a/keps/sig-release/3000-artifact-distribution/README.md +++ b/keps/sig-release/3000-artifact-distribution/README.md @@ -1,4 +1,4 @@ -# KEP/MST-3000: Artifact Promotion and Distribution Policy +# KEP 3000: Artifact Distribution Policy - [Summary](#summary) @@ -12,7 +12,7 @@ - [Notes/Constraints/Caveats](#notesconstraintscaveats) - [Risks and Mitigations](#risks-and-mitigations) - [Design Details](#design-details) - - [Release Promotion](#release-promotion) + - [Artifact Promotion](#artifact-promotion) - [Policy](#policy) - [Process](#process) - [Artifact Distribution](#artifact-distribution) @@ -21,6 +21,7 @@ - [Alternatives / Background](#alternatives--background) - [How much is this going to save us?](#how-much-is-this-going-to-save-us) - [Infrastructure Needed](#infrastructure-needed) +- [Hack on this doc](#hack-on-this-doc) ## Summary @@ -35,7 +36,7 @@ This KEP should cover the policy and distribution mechanisms we will put in plac Currently we push to a single provider, and distributing to the rest of community comes at great cost nearing $150k/month (mostly egress) in donations. -Additonally some of our community members are unable to access the official release artifacts due to country level firewalls that do not them connect to Google services. +Additionally, some of our community members are unable to access the official release artifacts due to country level firewalls that do not them connect to Google services. Ideally we can dramatically reduce cost and allow everyone in the world to download the artifacts released by our community. @@ -53,7 +54,7 @@ Anything related to creation of artifacts, bom, digital signatures. There are two intertwined concepts that are part of this proposal. -First, the policy and procedures to promote/upload our artifacts to multiple providers. Our existing processes upload only to GCS buckets. Ideally we extend the existing software/promotion process to push directly to multiple providers. Alternatively we use a second process to syncronize artifacts from our existing production buckets to similar contructs at other providers. +First, the policy and procedures to promote/upload our artifacts to multiple providers. Our existing processes upload only to GCS buckets. Ideally we extend the existing software/promotion process to push directly to multiple providers. Alternatively we use a second process to synchronize artifacts from our existing production buckets to similar constructs at other providers. Additionally we require a registry and artifact url-redirection solution to the local cloud provider or country. @@ -65,7 +66,7 @@ Additionally we require a registry and artifact url-redirection solution to the As a SIG Release volunteer I want to promote our binaries/images to multiple clouds -Given a promotion / mainifest +Given a promotion / manifest When my PR is merged Then the promotion process occurs ``` @@ -74,7 +75,7 @@ Then the promotion process occurs ```feature As a CLOUD end-user -I want to install kubernetes +I want to install Kubernetes Given some compute resources at CLOUD When I use kubeadm to deploy Kubernetes @@ -89,11 +90,11 @@ There has been a lot of activity around the technology and tooling for both goal ### Risks and Mitigations -This is the primary pipeline for delivering Kubernetes worldwide. Ensuring the appropriate SLAs and support as well as artifact integrite are crucial. +This is the primary pipeline for delivering Kubernetes worldwide. Ensuring the appropriate SLAs and support as well as artifact integrity is crucial. ## Design Details -### Release Promotion +### Artifact Promotion #### Policy @@ -101,7 +102,7 @@ This is the primary pipeline for delivering Kubernetes worldwide. Ensuring the a #### Process -Currently the promotion process is primarily driven by the CIP/[promo-tool#kpromo](https://github.com/kubernetes-sigs/promo-tools#kpromo)? +Currently the promotion process is primarily driven by the CIP/[promo-tool#kpromo](https://github.com/kubernetes-sigs/promo-tools#kpromo)? ### Artifact Distribution @@ -110,8 +111,9 @@ Currently the promotion process is primarily driven by the CIP/[promo-tool#kprom #### Process ## Alternatives / Background + - Apache has a widespread mirror network - - @dims has experince here + - @dims has experience here - http://ws.apache.org/mirrors.cgi - https://infra.apache.org/mirrors.html - [Umbrella issue: k8s.gcr.io => registry.k8s.io solution k/k8s.io#1834 @@ -119,14 +121,15 @@ Currently the promotion process is primarily driven by the CIP/[promo-tool#kprom - [ii/registry.k8s.io Implementation proposals](https://github.com/ii/registry.k8s.io#registryk8sio) - [ii.nz/blog :: Building a data pipline for displaying Kubernetes public artifact traffic ](https://ii.nz/post/building-a-data-pipline-for-displaying-kubernetes-public-artifact-traffic/) -### How much is this going to save us? -Cost of K8s Artifact hosting - Data Studio Graphs -![](https://i.imgur.com/LAn4UIE.png) +### How much is this going to save us? +![Cost of K8s Artifact hosting - Data Studio Graphs](https://i.imgur.com/LAn4UIE.png) ## Infrastructure Needed -It would be good to request some donations for some larger providers, including one in China, via cncf.io/credits +It would be good to request some donations for some larger providers, including one in China, via [Cloud Native Credits program](https://www.cncf.io/credits/). + +## Hack on this doc [![hackmd-github-sync-badge](https://hackmd.io/KjHufZssQR654ShkZFUzyA/badge)](https://hackmd.io/KjHufZssQR654ShkZFUzyA) From 300cc1d263130abd429ed4f9b2359d32fefdc7d6 Mon Sep 17 00:00:00 2001 From: Caleb Woodbine Date: Tue, 26 Apr 2022 15:42:49 +1200 Subject: [PATCH 04/17] Update kep according to hackmd content --- .../3000-artifact-distribution/README.md | 76 +++++++++++++++---- .../3000-artifact-distribution/kep.yaml | 1 + 2 files changed, 63 insertions(+), 14 deletions(-) diff --git a/keps/sig-release/3000-artifact-distribution/README.md b/keps/sig-release/3000-artifact-distribution/README.md index e4356d384af..624b50e7916 100644 --- a/keps/sig-release/3000-artifact-distribution/README.md +++ b/keps/sig-release/3000-artifact-distribution/README.md @@ -1,6 +1,7 @@ -# KEP 3000: Artifact Distribution Policy +# KEP/MST-3000: Image Promotion and Distribution Policy + - [Summary](#summary) - [Motivation](#motivation) - [Goals](#goals) @@ -12,7 +13,7 @@ - [Notes/Constraints/Caveats](#notesconstraintscaveats) - [Risks and Mitigations](#risks-and-mitigations) - [Design Details](#design-details) - - [Artifact Promotion](#artifact-promotion) + - [Release Promotion](#release-promotion) - [Policy](#policy) - [Process](#process) - [Artifact Distribution](#artifact-distribution) @@ -21,7 +22,6 @@ - [Alternatives / Background](#alternatives--background) - [How much is this going to save us?](#how-much-is-this-going-to-save-us) - [Infrastructure Needed](#infrastructure-needed) -- [Hack on this doc](#hack-on-this-doc) ## Summary @@ -32,14 +32,30 @@ The global community should be routed to the appropriate mirror for their countr This KEP should cover the policy and distribution mechanisms we will put in place to allow creating a globally distributed, multi-cloud and country solution. +## Background (from wiki) + ## Motivation -Currently we push to a single provider, and distributing to the rest of community comes at great cost nearing $150k/month (mostly egress) in donations. +For a few years now, we have been using k8s.gcr.io in all our repositories as default repository for downloading images from. -Additionally, some of our community members are unable to access the official release artifacts due to country level firewalls that do not them connect to Google services. +The cost of distributing Kubernetes comes at great cost nearing $150kUSD/month (mostly egress) in donations. + +Additionally some of our community members are unable to access the official release artifacts due to country level firewalls that do not them connect to Google services. Ideally we can dramatically reduce cost and allow everyone in the world to download the artifacts released by our community. +We are now used to using the [image promoter process](https://github.com/kubernetes/enhancements/tree/master/keps/sig-release/1734-k8s-image-promoter) to promote images to the official kubernetes container registry using the infrastructure (GCR staging repos etc) provided by [sig-k8s-infra](https://github.com/kubernetes/k8s.io/tree/main/k8s.gcr.io) + +## Why a new domain? + +So far we (all kubernetes project) are using GCP as our default infrastructure provider for all things like GCS, GCR, GKE based prow clusters etc. Google has graciously sponsored a lot of our infrastructure costs as well. However for about a year or so we are finding that our costs are sky-rocketing because the community usage of this infrastructure has been from other cloud providers like AWS, Azure etc. So in conjunction with CNCF staff we are trying to put together a plan to host copies of images and binaries nearer to where they are used rather than incur cross-cloud costs. + +One part of this plan is to setup a proxy OCI service, that can identify where the traffic is coming from and redirect to the nearest image layer/repository. This is why we are setting up a new service using what we call an [oci-proxy](https://github.com/kubernetes-sigs/oci-proxy) for everyone to use. This proxy will identify traffic coming from, for example, a certain AWS region, then will setup a HTTP redirect to a source in that AWS region. If we get traffic from GKE/GCP or we don't know where the traffic is coming from, it will still redirect to the current infrastructure (k8s.gcr.io). + +## How can we help? + +When Kubernetes master opens up for v1.25 development, we need to update all default urls in our code and test harness to the new registry url. As a team sig-k8s-infra is signing up to ensure that this oci-proxy based registry.k8s.io will be as robust and available as the current setup. As a backup, we will continue to run the current k8s.gcr.io as well. So do not worry about that going away. Turning on traffic to the new url will help us monitor and fix things if/when they break and we will be able to tune traffic and lower our costs of operation. + ### Goals A policy and procedure for use by SIG Release to promote container images and release binaries to multiple registries and mirrors. @@ -48,7 +64,17 @@ A solution to allow redirection to appropriate mirrors to lower cost and allow a ### Non-Goals -Anything related to creation of artifacts, bom, digital signatures. +Anything related to creation of artifacts, bom, digital signatures, staging buckets. + +### What is not in scope + +- Currently we focus on AWS only. We are getting a lot of help from AWS in terms of technical details as well as targeted infrastructure costs for standing up and running this infrastructure + +### What are good goals to shoot for + +- In terms of cost reduction, monitor GCP infrastructure and get to the point where we fully avoid serving large binary image layers from GCR/GCS +- We can add other AWS regions and clouds as needed in well known documented way +- Seamless transition for the community from the old k8s.gcr.io to registry.k8s.io with same rock solid stability as we now have with k8s.gcr.io ## Proposal @@ -58,6 +84,19 @@ First, the policy and procedures to promote/upload our artifacts to multiple pro Additionally we require a registry and artifact url-redirection solution to the local cloud provider or country. +## What exactly are you doing? + +- We are setting up an AWS account with an IAM role and s3 buckets in AWS regions where we see a large percentage of source image pull traffic +- We will iterate on a sandbox url (registry.sandbox.k8s.io) for our experiments and ONLY promote things to (registry.k8s.io) when we have complete confidence +- both registry and registry.sandbox are serving traffic using oci-proxy on google cloud run +- oci-proxy will be updated to identify incoming traffic from AWS regions based on IP ranges so we can route traffic to s3 buckets in that region. If a specific AWS region do not currently host s3 buckets, we will redirect to the nearest region which does have s3 buckets (tradeoff between storage and network costs) +- We will bulk sync existing image layers to these s3 layers as a starting point (from GCS/GCR) +- We will update image-promoter to push to these s3 buckets as well in addition to the current setup +- We will set up monitoring/reporting to check on new costs we incur on the AWS infrastructure and update what we do in GCP infrastructure as well to include the new components +- We will have a plan in place on how we could add additional AWS regions in the future +- We will have CI jobs that will run against registry.sandbox as well to monitor stability before we promote code to registry +- We will automate the deployment/monitoring and testing of code landing in the oci-proxy repository + ### User Stories #### SIG Release - Image Promotion @@ -94,7 +133,7 @@ This is the primary pipeline for delivering Kubernetes worldwide. Ensuring the a ## Design Details -### Artifact Promotion +### Release Promotion #### Policy @@ -112,24 +151,33 @@ Currently the promotion process is primarily driven by the CIP/[promo-tool#kprom ## Alternatives / Background +- Original KEP + - https://github.com/kubernetes/enhancements/tree/master/keps/sig-release/1734-k8s-image-promoter +- Oras + - https://github.com/oras-project/oras +- KubeCon Talk + - https://www.youtube.com/watch?v=F2IFjz7sr9Q - Apache has a widespread mirror network - - @dims has experience here + - @dims has experince here - http://ws.apache.org/mirrors.cgi - https://infra.apache.org/mirrors.html - [Umbrella issue: k8s.gcr.io => registry.k8s.io solution k/k8s.io#1834 -](https://github.com/kubernetes/k8s.io/issues/1834) + ](https://github.com/kubernetes/k8s.io/issues/1834) - [ii/registry.k8s.io Implementation proposals](https://github.com/ii/registry.k8s.io#registryk8sio) - [ii.nz/blog :: Building a data pipline for displaying Kubernetes public artifact traffic -](https://ii.nz/post/building-a-data-pipline-for-displaying-kubernetes-public-artifact-traffic/) + ](https://ii.nz/post/building-a-data-pipline-for-displaying-kubernetes-public-artifact-traffic/) ### How much is this going to save us? -![Cost of K8s Artifact hosting - Data Studio Graphs](https://i.imgur.com/LAn4UIE.png) +Cost of K8s Artifact hosting - Data Studio Graphs + +![](https://i.imgur.com/LAn4UIE.png) ## Infrastructure Needed -It would be good to request some donations for some larger providers, including one in China, via [Cloud Native Credits program](https://www.cncf.io/credits/). +It would be good to request some donations for some larger providers, including one in China, via cncf.io/credits -## Hack on this doc +## Hack this doc -[![hackmd-github-sync-badge](https://hackmd.io/KjHufZssQR654ShkZFUzyA/badge)](https://hackmd.io/KjHufZssQR654ShkZFUzyA) +- [![hackmd-github-sync-badge](https://hackmd.io/KjHufZssQR654ShkZFUzyA/badge)](https://hackmd.io/KjHufZssQR654ShkZFUzyA) +- [kubernetes/enhancements!3079](https://github.com/kubernetes/enhancements/pull/3079) diff --git a/keps/sig-release/3000-artifact-distribution/kep.yaml b/keps/sig-release/3000-artifact-distribution/kep.yaml index 35ed2731829..408b45cebf0 100644 --- a/keps/sig-release/3000-artifact-distribution/kep.yaml +++ b/keps/sig-release/3000-artifact-distribution/kep.yaml @@ -2,6 +2,7 @@ title: Artifact Distribution Policy kep-number: 3000 authors: - "@hh" + - "@BobyMCbobs" owning-sig: sig-release participating-sigs: - sig-k8s-infra From b5ffc451f418c66187ba4b72de2e8ad86ad9e251 Mon Sep 17 00:00:00 2001 From: Caleb Woodbine Date: Tue, 26 Apr 2022 15:47:52 +1200 Subject: [PATCH 05/17] Update toc --- keps/sig-release/3000-artifact-distribution/README.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/keps/sig-release/3000-artifact-distribution/README.md b/keps/sig-release/3000-artifact-distribution/README.md index 624b50e7916..fe3a3b15559 100644 --- a/keps/sig-release/3000-artifact-distribution/README.md +++ b/keps/sig-release/3000-artifact-distribution/README.md @@ -1,12 +1,17 @@ # KEP/MST-3000: Image Promotion and Distribution Policy - - [Summary](#summary) +- [Background (from wiki)](#background-from-wiki) - [Motivation](#motivation) +- [Why a new domain?](#why-a-new-domain) +- [How can we help?](#how-can-we-help) - [Goals](#goals) - [Non-Goals](#non-goals) + - [What is not in scope](#what-is-not-in-scope) + - [What are good goals to shoot for](#what-are-good-goals-to-shoot-for) - [Proposal](#proposal) +- [What exactly are you doing?](#what-exactly-are-you-doing) - [User Stories](#user-stories) - [SIG Release - Image Promotion](#sig-release---image-promotion) - [Cloud Customer - Installing K8s via kubeadm](#cloud-customer---installing-k8s-via-kubeadm) @@ -22,6 +27,7 @@ - [Alternatives / Background](#alternatives--background) - [How much is this going to save us?](#how-much-is-this-going-to-save-us) - [Infrastructure Needed](#infrastructure-needed) +- [Hack this doc](#hack-this-doc) ## Summary From 6e68a6c6b562879d9ac4943dcee0f8d0bf98af35 Mon Sep 17 00:00:00 2001 From: Caleb Woodbine Date: Wed, 27 Apr 2022 17:06:16 +1200 Subject: [PATCH 06/17] Update process --- keps/sig-release/3000-artifact-distribution/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/keps/sig-release/3000-artifact-distribution/README.md b/keps/sig-release/3000-artifact-distribution/README.md index fe3a3b15559..eacc54768f9 100644 --- a/keps/sig-release/3000-artifact-distribution/README.md +++ b/keps/sig-release/3000-artifact-distribution/README.md @@ -155,6 +155,8 @@ Currently the promotion process is primarily driven by the CIP/[promo-tool#kprom #### Process +Artifacts will be written to S3 style storage or CDNs provided by cloud providers through a tool in the promo-tools suite. + ## Alternatives / Background - Original KEP From 35d575b5d59fe65f1878be31e41b301b38e1cb11 Mon Sep 17 00:00:00 2001 From: Caleb Woodbine Date: Wed, 27 Apr 2022 17:06:21 +1200 Subject: [PATCH 07/17] Update registry-sandbox references --- keps/sig-release/3000-artifact-distribution/README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/keps/sig-release/3000-artifact-distribution/README.md b/keps/sig-release/3000-artifact-distribution/README.md index eacc54768f9..cb13ae8cbb5 100644 --- a/keps/sig-release/3000-artifact-distribution/README.md +++ b/keps/sig-release/3000-artifact-distribution/README.md @@ -1,6 +1,7 @@ # KEP/MST-3000: Image Promotion and Distribution Policy + - [Summary](#summary) - [Background (from wiki)](#background-from-wiki) - [Motivation](#motivation) @@ -94,13 +95,13 @@ Additionally we require a registry and artifact url-redirection solution to the - We are setting up an AWS account with an IAM role and s3 buckets in AWS regions where we see a large percentage of source image pull traffic - We will iterate on a sandbox url (registry.sandbox.k8s.io) for our experiments and ONLY promote things to (registry.k8s.io) when we have complete confidence -- both registry and registry.sandbox are serving traffic using oci-proxy on google cloud run +- both registry and registry-sandbox are serving traffic using oci-proxy on google cloud run - oci-proxy will be updated to identify incoming traffic from AWS regions based on IP ranges so we can route traffic to s3 buckets in that region. If a specific AWS region do not currently host s3 buckets, we will redirect to the nearest region which does have s3 buckets (tradeoff between storage and network costs) - We will bulk sync existing image layers to these s3 layers as a starting point (from GCS/GCR) - We will update image-promoter to push to these s3 buckets as well in addition to the current setup - We will set up monitoring/reporting to check on new costs we incur on the AWS infrastructure and update what we do in GCP infrastructure as well to include the new components - We will have a plan in place on how we could add additional AWS regions in the future -- We will have CI jobs that will run against registry.sandbox as well to monitor stability before we promote code to registry +- We will have CI jobs that will run against registry-sandbox.k8s.io as well to monitor stability before we promote code to registry - We will automate the deployment/monitoring and testing of code landing in the oci-proxy repository ### User Stories From 6a1b9bfa8d27014630d1f55a3d5c62d58319e4e3 Mon Sep 17 00:00:00 2001 From: Caleb Woodbine Date: Thu, 28 Apr 2022 16:26:30 +1200 Subject: [PATCH 08/17] Update readme --- keps/sig-release/3000-artifact-distribution/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/keps/sig-release/3000-artifact-distribution/README.md b/keps/sig-release/3000-artifact-distribution/README.md index cb13ae8cbb5..e67053d136e 100644 --- a/keps/sig-release/3000-artifact-distribution/README.md +++ b/keps/sig-release/3000-artifact-distribution/README.md @@ -117,15 +117,15 @@ When my PR is merged Then the promotion process occurs ``` -#### Cloud Customer - Installing K8s via kubeadm +#### Cloud Customer - pulling an official container image ```feature As a CLOUD end-user I want to install Kubernetes Given some compute resources at CLOUD -When I use kubeadm to deploy Kubernetes -Then I will be redirected to a local CLOUD registry +When I pull an official Kubernetes container image +Then I am redirected to a cloud provider backed bucket (set) or CDN otherwise fall back to k8s.gcr.io ``` ### Notes/Constraints/Caveats From f960d6f21b89a3d931d430e6d5e5ac18ce041107 Mon Sep 17 00:00:00 2001 From: Caleb Woodbine Date: Thu, 28 Apr 2022 16:26:39 +1200 Subject: [PATCH 09/17] Update readme --- .../3000-artifact-distribution/README.md | 31 ++++++++++--------- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/keps/sig-release/3000-artifact-distribution/README.md b/keps/sig-release/3000-artifact-distribution/README.md index e67053d136e..ccf6d8215b1 100644 --- a/keps/sig-release/3000-artifact-distribution/README.md +++ b/keps/sig-release/3000-artifact-distribution/README.md @@ -1,4 +1,4 @@ -# KEP/MST-3000: Image Promotion and Distribution Policy +# KEP 3000: Image Promotion and Distribution Policy @@ -47,9 +47,9 @@ For a few years now, we have been using k8s.gcr.io in all our repositories as de The cost of distributing Kubernetes comes at great cost nearing $150kUSD/month (mostly egress) in donations. -Additionally some of our community members are unable to access the official release artifacts due to country level firewalls that do not them connect to Google services. +Additionally some of our community members are unable to access the official release container images due to country level firewalls that do not them connect to Google services. -Ideally we can dramatically reduce cost and allow everyone in the world to download the artifacts released by our community. +Ideally we can dramatically reduce cost and allow everyone in the world to download the container iamges released by our community. We are now used to using the [image promoter process](https://github.com/kubernetes/enhancements/tree/master/keps/sig-release/1734-k8s-image-promoter) to promote images to the official kubernetes container registry using the infrastructure (GCR staging repos etc) provided by [sig-k8s-infra](https://github.com/kubernetes/k8s.io/tree/main/k8s.gcr.io) @@ -57,7 +57,7 @@ We are now used to using the [image promoter process](https://github.com/kuberne So far we (all kubernetes project) are using GCP as our default infrastructure provider for all things like GCS, GCR, GKE based prow clusters etc. Google has graciously sponsored a lot of our infrastructure costs as well. However for about a year or so we are finding that our costs are sky-rocketing because the community usage of this infrastructure has been from other cloud providers like AWS, Azure etc. So in conjunction with CNCF staff we are trying to put together a plan to host copies of images and binaries nearer to where they are used rather than incur cross-cloud costs. -One part of this plan is to setup a proxy OCI service, that can identify where the traffic is coming from and redirect to the nearest image layer/repository. This is why we are setting up a new service using what we call an [oci-proxy](https://github.com/kubernetes-sigs/oci-proxy) for everyone to use. This proxy will identify traffic coming from, for example, a certain AWS region, then will setup a HTTP redirect to a source in that AWS region. If we get traffic from GKE/GCP or we don't know where the traffic is coming from, it will still redirect to the current infrastructure (k8s.gcr.io). +One part of this plan is to setup a redirecting web service, that can identify where the traffic is coming from and redirect to the nearest image layer/repository. This is why we are setting up a new service using what we call an [oci-proxy](https://github.com/kubernetes-sigs/oci-proxy) for everyone to use. This redirector will identify traffic coming from, for example, a certain AWS region, then will setup a HTTP redirect to a source in that AWS region. If we get traffic from GKE/GCP or we don't know where the traffic is coming from, it will still redirect to the current infrastructure (k8s.gcr.io). ## How can we help? @@ -71,7 +71,7 @@ A solution to allow redirection to appropriate mirrors to lower cost and allow a ### Non-Goals -Anything related to creation of artifacts, bom, digital signatures, staging buckets. +Anything related to creation of artifacts, bom, staging buckets. ### What is not in scope @@ -87,14 +87,14 @@ Anything related to creation of artifacts, bom, digital signatures, staging buck There are two intertwined concepts that are part of this proposal. -First, the policy and procedures to promote/upload our artifacts to multiple providers. Our existing processes upload only to GCS buckets. Ideally we extend the existing software/promotion process to push directly to multiple providers. Alternatively we use a second process to synchronize artifacts from our existing production buckets to similar constructs at other providers. +First, the policy and procedures to promote/upload our container images to multiple providers. Our existing processes upload only to GCS buckets. Ideally we extend the existing software/promotion process to push directly to multiple providers. Alternatively we use a second process to synchronize container images from our existing production buckets to similar constructs at other providers. Additionally we require a registry and artifact url-redirection solution to the local cloud provider or country. ## What exactly are you doing? - We are setting up an AWS account with an IAM role and s3 buckets in AWS regions where we see a large percentage of source image pull traffic -- We will iterate on a sandbox url (registry.sandbox.k8s.io) for our experiments and ONLY promote things to (registry.k8s.io) when we have complete confidence +- We will iterate on a sandbox url (registry-sandbox.k8s.io) for our experiments and ONLY promote things to (registry.k8s.io) when we have complete confidence - both registry and registry-sandbox are serving traffic using oci-proxy on google cloud run - oci-proxy will be updated to identify incoming traffic from AWS regions based on IP ranges so we can route traffic to s3 buckets in that region. If a specific AWS region do not currently host s3 buckets, we will redirect to the nearest region which does have s3 buckets (tradeoff between storage and network costs) - We will bulk sync existing image layers to these s3 layers as a starting point (from GCS/GCR) @@ -109,8 +109,9 @@ Additionally we require a registry and artifact url-redirection solution to the #### SIG Release - Image Promotion ```feature -As a SIG Release volunteer -I want to promote our binaries/images to multiple clouds +Scenario: images are promoted + As a SIG Release volunteer + I want to promote our binaries/images to multiple clouds Given a promotion / manifest When my PR is merged @@ -120,12 +121,12 @@ Then the promotion process occurs #### Cloud Customer - pulling an official container image ```feature -As a CLOUD end-user -I want to install Kubernetes +Scenario: use Kubernetes container images + I want to be able to pull and use Kubernetes container images -Given some compute resources at CLOUD -When I pull an official Kubernetes container image -Then I am redirected to a cloud provider backed bucket (set) or CDN otherwise fall back to k8s.gcr.io + Given some compute resources at cloud + When I pull an official Kubernetes container image from registry.k8s.io + Then I am redirected to a close-by cloud provider backed bucket (set) / CDN otherwise fall back to k8s.gcr.io ``` ### Notes/Constraints/Caveats @@ -156,7 +157,7 @@ Currently the promotion process is primarily driven by the CIP/[promo-tool#kprom #### Process -Artifacts will be written to S3 style storage or CDNs provided by cloud providers through a tool in the promo-tools suite. +Container images will be written to S3 style storage or CDNs provided by cloud providers through a tool in the promo-tools suite. ## Alternatives / Background From 0e5fafb10766746015900d914606e9a5ea657733 Mon Sep 17 00:00:00 2001 From: Caleb Woodbine Date: Thu, 28 Apr 2022 16:26:45 +1200 Subject: [PATCH 10/17] Update kep.yaml --- keps/sig-release/3000-artifact-distribution/kep.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/keps/sig-release/3000-artifact-distribution/kep.yaml b/keps/sig-release/3000-artifact-distribution/kep.yaml index 408b45cebf0..54aa71bd5cb 100644 --- a/keps/sig-release/3000-artifact-distribution/kep.yaml +++ b/keps/sig-release/3000-artifact-distribution/kep.yaml @@ -19,6 +19,6 @@ approvers: stage: alpha latest-milestone: "v1.24" milestone: - alpha: "v1.24" - beta: "v1.25" - stable: "v1.26" + alpha: "v1.25" + beta: "v1.26" + stable: "v1.27" From f07424500a7d9a1c82d620d5a4bb27254046827c Mon Sep 17 00:00:00 2001 From: Caleb Woodbine Date: Fri, 6 May 2022 14:52:34 +1200 Subject: [PATCH 11/17] Add request handling source: Ben's doc (kubernetes-sigs/oci-proxy/cmd/archeio/docs/request-handling.md) --- .../3000-artifact-distribution/README.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/keps/sig-release/3000-artifact-distribution/README.md b/keps/sig-release/3000-artifact-distribution/README.md index ccf6d8215b1..1092aca38ed 100644 --- a/keps/sig-release/3000-artifact-distribution/README.md +++ b/keps/sig-release/3000-artifact-distribution/README.md @@ -104,6 +104,21 @@ Additionally we require a registry and artifact url-redirection solution to the - We will have CI jobs that will run against registry-sandbox.k8s.io as well to monitor stability before we promote code to registry - We will automate the deployment/monitoring and testing of code landing in the oci-proxy repository +### registry.k8s.io request handling + +Requests to [registry.k8s.io](https://registry.k8s.io) follows the following flow: + +1. If it's a request for `/`: redirect to our wiki page about the project +2. If it's not a request for `/` and does not start with `/v2/`: 404 error +3. For registry API requests, all of which start with `/v2/`: + +- If it's not a blob request: redirect to _Upstream Registry_ +- If it's not a known AWS IP: redirect to _Upstream Registry_ +- If it's a known AWS IP AND HEAD request for the layer succeeeds in S3: redirect to S3 +- If it's a known AWS IP AND HEAD fails: redirect to _Upstream Registry_ + +Currently the _Upstream Registry_ is https://k8s.gcr.io. + ### User Stories #### SIG Release - Image Promotion From 4a7709974b08a1730d9d1643306d0d3f550d439e Mon Sep 17 00:00:00 2001 From: Caleb Woodbine Date: Tue, 10 May 2022 09:29:05 +1200 Subject: [PATCH 12/17] Fix spelling and update goal --- .../3000-artifact-distribution/README.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/keps/sig-release/3000-artifact-distribution/README.md b/keps/sig-release/3000-artifact-distribution/README.md index 1092aca38ed..17e3c5b2458 100644 --- a/keps/sig-release/3000-artifact-distribution/README.md +++ b/keps/sig-release/3000-artifact-distribution/README.md @@ -49,7 +49,7 @@ The cost of distributing Kubernetes comes at great cost nearing $150kUSD/month ( Additionally some of our community members are unable to access the official release container images due to country level firewalls that do not them connect to Google services. -Ideally we can dramatically reduce cost and allow everyone in the world to download the container iamges released by our community. +Ideally we can dramatically reduce cost and allow everyone in the world to download the container images released by our community. We are now used to using the [image promoter process](https://github.com/kubernetes/enhancements/tree/master/keps/sig-release/1734-k8s-image-promoter) to promote images to the official kubernetes container registry using the infrastructure (GCR staging repos etc) provided by [sig-k8s-infra](https://github.com/kubernetes/k8s.io/tree/main/k8s.gcr.io) @@ -65,7 +65,7 @@ When Kubernetes master opens up for v1.25 development, we need to update all def ### Goals -A policy and procedure for use by SIG Release to promote container images and release binaries to multiple registries and mirrors. +A policy and procedure for use by SIG Release to promote container images to multiple registries and mirrors. A solution to allow redirection to appropriate mirrors to lower cost and allow access from any cloud or country globally. @@ -114,7 +114,7 @@ Requests to [registry.k8s.io](https://registry.k8s.io) follows the following flo - If it's not a blob request: redirect to _Upstream Registry_ - If it's not a known AWS IP: redirect to _Upstream Registry_ -- If it's a known AWS IP AND HEAD request for the layer succeeeds in S3: redirect to S3 +- If it's a known AWS IP AND HEAD request for the layer succeeds in S3: redirect to S3 - If it's a known AWS IP AND HEAD fails: redirect to _Upstream Registry_ Currently the _Upstream Registry_ is https://k8s.gcr.io. @@ -126,11 +126,11 @@ Currently the _Upstream Registry_ is https://k8s.gcr.io. ```feature Scenario: images are promoted As a SIG Release volunteer - I want to promote our binaries/images to multiple clouds + I want to promote our images to multiple clouds -Given a promotion / manifest -When my PR is merged -Then the promotion process occurs + Given a promotion / manifest + When my PR is merged + Then the promotion process occurs ``` #### Cloud Customer - pulling an official container image @@ -183,13 +183,13 @@ Container images will be written to S3 style storage or CDNs provided by cloud p - KubeCon Talk - https://www.youtube.com/watch?v=F2IFjz7sr9Q - Apache has a widespread mirror network - - @dims has experince here + - @dims has experience here - http://ws.apache.org/mirrors.cgi - https://infra.apache.org/mirrors.html - [Umbrella issue: k8s.gcr.io => registry.k8s.io solution k/k8s.io#1834 ](https://github.com/kubernetes/k8s.io/issues/1834) - [ii/registry.k8s.io Implementation proposals](https://github.com/ii/registry.k8s.io#registryk8sio) -- [ii.nz/blog :: Building a data pipline for displaying Kubernetes public artifact traffic +- [ii.nz/blog :: Building a data pipeline for displaying Kubernetes public artifact traffic ](https://ii.nz/post/building-a-data-pipline-for-displaying-kubernetes-public-artifact-traffic/) ### How much is this going to save us? From 2abfc73e62947c7c4104ac744a85c148064ff054 Mon Sep 17 00:00:00 2001 From: Caleb Woodbine Date: Wed, 11 May 2022 08:41:12 +1200 Subject: [PATCH 13/17] Fix toc --- keps/sig-release/3000-artifact-distribution/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/keps/sig-release/3000-artifact-distribution/README.md b/keps/sig-release/3000-artifact-distribution/README.md index 17e3c5b2458..07b5ea8fe77 100644 --- a/keps/sig-release/3000-artifact-distribution/README.md +++ b/keps/sig-release/3000-artifact-distribution/README.md @@ -1,7 +1,6 @@ # KEP 3000: Image Promotion and Distribution Policy - - [Summary](#summary) - [Background (from wiki)](#background-from-wiki) - [Motivation](#motivation) @@ -13,9 +12,10 @@ - [What are good goals to shoot for](#what-are-good-goals-to-shoot-for) - [Proposal](#proposal) - [What exactly are you doing?](#what-exactly-are-you-doing) + - [registry.k8s.io request handling](#registryk8sio-request-handling) - [User Stories](#user-stories) - [SIG Release - Image Promotion](#sig-release---image-promotion) - - [Cloud Customer - Installing K8s via kubeadm](#cloud-customer---installing-k8s-via-kubeadm) + - [Cloud Customer - pulling an official container image](#cloud-customer---pulling-an-official-container-image) - [Notes/Constraints/Caveats](#notesconstraintscaveats) - [Risks and Mitigations](#risks-and-mitigations) - [Design Details](#design-details) From 6bfecc8a53a8bb41ab2501981de5fb0aad4e5700 Mon Sep 17 00:00:00 2001 From: Caleb Woodbine Date: Wed, 11 May 2022 08:50:30 +1200 Subject: [PATCH 14/17] Update approvers --- keps/sig-release/3000-artifact-distribution/kep.yaml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/keps/sig-release/3000-artifact-distribution/kep.yaml b/keps/sig-release/3000-artifact-distribution/kep.yaml index 54aa71bd5cb..d895dfcd9f4 100644 --- a/keps/sig-release/3000-artifact-distribution/kep.yaml +++ b/keps/sig-release/3000-artifact-distribution/kep.yaml @@ -15,7 +15,9 @@ reviewers: - "@thockin" approvers: - "@ameukam" + - "@dims" - "@justaugustus" + - "@saschagrunert" stage: alpha latest-milestone: "v1.24" milestone: From 7030358b8e184b964dc4946134a71610eaff37a1 Mon Sep 17 00:00:00 2001 From: Caleb Woodbine Date: Wed, 11 May 2022 09:58:33 +1200 Subject: [PATCH 15/17] Update cost saving analysis section notes Co-Authored-By: Riaan Kleinhans --- keps/sig-release/3000-artifact-distribution/README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/keps/sig-release/3000-artifact-distribution/README.md b/keps/sig-release/3000-artifact-distribution/README.md index 07b5ea8fe77..1a5bdb40bc4 100644 --- a/keps/sig-release/3000-artifact-distribution/README.md +++ b/keps/sig-release/3000-artifact-distribution/README.md @@ -1,6 +1,7 @@ # KEP 3000: Image Promotion and Distribution Policy + - [Summary](#summary) - [Background (from wiki)](#background-from-wiki) - [Motivation](#motivation) @@ -198,6 +199,8 @@ Cost of K8s Artifact hosting - Data Studio Graphs ![](https://i.imgur.com/LAn4UIE.png) +Analysis has been done on usage patterns related to providers. AWS participated in this process and have a keen interest to help drive down cost by providing artifacts directly to their clients consuming resources from the public registry. + ## Infrastructure Needed It would be good to request some donations for some larger providers, including one in China, via cncf.io/credits From f15fdd6309400d184830d3fccde4192396fabd8c Mon Sep 17 00:00:00 2001 From: Hippie Hacker Date: Wed, 11 May 2022 15:59:02 +1200 Subject: [PATCH 16/17] Removing un-necessary sections --- .../3000-artifact-distribution/README.md | 63 ------------------- 1 file changed, 63 deletions(-) diff --git a/keps/sig-release/3000-artifact-distribution/README.md b/keps/sig-release/3000-artifact-distribution/README.md index 1a5bdb40bc4..1ff0e06499c 100644 --- a/keps/sig-release/3000-artifact-distribution/README.md +++ b/keps/sig-release/3000-artifact-distribution/README.md @@ -34,16 +34,6 @@ ## Summary -The container images and release binaries produced by our community need a clear path to be hosted by multiple service/cloud providers. - -The global community should be routed to the appropriate mirror for their country or cloud provider to ensure cost effective worldwide access. - -This KEP should cover the policy and distribution mechanisms we will put in place to allow creating a globally distributed, multi-cloud and country solution. - -## Background (from wiki) - -## Motivation - For a few years now, we have been using k8s.gcr.io in all our repositories as default repository for downloading images from. The cost of distributing Kubernetes comes at great cost nearing $150kUSD/month (mostly egress) in donations. @@ -120,31 +110,6 @@ Requests to [registry.k8s.io](https://registry.k8s.io) follows the following flo Currently the _Upstream Registry_ is https://k8s.gcr.io. -### User Stories - -#### SIG Release - Image Promotion - -```feature -Scenario: images are promoted - As a SIG Release volunteer - I want to promote our images to multiple clouds - - Given a promotion / manifest - When my PR is merged - Then the promotion process occurs -``` - -#### Cloud Customer - pulling an official container image - -```feature -Scenario: use Kubernetes container images - I want to be able to pull and use Kubernetes container images - - Given some compute resources at cloud - When I pull an official Kubernetes container image from registry.k8s.io - Then I am redirected to a close-by cloud provider backed bucket (set) / CDN otherwise fall back to k8s.gcr.io -``` - ### Notes/Constraints/Caveats The primary purpose of the KEP is getting consensus on the agreed policy and procedure to unblock our community and move forward together. @@ -155,26 +120,6 @@ There has been a lot of activity around the technology and tooling for both goal This is the primary pipeline for delivering Kubernetes worldwide. Ensuring the appropriate SLAs and support as well as artifact integrity is crucial. -## Design Details - -### Release Promotion - -#### Policy - -(more details needed, #sig-release-eng?) - -#### Process - -Currently the promotion process is primarily driven by the CIP/[promo-tool#kpromo](https://github.com/kubernetes-sigs/promo-tools#kpromo)? - -### Artifact Distribution - -#### Policy - -#### Process - -Container images will be written to S3 style storage or CDNs provided by cloud providers through a tool in the promo-tools suite. - ## Alternatives / Background - Original KEP @@ -201,11 +146,3 @@ Cost of K8s Artifact hosting - Data Studio Graphs Analysis has been done on usage patterns related to providers. AWS participated in this process and have a keen interest to help drive down cost by providing artifacts directly to their clients consuming resources from the public registry. -## Infrastructure Needed - -It would be good to request some donations for some larger providers, including one in China, via cncf.io/credits - -## Hack this doc - -- [![hackmd-github-sync-badge](https://hackmd.io/KjHufZssQR654ShkZFUzyA/badge)](https://hackmd.io/KjHufZssQR654ShkZFUzyA) -- [kubernetes/enhancements!3079](https://github.com/kubernetes/enhancements/pull/3079) From 4b14ce8bf136ebc5bd40ef0e327c6d92d9a613d8 Mon Sep 17 00:00:00 2001 From: Hippie Hacker Date: Wed, 11 May 2022 15:59:34 +1200 Subject: [PATCH 17/17] update ToC --- .../3000-artifact-distribution/README.md | 15 --------------- 1 file changed, 15 deletions(-) diff --git a/keps/sig-release/3000-artifact-distribution/README.md b/keps/sig-release/3000-artifact-distribution/README.md index 1ff0e06499c..a0afe0a0698 100644 --- a/keps/sig-release/3000-artifact-distribution/README.md +++ b/keps/sig-release/3000-artifact-distribution/README.md @@ -1,10 +1,7 @@ # KEP 3000: Image Promotion and Distribution Policy - - [Summary](#summary) -- [Background (from wiki)](#background-from-wiki) -- [Motivation](#motivation) - [Why a new domain?](#why-a-new-domain) - [How can we help?](#how-can-we-help) - [Goals](#goals) @@ -14,22 +11,10 @@ - [Proposal](#proposal) - [What exactly are you doing?](#what-exactly-are-you-doing) - [registry.k8s.io request handling](#registryk8sio-request-handling) - - [User Stories](#user-stories) - - [SIG Release - Image Promotion](#sig-release---image-promotion) - - [Cloud Customer - pulling an official container image](#cloud-customer---pulling-an-official-container-image) - [Notes/Constraints/Caveats](#notesconstraintscaveats) - [Risks and Mitigations](#risks-and-mitigations) -- [Design Details](#design-details) - - [Release Promotion](#release-promotion) - - [Policy](#policy) - - [Process](#process) - - [Artifact Distribution](#artifact-distribution) - - [Policy](#policy-1) - - [Process](#process-1) - [Alternatives / Background](#alternatives--background) - [How much is this going to save us?](#how-much-is-this-going-to-save-us) -- [Infrastructure Needed](#infrastructure-needed) -- [Hack this doc](#hack-this-doc) ## Summary