Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add more S3 regions #172

Open
BenTheElder opened this issue Mar 11, 2023 · 21 comments
Open

Add more S3 regions #172

BenTheElder opened this issue Mar 11, 2023 · 21 comments
Assignees
@ameukam
Labels
lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra.
Milestone
v1.31

Comments

@BenTheElder
Copy link
Member

South America is the obvious missing one, but we should also consider adding others.

xref: kubernetes/k8s.io#4739 (comment) and discussion in https://kubernetes.slack.com/archives/CCK68P2Q2/p1678504635523609

/sig k8s-infra
/priority important-soon

When we add a bucket it will have to be added to the terraform config in k8s.io including the s3 to s3 replication, then once the bucket is reasonably well populated we'll want to add it to both https://github.com/kubernetes/registry.k8s.io/blob/8408d0501a88b3d2531ff54b14eeb0e3c900a4f3/cmd/archeio/app/buckets.go (AWS clients) and https://github.com/kubernetes/k8s.io/blob/5821271f875cc970b5701983e5695a025b5d952f/infra/gcp/terraform/k8s-infra-oci-proxy-prod/terraform.tfvars (non-GCP non-AWS clients)

This is much more relevant after #147
@k8s-ci-robot k8s-ci-robot added sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Mar 11, 2023
@BenTheElder BenTheElder mentioned this issue Mar 11, 2023
Merged
@ameukam
Copy link
Member

ameukam commented Mar 11, 2023

/assign

/area infra/aws
/milestone v1.27

@k8s-ci-robot
Copy link
Contributor

@ameukam: The label(s) area/infra/aws cannot be applied, because the repository doesn't have them.

In response to this:

/assign

/area infra/aws
/milestone v1.27

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@BenTheElder
Copy link
Member Author

We should ideally have:

  1. All regions with notable client usage within AWS, for the AWS IP / region => bucket mapping
  2. All regions that map closely to our Cloud Run / AR GCP regions, for external clients with the Cloud Run instance => bucket mapping

We started with a good approximation of 1) in #39 / #72, but it might be worth revisiting.

For 2) we have an okay approximation reusing the same set from 1), but we know there's at least the South America gap and should definitely re-evaluate and select the remaining regions that would most closely match now that we've moved to sending non-AWS-non-GCP traffic to S3 generally.

@ameukam
Copy link
Member

ameukam commented Mar 13, 2023

List of remaining regions we can add:

  • ap-east-1
  • ap-northeast-2
  • ap-southeast-1
  • ap-southeast-3
  • ap-southeast-4
  • eu-central-1
  • eu-central-2
  • ca-central-1
  • sa-east-1

@BenTheElder
Copy link
Member Author

There's an inflection point where adding a region is wasteful.

If we are matching the IP to region with 1) for users within AWS and it only represents a very small amount of traffic within AWS then it's probably not worth it for 1), more regions than Cloud Run is not useful for 2) since we lean on the GCLB routing to determine approximate region for external traffic.

@BenTheElder
Copy link
Member Author

OK so we looked at the costs today and storage costs are likely going to be smaller than traffic in basically any AWS region with actual traffic.

So just adding them for 1) is going to be generally worthwhile.

xref: https://kubernetes.slack.com/archives/CCK68P2Q2/p1678915249434949

@BenTheElder
Copy link
Member Author

ref: #181 for hold a bit on production changes

@BenTheElder BenTheElder mentioned this issue Mar 23, 2023
Open
@ameukam ameukam added this to the v1.28 milestone Mar 23, 2023
@dims
Copy link
Member

dims commented Mar 25, 2023

cc @rothgar

@ameukam
Copy link
Member

ameukam commented Mar 27, 2023 •
edited
Loading

New AR regions added:

@ameukam ameukam mentioned this issue Mar 27, 2023
Merged
@dims
Copy link
Member

dims commented Apr 11, 2023

xref - some data from @TerryHowe here:
https://kubernetes.slack.com/archives/CCK68P2Q2/p1681223002601649

@TerryHowe
Copy link
Member

Adding more regions would be easy, but cloudfront should maybe be considered.

@BenTheElder
Copy link
Member Author

There's another issue tracking CloudFront.

I think the idea is we still want to route in-AWS traffic with S3 in any region where cost to store < cost to serve cross region / cloud front. Which should be many regions.

CloudFront is probably the right answer for egress otherwise?

@BenTheElder
Copy link
Member Author

Agree it should be easy, but we've avoided anything altering production behavior through securing the GCR redirect #181

@BenTheElder BenTheElder mentioned this issue Apr 27, 2023
Closed
2 tasks
@ameukam ameukam mentioned this issue Apr 27, 2023
Closed
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 11, 2023
@ameukam
Copy link
Member

ameukam commented Jul 11, 2023

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 11, 2023
@ameukam ameukam modified the milestones: v1.28, v1.29 Sep 4, 2023
ameukam added a commit to ameukam/k8s.io that referenced this issue Oct 10, 2023
@ameukam
AWS: Deny non-SSL transport and define bucket ownership
c788f79 
Related:
  - kubernetes/registry.k8s.io#172

Attach a policy ensuring non-TLS connections are denied.
Ensure bucket ownership for each bucket.
Ensure account policy for public buckets are disabled.
Change logic to generate bucket name.

Signed-off-by: Arnaud Meukam <[email protected]>
@ameukam ameukam mentioned this issue Oct 10, 2023
Merged
ameukam added a commit to ameukam/k8s.io that referenced this issue Oct 13, 2023
@ameukam
AWS: Define policy access for S3 buckets.
d747853 
Related to:
  -  kubernetes/registry.k8s.io#172

Define and attach a policy that make a S3 bucket world-readable.

Signed-off-by: Arnaud Meukam <[email protected]>
This was referenced Oct 13, 2023
Merged
Merged
ameukam added a commit to ameukam/k8s.io that referenced this issue Oct 18, 2023
@ameukam
AWS: Add a replication role for registry.k8s.io
7ef0533 
Related to:
 - kubernetes/registry.k8s.io#172

Define permissions needed to use S3 replication in the same account.

Signed-off-by: Arnaud Meukam <[email protected]>
@ameukam ameukam mentioned this issue Oct 18, 2023
Merged
@ameukam
Copy link
Member

ameukam commented Dec 7, 2023

Egress cost for the month of November 2023:

image

@ameukam ameukam mentioned this issue Dec 16, 2023
Merged
@ameukam ameukam modified the milestones: v1.29, v1.30 Jan 29, 2024
@ameukam ameukam moved this from 🔖 Ready to 🏗 In progress in AWS Infrastructure (SIG K8s Infra) Jan 29, 2024
@ameukam ameukam moved this from Backlog to Implementation in registry.k8s.io (SIG K8S Infra) Jan 29, 2024
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 28, 2024
@ameukam
Copy link
Member

ameukam commented Apr 29, 2024

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 29, 2024
@ameukam
Copy link
Member

ameukam commented Apr 29, 2024

/milestone v1.31

@ameukam ameukam modified the milestones: v1.30, v1.31 Apr 29, 2024
@BenTheElder BenTheElder added priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. and removed priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Jul 25, 2024
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 23, 2024
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle rotten
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Nov 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ameukam ameukam

Labels
lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra.
Projects
AWS Infrastructure (SIG K8s Infra)
Status: 🏗 In progress
registry.k8s.io (SIG K8S Infra)
Status: Implementation
Milestone
v1.31
Development

No branches or pull requests
6 participants
@dims @TerryHowe @BenTheElder @ameukam @k8s-ci-robot @k8s-triage-robot