Migrate away from google.com gcp project k8s-federated-conformance

[spiffxp]

Part of umbrella issue to migrate away from google.com gcp projects: #1469

Umbrella issue for migrating or removing dependence on all of the kubernetes project infra that lives under k8s-federated-conformance. For each of these buckets there may also exist corresponding service accounts.

• gs://k8s-conformance/
• gs://k8s-conformance-alibaba/
• gs://k8s-conformance-arm/
• gs://k8s-conformance-baiducloud/
• gs://k8s-conformance-demo/
• gs://k8s-conformance-digitalocean/
• gs://k8s-conformance-kind-arm64-openlab/
• gs://k8s-conformance-openstack/
• gs://k8s-conformance-oracle-cloud/

Of note: we have a system already in place to support this (ref: https://github.com/kubernetes/k8s.io/blob/master/infra/gcp/ensure-conformance-storage.sh), so this is mostly a matter of contacting and pushing migration along

[BenTheElder]

I audited each of these by checking the contact info I have listed on the associated service account keys and gsutil ls -l gs://${BUCKET}/** | sort -k 2 to inspect timestamps + keys of uploaded entries.

• gs://k8s-conformance: delete
  • This one is literally empty, nothing to do, we can just delete it.
  • There's no contact / account for this, I created this and migrated initial uploaders to individual buckets. Anything in it aged out.
• gs://k8s-conformance-alibaba: delete
  • 2020-10-14T13:47:10Z is the most recent entry. I do not believe this is being used for CI results currently, and we did not offer this bucket for any other purpose.
  • They can reapply for a wg-k8s-infra bucket with the updated instructions if they wish to publish continuous conformance results in the future
  • We can delete this.
  • Contact is @aoxn -- FYI this is not the intended use of this, if you're interested in running results against Kubernetes at HEAD and submitting them please re-apply through the new instructions for this repo.
• gs://k8s-conformance-arm: migrate delete
  • This one is active, as of 2021-02-12T21:22:47Z
  • @lubinsz is the contact for this one
  • We should help migrate this one.
  • Lack of response, ultimately deleted
• gs://k8s-conformance-baiducloud: delete
  • Same story as conformance-alibaba, this isn't running continuous results for Kubernetes, it's just some sporadic uploads at old versions (newest is 1.13 in march last year). That's not the purpose of this program, they should be submitting to the CNCF conformance results instead
  • We can delete this.
• gs://k8s-conformance-demo: delete
  • This was provided to @andrewsykim when circulating the concept to SIG cloud provider, to show how this works and how easy it can be. This serves no purpose now.
  • We can delete this.
• gs://k8s-conformance-digitalocean: delete
  • This was last updated 2018-10-08T16:46:32Z. It was only uploaded a few times 2018-09 onwards.|
  • @andrewsykim was the contact but has since changed employers, however it's unused anyhow.
  • We can delete this.
• gs://k8s-conformance-kind-arm64-openlab: delete
  • @dims and Melvin Hillsman (I have email but not github handy) are contacts on this one
  • Last uploaded 2020-05-1308:42:58Z. Did upload for some time but has been inactive since then
  • We can delete this, if they're interested in fixing the infra they can apply for a new bucket/key.
• gs://k8s-conformance-openstack: migrate
  • @dims is again the listed contact
  • This one is quite active, last result 2021-02-12T17:36:21Z and it even has junit_runner.xml etc, and against k8s 1.20
  • We should help migrate this one.
• gs://k8s-conformance-oracle-cloud: delete
  • I have contact emails but not GitHub
  • This was last updated 2020-03-11T11:27:56Z, the highest version I see is 1.15.
  • Does not appear to be in current use for nearly a year ... they can reapply.
  • We can delete this.

[BenTheElder]

I have taken no action yet, but proposed action and detailed reasoning per bucket is above. TLDR:

• Migrate gs://k8s-conformance-openstack and gs://k8s-conformance-arm which are still active, following guide at:
  https://github.com/kubernetes/test-infra/tree/master/testgrid/conformance#usage-guide
  • @dims @lubinsz to get in touch with the appropriate teams
• Delete everything else ~now as it's all inactive anyhow.
• Delete these two buckets once migrated.
• Wind down this project.

[dims]

@BenTheElder both buckets have the same name? (or a typo)

Either way, +1 to removing the one that lists Melvin as well.

[BenTheElder]

fixed! yep they are in the same order as @spiffxp's opening comment, bad re-formatting -- the first bucket should have read k8s-conformance-kind-arm64-openlab which is inactive.

[chrigl]

@BenTheElder I will take care of migrating gs://k8s-conformance-openstack

[BenTheElder]

Thank you!!

[spiffxp]

I have taken no action yet, but proposed action and detailed reasoning per bucket is above

/assign @BenTheElder
this plan LGTM, I'm gonna put in our current milestone

/milestone v1.21

[BenTheElder]

Taking action on the unused buckets, will update the checklist.

[BenTheElder]

We are down to the two marked to migrate, and it appears @chrigl already migrated openstack (Thank you!), leaving that one alone until this is confirmed though 🙂

[chrigl]

@BenTheElder At least from test runs, there should no new content go into gs://k8s-conformance-openstack

[spiffxp]

/milestone v1.22

[spiffxp]

/assign @chrigl
Keeping track of who's helped close this out

[BenTheElder]

Small update: deleting the old k8s-conformance-openstack bucket and creds now.

I think we need to set a responsiveness deadline for the remaining bucket. pinging again @lubinsz

Worst case I don't think it's a big deal if we lose older results from an unresponsive bucket and a new bucket can be easily requested.

[BenTheElder]

Only gs://k8s-conformance-arm + matching service account key remains. All other buckets have been migrated or otherwise cleaned up.

[BenTheElder]

Spoke to @spiffxp, setting a deadline of 4 weeks and reaching out today by:

• github
• slack
• email

After 4 weeks (people could be on vacation right now, and this is not maximally urgent) we will soft delete by revoking the service account permissions.
I'll ping again then and after say another week actually delete the bucket, to be replaced by a new request to this project.

[BenTheElder]

The 4 weeks deadline has more than elapsed, it has been 6 weeks.

Time to soft-revoke permissions.

[BenTheElder]

Last upload is 2021-05-14 as-is.

[BenTheElder]

I have deleted "Storage Object Creator" from the associated service account for gs://k8s-conformance-arm.

[spiffxp]

/milestone v1.23

[BenTheElder]

We'll just miss it, tomorrow I intend to delete the remaining bucket and close this out. It'll have been a week without the ability to write to the bucket.

[BenTheElder]

Bucket is deleting.

[BenTheElder]

This is done.
/close

[k8s-ci-robot]

@BenTheElder: Closing this issue.

In response to this:

This is done.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.