Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable large github runners for etcd-io org #4590

Closed
jmhbnz opened this issue Nov 18, 2023 · 11 comments
Closed

Enable large github runners for etcd-io org #4590

jmhbnz opened this issue Nov 18, 2023 · 11 comments

Comments

@jmhbnz
Copy link
Member

jmhbnz commented Nov 18, 2023

Organization or Repo

etcd-io/etcd

User affected

@serathius, @ahrtr

Describe the issue

After migrating the etcd-io org out from under the CNCF enterprise account to under the Kubernetes enterprise account, large runners need to be re-enabled: https://docs.github.com/en/actions/using-github-hosted-runners/about-larger-runners/about-larger-runners

etcd ci workflows are currently unable to complete, refer: https://github.com/etcd-io/etcd/actions/runs/6915180325/job/18813834949?pr=16959

etcd requires large runners ubuntu-latest-8-cores for robustness testing.

CNCF projects have accesses to larger instances, but it has to be explicitly enabled by an admin of https://github.com/etcd-io :

@ahrtr
Copy link
Member

ahrtr commented Nov 18, 2023

  • Click Selected repositories, and choose the etcd repo

bbolt also needs larger runner.

@mrbobbytables
Copy link
Member

I had to create the large runner definition^^;; I'm guessing those were created previously in the CNCF enterprise account. Let me know if you have any issues.

@jmhbnz
Copy link
Member Author

jmhbnz commented Nov 19, 2023

I had to create the large runner definition^^;; I'm guessing those were created previously in the CNCF enterprise account. Let me know if you have any issues.

Confirming the larger runners are now working once more. Thanks @mrbobbytables 🙏🏻

@jmhbnz jmhbnz closed this as completed Nov 19, 2023
@ahrtr
Copy link
Member

ahrtr commented Nov 20, 2023

Thanks @mrbobbytables for the quick fix.

@ahrtr
Copy link
Member

ahrtr commented Nov 20, 2023

I see that one bbolt workflow (runs on ubuntu-latest-8-cores) is getting stucked for almost 2 hours https://github.com/etcd-io/bbolt/actions/runs/6928339142/job/18843944830

@jmhbnz @mrbobbytables

@serathius
Copy link
Contributor

serathius commented Nov 20, 2023

Observed same on the robustness tests.
image

@mrbobbytables
Copy link
Member

sorry about all this back and forth =/
I know the issue now - I'm going to delete the runners scoped in the etcd org and create them at the k8s enterprise org and still grant access to the same repos. It's where we can easily centrally manage the limits and such.

@mrbobbytables
Copy link
Member

Okay, I've updated it - I've set the billing limit at 2k for now. Let me know if you have problems. We MAY need an azure subscription for it to all work correctly >_> I've pinged a few folk but with the US holiday things might not get looked at till next week. The GitHub admin team also has a meeting with GitHub next Tuesday so any issues we encounter we can surface there as an escalation.

/reopen
(till we know everything is good)

@k8s-ci-robot
Copy link
Contributor

@mrbobbytables: Reopened this issue.

In response to this:

Okay, I've updated it - I've set the billing limit at 2k for now. Let me know if you have problems. We MAY need an azure subscription for it to all work correctly >_> I've pinged a few folk but with the US holiday things might not get looked at till next week. The GitHub admin team also has a meeting with GitHub next Tuesday so any issues we encounter we can surface there as an escalation.

/reopen
(till we know everything is good)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot reopened this Nov 20, 2023
@ahrtr
Copy link
Member

ahrtr commented Nov 20, 2023

Thanks @mrbobbytables for the quick fix. Confirmed that the bbolt workflow is running well right now.

@jmhbnz
Copy link
Member Author

jmhbnz commented Jan 3, 2024

Large ubuntu-latest-8-cores runners have been working nicely since 21st November 🙏🏻

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants