Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for volumeClaimTemplates #612

Closed
mumoshu opened this issue Jun 8, 2021 · 8 comments
Closed

Add support for volumeClaimTemplates #612

mumoshu opened this issue Jun 8, 2021 · 8 comments
Labels
enhancement New feature or request
Milestone
v0.22.0

Comments

@mumoshu
Copy link
Collaborator

mumoshu commented Jun 8, 2021

Like Kubernetes StatefulSet, investigate adding support for volumeClaimTemplates to e.g. Runner spec, so that you can use two or more pods per node while leveraging docker layer caching and other custom caching using Kubernetes persistent volumes.

Theoretically, any functionality that can dynamically resize pool of PVCs and PVs and attach PVs to each runner pod should work. It's basically what volumeClaimTemplates does and I thought it's better to start modeling it after that.
@mumoshu mumoshu added the enhancement New feature or request label Jun 8, 2021
@mumoshu mumoshu mentioned this issue Jun 8, 2021
Closed
@inahga
Copy link
Contributor

inahga commented Jun 10, 2021

I had thought generic ephemeral volumes would fulfill this need https://kubernetes.io/docs/concepts/storage/ephemeral-volumes/#generic-ephemeral-volumes. Unfortunately it's alpha in Kubernetes 1.20, and beta in 1.21.

@mumoshu
Copy link
Collaborator Author

mumoshu commented Jun 10, 2021 •
edited
Loading

@inahga Hey! Thanks for the info. But I'm not sure how ephemeral volumes help. If it is ReadWriteOnce, any data doesn't persist between pod recreations, right? That defeats our goal of caching data. If it is ReadWriteMany, data is shared by you cant actually use it for docker layer caching, as /var/lib/docker is not designed to be shared by multiple dockerd processes.

I may be be wrong though. Could you confirm?

@inahga
Copy link
Contributor

inahga commented Jun 10, 2021

If it is ReadWriteOnce, any data doesn't persist between pod recreations, right?

Ah I think you're right. I was thinking you could set the reclaimPolicy of the PVC's storageClass to retain, but it seems then the backing volume won't be reused, rather it'll just sit there until manual deletion. https://kubernetes.io/docs/concepts/storage/persistent-volumes/#retain

Still though it might depend on the storage driver. I think sig-local-storage-provisioner for instance you can avoid scrubbing any released volumes--but I doubt other storage drivers work that way.

Either way thanks for clarifying the use case, looks like your proposed approach should fit it well.

@Puneeth-n
Copy link
Contributor

@mumoshu This would be dope :) I am currently using emptyDir and running 1 runner per instance. The dockerVolumeMounts feature finally enabled me to switch to GHA from Jenkins for our heavier workloads and huge docker images 😅

@mumoshu
Copy link
Collaborator Author

mumoshu commented Jun 11, 2021 •
edited
Loading

@Puneeth-n I've just read your story here #452 (comment). Congrats! 🎉

@Puneeth-n
Copy link
Contributor

The only reason we had Jenkins was because of the docker layer caching as our base images are huge. Huge base images + lots of services in the monorepo that leverage on these base images made it very difficult for me to move away from Jenkins.

Now I wast GitHub to allow different parameter types like choices in workflow_dispatch so that I can completely get rid of Jenkins

@AndreaSoldino-TomTom
Copy link

@Puneeth-n thank you for your kind words! Happy to see this work improving your quality of life 🎉

@mumoshu mumoshu mentioned this issue Jun 12, 2021
Merged
mumoshu added a commit that referenced this issue Jun 13, 2021
@mumoshu
feat: RunnerSet backed by StatefulSet
dc0e1bf 
Unlike a runner deployment, a runner set can manage a set of stateful runners by combining a statefulset and an admission webhook that mutates statefulset-managed pods with required envvars and registration tokens.

Resolves #613
Ref #612
mumoshu added a commit that referenced this issue Jun 22, 2021
@mumoshu @toast-gear @callum-tait-pbx
feat: RunnerSet backed by StatefulSet (#629)
9e4dbf4 
* feat: RunnerSet backed by StatefulSet

Unlike a runner deployment, a runner set can manage a set of stateful runners by combining a statefulset and an admission webhook that mutates statefulset-managed pods with required envvars and registration tokens.

Resolves #613
Ref #612

* Upgrade controller-runtime to 0.9.0

* Bump Go to 1.16.x following controller-runtime 0.9.0

* Upgrade kubebuilder to 2.3.2 for updated etcd and apiserver following local setup

* Fix startup failure due to missing LeaderElectionID

* Fix the issue that any pods become unable to start once actions-runner-controller got failed after the mutating webhook has been registered

* Allow force-updating statefulset

* Fix runner container missing work and certs-client volume mounts and DOCKER_HOST and DOCKER_TLS_VERIFY envvars when dockerdWithinRunner=false

* Fix runnerset-controller not applying statefulset.spec.template.spec changes when there were no changes in runnerset spec

* Enable running acceptance tests against arbitrary kind cluster

* RunnerSet supports non-ephemeral runners only today

* fix: docker-build from root Makefile on intel mac

* fix: arch check fixes for mac and ARM

* ci: aligning test data format and patching checks

* fix: removing namespace in test data

* chore: adding more ignores

* chore: removing leading space in shebang

* Re-add metrics to org hra testdata

* Bump cert-manager to v1.1.1 and fix deploy.sh

Co-authored-by: toast-gear <[email protected]>
Co-authored-by: Callum James Tait <[email protected]>
mumoshu added a commit that referenced this issue Jun 22, 2021
@mumoshu
Add HRA support for RunnerSet
a69d39e 
Ref #629
Ref #613
Ref #612
@mumoshu mumoshu mentioned this issue Jun 22, 2021
Merged
mumoshu added a commit that referenced this issue Jun 23, 2021
@mumoshu
Add HRA support for RunnerSet
9a40b5c 
Ref #629
Ref #613
Ref #612
mumoshu added a commit that referenced this issue Jun 23, 2021
@mumoshu
Add HRA support for RunnerSet (#647)
98da4c2 
`HRA.Spec.ScaleTargetRef.Kind` is added to denote that the scale-target is a RunnerSet.

It defaults to `RunnerDeployment` for backward compatibility.

```
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: myhra
spec:
  scaleTargetRef:
    kind: RunnerSet
    name: myrunnerset
```

Ref #629
Ref #613
Ref #612
mumoshu added a commit that referenced this issue Jun 24, 2021
@mumoshu
RunnerSet: Automatic-recovery from registration timeout and deregistr…
71db232 
…ation on pod termination

Ref #629
Ref #613
Ref #612
@mumoshu mumoshu mentioned this issue Jun 24, 2021
Merged
mumoshu added a commit that referenced this issue Jun 24, 2021
@mumoshu
RunnerSet: Automatic-recovery from registration timeout and deregistr…
acb9061 
…ation on pod termination (#652)

Ref #629
Ref #613
Ref #612
mumoshu added a commit that referenced this issue Jun 25, 2021
@mumoshu
doc: Describe RunnerSet
5b62b9b 
Ref #629
Ref #613
Ref #612
@mumoshu mumoshu mentioned this issue Jun 25, 2021
Merged
mumoshu added a commit that referenced this issue Jun 25, 2021
@mumoshu
doc: Describe RunnerSet (#654)
3b45d1b 
Ref #629
Ref #613
Ref #612
@mumoshu
Copy link
Collaborator Author

mumoshu commented Mar 14, 2022

ARC v0.22.0 makes RunnerSet even more reliable so I can generally recommend v0.22.0 + RunnerSet if you do need voluemClaimTemplates!

@mumoshu mumoshu closed this as completed Mar 14, 2022
@mumoshu mumoshu added this to the v0.22.0 milestone Mar 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
v0.22.0
Development

No branches or pull requests
4 participants
@mumoshu @Puneeth-n @AndreaSoldino-TomTom @inahga