v0.20.0: Enhanced ephemeral runners and webhook-based autoscaling, New RunnerSet API

[mumoshu]

---

This discussion was created from the release v0.20.0: Enhanced ephemeral runners and webhook-based autoscaling, New RunnerSet API.

[peimanja]

Hey! Thanks for great project and keep shipping new features. I was waiting for so long for webhook-based autoscaling and it's finally here.

So we have different org self-hosted runners. we leverage groups as well as labels to set these runners. Before this we could not really use the web-hook server for auto-scaling since labels were not exposed.

So now I just upgraded to lates Helm chart: v0.13.0 and controller version V0.20.0.

For testing I have this Runner Deployment

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: infra-generic-xs-runner
spec:
  template:
    spec:
      dockerRegistryMirror: 'https://mirror.gcr.io/'
      group: infra
      image: '<runner-image>'
      labels:
        - size/xs
        - feature/generic
        - env/dev-app
        - group/infra
      organization: <my-org>
      resources:
        limits:
          cpu: 500m
          memory: 500Mi
        requests:
          cpu: 500m
          memory: 500Mi
      securityContext:
        fsGroup: 1000
      serviceAccountName: infra-runner

and HorizontalRunnerAutoscaler as follow:

apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
spec:
  scaleTargetRef:
    name: infra-generic-xs-runner
  scaleUpTriggers:
    - duration: 15m
      githubEvent: {}

I can see the events are coming to the server but it's not able to successfully find scaleTargetRef for the event. Am I missing something here?

2021-09-22T03:05:31.523Z	DEBUG	controllers.Runner	Found 0 HRAs by key	{"key": "<my-org>/gitops-qa"}
2021-09-22T03:05:31.523Z	DEBUG	controllers.Runner	Found 1 HRAs by key	{"key": "<my-org>"}
2021-09-22T03:05:31.523Z	DEBUG	controllers.Runner	no repository runner or organizational runner found	{"event": "workflow_job", "hookID": "319438573", "delivery": "f2147290-1b51-11ec-99eb-9732fc2c20d8", "workflowJob.status": "queued", "workflowJob.labels": ["feature/generic", "size/xs", "env/dev-app", "group/infra"], "repository.name": "gitops-qa", "repository.owner.login": "<my-org>", "repository.owner.type": "Organization", "action": "queued", "repository": "<my-org>/gitops-qa", "organization": "<my-org>"}
2021-09-22T03:05:31.523Z	INFO	controllers.Runner	Scale target not found. If this is unexpected, ensure that there is exactly one repository-wide or organizational runner deployment that matches this webhook event	{"event": "workflow_job", "hookID": "319438573", "delivery": "f2147290-1b51-11ec-99eb-9732fc2c20d8", "workflowJob.status": "queued", "workflowJob.labels": ["feature/generic", "size/xs", "env/dev-app", "group/infra"], "repository.name": "gitops-qa", "repository.owner.login": "<my-org>", "repository.owner.type": "Organization", "action": "queued"}

[peimanja]

@mumoshu any idea?

[peimanja]

Also started to seeing these errors on the webhook server

021-09-22T21:49:08.221Z	INFO	controllers.Runner	finding organizational runner	{"event": "workflow_job", "hookID": "290639658", "delivery": "e98f45d0-1bee-11ec-8b9e-dbe85cf41894", "workflowJob.status": "queued", "workflowJob.labels": ["feature/generic", "size/nano", "env/infra-mgmt", "group/default"], "repository.name": "gitops-qa", "repository.owner.login": "<my-org>", "repository.owner.type": "Organization", "action": "queued", "organization": "<my-org>"}
2021-09-22T21:49:08.221Z	ERROR	controllers.Runner	handling check_run event	{"event": "workflow_job", "hookID": "290639658", "delivery": "e98f45d0-1bee-11ec-8b9e-dbe85cf41894", "workflowJob.status": "queued", "workflowJob.labels": ["feature/generic", "size/nano", "env/infra-mgmt", "group/default"], "repository.name": "gitops-qa", "repository.owner.login": "<my-org>", "repository.owner.type": "Organization", "action": "queued", "error": "cache contained <nil>, which is not an Object"}
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2046
net/http.(*ServeMux).ServeHTTP
	/usr/local/go/src/net/http/server.go:2424
net/http.serverHandler.ServeHTTP
	/usr/local/go/src/net/http/server.go:2878
net/http.(*conn).serve
	/usr/local/go/src/net/http/server.go:1929

[blz-ea]

Can this PR #850 be a probable solution ?

[peimanja]

Can this PR #850 be a probable solution ?

Oh that might be it!

[kszymans]

Exactly same story here. The only difference is that I have 2 RunnerDeployment for same Organisation, but different scaleUpTriggers. One uses checkRun event and the other one workflow_job

[kszymans]

Hey. I can confirm that starting with the version v0.20.2 above issues are gone. Thanks!

[peimanja]

Same here. All good now

[crackedupcorson]

EDIT - I resolved this by re-adding a missing variable for the RunnerDeployment namespace

I'm also encountering this issue, on the latest version. We're hosting this on ArgoCD, I'm maintaining the CRDs myself.

Chart: 0.15.1
Controller Image: v0.20.4-898ad3c
Github Enterprise 3.2.6
ArgoCD: 2.0.5.x

{"level":"error","ts":1643277447.260952,"logger":"controllers.Runner","msg":"handling check_run event","event":"workflow_job","hookID":"1925679","delivery":"8827ba30-7f57-11ec-9aed-32dea444aee0","workflowJob.status":"queued","workflowJob.labels":[],"repository.name":"sre-bot","repository.owner.login":"REDACTED","repository.owner.type":"Organization","enterprise.slug":"REDACTED","action":"queued","error":"cache contained <nil>, which is not an Object","stacktrace":"net/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2047\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2425\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2879\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1930"}
2
{"level":"info","ts":1643277447.2815416,"logger":"controllers.Runner","msg":"finding organizational runner","event":"check_run","hookID":"1925679","delivery":"882681b0-7f57-11ec-8c8e-835790724cc9","organization":"REDACTED"}

I've also seen this error a lot, have I set my webhook up incorrectly?

{"level":"info","ts":1643277665.5115302,"logger":"controllers.Runner","msg":"unknown event type","event":"status","hookID":"1925679","delivery":"0a2d3c4e-7f58-11ec-9c38-0867c58918d1","eventType":"status"}

Here's the HRA:

apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: {{ .Values.blablah.scalerName}}
spec:
  minReplicas: 1
  maxReplicas: 8
  scaleDownDelaySecondsAfterScaleOut: {{ .Values.blablah.scaleDownDelaySecondsAfterScaleOut }}
  scaleTargetRef:
    name: {{ .Values.blablah.deploymentName}}
    kind: RunnerDeployment
  scaleUpTriggers:
    - githubEvent: {}
      amount: 2
      duration: "20m" 

I can add the matching RunnerDeployment spec if needed, but I'm unsure what's causing this.
I've got a repository level webhook, do I need an org-level webhook to match the cache record it's expecting?

[crackedupcorson]

Update - this happens on repository and organizational level webhooks. Both give the same error

[crackedupcorson]

Gahh, this was a simple error.
I had multiple runnerdeployments and had cleaned up the yaml to remove one, and removed the namespace variable from my values. I re-added it, and the scaler kicked in!

[gopisaba]

@crackedupcorson - I have similar issue with our HRA now. What missing variable needs to be added? Thanks!

[crackedupcorson]

I had removed the "namespace" variable from from my values.yaml - this was used for the metadata.namespace in the RunnerDeployment

[gopisaba]

Thanks @crackedupcorson. will check my values.yaml.

[shichengripple001]

2022-02-22T08:47:02.648Z INFO controllers.Runner Scale target not found. If this is unexpected, ensure that there is exactly one repository-wide or organizational runner deployment that matches this webhook event {"event": "push", "hookID": "344568695", "delivery": "00b47cfe-93bc-11ec-92ac-fbb8ba3cae1f"}

I am having the same issue. i am using repository runner. has this issue been solved. or it is configuration issue?

it also says, but my runner is running fine
2022-02-22T08:58:56.556Z DEBUG controllers.Runner no repository runner found

[shadow9911]

this issue is still present with us. namespace is delared in the yaml, if we reapply it the issue goes away for "some time". We're unable to reproduce the issue every time

[toast-gear]

Please upgrade to the latest version, then if the issue is still present raise a issue including your:

• deployment method
• controller version
• chart version if relevant
• yaml for HRA, runnerdeployment/runnerset, webhook server
• Any relevant controller configuration e.g. non-default flags
• controller logs
• webhook server logs