Controller creates a lot of runners

[vterdunov]

Hi, thanks for the project.

I faced with issue after the operator was updated using helm chart 0.10.4->0.11.0.
Controller started to create a lot of runners pods.

k -n actions-runner-system get po | wc -l
213

I have two runner deployemnts and corresponding autoscalers. There are identical, just for different repos. E.g:

---
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: product-x-backend-runners
  namespace: actions-runner-system
spec:
  template:
    spec:
      repository: my-org/my-repo
      labels:
      - k8s

---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: product-x-backend-runners-autoscaler
  namespace: actions-runner-system
spec:
  scaleTargetRef:
    name: product-x-backend-runners
  minReplicas: 1
  maxReplicas: 4
  metrics:
  - type: TotalNumberOfQueuedAndInProgressWorkflowRuns
    repositoryNames:
    - my-org/my-repo

[mumoshu]

@vterdunov Hey! Thanks for reporting.

How have you upgraded your CRDs? Exact steps or tools used would be helpful.

Anyway, and unfortunately, you seem to have fallen into the same scenario as #418 (comment).

Please read the thread and try upgrading the CRD, if that's the case.

I haven't had chance to test it specifically, but I think kubectl apply against CRDs didn't work and you need to use kubectl replace to upgrade CRDs. If it still didn't work, I'd suggest using kubectl delete and then kubectl create to upgrade your CRDs.

Also note that I'm not talking about custom resources, like runner, runnerreplicaset, runnerdeployments. I'm talking about CRDs. Your custom resources, like runners, are very likely to be in invalid states already, and trying to kubectl delete runner $RUNNER for example won't work. So I believe upgrading CRDs the way to go.

[vterdunov]

@mumoshu thanks for quick answer! I've managed with the issue. I've updated the CRD (uninstall and install the new version).

It seems to me that the need to update files should be reflected in documentations and release notes. Like prometheus-kube-stack do.

[mumoshu]

@vterdunov Thanks for confirming!

Ah, I missed you mentioned our helm chart so you probably used helm upgrade --install to upgrade the controller only, right?

Then, you would have definitely had to use kubectl to upgrade CRDs.

Re your suggestion- Yes, it should have been noted in the release note. Thanks!

[vterdunov]

@mumoshu Yes, i use Flux2 to manage Helm charts. And yes, i use the official chart. Thanks again!

[hammadzz]

@vterdunov Hey! Thanks for reporting.

How have you upgraded your CRDs? Exact steps or tools used would be helpful.

Anyway, and unfortunately, you seem to have fallen into the same scenario as #418 (comment).

Please read the thread and try upgrading the CRD, if that's the case.

I haven't had chance to test it specifically, but I think kubectl apply against CRDs didn't work and you need to use kubectl replace to upgrade CRDs. If it still didn't work, I'd suggest using kubectl delete and then kubectl create to upgrade your CRDs.

Also note that I'm not talking about custom resources, like runner, runnerreplicaset, runnerdeployments. I'm talking about CRDs. Your custom resources, like runners, are very likely to be in invalid states already, and trying to kubectl delete runner $RUNNER for example won't work. So I believe upgrading CRDs the way to go.

The docs need to be updated, kubectl replace would have saved me. But I followed the docs here and ended up with endless runners being created and registered on Github.

[mumoshu]

@hammadzz Thanks for reporting and sorry for your frustration. I hear you, but how should it be added to the UPGRADING doc?

Do you think it's a good idea to mention about kubectl replace there?

I thought you won't end up in such a situation as long as you started by reading and following the UPGRADING doc.

[hammadzz]

@mumoshu my best guess is that doing apply instead of replace caused the issue. Even so I am not certain if it would always work, for example if resource names have changed. If they have changed then I assume replace would have overcome the issue I faced.

In the end I had to uninstall the helm chart and delete all CRDs. One of them required this step to be able to delete it.

kubectl patch crd/runners.actions.summerwind.dev -p '{"metadata":{"finalizers":[]}}' --type=merge

[mumoshu]

@hammadzz Thanks! Could you let me reproduce it. What version of the chart were you using before the upgrade, and to which version did you upgrade it?

[hammadzz]

@mumoshu my current version was 0.10.5 and upgrade version was 0.12.7

For context, I was upgrading due to the runner showing offline and on checking the runner container it was missing /runner/.runner config. Discussed in issue #288

[XBeg9]

I had the same issue with crd/runners.actions.summerwind.dev, it's always stuck on deleting and whole infrastructure as code piece fails to delete by timeout until you run kubectl patch crd/runners.actions.summerwind.dev -p '{"metadata":{"finalizers":[]}}' --type=merge in the middle