Help with Actions Runner Controller Service

[crackedupcorson]

Hey

I need some help with the controller service not responding after change in config within thevalues yaml (using Argo)
I'm running into an issues with getting the actions-runner-controller-webhook to run, and as a result, the RunnerDeployment and HRA won't become active.

I'm working on a POC to help see if this tool could be a viable option for wide use within our org, and had got it running and had a sample workflow executing - it's a fantastic tool and and the docs are great!
However, I wanted to go a step further and connect it up to an IAM role within the existing service account so that the runner could execute some terragrunt plans for PRs, and upon updating the values and deployment spec (for Argo to then apply), it lead to an unexpected error where the deployment webhook service seemingly had no endpoint

Internal error occurred: failed calling webhook "mutate.runnerdeployment.actions.summerwind.dev": Post https://actions- runner-controller-webhook.actions-runner-system.svc:443/mutate-actions-summerwind-dev-v1alpha1-runnerdeployment?timeout=30s: no endpoints available for service "actions-runner-controller-webhook"

A few key details

• We're running this through ArgoCD, version 1.7.3
• We're using EKS, version 1.16
• We've installed the cert-manager in the same cluster, using ArgoCD too
  Here is our values-yaml, indented to actions-runner-controller as argo uses that as we're using umbrella/proxy charts within the same directory

  githubEnterpriseServerURL: redacted
  name: actions-runner-controller
  syncPeriod: 5m
  githubAPICacheDuration: 30s
  logLevel: info
  podAnnotations:
    prometheus.io/scrape: 'true'
    prometheus.io/port: '8073'
    prometheus.io/path: '/app/management/prometheus'
  serviceAccount:
    create: true
    #if not set and create=true, a name is generated using fullname template
    name: actions-runner-service-account
    annotations:
      eks.amazonaws.com/role-arn: redacted
  authSecret:
    create: true 
    name: "controller-manager"
    env:
      - name: "redacted but can confirm this worked"
        valueFrom:
          secretKeyRef:
            name: controller-manager
            key: github_token
  service:
    type: ClusterIP
    port: 443
  metrics:
    serviceMonitor: false
    port: 8443
    proxy:
      enabled: true
      image:
        repository: quay.io/brancz/kube-rbac-proxy
        tag: v0.10.0  
  image:
    repository: summerwind/actions-runner-controller
    DinDSidecarRepositoryAndTag: "docker:dind"
    pullPolicy: IfNotPresent
  imagePullSecrets:
    - name: controller-manager
  scope:
    # If true, the controller will only watch custom resources in a single namespace
    singleNamespace: true
    # If `scope.singleNamespace=true`, the controller will only watch custom resources in this namespace
    # The default value is "", which means the namespace of the controller
    watchNamespace: ""
  spec:
    minReplicas: 2
    maxReplicas: 10
    scaleUpThreshold: '0.75'
    scaleDownThreshold: '0.25'
    scaleUpFactor: 2
    scaleDownFactor: 1
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      limits:
        cpu: 100m
        memory: 128Mi

Here is our RunnerDeployment, albeit a bit redacted

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: actions-runner-deployment
  namespace: actions-runner-system
#Defines the RunnerDeployment spec
spec:
  template:
    #Defines the Runner spec
    spec:
      labels:
        - redacted-label
      #repository:  bla
      serviceAccountName: actions-runner-service-account
      securityContext:
        fsGroup: 1000
      organization: redacted
      #enterprise: redacted
      image: "custom ECR url - we've published an image to ECR with tgenv and tfenv installed"
      imagePullPolicy: Always
      dockerdWithinRunnerContainer: true
      resources:
        limits:
          cpu:  0.5
          memory: 528Mi
        requests:
          cpu:  0.2
          memory: 256Mi
      dockerdContainerResources:
        limits:
          cpu: 0.6
          memory: 600Mi
        requests: 
          cpu: 0.3
          memory: 300Mi

Here is our HRA, redacted too

apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: actions-runner-controller-scaler
spec:
  #This setting prevents periodic loop of scaling up and down. By default, it doesn't scale down \
  #Until the grace period of 10 minutes passes after a scale up.
  scaleDownDelaySecondsAfterScaleOut: 60
  scaleTargetRef:
    name: actions-runner-deployment
    kind: RunnerDeployment
  minReplicas: 1
  maxReplicas: 3
  metrics:
  - type: PercentageRunnersBusy
    scaleUpThreshold: '0.85'    # The percentage of busy runners at which the number of desired runners are re-evaluated to scale up
    scaleDownThreshold: '0.15'   # The percentage of busy runners at which the number of desired runners are re-evaluated to scale down
    scaleUpFactor: '1.5'        # The scale up multiplier factor applied to desired count
    scaleDownFactor: '0.5'
  - type: TotalNumberOfQueuedAndInProgressWorkflowRuns
    #list of repository names to be used for calculating the metric.
    repositoryNames:
      - redacted
      - redacted

Troubleshooting I've done so far

I've been able to get the status of the two services associated with this chart. should these services not expose an endpoint?

kubectl -n actions-runner-system get svc
NAME                                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
actions-runner-controller-metrics-service   ClusterIP   redacted   <none>        8443/TCP   169m
actions-runner-controller-webhook           ClusterIP   redacted    <none>        443/TCP    131m

The controller pod is going into crashloopbackoff, which I guess is because the RunnerDeployment isn't completing so it's loading the default rather than the github_token (which this was was working with previously)? Here's the error:

Authentication failed: using private key of size 53 (/etc/actions-runner-controller/github_app_private_key...): could not parse private key: Invalid Key: Key must be PEM encoded PKCS1 or PKCS8 private key

I have a theory (from a quick investigation of the error on google) that there's an issue with argo/cert-manager and introducing IAM into the service account but I cannot be sure - we introduced OIDC as part of this POC to see if we could use GitHub actions for some terragrunt/terraform work