Runners and Istio

[js-timbirkett]

Hi 👋

We deploy most of our services onto an Istio service mesh. We'd like to put the runners on "the mesh" so that they can take advantage of MUTUAL_TLS and reach services without needing to expose the service through ingressgateway for integration and e2e tests.

The trouble is that I don't think the Istio sidecar comes up before the runner tries to execute entrypoint.sh this causes the good old:

│ runner Starting Runner listener with startup type: service                                                                                                                                                          │
│ runner Started listener process                                                                                                                                                                                     │
│ runner An error occurred: Not configured                                                                                                                                                                            │
│ runner Runner listener exited with error code 2                                                                                                                                                                     │
│ runner Runner listener exit with retryable error, re-launch runner in 5 seconds.

loop of death.

Shelling into the pod and running ./entrypoint.sh registers the runner as you'd expect at startup and running the runner with Istio sidecar injection also works.

There are a couple of things that could be done in the (top of the) entrypoint script:

• Allow an env variable configured sleep
• Add a wait-for-ready style loop

Is anyone else running this on Istio?

[mumoshu]

@js-timbirkett Hey! Thanks for bringing this up. Honestly speaking I have no much experience with Istio and things like this is why I'm still taking me away from using any kind of sidecar-based service mesh :) I was looking forward to kubernetes/enhancements#753 but it postponed...

I think I'm fine with the first option, sleeping. But I'm interested in the second option, too.

How would the second work? How do you usually wait-for-ready when using Istio? Any idea integrating it in a generic way into actions-runner-controller?

[js-timbirkett]

Hey @mumoshu - thanks for the quick reply. Back in the days of docker-compose you'd often end up having dependencies like databases that you'd want to be available before starting your application.

What you'd end up doing is adding something like:

• https://github.com/ufoscout/docker-compose-wait
• https://github.com/nickjj/wait-until
• https://github.com/vishnubob/wait-for-it

These scripts pretty much run a while loop and attempt to connect to a host and port with a smol sleep.

Istio takes a few hundred milliseconds before network traffic can flow through the sidecar proxy which is fine for java apps (long startup times), or apps that crash and restart when failing to connect to a thing.

[mabushey]

I'm using the runner with Istio 1.10.0 just fine. Add this to your istio operator config:

  meshConfig:
    defaultConfig:
      holdApplicationUntilProxyStarts: true

[mumoshu]

@mabushey Great! Thanks a lot for sharing ☺️

[js-timbirkett]

Thanks @mabushey - only another 4 Istio upgrades (* 3 cluster) to go for me :-|