Health check endpoint when running as sidecar

[a-martynovich]

Question

I'm running alloydb-auth-proxy as a "native" sidecar container in Kubernetes. My main container needs to connect to AlloyDB immediately at startup. The obvious solution is to wait until alloydb-auth-proxy is ready and then start the main container.

Kubernetes provides a startupProbe which can open a TCP connection to the specified port, and I can use that to check if alloydb-auth-proxy has started. But the probe requires the pod to expose the port, and I don't want to expose the port since nobody else needs to connect to it other than the main container in the pod (all traffic between sidecar and main should stay inside the pod).

Is there any other way to make a readiness check for alloydb-auth-proxy container? A /healthz endpoint maybe? There's no shell so scripting isn't an option.

Code

spec:
  initContainers:
  - name: alloydb-proxy
    image: gcr.io/alloydb-connectors/alloydb-auth-proxy
    restartPolicy: Always
    envFrom:
    - configMapRef:
        name: ${CONFIGMAP_REF}
    args:
      - "--auto-iam-authn"
      - "--port=5432"
      - "$(DATABASE_URI)"
    ports:
      - containerPort: 5432
        protocol: TCP
  containers:
  - name: main
    env:
      - name: DATABASE_URL
        value: postgresql://localhost:5432/postgres

Additional Details

No response

[enocom]

Reading through the docs on sidecar containers, I wonder have you just tried your deployment as written above? Sidecar containers (and init containers) start before the main container (see the docs). So your application should be able to just connect and the Proxy will be ready.

If you do have problems with this approach, though, I'd be curious to hear about it.

The Auth Proxy has a built-in wait command that might also be useful -- although it was built prior to first class sidecar container support. The best documentation is in the help message (run ./alloydb-auth-proxy wait --help), but you can see that same message here:

alloydb-auth-proxy/cmd/root.go

Lines 400 to 427 in 4a501f6

	Sometimes it is necessary to wait for the Proxy to start.
	To help ensure the Proxy is up and ready, the Proxy includes a wait
	subcommand with an optional --max flag to set the maximum time to wait.
	Invoke the wait command, like this:
	./alloydb-auth-proxy wait
	By default, the Proxy will wait up to the maximum time for the startup
	endpoint to respond. The wait command requires that the Proxy be started in
	another process with the HTTP health check enabled. If an alternate health
	check port or address is used, as in:
	./alloydb-auth-proxy <INSTANCE_URI> \
	--http-address 0.0.0.0 \
	--http-port 9191
	Then the wait command must also be told to use the same custom values:
	./alloydb-auth-proxy wait \
	--http-address 0.0.0.0 \
	--http-port 9191
	By default the wait command will wait 30 seconds. To alter this value,
	use:
	./alloydb-auth-proxy wait --max 10s

If your deployment above doesn't work (please let me know), then you could use wait like this:

• start the Auth Proxy like so: ./alloydb-auth-proxy <INSTANCE_URI> --health-check
• run the wait command: ./alloydb-auth-proxy wait
• then when your main container starts the Proxy should be up and running.

[a-martynovich]

Oh wow, this actually worked! Here's what I used:

spec:
  initContainers:
  - name: alloydb-proxy
    image: gcr.io/alloydb-connectors/alloydb-auth-proxy
    restartPolicy: Always
    startupProbe:
      exec:
        command: ["/alloydb-auth-proxy", "wait"]
    args:
      - "--auto-iam-authn"
      - "--port=5432"
      - "--health-check"
      - "$(DATABASE_URI)"
    ports:
      - containerPort: 5432
        protocol: TCP
    ...
  containers:
  - name: postgres
     image: "postgres",
    env:
      - name: DATABASE_URL
        value: postgresql://localhost:5432/postgres
...

I can see the connection to health check port in the logs, meaning that it's actually happening.

As for the reasoning, and in response to

I wonder have you just tried your deployment as written above? Sidecar containers (and init containers) start before the main container (see the docs). So your application should be able to just connect and the Proxy will be ready.

The issue is that without the health/readiness check of an init container it is considered ready immediately when its process starts up, but it takes another few milliseconds to actually establish the AlloyDB connection. The main container may start up faster and start connecting to alloydb-proxy container before that. This is what I had and this is why I needed a health check endpoint. So, thanks for solving this!

[enocom]

Nice -- the startup probe configuration looks great.

FWIW the Auth Proxy establishes a connection to AlloyDB lazily so I'd expect once the process was up, it would be ready to receive connection attempts. But in any case, if wait is working for you, then I'm happy to hear it.