Proxy exits with wrong code after SIGTERM

[cdaguerre]

Bug Description

Graceful termination seems impossible in v2. Whether I set --max-sigterm-delay or not doesn't matter, process always has exit code 137, even when there are no active connections.

Example code (or command)

- name: cloudsql-proxy
  image: {{ $root.Values.sqlProxy.image }}
  command:
    - '/kubexit/kubexit'
    - '/cloud-sql-proxy'
    - '--address=127.0.0.1'
    - '--max-sigterm-delay=10s'
    - '--structured-logs'
    - '--prometheus'
    {{- range $k, $v := $root.Values.sqlProxy.connections }}
    - '{{ $v.dsn }}?port={{ $v.port }}'
    {{- end }}
  lifecycle:
    preStop:
      exec:
        command: ['sleep', '10']

Stacktrace

How to reproduce

1. ?
2. ?

Environment

1. OS type and version: GKE 1.24.4-gke.800
2. Cloud SQL Proxy version (./cloud-sql-proxy --version): 2.0.0-preview.2

[enocom]

Thanks for the issue @cdaguerre.

Looking at these exit codes again, I see that SIGTERM is conventionally 143 and SIGKILL is 137.

I think what this means is that we should be returning exit code 143 instead of 137 here: https://github.com/GoogleCloudPlatform/cloud-sql-proxy/blob/main/cmd/errors.go#L29.

Cf. https://komodor.com/learn/sigterm-signal-15-exit-code-143-linux-graceful-termination/

Is that your expectation?

[cdaguerre]

Hey @enocom
Actually in v1, even when the proxy received a SIGTERM, the exit code would be 0 if there were no more active connections.
The use case is for running the proxy as a sidecar in kubernetes Jobs that run to completion.
You need to have a way to kill the cloudsql-proxy sidecar container when the main container finished.
Any exit code other then 0 from one of the containers causes the job to be considered failed.

Basically, ideal behaviour to gracefully terminate on SIGTERM taking the --max-sigterm-delay into account.

• Exit 0 as soon as there is no active connection anymore
• After sigterm delay,
  • exit 137 (SIGKILL) if there is an active connection
  • exit 0 if there is no active connection (probably already exited before delay...)

EDIT: added link to v1 code:

cloud-sql-proxy/proxy/proxy/client.go

Line 630 in 6af6a92

// Shutdown waits up to a given amount of time for all active connections to

[enocom]

Thanks for the clarification.

There are two issues here:

1. The v2 proxy uses the wrong exit code for SIGTERM. We'll fix that here.
2. There's no way to cleanly shutdown the proxy without some wrapper script. That's tracked here: Provide a way to shutdown the process from another container #828.

Since we've made this worse in v2 with the new non-0 exit code, I'll prioritize adding a /quitquitquit endpoint to solve this.

[adamstrawson]

Hey @enocom Just to dig up this issue again, it seems that we're hitting the same behaviour that @cdaguerre mentions here despite having no open connections (exit code: 143)

We use cloud-sql-proxy on Kubernetes Jobs, wrapped around timeout so that it gracefully shuts down the container after completion. Because of this, and the 143 exit code, kubernetes sees the job as failed and puts it into an Error state.

terminated - Error (exit code: 143)

- name: cloud-sql-proxy
  image: gcr.io/cloud-sql-connectors/cloud-sql-proxy:2.0.0-buster
  imagePullPolicy: Always
  command: ["/bin/sh", "-c"]
  args:
  - timeout --preserve-status -s SIGTERM 60s /cloud-sql-proxy \
  --max-sigterm-delay=60s --structured-logs --health-check \
  --http-address=0.0.0.0 --port=5432 --private-ip \
  {INSTANCE_CONNECTION}

Prior to v2, this behaved gracefully as expected.

Happy to open as a new issue if you'd prefer.

[enocom]

We recently added support for a /quitquitquit endpoint to support exactly this use case. See here for details: https://github.com/GoogleCloudPlatform/cloud-sql-proxy#localhost-admin-server.