Use sidecar to monitor events on timescaledb container, then cleanup …

…(hopefully last pattern to be tried)

Given a few peculiarities with how liveness probes work, and the fact
that the underlying timescaledb container could end up being removed by
the underlying kubelet before the liveness probe has a change to perform
its cleanup logic, it appears as though this pattern provides the most
robust approach.

The fact that we are monitoring the K8s API for events related to the
specific target container, and that we are using timestamps of the
events to maintain event stream offsets, should ensure that this pattern
is the most robust pattern for guaranteeing that a specific cleanup
routine is performed when the timescaledb container exits, all without
touching the timescaledb container itself.

charts/timescaledb-single/scripts/sidecar_events_monitor.sh

@@ -0,0 +1,68 @@
+#!/bin/bash
+#
+# Monitor Kubernetes events related to this pod (determined via HOSTNAME env var),
+# in order to determine when some target container of this pod has stopped according
+# to Kubernetes.
+
+# Get the pod service account token.
+TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
+# Set the Kubernetes API server address.
+API_SERVER="https://kubernetes.default.svc"
+# Get the pod name.
+POD_NAME=$HOSTNAME
+# Construct the authorization header
+AUTH_HEADER="Authorization: Bearer $TOKEN"
+# Last seen event timestamp (for paging).
+LAST_EVENT_TIME="$(date +'%Y-%m-%dT%H:%M:%SZ')"
+# The namespace in which this pod is running.
+TARGET_NS="${1}"
+# The container for which events are being monitored, in order to
+# determine when the target container has stopped according to Kubernetes.
+TARGET_CONTAINER="${2}"
+
+function process_events() {
+    while true; do
+        # Construct the URL with paging parameters.
+        URL="$API_SERVER/api/v1/namespaces/savannah-system/events?fieldSelector=involvedObject.name=$POD_NAME&limit=500&since=$LAST_EVENT_TIME"
+
+        # Fetch a batch of events for the current pod.
+        EVENTS=$(curl -s -k -H "$AUTH_HEADER" "$URL")
+
+        # Check for "ContainerStopped" event related to the "timescaledb" container.
+        if [[ $(echo "$EVENTS" | jq -r '.items[] | select(.reason == "ContainerStopped" and .involvedObject.name == "timescaledb")') ]]; then
+            echo "timescaledb container in pod $POD_NAME has stopped, will call linkerd-shutdown now"
+        fi
+
+        # Extract timestamp of the latest event for next iteration.
+        LAST_EVENT_TIME=$(echo "$EVENTS" | jq -r '.items[0].lastTimestamp')
+
+        # Sleep for a short interval before checking again.
+        sleep 5
+    done
+
+    # Execute custom cleanup routine.
+    custom_cleanup
+}
+
+function custom_cleanup() {
+    echo "executing custom cleanup routine: shutting down local linkerd-proxy"
+    curl -s -m 5 -X POST http://localhost:4191/shutdown
+}
+
+function main() {
+    # This trap ensures that this container will not shutdown based on SIGTERM.
+    # It will do its absolute damnedest to detect when the target container is shutdown first
+    # in order to ensure that it can execute its custom shutdown logic.
+    trap process_events SIGTERM
+    if [[ $TARGET_NS == "" || $TARGET_CONTAINER == "" ]]; then
+        echo "missing input, can not proceed"
+        echo "usage: $0 <namespace> <pod>"
+        exit 1
+    fi
+    process_events
+}
+
+# Execute main if this script is being executed, not sourced.
+#
+# Though this script should never be sourced, we employ some defensive programming here.
+if [ "${BASH_SOURCE[0]}" == "$0" ]; then main "$@"; fi

charts/timescaledb-single/templates/role-timescaledb.yaml

@@ -44,4 +44,12 @@ rules:
   - patch
   - update
   - watch
+{{- if .Values.linkerd.adminShutdownOnExit }}
+- apiGroups: [""]
+  resources: ["events"]
+  verbs:
+  - get
+  - list
+  - watch
+{{- end }}
 {{- end }}

charts/timescaledb-single/templates/statefulset-timescaledb.yaml

@@ -102,6 +102,21 @@ spec:
       # we can still serve clients.
       terminationGracePeriodSeconds: 600
       containers:
+      {{- if .Values.linkerd.adminShutdownOnExit }}
+      - name: linkerd-shutdown
+        securityContext:
+          allowPrivilegeEscalation: false
+        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
+        imagePullPolicy: {{ .Values.image.pullPolicy }}
+        command:
+          - "{{ template "scripts_dir" . }}/sidecar_events_monitor.sh"
+          - "{{ $.Release.Namespace }}"
+          - "timescaledb"
+        volumeMounts:
+        - mountPath: {{ template "scripts_dir" . }}
+          name: timescaledb-scripts
+          readOnly: true
+      {{- end }}
       - name: timescaledb
         securityContext:
           allowPrivilegeEscalation: false
@@ -271,18 +286,6 @@ spec:
           successThreshold: {{ .Values.readinessProbe.successThreshold }}
           failureThreshold: {{ .Values.readinessProbe.failureThreshold }}
         {{- end }}
-        {{- if .Values.livenessProbe.enabled }}
-        livenessProbe:
-          exec:
-            command:
-              - "{{ template "scripts_dir" . }}/liveness_probe.sh"
-              - "{{ if .Values.linkerd.adminShutdownOnExit }}1{{else}}0{{end}}"
-          initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }}
-          periodSeconds: {{ .Values.livenessProbe.periodSeconds }}
-          timeoutSeconds: {{ .Values.livenessProbe.timeoutSeconds }}
-          successThreshold: {{ .Values.livenessProbe.successThreshold }}
-          failureThreshold: {{ .Values.livenessProbe.failureThreshold }}
-        {{- end }}
         volumeMounts:
         - name: storage-volume
           mountPath: {{ .Values.persistentVolumes.data.mountPath | quote }}

charts/timescaledb-single/values.yaml

@@ -362,14 +362,6 @@ readinessProbe:
   failureThreshold: 6
   successThreshold: 1
 
-livenessProbe:
-  enabled: false
-  initialDelaySeconds: 15
-  periodSeconds: 10
-  timeoutSeconds: 10
-  failureThreshold: 1
-  successThreshold: 1
-
 persistentVolumes:
   # For sanity reasons, the actual PGDATA and wal directory will be subdirectories of the Volume mounts,
   # this allows Patroni/a human/an automated operator to move directories during bootstrap, which cannot