[opentelemetry-collector] Support automated collector configuration reloading

[chunter0]

Now that collector supports reloading configuration when a SIGHUP is received, can we expose shareProcessNamespace field on Deployment/Daemonset templates. Adding this will allow configuration reloads to be handled by automation from a sidecar through extraContainers field.

Quick Example

1. Install with extraContainers

cat << EOF | helm upgrade -i collector open-telemetry/opentelemetry-collector -f -

mode: daemonset

image:
  repository: otel/opentelemetry-collector

extraContainers:
- name: reloader
  args:
  - sleep
  - infinity
  image: alpine:3.14
EOF

From the sidecar we don't see any processes for otel-collector

$ kc exec daemonsets/collector-opentelemetry-collector-agent -c reloader -- ps ax
PID   USER     TIME  COMMAND
    1 root      0:00 sleep infinity
   13 root      0:00 ps ax

After updating daemonset to include shareProcesses

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: collector-opentelemetry-collector-agent
spec:
  template:
    spec:
      shareProcessNamespace: true

We'll now see collector process from sidecar so we can automate detecting config changes

$ kc exec daemonsets/collector-opentelemetry-collector-agent -c reloader -- ps ax
PID   USER     TIME  COMMAND
    1 65535     0:00 /pause
    7 10001     0:00 /otelcol --config=/conf/relay.yaml
   43 root      0:00 sleep infinity
   64 root      0:00 ps ax

This is useful when exporters connection endpoints have changed and helm doesn't re-deploy for deployments/daemonsets since configuration is loaded via a file and not command-line args.

A local test using sidecar shows expected logs and the service is properly reloaded 👍

kc exec -n gloo-mesh daemonsets/collector-opentelemetry-collector-agent -c reloader -- sh -c 'kill -HUP $(pidof otelcol)'

2024-06-07T15:13:56.265Z        info    [email protected]/collector.go:323       Received signal from OS {"signal": "hangup"}
2024-06-07T15:13:56.265Z        warn    [email protected]/collector.go:232       Config updated, restart service
2024-06-07T15:13:56.265Z        info    [email protected]/service.go:243 Starting shutdown...
2024-06-07T15:13:56.265Z        info    healthcheck/handler.go:132      Health Check state change       {"kind": "extension", "name": "health_check", "status": "unavailable"}
2024-06-07T15:13:56.265Z        info    extensions/extensions.go:59     Stopping extensions...
2024-06-07T15:13:56.265Z        info    [email protected]/service.go:257 Shutdown complete.
2024-06-07T15:13:56.267Z        info    [email protected]/service.go:113 Setting up own telemetry...

[jaronoff97]

We also added this capability into the operator, so I think it makes sense to enable it here