Ensure that we can do rolling updates/replicas

[erkannt]

If we want to have rolling updates/replicas of reviewer we need to ensure several things:

• Multiple pods can use RDS/s3 simultaneously in a safe manner
• File upload and progress state get handled by by same pod
• Applications need to handle SIGTERM
• Ensure you don’t end in bad state if SIGKILL or websocket connection gets terminated due to nginx config reload (see this article)

You can manage where traffic goes with

• Readiness probe state (k8s ingress will send traffic)
• Cookie based session affinity (ingress annotation)
• Chart can set grace period for SIGTERM
• Ingress annotation how long to send existing connections

This ticket is result of discussion ticket #907

DoD:

• connections to submission are sticky
• submission pods are kept alive to handle existing connections during rollout
• per app replica and disruptionbudget
• ingress sends existing connections to old pod
• grace-period can be set via values.yaml

I have successfully uploaded large files while performing a rollout. I haven't looked deeper if all subsystems are behaving as expected.

Tasks:

• replicas/disruptions budget in chart and deployments
• pod grace period
• pre stop hook <<< requires Create http endpoint for preStop hook #1310 submission handles SIGTERM gracefully, keeping uploads alive
• ingress grace period
• ingress stickyness
• evaluate readyness and liveness probes >>> Check Readiness, Liveness and graceful shutdown of submission, client and continuum-adaptor #1306
• test survival of connection during rollout
• investigate replicacount/canary interaction

[erkannt]

Blocked by #1310