Meaningful ECS service health checks

[jamieparkinson]

Some of our web application / API services expose health check endpoints, but none of these actually perform a meaningful check for service health.

There are 2 primary consequences of this:

• ECS doesn't know that it needs to restart applications that have got into a broken state (we have had to do this manually sometimes)
• During deployments we see a small number of failed requests due to the service not being able to serve requests at the time it receives them (see Fix catalogue-api request errors during deployment #10513)

Here are the services which need health checks adding/improving:

• Catalogue API: Add option for HTTP healthcheck, enable for works catalogue-api#736
• Content API: Adds a simple healthcheck endpoint for content-api content-api#94
• Items API: Adds a simple healthcheck endpoint for the items service catalogue-api#739
• Requests API: Adds a simple healthcheck endpoint for the requests service catalogue-api#747
• Concepts API: Adds a simple healthcheck endpoint for the concepts service catalogue-api#749
• wc.org:
  • Content (wc.org) webapp: Adds a simple healthcheck endpoint to the content app #10612
  • Identity (wc.org) webapp: Adds a simple healthcheck endpoint to the identity app #10617
  • Remove healthcheck endpoints from nginx configuration for frontend platform-infrastructure#413
  • Updates the name of the nginx frontend container image  terraform-aws-ecs-service#52
  • Use updated container image for nginx in content & identity #10618
• Storage service: Adds a simple healthcheck endpoint to the bags and ingest apis storage-service#1115
• An RFC for where we might want to end up: RFC 050: health check principles docs#98

[kenoir]

Flagging that for wellcomecollection.org the healthcheck endpoints are HTTP but still terminate at the nginx container: https://github.com/wellcomecollection/platform-infrastructure/blob/main/images/dockerfiles/nginx/frontend.nginx.conf#L24