fix: Address race condition disrupting graceful shutdown process

[davidalger]

What this PR does / why we need it:

This PR resolves a race condition created when the original implementation of /shutdown/ready was altered in an attempt to safeguard against CPU usage from excessive polling if a malicious actor issued a large number of requests to the /shutdown/ready endpoint of active proxy pods. In reality that is likely a very low risk compared to the tradeoff of the race condition.

Given this change has been proven to result in a race condition that causes real harm (breaks the graceful drain), I'm proposing we revert to the original implementation where /shutdown/ready blocks until the readiness indicator turns up or until it's timeout is reached preventing the race condition.

More than a few times now on clusters with small nodes but which are practically idle I've witnessed the process fail due to the exec in the preStop taking longer than 2 seconds to start and touch the shutdown active file. This can be seen in logs where the message shutdown active file not found turns up and Envoy receives the SIGTERM early. Effects are that Envoy doesn't exit gracefully and the pod doesn't completely terminate until the graceful drain timeout elapses.

envoy [2024-03-09 21:02:51.657][1][warning][main] [source/server/server.cc:881] caught ENVOY_SIGTERM
envoy [2024-03-09 21:02:51.914][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:155] DeltaAggregatedResources gRPC config stream to xds_cluster closed: 13, 
shutdown-manager 2024-03-09T21:02:49.561Z    INFO    shutdown-manager    envoy/shutdown_manager.go:94    received shutdown ready request
shutdown-manager 2024-03-09T21:02:51.619Z    INFO    shutdown-manager    envoy/shutdown_manager.go:102    shutdown active file not found
shutdown-manager 2024-03-09T21:02:51.663Z    INFO    shutdown-manager    envoy/shutdown_manager.go:148    initiating graceful drain with 5 second minimum drain period and 600 second timeout
shutdown-manager 2024-03-09T21:02:51.931Z    ERROR    shutdown-manager    envoy/shutdown_manager.go:153    error failing active health checks    {"error": "Post \"http://127.0.0.1:19000/healthcheck/fail\": read tcp 127.0.0.1:58214->127.0.0.1:19000: read: connection reset by peer"}
shutdown-manager 2024-03-09T21:02:51.933Z    ERROR    shutdown-manager    envoy/shutdown_manager.go:158    error initiating graceful drain    {"error": "Post \"http://127.0.0.1:19000/drain_listeners?graceful&skip_exit\": dial tcp 127.0.0.1:19000: connect: connection refused"}
shutdown-manager 2024-03-09T21:02:51.933Z    ERROR    shutdown-manager    envoy/shutdown_manager.go:168    error getting total connections    {"error": "Get \"http://127.0.0.1:19000//stats?filter=^server\\\\.total_connections$&format=json\": dial tcp 127.0.0.1:19000: connect: connection refused"}
shutdown-manager 2024-03-09T21:02:52.935Z    ERROR    shutdown-manager    envoy/shutdown_manager.go:168    error getting total connections    {"error": "Get \"http://127.0.0.1:19000//stats?filter=^server\\\\.total_connections$&format=json\": dial tcp 127.0.0.1:19000: connect: connection refused"}
shutdown-manager 2024-03-09T21:02:53.936Z    ERROR    shutdown-manager    envoy/shutdown_manager.go:168    error getting total connections    {"error": "Get \"http://127.0.0.1:19000//stats?filter=^server\\\\.total_connections$&format=json\": dial tcp 127.0.0.1:19000: connect: connection refused"}
shutdown-manager 2024-03-09T21:02:54.936Z    ERROR    shutdown-manager    envoy/shutdown_manager.go:168    error getting total connections    {"error": "Get \"http://127.0.0.1:19000//stats?filter=^server\\\\.total_connections$&format=json\": dial tcp 127.0.0.1:19000: connect: connection refused"}
shutdown-manager 2024-03-09T21:02:55.937Z    ERROR    shutdown-manager    envoy/shutdown_manager.go:168    error getting total connections    {"error": "Get \"http://127.0.0.1:19000//stats?filter=^server\\\\.total_connections$&format=json\": dial tcp 127.0.0.1:19000: connect: connection refused"}
shutdown-manager 2024-03-09T21:02:56.938Z    ERROR    shutdown-manager    envoy/shutdown_manager.go:168    error getting total connections    {"error": "Get \"http://127.0.0.1:19000//stats?filter=^server\\\\.total_connections$&format=json\": dial tcp 127.0.0.1:19000: connect: connection refused"}
shutdown-manager 2024-03-09T21:02:56.938Z    INFO    shutdown-manager    envoy/shutdown_manager.go:172    minimum drain period reached; will exit when total connections reaches 0
shutdown-manager 2024-03-09T21:02:57.939Z    ERROR    shutdown-manager    envoy/shutdown_manager.go:168    error getting total connections    {"error": "Get \"http://127.0.0.1:19000//stats?filter=^server\\\\.total_connections$&format=json\": dial tcp 127.0.0.1:19000: connect: connection refused"}
shutdown-manager 2024-03-09T21:02:58.941Z    ERROR    shutdown-manager    envoy/shutdown_manager.go:168    error getting total connections    {"error": "Get \"http://127.0.0.1:19000//stats?filter=^server\\\\.total_connections$&format=json\": dial tcp 127.0.0.1:19000: connect: connection refused"}
shutdown-manager 2024-03-09T21:02:59.941Z    ERROR    shutdown-manager    envoy/shutdown_manager.go:168    error getting total connections    {"error": "Get \"http://127.0.0.1:19000//stats?filter=^server\\\\.total_connections$&format=json\": dial tcp 127.0.0.1:19000: connect: connection refused"}
shutdown-manager 2024-03-09T21:03:00.942Z    ERROR    shutdown-manager    envoy/shutdown_manager.go:168    error getting total connections    {"error": "Get \"http://127.0.0.1:19000//stats?filter=^server\\\\.total_connections$&format=json\": dial tcp 127.0.0.1:19000: connect: connection refused"}
shutdown-manager 2024-03-09T21:03:01.944Z    ERROR    shutdown-manager    envoy/shutdown_manager.go:168    error getting total connections    {"error": "Get \"http://127.0.0.1:19000//stats?filter=^server\\\\.total_connections$&format=json\": dial tcp 127.0.0.1:19000: connect: connection refused"}

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 63.88%. Comparing base (69010d2) to head (7ee0787).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2864   +/-   ##
=======================================
  Coverage   63.88%   63.88%           
=======================================
  Files         124      124           
  Lines       20927    20927           
=======================================
  Hits        13369    13369           
  Misses       6703     6703           
  Partials      855      855           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[arkodg]

cc @guydc who has been trying to test this with #2839