flaky: e2e TestEGUpgrade/EnvoyShutdown

[shawnh2]

https://github.com/envoyproxy/gateway/actions/runs/8811709532/job/24187972908?pr=2665#step:6:17406

[shawnh2]

still fail: https://github.com/envoyproxy/gateway/actions/runs/9655504860/job/26631669933?pr=3599#step:6:19590

cc @guydc

[arkodg]

can we bump initDelaySeconds to 10 ?

[guydc]

Sorry for the late reply, I missed that CC.

The issue this time appears to be connection reset/refused and not 404s:

{"ts":1719283813.373276,"level":"err","r":177,"file":"http_client.go","line":1098,"msg":"Read error","err":"read tcp 172.18.0.1:35884->172.18.255.208:80: read: connection reset by peer","size":0,"header_len":0,"dest":{"IP":"172.18.255.208","Port":80,"Zone":""},"url":"http://172.18.255.208/envoy-shutdown","thread":11,"run":0}
{"ts":1719283813.373384,"level":"err","r":313,"file":"http_client.go","line":1098,"msg":"Read error","err":"read tcp 172.18.0.1:35904->172.18.255.208:80: read: connection reset by peer","size":0,"header_len":0,"dest":{"IP":"172.18.255.208","Port":80,"Zone":""},"url":"http://172.18.255.208/envoy-shutdown","thread":35,"run":0}
{"ts":1719283813.373564,"level":"err","r":174,"file":"http_client.go","line":1098,"msg":"Read error","err":"read tcp 172.18.0.1:35872->172.18.255.208:80: read: connection reset by peer","size":0,"header_len":0,"dest":{"IP":"172.18.255.208","Port":80,"Zone":""},"url":"http://172.18.255.208/envoy-shutdown","thread":8,"run":0}
{"ts":1719283813.374018,"level":"err","r":177,"file":"http_client.go","line":951,"msg":"Unable to connect","dest":{"IP":"172.18.255.208","Port":80,"Zone":""},"err":"dial tcp 172.18.255.208:80: connect: connection refused","numfd":58,"thread":11,"run":0}

I'm not sure that increasing the readiness time will help in this case. This could be an issue with metallb/kube-proxy forwarding requests to terminated pods.

[shawnh2]

closed in favour of #3764