Console losing GPRC streaming connection

[wesbillman]

After about 20s on a deployed FTL instance, the GRPC connection seems to timeout (or have some other streaming connection error)

Note this does not happen in local development with ftl dev

useSchema - streamSchema - Connect error: ConnectError: [unknown] missing EndStreamResponse
    at Zt.from (index-7hTRnJxG.js:279:9628)
    at index-7hTRnJxG.js:280:41319Caused by: missing EndStreamResponse
Console service - streamEvents - Connect error: ConnectError: [unknown] missing EndStreamResponse
    at Zt.from (index-7hTRnJxG.js:279:9628)
    at index-7hTRnJxG.js:280:41319Caused by: missing EndStreamResponse

This can lead to other issues as reported here: #3260

[stuartwdouglas]

Maybe something in the network setup is killing idle connections. Does it still happen if you send a ping every 10s?

[alecthomas]

These kinds of timeouts are really common in deployed environments, they'll typically be some Envoy setting, or LB setting. They're always annoying.

[wesbillman]

I don't think we have a mechanism currently to send a ping from the console. I could setup a service on the client side to do this if it would be helpful though. But yeah, I think it's some setting/config that will need to be figured out :)