[BUG] [FlyteAdmin] Notifications SQS subscriber stops processing messages when "connection reset by peer"

[rstanevich]

Describe the bug
Notifications SQS subscriber stopped process messages

Expected behavior
Gracefully reconnecting if the application is running

Flyte component

• Overall
• Flyte Setup and Installation scripts
• Flyte Documentation
• Flyte communication (slack/email etc)
• FlytePropeller
• FlyteIDL (Flyte specification language)
• Flytekit (Python SDK)
• FlyteAdmin (Control Plane service)
• FlytePlugins
• DataCatalog
• FlyteStdlib (common libraries)
• FlyteConsole (UI)
• Other

To Reproduce
Steps to reproduce the behavior:

1. Run flyte with enabled notifications
2. Wait for this happens

Environment
Flyte component

• Sandbox (local or on one machine)
• Cloud hosted
  • AWS
  • GCP
  • Azure
• Baremetal
• Other

Additional context
Logs:

{"json":{"src":"base.go:103"},"level":"error","msg":"error with starting processor err: [RequestError: send request failed\ncaused by: Post https://sqs.us-east-1.amazonaws.com/: read tcp 10.200.8.116:59882-\u003e52.46.137.144:443: read: connection reset by peer] ","ts":"2020-07-03T10:35:10Z"}
{"json":{"src":"processor.go:113"},"level":"warning","msg":"The stream for the subscriber channel closed with err: RequestError: send request failed\ncaused by: Post https://sqs.us-east-1.amazonaws.com/: read tcp 10.200.8.116:59882-\u003e52.46.137.144:443: read: connection reset by peer","ts":"2020-07-03T10:35:10Z"}

I guess solution will be similar to this one: flyteorg/flyteadmin#92

[kumare3]

@rstanevich thank you for opening up the bug, I will add it to the next milestone. Do you think thats ok, or is it affecting everyday and we should fix it ASAP?

[katrogan]

thanks for opening @rstanevich - is this a recent issue?

[rstanevich]

thank you @kumare3 and @katrogan

Honestly, we noticed it first time about notifications SQS subscriber.
We are running flyteadmin in multiple replicas and we have alert for the oldest message in SQS. Then we can just restart flyteadmin pods.
So, it looks not urgent at least for us.

[kumare3]

thank you @kumare3 and @katrogan

Honestly, we noticed it first time about notifications SQS subscriber.
We are running flyteadmin in multiple replicas and we have alert for the oldest message in SQS. Then we can just restart flyteadmin pods.
So, it looks not urgent at least for us.

Thank you for the reply. BTW, we have Flyte's very own Flash @katrogan who already has a PR for this

[kumare3]

@rstanevich do you have an alert if FlyteAdmin crashes? What would be preferrable, that you notice a FlyteAdmin crash or it continues to limp along and you notice older messages?

[rstanevich]

@kumare3
Yes, we have alerts on crashing any pods in cluster.
Do you mean restarting container if sqs subscriber stopped?
actually, I don't know what is better. It is pretty good now... maybe I think connection retry should be executed unlimited times
with some interval.

[kumare3]

@rstanevich I think we found a very good way of solving this problem. @katrogan will merge the PR soon. Thank you for raising the issue.

[kumare3]

it is merged and will be part of the next release