-
Notifications
You must be signed in to change notification settings - Fork 671
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] [FlyteAdmin] [scheduledWorkflowExecutor] SQS subscriber client stopped working #198
Comments
Hi @rstanevich thank you for the bug. This is highly unexpected and undesirable behavior. What version of FlyteAdmin are you running? |
Thanks @kumare3 for your response! |
looks like I duplicated this issue #88 |
Yes @katrogan confirmed she is on it. I will close this one and keep the root open @rstanevich |
It's on the backlog but I'm not actively taking a look. @anandswaminathan i honestly don't remember anything from when i looked at this when l5 ran into it. feel free to take this over if you want |
I'd like to show our case which strange enough from the logs:
we see that at But in 30mins new message appeared in SQS and it hasn't been handled by this subscriber. The messages from the queue started processing after restarting Do you have an idea what could happened with this client. I am also trying figuring out what the reason. I am not familiar with gizmo client, maybe the client there is some |
" (flyteorg#207) * Revert "Adopt flyteidl's ordered variable map change (flyteorg#198)" This reverts commit d76eb15 Signed-off-by: Sean Lin <sean@union.ai>
Signed-off-by: Ketan Umare <ketan.umare@gmail.com>
* feat: add workflow versions table Signed-off-by: csirius <davidtruong.dev@gmail.com> * feat: workflow version details page Signed-off-by: csirius <davidtruong.dev@gmail.com>
* feat: add workflow versions table Signed-off-by: csirius <davidtruong.dev@gmail.com> Signed-off-by: Jason Porter <jason@union.ai> * chore(release): Release 0.25.0 [skip ci] # [0.25.0](http://github.com/lyft/flyteconsole/compare/v0.24.0...v0.25.0) (2021-08-31) ### Features * add workflow versions table ([flyteorg#193](http://github.com/lyft/flyteconsole/issues/193)) ([6fff87e](http://github.com/lyft/flyteconsole/commit/6fff87e40007fd15faae634eb6402045c067dd2c)) Signed-off-by: Jason Porter <jason@union.ai> * improvement: show proper error message for aborted workflows (flyteorg#195) * improvement: show proper error message for aborted workflows Signed-off-by: Pianist038801 <steven@union.ai> * improvement: show abort message in the execution list Signed-off-by: Pianist038801 <steven@union.ai> Co-authored-by: Pianist038801 <steven@union.ai> Signed-off-by: Jason Porter <jason@union.ai> * Fix/versions executions gap (flyteorg#197) * feat: add workflow versions table Signed-off-by: csirius <davidtruong.dev@gmail.com> * fix: space between versions and executions table Signed-off-by: csirius <davidtruong.dev@gmail.com> Signed-off-by: Jason Porter <jason@union.ai> * feat: workflow version details page Signed-off-by: csirius <davidtruong.dev@gmail.com> Signed-off-by: Jason Porter <jason@union.ai> * Pre merge checkin Signed-off-by: Jason Porter <jason@union.ai> * fix: this is rfc, do not deploy until resolved; details in comments (flyteorg#172) * fix: this is rfc, do not deploy until resolved; details in comments Signed-off-by: Jason Porter <jason@union.ai> * fix: this is rfc, do not deploy until resolved; details in comments note: deploying with caution; the believe if that it was incorrect to use meta. Signed-off-by: Jason Porter <jason@union.ai> * fix: show field types on json launch form (flyteorg#199) Signed-off-by: Pianist038801 <steven@union.ai> Co-authored-by: Pianist038801 <steven@union.ai> Signed-off-by: Jason Porter <jason@union.ai> * chore(release): Release 0.25.1 [skip ci] ## [0.25.1](http://github.com/lyft/flyteconsole/compare/v0.25.0...v0.25.1) (2021-09-13) ### Bug Fixes * show field types on json launch form ([flyteorg#199](http://github.com/lyft/flyteconsole/issues/199)) ([a42b9f8](http://github.com/lyft/flyteconsole/commit/a42b9f8520fcd24dee752111e606ad9ae9bd88f5)) * this is rfc, do not deploy until resolved; details in comments ([flyteorg#172](http://github.com/lyft/flyteconsole/issues/172)) ([67dd183](http://github.com/lyft/flyteconsole/commit/67dd18397caf40e350da40e0672e500eaa9f338a)) Signed-off-by: Jason Porter <jason@union.ai> * Minor fixes Signed-off-by: Jason Porter <jason@union.ai> * Feat/version details (flyteorg#198) * feat: add workflow versions table Signed-off-by: csirius <davidtruong.dev@gmail.com> * feat: workflow version details page Signed-off-by: csirius <davidtruong.dev@gmail.com> Signed-off-by: Jason Porter <jason@union.ai> * fix: repopulate struct input fields on relaunch form (flyteorg#201) Signed-off-by: Pianist038801 <steven@union.ai> Co-authored-by: Pianist038801 <steven@union.ai> Signed-off-by: Jason Porter <jason@union.ai> * Graph ux feature add legend (flyteorg#196) * Checkin and merge to master Signed-off-by: Jason Porter <jason@union.ai> * fixed one more Signed-off-by: Jason Porter <jason@union.ai> * chore(release): Release 0.25.2 [skip ci] ## [0.25.2](http://github.com/lyft/flyteconsole/compare/v0.25.1...v0.25.2) (2021-09-16) ### Bug Fixes * repopulate struct input fields on relaunch form ([flyteorg#201](http://github.com/lyft/flyteconsole/issues/201)) ([950e080](http://github.com/lyft/flyteconsole/commit/950e080a2c52c6294630f0ffd1410b0e45b50a8d)) Signed-off-by: Jason Porter <jason@union.ai> * final before pr Signed-off-by: Jason Porter <jason@union.ai> * Fixed issues created by bad rebase Signed-off-by: Jason Porter <jason@union.ai> * Fixed more merge issues Signed-off-by: Jason Porter <jason@union.ai> Co-authored-by: csirius <davidtruong.dev@gmail.com> Co-authored-by: flyte bot <admin@flyte.org> Co-authored-by: pianist <26953709+Pianist038801@users.noreply.github.com> Co-authored-by: Pianist038801 <steven@union.ai> Co-authored-by: csirius <85753828+csirius@users.noreply.github.com>
Signed-off-by: Ketan Umare <ketan.umare@gmail.com>
" (flyteorg#207) * Revert "Adopt flyteidl's ordered variable map change (flyteorg#198)" This reverts commit 5a8e120 Signed-off-by: Sean Lin <sean@union.ai>
Signed-off-by: Ketan Umare <ketan.umare@gmail.com>
Signed-off-by: Ketan Umare <ketan.umare@gmail.com>
Signed-off-by: Ketan Umare <ketan.umare@gmail.com>
Describe the bug
Flyteadmin scheduledWorkflowExecutor worked successfully ~2 weeks with no redeployment.
But one day:
What logs I'd got before the scheduler stopped working:
https://github.com/lyft/flyteadmin/blob/60b4c876ea105d4c79e3cad7d56fde6b9c208bcd/pkg/rpc/adminservice/base.go#L138L139
Normally, this log doesn't appear when FlyteAdmin starts. And if I understand correctly this log shouldn't appear because this row is not reachable.
Unfortunately, there is no more logs.
Expected behavior
https://github.com/nytimes/gizmo/blob/master/pubsub/pubsub.go#L44
Flyte component
To Reproduce
Steps to reproduce the behavior:
{"json":{"src":"base.go:138"},"level":"info","msg":"Successfully started running the scheduled workflow executor","ts":"2020-03-07T05:39:04Z"}
Environment
Flyte component
Others
Restarting of the FlyteAdmin pod initialized new scheduledWorkflowExecutor and the SQS events were executed.
Thank you!
The text was updated successfully, but these errors were encountered: