Enable redis server as coordination backend for platform development

[m4dcoder]

A lot of workflow use cases that contain parallel branches, join task, and with items task require coordination service to be configured to be executed properly. For platform development, setup coordination service using redis server as backend by default to avoid confusion in troubleshooting and debugging workflows.

[punkrokk]

I feel like this opens a can of worms. For example, how will we test that things continue to function where coordination is not present. If we are developing for the other? JP Bourget / @punkrokk

…

On Oct 4, 2020, at 10:51 PM, W Chan ***@***.***> wrote:  A lot of workflow use cases that contain parallel branches, join task, and with items task require coordination service to be configured to be executed properly. For platform development, setup coordination service using redis server as backend by default to avoid confusion in troubleshooting and debugging workflows. You can view, comment on, or merge this pull request online at: #5052 Commit Summary Enable redis server as coordination backend for platform development File Changes M Makefile (5) M conf/st2.dev.conf (2) Patch Links: https://github.com/StackStorm/st2/pull/5052.patch https://github.com/StackStorm/st2/pull/5052.diff — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[m4dcoder]

@punkrokk I'm not sure I know what you mean. Coordination service is really a requirement given the number of workflow use cases that depend on it to work correctly. The only workflow use case that does not require coordination service are sequential workflows. For sequential workflows, the workflow engine does not invoke the coordination service whether it is configured or not. So configuring the coordination service has no bearing on testing these workflows.

[amanda11]

I think in the documentation at the moment then it states that the coordination service is only required for the concurrency policy or HA.
I don't think it states it is needed for the other examples that have been listed, so do we also need a doc update?

And then does it also follow that if it's used in dev, should we have a default coordination service installed by one liner installer and others?

[arm4b]

Indeed coordination was really documented to be a requirement for Policies and HA only.

If the reality now that [coordination] is required for any normal Orquesta Workflow execution, - apart of just documentation update it should be added to the default StackStorm architecture and all the installers/deployment methods we have.

[m4dcoder]

It really is required for any non-sequential workflows to function properly. The documentation should be updated. We will save a lot of user headaches just including a coordination service backend as part of the StackStorm install. For this PR here though, which is specifically to ensure development environment is setup properly at least, I'm proposing redis server as the backend.

[guzzijones]

YES omg YES. I have been wanting this as well. Orquesta workflows don't work correctly if they are non-sequential without a coordination backend.

[punkrokk]

Ok, I see your point. I am also seeing situations where concurrency is enabled absent coordination and the scheduler service is very unhappy. (@m4dcoder )

[arm4b]

Per team feedback and as a follow-up,

The change here is part of the whole story of enabling [coordination] backend across the StackStorm fleet. From what I understand the following might be involved:

• documentation update signalizing that [coordination] is now required/enabled
• scripted installer changes to include Redis in default installation

That sounds like a major item for a release.
I believe we all care about the user experience making sure Orquesta workflows are executing fine in production, not only in dev env.

@m4dcoder Do you think you can target next v3.5.0 release with these major set of changes?

[cognifloyd]

Once merge conflicts are resolved, LGTM.

[arm4b]

I believe we'll need to include redis in main pip requirements https://github.com/StackStorm/st2/blob/master/requirements.txt

Besides of that, it's all good as the target is to ship the Redis as a default st2 coordination backend in v3.5.0 👍

[punkrokk]

Eugen is correct. JP Bourget / @punkrokk

…

On Apr 1, 2021, at 1:26 PM, Eugen Cusmaunsa ***@***.***> wrote:  @armab commented on this pull request. I believe we'll need to include redis in main pip requirements https://github.com/StackStorm/st2/blob/master/requirements.txt Besides of that, it's all good as the target is to ship the Redis as a default st2 coordination backend in v3.5.0 👍 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[m4dcoder]

Replaced this PR with #5226.