Chaining custom commands in cluster mode #536

manast · 2017-10-25T10:57:32Z

According to the documentation "Chaining custom commands in the pipeline is not supported in Cluster mode.".
I am wondering, is this a limitation of ioredis or of redis itself? and do you have any information or link on why this limitation exists?

stale · 2017-11-24T11:04:56Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 7 days if no further activity occurs, but feel free to re-open a closed issue if needed.

manast · 2017-11-24T12:40:29Z

@luin please, could you be so kind to share any insight you have in this issue?

luin · 2017-11-24T17:23:52Z

Sorry for the late response.

That's not a limitation of Redis. It's just not easy to support chaining custom commands in the cluster setup: ioredis always try evalsha first when sending custom commands, and when the command fails, eval will be sent instead. However, when it comes to pipelines, to make sure all commands in a pipeline will be received by Redis in order, ioredis sends script exists command to check if all lua scripts have loaded in the target redis node and will load the scripts which haven't loaded before sending the pipeline.

In the cluster setup, the nodes may change (e.g. failover happens) during the process of loading scripts, so it's hard to make sure that the scripts are loaded to the correct redis node. That's why this feature haven't been implemented.

manast · 2017-11-26T15:06:02Z

thanks for the throughout explanation. When calling defineCommand, why can't we just call SCRIPT LOAD and then after that just assume that the command is available?

luin · 2017-11-26T16:27:38Z

@manast Script cache may be cleared by the SCRIPT FLUSH command. Despite that, calling SCRIPT LOAD when defining commands indeed solves the problem (we also need to resend all SCRIPT LOAD again when a new connection has been made). Good idea! 👍

stale · 2017-12-26T17:23:18Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 7 days if no further activity occurs, but feel free to re-open a closed issue if needed.

manast · 2017-12-27T08:17:43Z

Bump! :)

stale · 2018-01-26T09:23:29Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 7 days if no further activity occurs, but feel free to re-open a closed issue if needed.

btd · 2018-01-26T10:29:11Z

Bump

stale · 2018-02-25T10:40:27Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 7 days if no further activity occurs, but feel free to re-open a closed issue if needed.

manast · 2018-02-25T12:45:13Z

Bump!

stale · 2018-03-27T13:31:54Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 7 days if no further activity occurs, but feel free to re-open a closed issue if needed.

btd · 2018-03-27T14:18:03Z

Bump

stale · 2018-04-26T15:10:32Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 7 days if no further activity occurs, but feel free to re-open a closed issue if needed.

manast · 2018-04-26T15:16:56Z

Bump because it is an important issue!

Rush · 2018-05-07T17:18:17Z

Bump - interested in it as well.

firedfox · 2019-03-11T02:29:04Z

Bump because we are using more Redis clusters in recent projects. This problem is getting more serious.

ajwootto · 2019-03-20T21:59:14Z

Also running into this.

btzsoft · 2019-04-16T10:56:55Z

we have the same issue.

botzill · 2019-04-16T10:57:50Z

Any updates on this issue? @luin

Thx.

jl9404 · 2019-04-29T18:25:52Z

@manast Script cache may be cleared by the SCRIPT FLUSH command. Despite that, calling SCRIPT LOAD when defining commands indeed solves the problem (we also need to resend all SCRIPT LOAD again when a new connection has been made). Good idea!

@luin wondering if we could always send SCRIPT LOAD to the pipeline so we can EVALSHA safely? Thx.

RPallas92 · 2019-07-29T09:54:55Z

Same issue here

andreimc · 2019-09-05T01:02:08Z

same issue here

kaikash · 2019-09-13T20:23:26Z

same issue here

atol-kyiv-ua · 2019-09-17T09:50:12Z

Any updates on this issue? @luin

NivLipetz · 2020-04-21T08:37:13Z

Hi @luin , are there any plans for this issue to be fixed or is it a lost cause?
Thanks!

shyimo · 2020-07-23T10:51:08Z

Hi @luin , any plans about adding this ?
we are now fully use Redis cluster and it's getting more serious now

Affects the following scripts: - moveToDelayed - moveToFinished - retryJob Allows for the `attemptsMade`, `stacktrace` and `message` fields to be updated through Lua scripts, rather than requiring a separate operation (or multiple operations) to set these fields. With the possibility of setting these fields within the context of a single script, we avoid the need to craft a transaction containing both HMSET and EVAL(SHA) commands to achieve atomicity. By shedding the need for a transaction, we are able to avoid redis/ioredis#536 and thus solve a common problem seen when targetting Redis clusters. This isn’t necessarily an ideal means of solving this problem, given we’re violating DRY principles by copying the same logic across scripts. It’d be ideal if the underlying ioredis restriction was removed (in a manner which doesn’t entail the performance cost of resubmitting scripts each time).

Fixes #536. Co-Authored-By: Matteo Collina <[email protected]>

# [4.19.0](v4.18.0...v4.19.0) (2020-10-23) ### Bug Fixes * Ensure delayed callbacks are always invoked. ([d6e78c3](d6e78c3)) ### Features * Add autopipeline for commands and allow multi slot pipelines. ([aba3c74](aba3c74)), closes [#536](#536)

ioredis-robot · 2020-10-23T15:59:03Z

🎉 This issue has been resolved in version 4.19.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

manast · 2020-11-19T11:14:17Z

FYI, I just upgraded to 4.19.0 and some of Bull tests are failing. I need to investigate more, but for now I will need to lock ioredis to 4.18.0 in order to not break bull users environments.

n0rick · 2022-05-28T20:05:28Z

sorry. useless comment.

# [4.19.0](redis/ioredis@v4.18.0...v4.19.0) (2020-10-23) ### Bug Fixes * Ensure delayed callbacks are always invoked. ([d6e78c3](redis/ioredis@d6e78c3)) ### Features * Add autopipeline for commands and allow multi slot pipelines. ([aba3c74](redis/ioredis@aba3c74)), closes [#536](redis/ioredis#536)

manast mentioned this issue Oct 25, 2017

Error: Sending custom commands in pipeline is not supported in Cluster mode when use Cluster Mode OptimalBits/bull#758

Open

stale bot added the wontfix label Nov 24, 2017

stale bot removed the wontfix label Nov 24, 2017

stale bot added the wontfix label Dec 26, 2017

stale bot removed the wontfix label Dec 27, 2017

stale bot added the wontfix label Jan 26, 2018

stale bot removed the wontfix label Jan 26, 2018

stale bot added the wontfix label Feb 25, 2018

stale bot removed the wontfix label Feb 25, 2018

stale bot added the wontfix label Mar 27, 2018

stale bot removed the wontfix label Mar 27, 2018

stale bot added the wontfix label Apr 26, 2018

stale bot removed the wontfix label Apr 26, 2018

luin added the pinned label May 8, 2018

r3nat mentioned this issue May 23, 2019

Chaining custom commands in cluster mode #876

Closed

ShogunPanda mentioned this issue Sep 30, 2020

Add autopipeline for commands and allow multi slot pipelines. #1201

Merged

AVVS closed this as completed in #1201 Oct 23, 2020

AVVS pushed a commit that referenced this issue Oct 23, 2020

feat: Add autopipeline for commands and allow multi slot pipelines.

aba3c74

Fixes #536. Co-Authored-By: Matteo Collina <[email protected]>

joebowbeer mentioned this issue Oct 23, 2020

Job.moveToFailed fails in Cluster mode taskforcesh/bullmq#257

Closed

ioredis-robot added the released label Oct 23, 2020

snyk-bot mentioned this issue Dec 5, 2020

[Snyk] Upgrade ioredis from 4.17.3 to 4.19.2 SalesforceCommerceCloud/commerce-sdk-core#16

Closed

snyk-bot mentioned this issue Jan 6, 2021

[Snyk] Upgrade ioredis from 4.17.3 to 4.22.0 SalesforceCommerceCloud/commerce-sdk-core#28

Closed

cc-prodsec mentioned this issue Mar 2, 2021

[Snyk] Upgrade ioredis from 4.17.3 to 4.22.0 SalesforceCommerceCloud/commerce-sdk-core#42

Closed

rahmed-hartree mentioned this issue Nov 7, 2023

Calling pipelined function evalsha is blocked when running redis in cluster mode redis/redis-py#2914

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Chaining custom commands in cluster mode #536

Chaining custom commands in cluster mode #536

manast commented Oct 25, 2017

stale bot commented Nov 24, 2017

manast commented Nov 24, 2017

luin commented Nov 24, 2017

manast commented Nov 26, 2017 •

edited

Loading

luin commented Nov 26, 2017 •

edited

Loading

stale bot commented Dec 26, 2017

manast commented Dec 27, 2017

stale bot commented Jan 26, 2018

btd commented Jan 26, 2018

stale bot commented Feb 25, 2018

manast commented Feb 25, 2018

stale bot commented Mar 27, 2018

btd commented Mar 27, 2018

stale bot commented Apr 26, 2018

manast commented Apr 26, 2018

Rush commented May 7, 2018

firedfox commented Mar 11, 2019

ajwootto commented Mar 20, 2019

btzsoft commented Apr 16, 2019

botzill commented Apr 16, 2019

jl9404 commented Apr 29, 2019 •

edited

Loading

RPallas92 commented Jul 29, 2019

andreimc commented Sep 5, 2019

kaikash commented Sep 13, 2019

atol-kyiv-ua commented Sep 17, 2019

NivLipetz commented Apr 21, 2020

shyimo commented Jul 23, 2020

ioredis-robot commented Oct 23, 2020

manast commented Nov 19, 2020

n0rick commented May 28, 2022 •

edited

Loading

Chaining custom commands in cluster mode #536

Chaining custom commands in cluster mode #536

Comments

manast commented Oct 25, 2017

stale bot commented Nov 24, 2017

manast commented Nov 24, 2017

luin commented Nov 24, 2017

manast commented Nov 26, 2017 • edited Loading

luin commented Nov 26, 2017 • edited Loading

stale bot commented Dec 26, 2017

manast commented Dec 27, 2017

stale bot commented Jan 26, 2018

btd commented Jan 26, 2018

stale bot commented Feb 25, 2018

manast commented Feb 25, 2018

stale bot commented Mar 27, 2018

btd commented Mar 27, 2018

stale bot commented Apr 26, 2018

manast commented Apr 26, 2018

Rush commented May 7, 2018

firedfox commented Mar 11, 2019

ajwootto commented Mar 20, 2019

btzsoft commented Apr 16, 2019

botzill commented Apr 16, 2019

jl9404 commented Apr 29, 2019 • edited Loading

RPallas92 commented Jul 29, 2019

andreimc commented Sep 5, 2019

kaikash commented Sep 13, 2019

atol-kyiv-ua commented Sep 17, 2019

NivLipetz commented Apr 21, 2020

shyimo commented Jul 23, 2020

ioredis-robot commented Oct 23, 2020

manast commented Nov 19, 2020

n0rick commented May 28, 2022 • edited Loading

manast commented Nov 26, 2017 •

edited

Loading

luin commented Nov 26, 2017 •

edited

Loading

jl9404 commented Apr 29, 2019 •

edited

Loading

n0rick commented May 28, 2022 •

edited

Loading