Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix replace conflict strategy #314

Closed
wants to merge 1 commit into from

Conversation

rorhug
Copy link

@rorhug rorhug commented Aug 1, 2018

After initially failing to get this feature working, I found that the strategy class wasn't in the hash of strategies.

The other thing this PR addresses is removing the client middleware is loaded on the server.

The client middleware checks if a lock is in place and enqueues in its absense. Client middleware on the server however is only hit when the server enqueues to itself which happens when sidekiq takes scheduled items and enqueues them on the work queue.

I was trying to use the :until_executing lock along with :replace strategy, but the server would just take the job and drop in Client::Middleware#call because the lock only gets released during the server middleware call.

All ears if the client middleware on the server is necessary for whatever other reason.

All the tests pass without it (bar the test which just checks the presence of it, which I fixed).

After initially failing to get this feature working, I found that the strategy class wasn't in the hash of strategies.

The other thing this PR addresses is removing the client middleware is loaded on the server.
The client middleware checks if a lock is in place and enqueues in its absense. Client middleware is only hit when
the server enqueues to itself which happens when sidekiq takes scheduled items and enqueues them on the work queue.
I was trying to use the `:until_executing` lock along with `:replace` strategy, but the server would just take the job
and drop in `Client::Middleware#call` because the lock only gets released during the server middleware call.

All ears if the client middleware on the server is necessary for whatever other reason.

All the tests pass without it (bar the test which just checks the presence of it, which I fixed).
@mhenrixon
Copy link
Owner

Hi @RoryDH, thank you for the contribution.

The problem with this PR is that if a job has a nested job that requires uniqueness that nested job will be pushed regardless.

I'll see about maybe pushing it directly to redis using Lua or some such.

@mhenrixon mhenrixon mentioned this pull request Aug 2, 2018
@mhenrixon
Copy link
Owner

If you have a look in #315 that should be enough to make the replace work. The thing is that what the replace strategy does is remove the other job from the queue, scheduled queue or retry queue and then also remove the digest and locks. Now the client should be able to push this job again. It has nothing to do with executing the job. That is done from the server process while it is locked from the client push.

@rorhug
Copy link
Author

rorhug commented Aug 2, 2018

Great thanks for explaining and making those changes!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants