Not unlocking automatically (version 6.0.0rc5)

[noozo]

Describe the bug
Getting this in the sidekiq console when running an unique job:

2018-07-17T10:27:01.023Z 23351 TID-ouun5sb77 PcnCsvExportWorker JID-572a7573d90f9bfb8d100d83 INFO: start                                     
2018-07-17T10:28:13.650Z 23351 TID-ouun5sb77 PcnCsvExportWorker JID-572a7573d90f9bfb8d100d83 FATAL: the unique_key: uniquejobs:ba1ec618e6851fdfdf1ee334fb6c352a needs to be unlocked manually                                                                                             
2018-07-17T10:28:13.650Z 23351 TID-ouun5sb77 PcnCsvExportWorker JID-572a7573d90f9bfb8d100d83 INFO: done: 72.627 sec

Expected behavior
Expected the lock to be released so i could enqueue further jobs after this one is finished

Current behavior
No further jobs can be queued.

Worker class

class PcnCsvExportWorker
  include ActionView::Helpers::NumberHelper
  include Sidekiq::Worker
  sidekiq_options queue: :export, retry: 5, unique_across_queues: true, unique: :until_executed, log_duplicate_payload: true

  def perform(file)
    ...
  end
end

[mhenrixon]

Interesting! If you have a look at #289 and #290 the problem with v6.0.0.rc5 was that jobs was always unlocked when a worker error was raised.

Using the else clause now only unlocks the error if the worker does not raise an error. What you are experiencing seems to be the opposite of that. If you try v6.0.0.rc6 that won't exactly help. The problem seems to be that Sidekiq shuts down in the middle of you processing the job in which case we can't really unlock the job because we don't know if it will be retried or not. Could that be the case?

[noozo]

Well, the job does finish, and sidekiq does not crash :(

[mhenrixon]

I can't replicate that behaviour @nocivus. I'll keep investigating but this should not be happening unless your worker raises an error, sidekiq is shutting down or the job can't be unlocked. It might be that the job refuses to get unlocked for some reason.

Would be helpful to get some more information on the key that sits in Redis specifically it's ttl for instance. Might I ask why you are using unique_across_queues? That configuration option is a little misleading and are mostly intended to be used when scheduling the same worker to multiple queues and wanting to ensure that uniqueness are taken into consideration regardless of the queue. You would still have to use the same worker class for all those jobs for that to work though.

[lephyrius]

I also get this error when I only have:
sidekiq_options unique: :until_executed

[mhenrixon]

Thank you for the feedback @lephyrius. I will dig into this right now and see if I can replicate it using some real world code.

[mhenrixon]

I really can't replicate the behaviour. The only time this would happen to me is when the worker crashes or sidekiq shuts down which is a required behaviour. When the worker crashes the job will be retried, when Sidekiq shuts down we don't really know if the worker completed or not and we will have to figure this out ourselves.

I'll think of a way to make cleaning up these jobs or marking them as need attention or something. Maybe add something to Sidekiq::Web to give you guys a better overview.

[mhenrixon]

@lephyrius @nocivus I've added a more convenient way of dealing with this situation using a custom tab for Unique Digests but I still haven't been able to create a scenario where your problem happens.

[noozo]

Hi Mika, don't sweat it. Ended up doing a database based :) locking mechanism for my simple purposes :) Thanks again! Pedro

…

On Wed, Jul 25, 2018 at 9:44 PM Mika Hel ***@***.***> wrote: @lephyrius <https://github.com/lephyrius> @nocivus <https://github.com/nocivus> I've added a more convenient way of dealing with this situation using a custom tab for Unique Digests but I still haven't been able to create a scenario where your problem happens. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#293 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAANGfpOW92W4ZnRe-oUV5ryQbKC6Yzrks5uKNiVgaJpZM4VSmPV> .