Sidekiq is not reliable out of the box?

[ryana]

Hi,

My jaw literally dropped just now reading this https://github.com/mperham/sidekiq/wiki/Reliability

Sidekiq uses BRPOP to pop a job off the queue in Redis. This is very efficient and simple but it has one drawback: the job is now removed from Redis. If Sidekiq crashes while processing that job, it is lost forever. This is not a problem for many but some businesses need absolute reliability when processing jobs.

I'm not sure what "many" you are talking about. I don't know a single person that feels this way.

I am constantly running into this:

• In production on Heroku, long running tasks are being killed upon deploy. Jobs lost.
• In development, ~ 20% of the time I get uninitialized constant ClosetWorker on the first couple jobs (even though my eager load paths are correct) and so I lose those
• In production on a capistrano/ubuntu deploy, some issue in Ruby 2.1.1 was causing seg faults and thus the worker was crashing. Jobs lost.

I get that Sidekiq Pro helps to make your living, but how can reliability be a premium feature? Any chance we could get that ported over for us leeches? ;)

[seuros]

Try the latest version, 3.2. the eager loading was fixed.

[ryana]

I'm on 3.2.0

[seuros]

I think that if your background job is critical, you have to bake your own solution.
For example once you start your job, save the params in a database table and delete them when the job if finished

def preform(params) 
save_params
#your job here
delete_params
end

in case your program crash, you can always relaunch it since you have the params now.

[jonhyman]

Or buy Sidekiq Pro. It's a great piece of software and easily worth it. Just think of Sidekiq Pro like two extra dynos that are there as part of your HA/DR plan.

[ryana]

@seuros yeah that's too much of a hack for my liking ;) also, it's not completely reliable, as there is a chance of losing the job between the Redis pop instruction and the saving of the params.

@jonhyman yeah man it looks great, but I use it in maybe 10 apps that are in production. I can't go drop $7,500 to cover them all, and then continue to up that by $750 every time I deploy something new.

[mperham]

Sidekiq was originally designed to operate in a similar manner to Resque, including many Redis data structures and how it fetched jobs with BRPOP. It wasn't for a few months after the initial Sidekiq release that I realized there was an alternative fetching strategy that might be valuable.

1. Sidekiq should not lose jobs in progress. Shorten your timeout so Sidekiq has more time to push them back to Redis before Heroku kills the process, e.g. -t 5 or move your long jobs to rake tasks.
2. This is a bug in the application code. We saw those occasionally and they were always due to deviations from the Rails class naming conventions. Autoloading can be painful but there's not much Sidekiq can do about it.
3. Yep, this is something only RPOPLPUSH can help with.

I have no plans to open source any Pro features. I build high-quality OSS-based products. High-quality takes time and effort.

[mperham]

@ryana Pro is $750 per year per company. If all ten apps are running for the same company, you just need one license. If you are running a consultancy and there's 10 apps for 10 clients, then you need 10 licenses - 1 per client.

[seuros]

@ryana , i don't think you have to spend 7500$ for your app if they are for the same client.

[ryana]

@mperham good to know about the $750/company. I guess technically all but 1 of those apps fall under the umbrella of one LLC (not clients of that LLC).... Interesting. I'll consider that.

Thanks!