Skip to content

Releases: teamintricately/resque-unique_in_queue

v2.0.2

03 Jun 21:13
da68689
Compare
Choose a tag to compare
Using SCAN instead of KEYS at remove_queue (#1)

*** Why is this change necessary?

If you have a big collection of keys to be fetched and deleted from
Redis two problems will occur:

1. The server will be blocked due to the use of KEYS command since it
   takes a long time to return for big collections;
2. Using the splat ("*") operator for a large array raises a
   SystemStackError

Ultimately the queue is removed but the keys remain there:

```
2.3.8 :057 > Resque.size :test_queue
 => 300001
2.3.8 :058 > Resque.remove_queue :test_queue
SystemStackError: stack level too deep
	from /root/project/vendor/bundle/ruby/2.3.0/gems/resque-unique_in_queue-2.0.1/lib/resque/uniqu
e_in_queue/queue.rb:68:in `cleanup'
	from /root/project/vendor/bundle/ruby/2.3.0/gems/resque-unique_in_queue-2.0.1/lib/resque/uniqu
e_in_queue/resque_ext/resque.rb:35:in `remove_queue_with_unique_in_queue_cleanup'
	from /root/project/lib/ext/hacks/resque.rb:115:in `remove_queue_rewrite'
	from (irb):58
	from /root/project/vendor/bundle/ruby/2.3.0/gems/railties-4.2.11.1/lib/rails/commands/console.
rb:110:in `start'
	from /root/project/vendor/bundle/ruby/2.3.0/gems/railties-4.2.11.1/lib/rails/commands/console.
rb:9:in `start'
	from /root/project/vendor/bundle/ruby/2.3.0/gems/railties-4.2.11.1/lib/rails/commands/commands
_tasks.rb:68:in `console'
	from /root/project/vendor/bundle/ruby/2.3.0/gems/railties-4.2.11.1/lib/rails/commands/commands
_tasks.rb:39:in `run_command!'
	from /root/project/vendor/bundle/ruby/2.3.0/gems/railties-4.2.11.1/lib/rails/commands.rb:17:in
 `<top (required)>'
	from script/rails:9:in `require'
	from script/rails:9:in `<main>'
2.3.8 :059 > Resque.size :test_queue
 => 0
2.3.8 :060 > Resque.redis.keys("#{Resque::UniqueInQueue.configuration&.unique_in_queue_key_base}:queue:test_queue:job:*").count
 => 300001
```

*** How does it address the issue?

1. Replacing KEYS by SCAN command
2. Removing splat operator.

*** What side effects does this change have?

The removal process for large collections will take a lot longer:

```
irb(main):001:0> Benchmark.measure { Resque.remove_queue :test_queue }
 => #<Benchmark::Tms:0x000055b793b3cd60 @label="", @real=167.7889306979996, @cstime=0.0, @cutime=0.0, @stime=0.7600000000000002, @utime=11.500000000000002, @total=12.260000000000002>
```