Skip to content

Commit

Permalink
Merge branch 'master' of github.com:mperham/sidekiq
Browse files Browse the repository at this point in the history
  • Loading branch information
mperham committed Jul 31, 2018
2 parents 3cadbce + 95f8fc1 commit 24ac099
Show file tree
Hide file tree
Showing 8 changed files with 50 additions and 13 deletions.
2 changes: 2 additions & 0 deletions Changes.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
HEAD
-----------

- **Decrease default concurrency from 25 to 10** [#3892]
- Smoother scheduling for large Sidekiq clusters [#3889]
- Switch Sidekiq::Testing impl from alias\_method to Module#prepend, for resiliency [#3852]
- Update Sidekiq APIs to use SCAN for scalability [#3848, ffiller]
- Remove concurrent-ruby gem dependency [#3830]
Expand Down
2 changes: 1 addition & 1 deletion lib/sidekiq.rb
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ module Sidekiq
DEFAULTS = {
queues: [],
labels: [],
concurrency: 25,
concurrency: 10,
require: '.',
environment: nil,
timeout: 8,
Expand Down
35 changes: 32 additions & 3 deletions lib/sidekiq/scheduled.rb
Original file line number Diff line number Diff line change
Expand Up @@ -97,9 +97,34 @@ def wait
sleep 5
end

# Calculates a random interval that is ±50% the desired average.
def random_poll_interval
poll_interval_average * rand + poll_interval_average.to_f / 2
# We want one Sidekiq process to schedule jobs every N seconds. We have M processes
# and **don't** want to coordinate.
#
# So in N*M second timespan, we want each process to schedule once. The basic loop is:
#
# * sleep a random amount within that N*M timespan
# * wake up and schedule
#
# We want to avoid one edge case: imagine a set of 2 processes, scheduling every 5 seconds,
# so N*M = 10. Each process decides to randomly sleep 8 seconds, now we've failed to meet
# that 5 second average. Thankfully each schedule cycle will sleep randomly so the next
# iteration could see each process sleep for 1 second, undercutting our average.
#
# So below 10 processes, we special case and ensure the processes sleep closer to the average.
# In the example above, each process should schedule every 10 seconds on average. We special
# case smaller clusters to add 50% so they would sleep somewhere between 5 and 15 seconds.
# As we run more processes, the scheduling interval average will approach an even spread
# between 0 and poll interval so we don't need this artifical boost.
#
if process_count < 10
# For small clusters, calculate a random interval that is ±50% the desired average.
poll_interval_average * rand + poll_interval_average.to_f / 2
else
# With 10+ processes, we should have enough randomness to get decent polling
# across the entire timespan
poll_interval_average * rand
end
end

# We do our best to tune the poll interval to the size of the active Sidekiq
Expand All @@ -123,9 +148,13 @@ def poll_interval_average
# This minimizes a single point of failure by dispersing check-ins but without taxing
# Redis if you run many Sidekiq processes.
def scaled_poll_interval
process_count * Sidekiq.options[:average_scheduled_poll_interval]
end

def process_count
pcount = Sidekiq::ProcessSet.new.size
pcount = 1 if pcount == 0
pcount * Sidekiq.options[:average_scheduled_poll_interval]
pcount
end

def initial_wait
Expand Down
2 changes: 1 addition & 1 deletion lib/sidekiq/version.rb
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# frozen_string_literal: true
module Sidekiq
VERSION = "5.1.4"
VERSION = "5.2.0"
end
2 changes: 1 addition & 1 deletion lib/sidekiq/web/helpers.rb
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ def job_params(job, score)
end

def parse_params(params)
score, jid = params.split("-")
score, jid = params.split("-", 2)
[score.to_f, jid]
end

Expand Down
12 changes: 9 additions & 3 deletions test/test_web.rb
Original file line number Diff line number Diff line change
Expand Up @@ -453,7 +453,7 @@ def perform(a, b)

post "/morgue/#{score}-", 'delete' => 'Delete'
assert_equal 302, last_response.status
assert_equal 0, s.size
assert_equal 1, s.size
end
end

Expand Down Expand Up @@ -515,6 +515,12 @@ def perform(a, b)
post "/morgue/#{job_params(*params)}", 'retry' => 'Retry'
assert_equal 302, last_response.status
assert_equal 'http://example.org/morgue', last_response.header['Location']
assert_equal 0, Sidekiq::DeadSet.new.size

params = add_dead('jid-with-hyphen')
post "/morgue/#{job_params(*params)}", 'retry' => 'Retry'
assert_equal 302, last_response.status
assert_equal 0, Sidekiq::DeadSet.new.size

get '/queues/foo'
assert_equal 200, last_response.status
Expand Down Expand Up @@ -556,15 +562,15 @@ def add_retry
[msg, score]
end

def add_dead
def add_dead(jid = SecureRandom.hex(12))
msg = { 'class' => 'HardWorker',
'args' => ['bob', 1, Time.now.to_f],
'queue' => 'foo',
'error_message' => 'Some fake message',
'error_class' => 'RuntimeError',
'retry_count' => 0,
'failed_at' => Time.now.utc,
'jid' => SecureRandom.hex(12) }
'jid' => jid }
score = Time.now.to_f
Sidekiq.redis do |conn|
conn.zadd('dead', score, Sidekiq.dump_json(msg))
Expand Down
6 changes: 3 additions & 3 deletions web/locales/es.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,11 @@ es:
Failed: Fallidas
Scheduled: Programadas
Retries: Reintentos
Enqueued: En Fila
Enqueued: En Cola
Worker: Trabajador
LivePoll: Sondeo en Vivo
StopPolling: Detener Sondeo
Queue: Fila
Queue: Cola
Class: Clase
Job: Trabajo
Arguments: Argumentos
Expand All @@ -27,7 +27,7 @@ es:
AddToQueue: Añadir a fila
AreYouSureDeleteJob: ¿Estás seguro de eliminar este trabajo?
AreYouSureDeleteQueue: ¿Estás seguro de eliminar la fila %{queue}?
Queues: Filas
Queues: Colas
Size: Tamaño
Actions: Acciones
NextRetry: Siguiente Intento
Expand Down
2 changes: 1 addition & 1 deletion web/views/layout.erb
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
<html dir="<%= text_direction %>">
<head>
<title><%= environment_title_prefix %><%= Sidekiq::NAME %></title>
<meta charset="utf8" />
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width,initial-scale=1.0" />

<link href="<%= root_path %>stylesheets/bootstrap.css" media="screen" rel="stylesheet" type="text/css" />
Expand Down

0 comments on commit 24ac099

Please sign in to comment.