Not processing unfinished jobs across server restarts using async_server mode on Iodine server

[daya]

First of all a bunch of thanks for doing such a good job 😍

FYI, I am using Iodine server instead of Puma and using async_server mode, to test if unfinished jobs will be retried across server restarts I did the following

class TestReliabilityJob < ApplicationJob
  queue_as :test_reliability

  def perform(num)
    begin
      if ENV["ALLOW_TO_PASS"]
        puts "I am allowed to finish, arg was #{num}"
      else
        raise "Forcibly not allowed, hence failing a test job"
      end
    rescue RealTimeSyncException => e
      puts "RealTimeSyncException thrown"
      puts e.message
      puts e.backtrace
      raise e
    end
  end
end

in application.rb I have

    config.good_job = {
        execution_mode: :async_server,
        queues: "test_reliability:1;cdr_sns:4",
        max_threads: 5,
        poll_interval: 30
    }

in application_job.rb

require 'real_time_sync'
class ApplicationJob < ActiveJob::Base
  retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY

  retry_on ::RealTimeSyncException, attempts: 100, wait: :exponentially_longer  do |_job, exception|
    puts exception.message
    puts exception.backtrace
  end

  JobTimeoutError = Class.new(StandardError)

  around_perform do |_job, block|
    # Timeout jobs after 10 minutes
    Timeout.timeout(10.minutes, JobTimeoutError) do
      block.call
    end
  rescue StandardError => e
    puts "Job Error: #{e.message}"
    puts e.backtrace
    raise
  end

end

then submitted the job

TestReliabilityJob.set(wait_until: 15.seconds).perform_later 911

The job was tried 3 times in after 15-20 seconds and showed up as unfinished on the dashboard

I stopped and restarted the server this time with

ALLOW_TO_PASS=true WEB_CONCURRENCY=5 RAILS_MAX_THREADS=10 GOOD_JOB_EXECUTION_MODE=async_server GOOD_JOB_MAX_THREADS=1 GOOD_JOB_POLL_INTERVAL=30 bundle exec rails s

Not sure what I am missing but the job is never retried.

So I am worried that if I try this approach in production then I am going to have a bunch of unfinished jobs that would never be retried 😒 . Could you enlighten me please?

Also is there a way to manually retry unfinished jobs in case I need to do it sometimes?

[daya]

Further investigation into the issue using good_job external worker

GOOD_JOB_QUEUES="test_reliability:1;cdr_sns:4" ALLOW_TO_PASS=true be good_job start

shows the unfinished jobs are re-processed and ultimately finished.

So this means async_server execution mode is not really production ready. What do you think @bensheldon ?

[bensheldon]

@daya thanks for opening this issue and documenting your configuration. This is strange and unexpected; unfinished jobs should be executed continuously until they are finished or errored.

Let me try to reproduce the problem with the configuration that you shared. I haven't used Iodine before. This is the code that async_server mode uses to detect that it is running within a server process:

good_job/lib/good_job/adapter.rb

Lines 148 to 154 in b436e4d

	def in_server_process?
	return @_in_server_process if defined? @_in_server_process
	@_in_server_process = Rails.const_defined?('Server') ||
	caller.grep(%r{config.ru}).any? || # EXAMPLE: config.ru:3:in `block in <main>' OR config.ru:3:in `new_from_string'
	(Concurrent.on_jruby? && caller.grep(%r{jruby/rack/rails_booter}).any?) # EXAMPLE: uri:classloader:/jruby/rack/rails_booter.rb:83:in `load_environment'
	end

[daya]

is there a way to manually retry unfinished jobs in case I need to do it sometimes?
@bensheldon ^^

[bensheldon]

is there a way to manually retry unfinished jobs in case I need to do it sometimes?

You can execute jobs with GoodJob.where(id: JOB_ID).perform_with_advisory_lock, but that's not intended to be a public API.

[bensheldon]

@daya I released GoodJob v1.9.3 which includes support for Iodine. Please let me know if it addresses the issue you experienced with async_server mode!

Also, I'm curious about the benefits your seeing from Iodine. I'd never heard of that server before. Thanks!

[daya]

@bensheldon thanks for such a quick turnaround.

The performance is better than puma. And apparently there is a better native support for websockets, better than ActionCable plus Redis free pubsub. I haven't yet used those features but they were tempting to select Iodine as I know I will need them later.