Add async_server option to run async only in Rails web server proce…

…ss (#230)

* Add `async_server` option to run async only in Rails web server process; renames `async` option to `async_all` with deprecation notice

* Update server-detection logic; remove `:async_all` and deprecation of `:async`

* Do not output "Process already exited" from ShellOut spec helper

* Refactor Configuration#execution_mode

.rubocop.yml

@@ -52,6 +52,10 @@ Rails/ApplicationJob:
 Rails/ApplicationRecord:
   Enabled: false
 
+Rails/Inquiry:
+  Exclude:
+    - spec/**/*
+
 RSpec/AnyInstance:
   Enabled: false
 

README.md

@@ -119,10 +119,10 @@ For more of the story of GoodJob, read the [introductory blog post](https://isla
     - GoodJob can also be configured to execute jobs within the web server process to save on resources. This is useful for low-workloads when economy is paramount.
 
         ```
-        $ GOOD_JOB_EXECUTION_MODE=async rails server
+        $ GOOD_JOB_EXECUTION_MODE=async_server rails server
         ```
 
-        Additional configuration is likely necessary, see the reference below for async configuration.
+        Additional configuration is likely necessary, see the reference below for f configuration.
 
 ## Compatibility
 
@@ -206,15 +206,15 @@ Additional configuration can be provided via `config.good_job.OPTION = ...` for
 config.active_job.queue_adapter = :good_job
 
 # Configure options individually...
-config.good_job.execution_mode = :async
+config.good_job.execution_mode = :async_server
 config.good_job.max_threads = 5
 config.good_job.poll_interval = 30 # seconds
 config.good_job.shutdown_timeout = 25 # seconds
 
 
 # ...or all at once.
 config.good_job = {
-  execution_mode: :async,
+  execution_mode: :async_server,
   max_threads: 5,
   poll_interval: 30,
   shutdown_timeout: 25,
@@ -226,10 +226,11 @@ Available configuration options are:
 - `execution_mode` (symbol) specifies how and where jobs should be executed. You can also set this with the environment variable `GOOD_JOB_EXECUTION_MODE`. It can be any one of:
     - `:inline` executes jobs immediately in whatever process queued them (usually the web server process). This should only be used in test and development environments.
     - `:external` causes the adapter to enqueue jobs, but not execute them. When using this option (the default for production environments), you’ll need to use the command-line tool to actually execute your jobs.
-    - `:async` causes the adapter to execute you jobs in separate threads in whatever process queued them (usually the web process). This is akin to running the command-line tool’s code inside your web server. It can be more economical for small workloads (you don’t need a separate machine or environment for running your jobs), but if your web server is under heavy load or your jobs require a lot of resources, you should choose `:external` instead.
-- `max_threads` (integer) sets the maximum number of threads to use when `execution_mode` is set to `:async`. You can also set this with the environment variable `GOOD_JOB_MAX_THREADS`.
-- `queues` (string) determines which queues to execute jobs from when `execution_mode` is set to `:async`. See the description of `good_job start` for more details on the format of this string. You can also set this with the environment variable `GOOD_JOB_QUEUES`.
-- `poll_interval` (integer) sets the number of seconds between polls for jobs when `execution_mode` is set to `:async`. You can also set this with the environment variable `GOOD_JOB_POLL_INTERVAL`.
+    - `:async_server` executes jobs in separate threads within the Rails webserver process (`bundle exec rails server`). It can be more economical for small workloads because you don’t need a separate machine or environment for running your jobs, but if your web server is under heavy load or your jobs require a lot of resources, you should choose `:external` instead.  When not in the Rails webserver, jobs will execute in `:external` mode to ensure jobs are not executed within `rails console`, `rails db:migrate`, `rails assets:prepare`, etc.
+    - `:async` executes jobs in separate threads in _any_ Rails process.
+- `max_threads` (integer) sets the maximum number of threads to use when `execution_mode` is set to `:async` or `:async_server`. You can also set this with the environment variable `GOOD_JOB_MAX_THREADS`.
+- `queues` (string) determines which queues to execute jobs from when `execution_mode` is set to `:async` or `:async_server`. See the description of `good_job start` for more details on the format of this string. You can also set this with the environment variable `GOOD_JOB_QUEUES`.
+- `poll_interval` (integer) sets the number of seconds between polls for jobs when `execution_mode` is set to `:async` or `:async_server`. You can also set this with the environment variable `GOOD_JOB_POLL_INTERVAL`.
 - `max_cache` (integer) sets the maximum number of scheduled jobs that will be stored in memory to reduce execution latency when also polling for scheduled jobs. Caching 10,000 scheduled jobs uses approximately 20MB of memory. You can also set this with the environment variable `GOOD_JOB_MAX_CACHE`.
 - `shutdown_timeout` (float) number of seconds to wait for jobs to finish when shutting down before stopping the thread. Defaults to forever: `-1`. You can also set this with the environment variable `GOOD_JOB_SHUTDOWN_TIMEOUT`.
 
@@ -485,11 +486,11 @@ GoodJob can execute jobs "async" in the same process as the webserver (e.g. `bin
     config.active_job.queue_adapter = :good_job
 
     # To change the execution mode
-    config.good_job.execution_mode = :async
+    config.good_job.execution_mode = :async_server
 
     # Or with more configuration
     config.good_job = {
-      execution_mode: :async,
+      execution_mode: :async_server,
       max_threads: 4,
       poll_interval: 30
     }
@@ -498,7 +499,7 @@ GoodJob can execute jobs "async" in the same process as the webserver (e.g. `bin
 - Or, with environment variables:
 
     ```bash
-    $ GOOD_JOB_EXECUTION_MODE=async GOOD_JOB_MAX_THREADS=4 GOOD_JOB_POLL_INTERVAL=30 bin/rails server
+    $ GOOD_JOB_EXECUTION_MODE=async_server GOOD_JOB_MAX_THREADS=4 GOOD_JOB_POLL_INTERVAL=30 bin/rails server
     ```
 
 Depending on your application configuration, you may need to take additional steps:

exe/good_job

@@ -1,5 +1,5 @@
 #!/usr/bin/env ruby
 require 'good_job/cli'
-GOOD_JOB_WITHIN_CLI = true
+GOOD_JOB_WITHIN_EXE = true
 GOOD_JOB_LOG_TO_STDOUT = true
 GoodJob::CLI.start(ARGV)

lib/active_job/queue_adapters/good_job_adapter.rb

@@ -2,10 +2,6 @@ module ActiveJob # :nodoc:
   module QueueAdapters # :nodoc:
     # See {GoodJob::Adapter} for details.
     class GoodJobAdapter < GoodJob::Adapter
-      def initialize(**options)
-        configuration = GoodJob::Configuration.new(options, env: ENV)
-        super(**options.merge(execution_mode: configuration.rails_execution_mode))
-      end
     end
   end
 end

lib/good_job/adapter.rb

@@ -4,13 +4,15 @@ module GoodJob
   #
   class Adapter
     # Valid execution modes.
-    EXECUTION_MODES = [:async, :external, :inline].freeze
+    EXECUTION_MODES = [:async, :async_server, :external, :inline].freeze
 
     # @param execution_mode [nil, Symbol] specifies how and where jobs should be executed. You can also set this with the environment variable +GOOD_JOB_EXECUTION_MODE+.
     #
     #  - +:inline+ executes jobs immediately in whatever process queued them (usually the web server process). This should only be used in test and development environments.
     #  - +:external+ causes the adapter to enqueue jobs, but not execute them. When using this option (the default for production environments), you'll need to use the command-line tool to actually execute your jobs.
-    #  - +:async+ causes the adapter to execute you jobs in separate threads in whatever process queued them (usually the web process). This is akin to running the command-line tool's code inside your web server. It can be more economical for small workloads (you don't need a separate machine or environment for running your jobs), but if your web server is under heavy load or your jobs require a lot of resources, you should choose `:external` instead.
+    #  - +:async_server+ executes jobs in separate threads within the Rails webserver process (`bundle exec rails server`). It can be more economical for small workloads because you don't need a separate machine or environment for running your jobs, but if your web server is under heavy load or your jobs require a lot of resources, you should choose +:external+ instead.
+    #    When not in the Rails webserver, jobs will execute in +:external+ mode to ensure jobs are not executed within `rails console`, `rails db:migrate`, `rails assets:prepare`, etc.
+    #  - +:async+ executes jobs in any Rails process.
     #
     #  The default value depends on the Rails environment:
     #
@@ -46,8 +48,7 @@ def initialize(execution_mode: nil, queues: nil, max_threads: nil, poll_interval
           poll_interval: poll_interval,
         }
       )
-
-      raise ArgumentError, "execution_mode: must be one of #{EXECUTION_MODES.join(', ')}." unless EXECUTION_MODES.include?(@configuration.execution_mode)
+      @configuration.validate!
 
       if execute_async? # rubocop:disable Style/GuardClause
         @notifier = GoodJob::Notifier.new
@@ -126,17 +127,30 @@ def shutdown(timeout: :default, wait: nil)
 
     # Whether in +:async+ execution mode.
     def execute_async?
-      @configuration.execution_mode == :async
+      @configuration.execution_mode == :async ||
+        @configuration.execution_mode == :async_server && in_server_process?
     end
 
     # Whether in +:external+ execution mode.
     def execute_externally?
-      @configuration.execution_mode == :external
+      @configuration.execution_mode == :external ||
+        @configuration.execution_mode == :async_server && !in_server_process?
     end
 
     # Whether in +:inline+ execution mode.
     def execute_inline?
       @configuration.execution_mode == :inline
     end
+
+    private
+
+    # Whether running in a web server process.
+    def in_server_process?
+      return @_in_server_process if defined? @_in_server_process
+
+      @_in_server_process = Rails.const_defined?('Server') ||
+                            caller.grep(%r{config.ru}).any? || # EXAMPLE: config.ru:3:in `block in <main>' OR config.ru:3:in `new_from_string'
+                            (Concurrent.on_jruby? && caller.grep(%r{jruby/rack/rails_booter}).any?) # EXAMPLE: uri:classloader:/jruby/rack/rails_booter.rb:83:in `load_environment'
+    end
   end
 end

lib/good_job/configuration.rb

@@ -5,6 +5,8 @@ module GoodJob
   # set options to get the final values for each option.
   #
   class Configuration
+    # Valid execution modes.
+    EXECUTION_MODES = [:async, :async_server, :external, :inline].freeze
     # Default number of threads to use per {Scheduler}
     DEFAULT_MAX_THREADS = 5
     # Default number of seconds between polls for jobs
@@ -13,7 +15,7 @@ class Configuration
     DEFAULT_MAX_CACHE = 10000
     # Default number of seconds to preserve jobs for {CLI#cleanup_preserved_jobs}
     DEFAULT_CLEANUP_PRESERVED_JOBS_BEFORE_SECONDS_AGO = 24 * 60 * 60
-    # Default to always wait for jobs to finish for {#shutdown}
+    # Default to always wait for jobs to finish for {Adapter#shutdown}
     DEFAULT_SHUTDOWN_TIMEOUT = -1
 
     # The options that were explicitly set when initializing +Configuration+.
@@ -35,40 +37,30 @@ def initialize(options, env: ENV)
       @env = env
     end
 
+    def validate!
+      raise ArgumentError, "GoodJob execution mode must be one of #{EXECUTION_MODES.join(', ')}. It was '#{execution_mode}' which is not valid." unless execution_mode.in?(EXECUTION_MODES)
+    end
+
     # Specifies how and where jobs should be executed. See {Adapter#initialize}
     # for more details on possible values.
-    #
-    # When running inside a Rails app, you may want to use
-    # {#rails_execution_mode}, which takes the current Rails environment into
-    # account when determining the final value.
-    #
-    # @param default [Symbol]
-    #   Value to use if none was specified in the configuration.
     # @return [Symbol]
-    def execution_mode(default: :external)
-      if defined?(GOOD_JOB_WITHIN_CLI) && GOOD_JOB_WITHIN_CLI
-        :external
-      elsif options[:execution_mode]
-        options[:execution_mode]
-      elsif rails_config[:execution_mode]
-        rails_config[:execution_mode]
-      elsif env['GOOD_JOB_EXECUTION_MODE'].present?
-        env['GOOD_JOB_EXECUTION_MODE'].to_sym
-      else
-        default
-      end
-    end
+    def execution_mode
+      @_execution_mode ||= begin
+        mode = if defined?(GOOD_JOB_WITHIN_EXE) && GOOD_JOB_WITHIN_EXE
+                 :external
+               else
+                 options[:execution_mode] ||
+                   rails_config[:execution_mode] ||
+                   env['GOOD_JOB_EXECUTION_MODE']
+               end
 
-    # Like {#execution_mode}, but takes the current Rails environment into
-    # account (e.g. in the +test+ environment, it falls back to +:inline+).
-    # @return [Symbol]
-    def rails_execution_mode
-      if execution_mode(default: nil)
-        execution_mode
-      elsif Rails.env.development? || Rails.env.test?
-        :inline
-      else
-        :external
+        if mode
+          mode.to_sym
+        elsif Rails.env.development? || Rails.env.test?
+          :inline
+        else
+          :external
+        end
       end
     end
 

spec/integration/adapter_spec.rb

@@ -2,7 +2,7 @@
 require 'rails_helper'
 
 RSpec.describe 'Adapter Integration' do
-  let(:adapter) { GoodJob::Adapter.new }
+  let(:adapter) { GoodJob::Adapter.new(execution_mode: :external) }
 
   around do |example|
     original_adapter = ActiveJob::Base.queue_adapter

spec/integration/server_spec.rb

@@ -29,4 +29,30 @@
       end
     end
   end
+
+  context 'when production async_server' do
+    let(:env) do
+      {
+        "RAILS_ENV" => "production",
+        "GOOD_JOB_EXECUTION_MODE" => "async_server",
+      }
+    end
+
+    it 'starts GoodJob when running webserver' do
+      ShellOut.command('bundle exec rails s', env: env) do |shell|
+        wait_until(max: 30) do
+          expect(shell.output).to include(/GoodJob started scheduler/)
+        end
+      end
+    end
+
+    it 'does not start GoodJob when running other commands' do
+      ShellOut.command('bundle exec rails db:version', env: env) do |shell|
+        wait_until(max: 30) do
+          expect(shell.output).to include(/Current version/)
+        end
+        expect(shell.output).not_to include(/GoodJob started scheduler/)
+      end
+    end
+  end
 end

spec/lib/good_job/adapter_spec.rb

@@ -2,7 +2,7 @@
 require 'rails_helper'
 
 RSpec.describe GoodJob::Adapter do
-  let(:adapter) { described_class.new }
+  let(:adapter) { described_class.new(execution_mode: :external) }
   let(:active_job) { instance_double(ActiveJob::Base) }
   let(:good_job) { instance_double(GoodJob::Job, queue_name: 'default', scheduled_at: nil) }
 
@@ -76,4 +76,40 @@
       adapter.shutdown
     end
   end
+
+  describe '#execute_async?' do
+    context 'when execution mode async_all' do
+      let(:adapter) { described_class.new(execution_mode: :async) }
+
+      it 'returns true' do
+        expect(adapter.execute_async?).to eq true
+      end
+    end
+
+    context 'when execution mode async_server' do
+      let(:adapter) { described_class.new(execution_mode: :async_server) }
+
+      context 'when Rails::Server is defined' do
+        before do
+          stub_const("Rails::Server", Class.new)
+        end
+
+        it 'returns true' do
+          expect(adapter.execute_async?).to eq true
+          expect(adapter.execute_externally?).to eq false
+        end
+      end
+
+      context 'when Rails::Server is not defined' do
+        before do
+          hide_const("Rails::Server")
+        end
+
+        it 'returns false' do
+          expect(adapter.execute_async?).to eq false
+          expect(adapter.execute_externally?).to eq true
+        end
+      end
+    end
+  end
 end

spec/lib/good_job/configuration_spec.rb

@@ -3,15 +3,25 @@
 
 RSpec.describe GoodJob::Configuration do
   describe '#execution_mode' do
-    it 'defaults to :external' do
-      configuration = described_class.new({})
-      expect(configuration.execution_mode).to eq :external
+    context 'when in development' do
+      before do
+        allow(Rails).to receive(:env) { "development".inquiry }
+      end
+
+      it 'defaults to :inline' do
+        configuration = described_class.new({})
+        expect(configuration.execution_mode).to eq :inline
+      end
     end
 
-    context 'when an explicit default is passed' do
-      it 'falls back to the default' do
+    context 'when in production' do
+      before do
+        allow(Rails).to receive(:env) { "production".inquiry }
+      end
+
+      it 'defaults to :external' do
         configuration = described_class.new({})
-        expect(configuration.execution_mode(default: :truck)).to eq :truck
+        expect(configuration.execution_mode).to eq :external
       end
     end
   end

spec/support/shell_out.rb

@@ -2,6 +2,7 @@
 
 class ShellOut
   WaitTimeout = Class.new(StandardError)
+  PROCESS_EXIT = "[PROCESS EXIT]".freeze
 
   def self.command(command, env: {}, &block)
     new.command(command, env: env, &block)
@@ -43,14 +44,14 @@ def command(command, env: {})
           Process.kill('TERM', pid)
           wait_thr.value
         rescue Errno::ESRCH
-          puts "Process already exited."
+          @output << PROCESS_EXIT
         end
       end
 
       stdout_future.value
       stderr_future.value
 
-      output
+      @output
     end
   end
 end