Web app comes to grinding halt with "Excon::Error::TooManyRequests"

[plindelauf]

We just migrated to the heroku-16 stack and with that upgraded to Workless 2(.2.0). However, once deployed to Heroku, our whole web app comes to a grinding halt after a few minutes with a "Excon::Error::TooManyRequests" exception. Judging from the stack trace, it seems that Workless accesses the Heroku API with every web request that comes in. Since we have a relatively high traffic app (many mico transactions), the API seems to run over its rate limit.

This has never been a problem with version 1 of Workless. Any way to get this working again like it did before?

Here's a stack trace:

2017-10-24T11:04:11.596159+00:00 app[web.6]:
2017-10-24T11:04:11.596237+00:00 app[web.6]: Excon::Error::TooManyRequests (Expected([200, 201, 202, 204, 206, 304]) <=> Actual(429 Too Many Requests)
2017-10-24T11:04:11.596238+00:00 app[web.6]: ):
2017-10-24T11:04:11.596289+00:00 app[web.6]:
2017-10-24T11:04:11.596426+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/excon-0.59.0/lib/excon/middlewares/expects.rb:7:in `response_call'
2017-10-24T11:04:11.596427+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/excon-0.59.0/lib/excon/middlewares/response_parser.rb:9:in `response_call'
2017-10-24T11:04:11.596429+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/excon-0.59.0/lib/excon/connection.rb:389:in `response'
2017-10-24T11:04:11.596430+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/excon-0.59.0/lib/excon/connection.rb:253:in `request'
2017-10-24T11:04:11.596430+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/heroics-0.0.24/lib/heroics/link.rb:111:in `request_with_cache'
2017-10-24T11:04:11.596431+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/heroics-0.0.24/lib/heroics/link.rb:66:in `run'
2017-10-24T11:04:11.596432+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/heroics-0.0.24/lib/heroics/resource.rb:28:in `method_missing'
2017-10-24T11:04:11.596433+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/platform-api-2.1.0/lib/platform-api/client.rb:1431:in `info'
2017-10-24T11:04:11.596433+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/workless-2.2.0/lib/workless/scalers/heroku.rb:24:in `workers'
2017-10-24T11:04:11.596434+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/workless-2.2.0/lib/workless/scalers/heroku.rb:12:in `up'
2017-10-24T11:04:11.596435+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/workless-2.2.0/lib/workless/middleware/workless_checker.rb:12:in `call'
2017-10-24T11:04:11.596436+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
2017-10-24T11:04:11.596436+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/rack-ssl-enforcer-0.2.9/lib/rack/ssl-enforcer.rb:57:in `call'
2017-10-24T11:04:11.596437+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
2017-10-24T11:04:11.596438+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/rack-attack-5.0.1/lib/rack/attack.rb:147:in `call'
2017-10-24T11:04:11.596439+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
2017-10-24T11:04:11.596439+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/warden-1.2.7/lib/warden/manager.rb:36:in `block in call'
2017-10-24T11:04:11.596440+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/warden-1.2.7/lib/warden/manager.rb:35:in `catch'
2017-10-24T11:04:11.596441+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/warden-1.2.7/lib/warden/manager.rb:35:in `call'
2017-10-24T11:04:11.596442+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
2017-10-24T11:04:11.596443+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/rack-2.0.3/lib/rack/etag.rb:25:in `call'
2017-10-24T11:04:11.596443+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
2017-10-24T11:04:11.596444+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/rack-2.0.3/lib/rack/conditional_get.rb:25:in `call'
2017-10-24T11:04:11.596445+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
2017-10-24T11:04:11.596445+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/rack-2.0.3/lib/rack/head.rb:12:in `call'
2017-10-24T11:04:11.596446+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
2017-10-24T11:04:11.596447+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/rack-2.0.3/lib/rack/session/abstract/id.rb:232:in `context'
2017-10-24T11:04:11.596447+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/rack-2.0.3/lib/rack/session/abstract/id.rb:226:in `call'
2017-10-24T11:04:11.596448+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
2017-10-24T11:04:11.596449+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/actionpack-5.0.3/lib/action_dispatch/middleware/cookies.rb:613:in `call'
2017-10-24T11:04:11.596450+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
2017-10-24T11:04:11.596450+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/actionpack-5.0.3/lib/action_dispatch/middleware/callbacks.rb:38:in `block in call'
2017-10-24T11:04:11.596451+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/activesupport-5.0.3/lib/active_support/callbacks.rb:97:in `__run_callbacks__'
2017-10-24T11:04:11.596452+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/activesupport-5.0.3/lib/active_support/callbacks.rb:750:in `_run_call_callbacks'
2017-10-24T11:04:11.596453+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/activesupport-5.0.3/lib/active_support/callbacks.rb:90:in `run_callbacks'
2017-10-24T11:04:11.596454+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/actionpack-5.0.3/lib/action_dispatch/middleware/callbacks.rb:36:in `call'
2017-10-24T11:04:11.596454+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
2017-10-24T11:04:11.596455+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/actionpack-5.0.3/lib/action_dispatch/middleware/remote_ip.rb:79:in `call'
2017-10-24T11:04:11.596456+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
2017-10-24T11:04:11.596456+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/actionpack-5.0.3/lib/action_dispatch/middleware/debug_exceptions.rb:49:in `call'
2017-10-24T11:04:11.596461+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
2017-10-24T11:04:11.596462+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/actionpack-5.0.3/lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'
2017-10-24T11:04:11.596462+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
2017-10-24T11:04:11.596463+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/railties-5.0.3/lib/rails/rack/logger.rb:36:in `call_app'
2017-10-24T11:04:11.596464+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/railties-5.0.3/lib/rails/rack/logger.rb:24:in `block in call'
2017-10-24T11:04:11.596465+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/activesupport-5.0.3/lib/active_support/tagged_logging.rb:69:in `block in tagged'
2017-10-24T11:04:11.596465+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/activesupport-5.0.3/lib/active_support/tagged_logging.rb:26:in `tagged'
2017-10-24T11:04:11.596466+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/activesupport-5.0.3/lib/active_support/tagged_logging.rb:69:in `tagged'
2017-10-24T11:04:11.596467+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/railties-5.0.3/lib/rails/rack/logger.rb:24:in `call'
2017-10-24T11:04:11.596467+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
2017-10-24T11:04:11.596468+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/request_store-1.3.2/lib/request_store/middleware.rb:9:in `call'
2017-10-24T11:04:11.596469+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
2017-10-24T11:04:11.596469+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/actionpack-5.0.3/lib/action_dispatch/middleware/request_id.rb:24:in `call'
2017-10-24T11:04:11.596470+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
2017-10-24T11:04:11.596471+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/rack-2.0.3/lib/rack/method_override.rb:22:in `call'
2017-10-24T11:04:11.596475+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
2017-10-24T11:04:11.596476+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/rack-2.0.3/lib/rack/runtime.rb:22:in `call'
2017-10-24T11:04:11.596476+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
2017-10-24T11:04:11.596477+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/activesupport-5.0.3/lib/active_support/cache/strategy/local_cache_middleware.rb:28:in `call'
2017-10-24T11:04:11.596478+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
2017-10-24T11:04:11.596479+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/dragonfly-1.0.12/lib/dragonfly/cookie_monster.rb:9:in `call'
2017-10-24T11:04:11.596479+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
2017-10-24T11:04:11.596480+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/actionpack-5.0.3/lib/action_dispatch/middleware/executor.rb:12:in `call'
2017-10-24T11:04:11.596481+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
2017-10-24T11:04:11.596482+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/actionpack-5.0.3/lib/action_dispatch/middleware/static.rb:136:in `call'
2017-10-24T11:04:11.596482+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
2017-10-24T11:04:11.596483+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/rack-2.0.3/lib/rack/sendfile.rb:111:in `call'
2017-10-24T11:04:11.596484+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
2017-10-24T11:04:11.596485+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/railties-5.0.3/lib/rails/engine.rb:522:in `call'
2017-10-24T11:04:11.596485+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/railties-5.0.3/lib/rails/railtie.rb:193:in `public_send'
2017-10-24T11:04:11.596486+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/railties-5.0.3/lib/rails/railtie.rb:193:in `method_missing'
2017-10-24T11:04:11.596487+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
2017-10-24T11:04:11.596487+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/rack-2.0.3/lib/rack/urlmap.rb:68:in `block in call'
2017-10-24T11:04:11.596488+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/rack-2.0.3/lib/rack/urlmap.rb:53:in `each'
2017-10-24T11:04:11.596489+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/rack-2.0.3/lib/rack/urlmap.rb:53:in `call'
2017-10-24T11:04:11.596490+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
2017-10-24T11:04:11.596490+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/rack-cors-0.4.1/lib/rack/cors.rb:81:in `call'
2017-10-24T11:04:11.596491+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
2017-10-24T11:04:11.596492+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/unicorn-5.3.1/lib/unicorn/http_server.rb:606:in `process_client'
2017-10-24T11:04:11.596496+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/unicorn-5.3.1/lib/unicorn/http_server.rb:702:in `worker_loop'
2017-10-24T11:04:11.596496+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/unicorn-5.3.1/lib/unicorn/http_server.rb:549:in `spawn_missing_workers'
2017-10-24T11:04:11.596497+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/unicorn-5.3.1/lib/unicorn/http_server.rb:142:in `start'
2017-10-24T11:04:11.596498+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/unicorn-5.3.1/bin/unicorn:126:in `<top (required)>'
2017-10-24T11:04:11.596499+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/bin/unicorn:22:in `load'
2017-10-24T11:04:11.596500+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/bin/unicorn:22:in `<top (required)>'
2017-10-24T11:04:11.596500+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/bundler-1.15.2/lib/bundler/cli/exec.rb:74:in `load'
2017-10-24T11:04:11.596501+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/bundler-1.15.2/lib/bundler/cli/exec.rb:74:in `kernel_load'
2017-10-24T11:04:11.596501+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/bundler-1.15.2/lib/bundler/cli/exec.rb:27:in `run'
2017-10-24T11:04:11.596502+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/bundler-1.15.2/lib/bundler/cli.rb:360:in `exec'
2017-10-24T11:04:11.596503+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/bundler-1.15.2/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
2017-10-24T11:04:11.596503+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/bundler-1.15.2/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
2017-10-24T11:04:11.596504+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/bundler-1.15.2/lib/bundler/vendor/thor/lib/thor.rb:369:in `dispatch'
2017-10-24T11:04:11.596505+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/bundler-1.15.2/lib/bundler/cli.rb:20:in `dispatch'
2017-10-24T11:04:11.596505+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/bundler-1.15.2/lib/bundler/vendor/thor/lib/thor/base.rb:444:in `start'
2017-10-24T11:04:11.596506+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/bundler-1.15.2/lib/bundler/cli.rb:10:in `start'
2017-10-24T11:04:11.596507+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/bundler-1.15.2/exe/bundle:30:in `block in <top (required)>'
2017-10-24T11:04:11.596508+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/bundler-1.15.2/lib/bundler/friendly_errors.rb:121:in `with_friendly_errors'
2017-10-24T11:04:11.596508+00:00 app[web.6]: vendor/bundle/ruby/2.3.0/gems/bundler-1.15.2/exe/bundle:22:in `<top (required)>'
2017-10-24T11:04:11.596509+00:00 app[web.6]: bin/bundle:3:in `load'
2017-10-24T11:04:11.596510+00:00 app[web.6]: bin/bundle:3:in `<main>'