RSpec Queue (RSpecQ) distributes and executes RSpec suites among parallel workers. It uses a centralized queue that workers connect to and pop off tests from. It ensures optimal scheduling of tests based on their run time, facilitating faster CI builds.
RSpecQ is inspired by test-queue and ci-queue.
- Run an RSpec suite among many workers (potentially located in different hosts) in a distributed fashion, facilitating faster CI builds.
- Consolidated, real-time reporting of a build's progress.
- Optimal scheduling of test execution by using timings statistics from previous runs and automatically scheduling slow spec files as individual examples. See Spec file splitting.
- Automatic retry of test failures before being considered legit, in order to rule out flakiness. Additionally, flaky tests are detected and provided to the user. See Requeues.
- Handles intermittent worker failures (e.g. network hiccups, faulty hardware etc.) by detecting non-responsive workers and requeing their jobs. See Worker failures
- Sentry integration for monitoring build-level events. See Sentry integration. See #2.
- Automatic termination of builds after a certain amount of failures. See Fail-fast.
A worker needs to be given a name and the build it will participate in.
Assuming there's a Redis instance listening at localhost, starting a worker
is as simple as:
$ rspecq --build=123 --worker=foo1 spec/To start more workers for the same build, use distinct worker IDs but the same build ID:
$ rspecq --build=123 --worker=foo2To view the progress of the build use --report:
$ rspecq --build=123 --reportFor detailed info use --help:
NAME:
rspecq - Optimally distribute and run RSpec suites among parallel workers
USAGE:
rspecq [<options>] [spec files or directories]
OPTIONS:
-b, --build ID A unique identifier for the build. Should be common among workers participating in the same build.
-w, --worker ID An identifier for the worker. Workers participating in the same build should have distinct IDs.
--seed SEED The RSpec seed. Passing the seed can be helpful in many ways i.e reproduction and testing.
-r, --redis HOST --redis is deprecated. Use --redis-host or --redis-url instead. Redis host to connect to (default: 127.0.0.1).
--redis-host HOST Redis host to connect to (default: 127.0.0.1).
--redis-url URL Redis URL to connect to (e.g.: redis://127.0.0.1:6379/0).
--update-timings Update the global job timings key with the timings of this build. Note: This key is used as the basis for job scheduling.
--file-split-threshold N Split spec files slower than N seconds and schedule them as individual examples.
--report Enable reporter mode: do not pull tests off the queue; instead print build progress and exit when it's finished.
Exits with a non-zero status code if there were any failures.
--report-timeout N Fail if build is not finished after N seconds. Only applicable if --report is enabled (default: 3600).
--max-requeues N Retry failed examples up to N times before considering them legit failures (default: 3).
--queue-wait-timeout N Time to wait for a queue to be ready before considering it failed (default: 30).
--fail-fast N Abort build with a non-zero status code after N failed examples.
--reproduction Enable reproduction mode: Publish files and examples in the exact order given in the command. Incompatible with --timings.
--tag TAG Run examples with the specified tag, or exclude examples by adding ~ before the tag. - e.g. ~slow - TAG is always converted to a symbol.
-h, --help Show this message.
-v, --version Print the version and exit.
You can set most options using ENV variables:
$ RSPECQ_BUILD=123 RSPECQ_WORKER=foo1 rspecq spec/| Name | Desc |
|---|---|
RSPECQ_BUILD |
Build ID |
RSPECQ_WORKER |
Worker ID |
RSPECQ_SEED |
RSpec seed |
RSPECQ_REDIS |
Redis HOST |
RSPECQ_UPDATE_TIMINGS |
Timings |
RSPECQ_FILE_SPLIT_THRESHOLD |
File split threshold |
RSPECQ_REPORT |
Report |
RSPECQ_REPORT_TIMEOUT |
Report Timeout |
RSPECQ_MAX_REQUEUES |
Max requests |
RSPECQ_QUEUE_WAIT_TIMEOUT |
Queue wait timeout |
RSPECQ_REDIS_URL |
Redis URL |
RSPECQ_FAIL_FAST |
Fail fast |
RSPECQ_REPORTER_RERUN_COMMAND_SKIP |
Do not report flaky test's rerun command |
RSpecQ can optionally emit build events to a
Sentry project by setting the
SENTRY_DSN environment variable.
This is convenient for monitoring important warnings/errors that may impact build times, such as the fact that no previous timings were found and therefore job scheduling was effectively random for a particular build.
The core design is almost identical to ci-queue so please refer to its README instead.
- Job: the smallest unit of work, which is usually a spec file
(e.g.
./spec/models/foo_spec.rb) but can also be an individual example (e.g../spec/models/foo_spec.rb[1:2:1]) if the file is too slow. - Queue: a collection of Redis-backed structures that hold all the necessary information for an RSpecQ build to run. This includes timing statistics, jobs to be executed, the failure reports and more.
- Build: a particular test suite run. Each build has its own Queue.
- Worker: an
rspecqprocess that, given a build id, consumes jobs off the build's queue and executes them using RSpec - Reporter: an
rspecqprocess that, given a build id, waits for the build's queue to be drained and prints the build summary report
Particularly slow spec files may set a limit to how fast a build can be. For example, a single file may need 10 minutes to run while all other files finish after 8 minutes. This would cause all but one workers to be sitting idle for 2 minutes.
To overcome this issue, RSpecQ can split files which their execution time is
above a certain threshold (set with the --file-split-threshold option)
and instead schedule them as individual examples.
Note: In the future, we'd like for the slow threshold to be calculated and set dynamically (see #3).
As a mitigation technique against flaky tests, if an example fails it will be
put back to the queue to be picked up by another worker. This will be repeated
up to a certain number of times (set with the --max-requeues option), after
which the example will be considered a legit failure and printed as such in the
final report.
Flaky tests are also detected and printed as such in the final report. They are also emitted to Sentry (see Sentry integration).
In order to prevent large suites running for a long time with a lot of failures, a threshold can be set to control the number of failed examples that will render the build unsuccessful. This is in par with RSpec's --fail-fast.
This feature is disabled by default, and can be controlled via the
--fail-fast command line option.
It's not uncommon for CI processes to encounter unrecoverable failures for various reasons: faulty hardware, network hiccups, segmentation faults in MRI etc.
For resiliency against such issues, workers emit a heartbeat after each
example they execute, to signal
that they're healthy and performing jobs as expected. If a worker hasn't
emitted a heartbeat for a given amount of time (set by WORKER_LIVENESS_SEC)
it is considered dead and its reserved job will be put back to the queue, to
be picked up by another healthy worker.
Update: ci-queue deprecated support for RSpec.
While evaluating ci-queue we experienced slow worker boot times (up to 3 minutes in some cases) combined with disk IO saturation and increased memory consumption. This is due to the fact that a worker in ci-queue has to load every spec file on boot. In applications with a large number of spec files this may result in a significant performance hit and in case of cloud environments, increased costs.
We also observed slower build times compared to our previous solution which scheduled whole spec files (as opposed to individual examples), due to big differences in runtimes of individual examples, something common in big RSpec suites.
We decided for RSpecQ to use whole spec files as its main unit of work (as opposed to ci-queue which uses individual examples). This means that an RSpecQ worker only loads the files needed and ends up with a subset of all the suite's files. (Note: RSpecQ also schedules individual examples, but only when this is deemed necessary, see Spec file splitting).
This kept boot and test run times considerably fast. As a side benefit, this
allows suites to keep using before(:all) hooks (which ci-queue explicitly
rejects).
The downside of this design is that it's more complicated, since the scheduling
of spec files happens based on timings calculated from previous runs. This
means that RSpecQ maintains a key with the timing of each job and updates it
on every run (if the --update-timings option was used). Also, RSpecQ has a
"slow file threshold" which, currently has to be set manually (but this can be
improved in the future).
Install the required dependencies:
$ bundle install
Then you can execute the tests after spinning up a Redis instance at
127.0.0.1:6379:
$ bundle exec rake
To enable verbose output in the tests:
$ RSPECQ_DEBUG=1 bundle exec rake
RSpecQ by design doesn't expire its keys from Redis. It is left to the user to configure the Redis server to do so; see Using Redis as an LRU cache for more info.
You can do this from a configuration file or with redis-cli.
RSpecQ is licensed under MIT. See LICENSE.