Parallel scenario execution

[joshski]

I'm looking into the possibility of running browser-based scenarios concurrently on cloud machines, for fast cross-browser testing.

So I would like to execute scenarios in parallel, but I would also like to execute each scenario multiple times (i.e. run the same scenario against lots of different devices concurrently). Ideally, the whole suite would execute in just over the time of the slowest scenario on the slowest device.

Historically there was some support for this in parallel_test, but this is broken in cucumber 2.0.

My first questions are:

1. Does the new cucumber API support asynchronous and/or concurrent scenario execution, or is it designed to execute scenarios in serial?
2. Do I need to fork a process for each scenario?
3. Applying this to a local web app (let's assume rails) how do I isolate the application state for each scenario run?

If anybody can point me in the right direction with respect to the current cucumber API, I'll give it a crack and see what happens.

[mattwynne]

Regarding the API, things are still in flux: we've cleaned up the internals a great deal and put many of then into https://github.com/cucumber/cucumber-ruby-core

We're trying to keep this as the 'inner hexagon' that can compile and run test cases, with the cucumber gem then being a client of that. The code in the cucumber gem is still a bit of a mess, but it's getting better.

So you might want to start with the core and build something from scratch. If you look at the README you'll see an example of how to build one in only a few lines.

The key abstraction you'll need to leverage is the Filter API. This isn't well documented, but you can think of it a bit like Rack middleware for a cucumber suite. The filters are arranged into a chain with the Gherkin compiler at the input. As test cases are compiled from the gherkin, they're passed into the chain. Filters can choose to either pass the test case through, pass a modified copy, or not pass it on. At the end of the chain is the Runner that executes the test cases.

I guess you'll need to implement something that supports the Filter protocol but does the distribution of work to various runners. You'll then need something that supports the Report API at the other end to collate the results.

Have a spelunk through the code in the core and see if that makes any sense. If you want to book some pairing time with me to go through this in more depth (assuming the code you're working on is OSS) just shout.

[joshski]

Thanks for your help @mattwynne!

I had a little play with it, got something working with core filters and the parallel gem and ended up with this slightly monstrous hack: https://github.com/featurist/cucumber-parallel

In this example I run cucumber (scenarios * contexts) + 2 times in total, in a single process that forks up to 50 others:

1. once as a dry run to capture scenario locations
2. once per scenario per context (a context might be a particular browser, for example)
3. once at the end with a filter that "replays" the results of the many scenario runs, so that normal cucumber formatters can be used.

My thinking was that the ideal parallel cucumber runner would behave just like cucumber, but faster. So I need to reuse the bits of cucumber-ruby that interpret CLI options, find gherkin files and so on. Without pulling the code apart that means building an alternative Cucumber::Cli::Main that doesn't try to Process.exit all the time.

Because I'm running cucumber so many times, I'm assuming this will repeat various bits of work like reading gherkin files and won't be as fast as if I did it with more filters and fewer passes.

I also think a separate "parallel formatter" API might make more sense, because the above implementation doesn't give feedback until all scenarios have been executed.

Anyway, I just wanted to post progress here to see if anybody had any more feedback about running parallel scenarios.

Does it make sense to have a cucumber_parallel binary in a separate gem, with a similar UI to cucumber? or is there an alternative way of packaging it?

If you run the same scenarios in multiple contexts, what kind of report would you like to see? A normal cucumber report, with aggregated errors, or something else?

[brasmusson]

@joshski When you talk about "scenario per context", it seems like you are basically talking about "suites" as they are defined in #821. So here we talk about two things, the ability to run the same scenario several times with different context, and the ability to run scenarios in parallel. In the longer run I think it would rather be "pickles" as they are defined in gherkin3 that should be distributed over different contexts and runners. Cucumber-Ruby already have a compile step like the one described in gherkin3 (even though "pickle" is not a term used in Cucumber-Ruby). But there will probably take some time to get there.

[mattwynne]

I would like it best if we can make the distribution of test cases to runners something that you can hook into and control from cucumber itself, so that you can do it using processes or even across machines or whatever, but all using the regular cucumber UI.

so rather than this being a binary, it might be a plug-in extension.

So I say carry on hacking on it outside for now, but try to help me understand where we need to put in those extension points for you so that it could be a plug-in in the end.

Make sense? I wish I had more time to focus on this Josh but not right now!

[t-morgan]

I looked into this a little bit and it looks like there are at least two places parallel execution could be started from within cucumber-ruby-core.

For features one possible location where the parallel threads could start is in Cucumber::Core's compile method.

For test cases, Cucumber::Core::Test::LocationsFilter seems to be the right spot to start parallel execution (I've tested this out a little bit and it does seem to work).

My idea is to check the configuration for the '--parallel' flag (probably feature by default and '--parallel scenario' or something of the sort to toggle) and have a parallel runner object take care of the work. This could at some point also take a node location for test distribution -- but I don't want to think too much about that at the moment.

Let me know what you think, if I'm on the right path or maybe should think of more options.

[mattwynne]

Note that there are some changes coming in the way this API works.

I think I would approach this by replacing the Core::Test::Runner as a DistributedRunner or some such that takes test cases and distributes them across the nodes, collates the results and then fires out result events on the bus just the same way as the existing Runner does. The slave nodes could use a real Core::Test::Runner.

The challenge will be in serializing test cases that include their step definitions, and re-hydrating those on the slave. They do contain a location for the source of the step definition block, so this is possible, but that's probably the hardest part of this. That and defining the protocol between the master and the slaves.

Does that make any sense?

[mattwynne]

On second thoughts there's an alternative where the ActivateSteps, AddBeforeHooks etc filters are run on the slaves. That would make the test cases easier to serialise / deserialise.

[brasmusson]

I agree, conceptually the slaves would be sent a pickle. Event though Cucumber-Ruby, in terms of Ruby classes, does not make any difference between pickles and test cases, they are useful when thinking about the design.

[t-morgan]

So, if we want to go the Filter route, I think it would be possible to add a PickleDistributor / ParallelFilter somewhere near the Quit filter. It might also be nice to use a config file to set the number of processors to use, location of distributed machines (if distributing as well as running in parallel) etc.

With filters the parallel part seems pretty straight forward to me. I'm just not sure about how to go about distribution -- is this something we would use DRb for? Or are there other/better ways to achieve this that you know of?

[mattwynne]

it feels like this deserves to be in a repo of its own, built as a plugin. I wonder what extension points we'd need to add so that cucumber was flexible enough for you to be able to plug this in?

I think it would be worth building a spike in another repo (or a gist) to explore this some more. It could be a variation of the example in the readme for the core. WDYT?

[t-morgan]

I have created a cucumber-threads repo to explore this more. Currently there is a working example using a Queue and some worker Threads.

* Working for very simple scenarios on MRI Ruby. More complex scenarios (especially those that share state) do not work properly in threads -- I will look into forking and/or DRb going forward.

[westlakem]

t-morgan informed me of this thread, as I have been working on a solution for some time. I thought I would give my insights into some of my findings.

SUMMARY

There are too many global variables to the cucumber_world to make this possible right now (including results, cucumber_runtime instance, etc). In order to do parallel testing, multiple ruby instances must be created unless those items global to the cucumber_world are abstracted out.

That being said, if we were to abstract all those parts out, we can already execute the test code as part of the cucumber pickle . (This was tested by using a before hook)

Before do |scenario|
  scenario.instance_variable_get(:@test_case).instance_variable_get(:@test_steps)[0].instance_variable_get(:@action).instance_variable_get(:@block).call
end

this could be called in a .each loop on the test_steps to execute all steps within the pickle. This is possible due to the block being an executable proc

CURRENT SOLUTION:

My current solution is a client/server configuration, that utilizes a sinatra server that feeds test locations to runners.

server:

pass in feature directory through command line, and parse for tests by feature or scenario (by using Dir.glob or Gherkin::Parser on a single file)

Then I start a sinatra server with a single endpoint to feed the tests to the runners

runners:

I mixin a run_without_setup method on the runtime

 def run_without_setup
    self.visitor = report
    @features = nil
    @filespecs = nil
    receiver = Cucumber::Core::Test::Runner.new(report)
    compile features, receiver, filters
  end

Then I manually start a Cucumber::Runtime, and load the language

I then get a test from the sinatra server and run it using cucumber CLI. For each test after that, i change the path in the runtime options, and run the run_without_setup command on the runtime.

Issues overcome:

1. Distributed parallel exeuction: All runners are connected to the sinatra server feeding it what test to run. Each client only asks for 1 test at a time (following the FAIR distribution pattern)
2. Multiple runtimes: You cannot have multiple Cucumber::Runtime environments in one ruby process. This means each individual runner must start it's own process (done by calling the CLI)
3. Each machine must have access to the code, and an environment to run the code in. We solved this by having a startup script that provides us what we need, with a future state of having docker containers with rvm already configured, and a shared data container with the must up-to-date code.
4. Multiple runners per machine: Since each runner is calling CLI, there can be as many running tests on one machine as it can handle. There more is overhead then ideal, as each runner needs its own ruby env.
5. There is no communication back to the server to report the test case: We use a formatter to collect all the data from the different runners and store it in a centralized location.

Drawbacks:

1. No error handling: Test distribution is fire and forget. If a runner crashes before the test is complete, it doesn't get counted. Have though about using a framework like rabbitmq and using manual ack to signal when a test is complete (in place of the sinatra server), but I haven't experimented with that yet.

[mattwynne]

Wow what great exploration @westlakem!

It seems to me that we need to do more work to make the code in cucumber-ruby more modular before this is going to be easy. The abstractions like the Runner and Filters from the core will make this easier, but as you've seen, there's still some pretty gnarly code in and around the runtime.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs.

[stale]

This issue has been automatically closed because of inactivity. You can support the Cucumber core team on opencollective.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.