shared binary

[charlierudolph]

The following is a proposal for how cucumber could be built in the future. There is binary that is responsible for all shared logic (CLI parsing, reading gherkin files, compiling pickles, basic infrastructure running, formatting results). This binary would be written in go or some other language that allows us to compile binaries for each OS. Then there are a series of language specific libraries which use the binary to run tests.

Logic for the language specific libraries:

• If needed, fetch operating system specific binary (this could also be done during the install process where possible)
  • Store in .cucumber/
• Execute binary as a child process (passing all command line options). Communicate with the binary over stdin / stdout via lines of JSON. The binary requests the language specific library to do the following things:
  • Load support code
    • Request includes: paths to load
    • Reply with
      • array of step definitions (id, expression text, expression type)
      • array of parameter types (id, regular expression)
  • Setup test run
    • Run before all hooks - reply with possible error when done
  • Setup test case
    • Run before each hooks - reply with possible error when done
  • Run test step
    • Request includes
      • Id
      • Arguments (each has a parameter type ids if it needs to be transformed)
    • Run step with given id - reply with possible error when done
  • Teardown test case
    • Run after each hooks - reply with possible error when done
  • Teardown test run
    • Run after all hooks - reply with possible error when done
  • Print to stdout
    • Request includes: text
  • Print to stderr
    • Request includes: text
  • Exit
    • Request includes: exit code
  • Generate step definition snippet
    • Request includes:
      • Text
      • Type - cucumber expression or regular expression
    • Reply with a step definition snippet

This is similar to the cucumber wire protocol but built in a way that the ruby dependency is removed and allows language specific code to be the entry point.

I want to spike this out on cucumber-js with the shared binary being written in node for the purposes of the spike. Primary goals would be to pull the language specific code and see how the interface feels.

I’m uncertain how custom formatters / plugins would fit into this but those could potentially be done with a formatter streaming to a named pipe.

Please bring up any issues you can think of. I’m sure there are features in some of the other cucumber flavors that I am unaware of.

[aslakhellesoy]

Thanks for bringing this up @charlierudolph! Inthink this would be a great topic for discussion at Cukenspace (preferrably Sunday as I'm busy Saturday).

I have a lot of ideas that are similar to yours and I'll dump them here when I find time, hopefully before cukenspace.

[brasmusson]

Neither Cucumber-Ruby not Cucumber-JVM have built in support for parallel execution, but there are third solutions for achieving that (parallel_cucumber, cucumber-jvm-parallel-plugin). In that direction, I have imagined what massive parallel execution on a computing grid could look like. We already have the Pickle abstraction and the Gherkin to Pickle compiler implemented in several languages, so it seems natural to use that as a foundation. I imagine a central controller dispatching Pickles Events to the different executors in the computing grid, the executors creates Test Cases from the Pickles, executes them and pass back a Test Case Result Event.

In a more detailed view I imagine that an executor is generating (the normal) Test Case Started/Finished, and Test Step Started/Finished Events, but over the network they are compressed to a Test Case Result Event, from which the original events can be recreated on the controller side.

    Controller            |               Network        |        Executor(s)
                          |         Pickle Event         | 
                          |      ---------------->       | 
                          |                              |    Test Case Started Event
                          |                              |    <--------------------
                          |                              |    Test Step Started Event
                          |                              |    <--------------------
                          |                              |    Test Step Finished Event
                          |                              |    <--------------------
                          |                              |    Test Case Finished Event
                          |                              |    <--------------------
                          |    Test Case Result Event    |
                          |    <--------------------     |
 Test Case Started Event  |                              |
 <--------------------    |                              |
 Test Step Started Event  |                              |
 <--------------------    |                              |
 Test Step Finished Event |                              |
 <--------------------    |                              |
 Test Case Finished Event |                              |
 <--------------------    |                              |                                   

This way less events will be sent over the network (which may or may not be important), and formatters listening to event of the controller have the events from test case execution serialized which definitely simplify their implementation.

With this structure it would be possible to

• execute suites in parallel, that is some executors execute end-to-end and others execute directly against the the domain layer.
• have the executors use different languages for step definitions, like in the suites case the end-to-end testing use one step definitions language, and the domain layer testing use another step definitions language.

[lxbzmy]

Cucumber is a good library connect tester and developer.
Using cucumber ,developer could write less code but add more test case by Tester.
I alway suggest my team and our customer use cucumber to improve poor unit test.

I dont like gauge because it can not embed in source code.it's not friendly for developer.

[mattwynne]

Some brief thoughts from me to share before we get to talk about this face-to-face.

• I like the idea of having less codeabases doing the same / similar thing in different languages
• I like the idea of making behaviour more consistent across implementations
• I like the idea of thinking broadly about architecture and protocols
• I worry that we'd inhibit contributions to Cucumber by using a lower-level language like Go for most of the guts of it.
• I worry that trying to build one giant binary that does everything might be a big project
• I worry that if we end up having to start servers or whatever in order to "serve" platform-specific glue code to the cross-platform binary, the overall UX might end up clunky. That's why SpecFlow turned out to be better than Cuke4Nuke.

I think I'd like to see us figure out where the seams are, and think about budding off smaller pieces, like a console formatter binary/ies that takes a stream of results NJSON and emits a stream of text, or a runtime binary that takes a stream of pickles as input, calls step definitions and emits results NDJSON. To me, building smaller, compose-able pieces like this would be a less risky approach. Gradually hollowing out the individual implementations, rather than trying to replace them in one go.

[charlierudolph]

I worry that if we end up having to start servers or whatever in order to "serve" platform-specific glue code to the cross-platform binary, the overall UX might end up clunky. That's why SpecFlow turned out to be better than Cuke4Nuke.

I'm was thinking of hosting the binaries on github attached to releases (example). The individual libraries can pack the binaries with their source code if they like. I was against that to start because it means a user has the binaries for all supported os on their machine when they only need one. Pulling the binary can be done during the install process where possible or it can be pulled down and cached locally on first run.

---

Attempting to split this large binary into smaller pieces I think we could potentially break it into the following:

• gherkin-parser binary
  • input
    • current working directory
    • feature paths
    • tag filters
    • name filters
  • output
    • streaming event protocol for events of type: 'source', 'gherkin-document', 'pickle', 'pickle-accepted', 'pickle-rejected''
• pickle-runner binary
  • input
    • accepted pickles
    • other runtime config (fail-fast, dry-run)
    • step definition configs (for each step definition it needs an id and the pattern)
    • hook definition configs (for each hook definition it needs an id and the tags)
    • parameter types (id and pattern)
  • interacts with calling process telling it to run hooks / steps and is sent back the results
  • output
    • streams event protocol for events of type: 'test-case-prepared', 'test-case-started', 'test-step-started', 'test-step-finished', 'test-case-finished'
• formatter-binary - probably one binary for each type or a single binary with all the built ins
  • input
    • stream of event protocol
  • output
    • stream of text

The first is mostly extracted with the gherkin library but requires implementations in every language and we could shave that down to 1. After implementing the event protocol on cucumber-js, a lot of logic moved into the formatter as that was the only place that needed it.

[mattwynne]

Yeah the interesting part of this will be building the pick-runner binary.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs.

[stale]

This issue has been automatically closed because of inactivity. You can support the Cucumber core team on opencollective.

[aslakhellesoy]

@charlierudolph shall we open this issue again?

Want to join the Slack discussion? https://cucumberbdd.slack.com/archives/C62D0FK0E/p1520406289000034

[aslakhellesoy]

This is currently in progress over at https://github.com/cucumber/cucumber-pickle-runner

/cc @mpkorstanje

[charlierudolph]

Currently, the pickle runner binary is a WIP I will be experimenting with on cucumber-js, once it gets to a workable state. I am happy to get feedback if the current api is not sufficient for a particular implementation.

[aslakhellesoy]

I'm closing this as cucumber-engine is now a thing.

Please see the roadmap for details about how we'll get there.

[charlierudolph]

Woohoo! So glad to have this rolling now!

[mattwynne]

Big thanks to you @charlierudolph for putting so much momentum behind it!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.