aruba 0.3.2 fails features (stdout buffering issue?)

[graaff]

I'm getting a few failures like this one with the released version of the aruba 0.3.2 gem. The intend of the feature seems to be fulfilled, just not the order in which the results are shown. I'm not sure to what extend this is considered to be a problem.

This is using "ruby 1.8.7 (2011-02-18 patchlevel 334) [x86_64-linux]". Please let me know if you need more details.

  Scenario: Detect output from all processes                    # features/output.feature:142
    When I run "ruby -e 'puts \"hello world!\"'"                # lib/aruba/cucumber.rb:94
    And I run "ruby -e 'puts gets.chomp.reverse'" interactively # lib/aruba/cucumber.rb:102
    And I type "hello"                                          # lib/aruba/cucumber.rb:106
    Then the output should contain exactly:                     # lib/aruba/cucumber.rb:138
      """
      hello world!
      olleh

      """
      expected: "hello world!\nolleh\n"
           got: "olleh\nhello world!\n" (using ==)
      Diff:
      
      @@ -1,3 +1,3 @@
      -hello world!
       olleh
      +hello world!
       (RSpec::Expectations::ExpectationNotMetError)
      ./features/support/../../lib/aruba/cucumber.rb:139:in `/^the output should contain exactly:$/'
      features/output.feature:146:in `Then the output should contain exactly:'

[msassak]

If you're seeing failures like this with your own features, can you post an example like the above but include the relevant scenario (or scenarios) and code from your own project? This is the first time I've seen an error like this, and that might help narrow it down.

[graaff]

I have not tried to write my own code yet, I'm only trying to run aruba's own features. It seems to me to be a really simple test case already since it only involves calling out to ruby.

[graaff]

I can also reproduce this with aruba HEAD. The problem is that @processes in aruba/api.rb is a Hash, and thus in all_stderr no ordering is guaranteed in ruby 1.8.7 and my ruby picks the "wrong" order.

One way to fix this is would be to add a @started_at timestamp to aruba/process.rb and use that to sort.

The real underlying problem seems to be that the output code doesn't multiplex the different streams, so if applications output different things at different times the output still won't be accurately reflected in aruba. Fixing that would also solve this issue.

[msassak]

What about storing the processes as an array of [name, process] arrays, and searching for them that way? That's pretty simple, and will keep order of operations on the processes deterministic when we need to do something to all of them. Care to submit a patch?

[hedgehog]

I can confirm this issue, I work around this by adding this step to my common_steps.rb:

Then /^the output from "([^"]*)" should contain exactly:$/ do |cmd, exact_output|
  output_from(cmd).should == exact_output
end

If you see this error:

NoMethodError: undefined method `output' for nil:NilClass
/home/hedge/.rvm/gems/ruby-1.9.2-p136@bbb-bdd-meta-bdd/gems/aruba-0.3.2/lib/aruba/api.rb:118:in `output_from'
/home/mv/Documents/Workspaces/b3/features/step_definitions/common_steps.rb:2:in `block in <top (required)>'

Make sure your feature description does this:

Then I run "b3 libraries"
And the output from "b3 libraries" should contain exactly:
"""
....
"""

[msassak]

@hedgehog How do you use that code to modify the behavior graaff is seeing?

[hedgehog]

To match a command with the output from that command. @graaff is seeing the side effect of Aruba's method of joining output contents from multiple commands.

This raises your point: Can/should Aruba guarantee the output from separate When I run ... steps are in the sequence that the commands are invoked (synchronous), or not?

I say no, no, no, no.
Make a README.md note that if output order matters then you should not be issuing the next command until the first completes, and its output is validated.

OR:

Consider adding a When /^I run the command list:$/ step that enforces, and records, run order using a table of commands.
As well as a `Then ^/the ordered output|stderr|stdout from the runlist should contain|exactly:$/

This will then make unambiguous that the multiple commands case needs to be treated differently.

[msassak]

@hedgehog I like the idea of running a list of commands in series, but I think that's a bit more complex than is called for at the moment. Right now there's a failing test, and I think it would be best to get that fixed before considering new features. Whether Aruba should have more complex opinions about the order of IO reads is a discussion for a different ticket, don't you think?

[hedgehog]

@msassak, that is the point, the example is failing simply becasue it describes behavior that Aruba doesn't (shouldn't) promise (or try) to deliver: output order by in runlist exec time.

You'll see in the pull request I take this into account:
hedgehog@5e4815e#L0R55

No?

[msassak]

I don't think that's the case. The example is failing because the implementation of the Hash class in 1.9 differs from its implementation in 1.8. The steps for detecting multiline output from a named process would be a good addition to Aruba, but I don't see how that is relevant to fixing this bug, or why, in a testing framework, we should choose the non-deterministic behavior over the deterministic one.

[hedgehog]

OK.

[msassak]

Fix pushed here: https://github.com/msassak/aruba/tree/48-ordered-output. I'll send a pull request soon.

[aslakhellesoy]

Closed by #61. 0.3.5 released. Thanks Mike!