Aruba should be up to date

[gonzalo-bulnes]

Please see this comment and the few next ones.

[honzajavorek]

The test suite is copied to each of the projects and it's not a real dependency (as in something which can be installed or is a git submodule), so it can be easily modified in one project and not the others. On the other hand, we definitely want to keep them the same as that's the whole purpose of the BDD test suite.

I'd propose we update Aruba in one of the reference hook projects - either in Ruby (@gonzalo-bulnes) or Python (me or @w-vi). Ideally, new Aruba would mean just changes in how the test suite is written, not changes in the code of the actual hooks handler under test. Once we're able to get it passing, we would then take the changes made to the tests and port them to dredd-hooks-template. Then, we would make Pull Requests with the changes to the all other projects (cc @ddelnano) in a similar manner I distributed required changes in .travis.yml in the past (#11, snikch/goodman#10, ddelnano/dredd-hooks-php#22, ungrim97/Dredd-Hooks#2, apiaryio/dredd-hooks-python#19).

What do you think @gonzalo-bulnes, @netmilk? Gonzalo, would you like to pioneer the Aruba upgrade?

[ddelnano]

👍

[gonzalo-bulnes]

Hi everyone!

Thoughts:

[...] new Aruba would mean just changes in how the test suite is written, not changes in the code of the actual hooks handler under test.

Yes, definitely. Any required change in the hook code should be regarded with great suspicion.

I'd propose we update Aruba in one of the reference hook projects [...] then take the changes made to the tests and port them to dredd-hooks-template [t]hen [...] make Pull Requests with the changes to the all other projects [...]

Sounds good to me 👍

Gonzalo, would you like to pioneer the Aruba upgrade?

Okay @honzajavorek I'll give it a try! : )

[gonzalo-bulnes]

It seems that updating Aruba is going to be much simpler than I expected, in a nutshell:

• @netmilk is this @debug tag used for anything at all in production or test environment?
• If the tag is dispensable (as I think it is) then the features failures were not caused by a change of behaviour in the Aruba API, but actually by a the addition of a new feature which involves a @debug tag and has nothing to do with our test suite - in other terms it's just a name clash, and the name that is clashing is not even something we're actually using.

The long story:

I spent some time wandering around the Aruba docs, and wondering what exactly the @debug tag was doing, without much success. I then decided to try updating Aruba by steps, just to get an idea of how bad things were, and it failed straight away at v0.7.0 (instead of v0.9.x as I expected).

The v0.7.0 release of Aruba introduced a lot of refactoring and features (more than I could fit in my head). That's when I went for git bisect (changelog).

Our test suite started failing when Aruba added support for a feature that uses an @debug tag. I quickly checked that our @debug tag was actually not doing anything visible when we were using Aruba v0.6.2, but started displaying the Dredd hooks output when moving to Aruba v0.7.0 - and causing the test to fail with messages in the lines of:

[...]

And the output should contain:
      """
      Yay! Failed in ruby!
      """
      expected "This is the debug launcher on STDOUT. If this output is unexpected, please check your setup.This is the debug launcher on STDERR. If this output is unexpected, please check your setup." to string includes: "Yay! Failed in ruby!" (RSpec::Expectations::ExpectationNotMetError)

What I understand, is that @debug in our features was used when writing them (I mean locally), but had no particular meaning for Aruba at that point. (Aruba did not support anything related to that tag before v0.7.0 and Cucumber itself doesn't seem to assign it any particular meaning either.) It was just a tag.

Somehow it stayed in place, and did not cause any trouble (nor did anything at all - unless I'm missing something) until Aruba just decided to use the exact same tag name for a new feature of theirs.

Removing the tag causes no harm to our test suite when using Aruba v0.6.2 and allows seamless updates up to v0.14.2 (the latest release of Aruba to date).

I'm going to create a PR for this, and if the timeline/explanation makes sense to you @honzajavorek, I think we can go forward with the update! The spike is here for easier review : )

[honzajavorek]

I don't know exactly what's the history behind the @debug tag (@netmilk's input welcome), but when I was previously replying with a link to the repo of the template, I noticed this in the list of issues: apiaryio/dredd-hooks-template#2

[gonzalo-bulnes]

I'm not sure either, and I'm a bit confuse about both using @announce (or @debug for that purpose) in the "production test suites" (if that makes any sense? - I mean: on a permanent basis, vs. for debug purpose).

First of all, the @announce tag is not necessary to test the content of the output - see how similar the @annouce feature and the "output" feature are.

Besides, while I confirm that @announce tag does work fine in our test suite (as @w-vi was suggesting), I think it is much more verbose than we need on a daily basis and I'm not sure that level of detail would really help anyone new to the hooks to understand them better. (I think the features themselves are pretty clear.)

With that in mind, unless I am missing a benefit of making these details visible, I would not replace @debug by @announce in our case, I would just remove @debug.

If debug is needed, however, I would definitely go with @announce - and I'll mention it in the corresponding issue : )

[honzajavorek]

I think the only missing piece is then to ask @netmilk what was the original purpose of @debug in this case. Just to be sure. Then I'd say it's good to go :)

[netmilk]

@gonzalo-bulnes @honzajavorek the @debug tag is an Aruba thing and can be removed from the features here. I recall I left it there undocumented because I had hard time to find it initially.

[gonzalo-bulnes]

Hi @honzajavorek! Just waiting for you green light : )

[honzajavorek]

I approve! Thanks!

[gonzalo-bulnes]

@ddelnano For info, we're moving to dredd-hooks-template : )

[ddelnano]

@gonzalo-bulnes just to be clear all we need to do is remove the @debug?? I didn't know what to expect for this change but I was expecting it to be a little more involved than that.

[gonzalo-bulnes]

Yes @ddelnano!

I was also expecting a lot more than that -I thought that how Aruba handles output had changed over time- but it happens that removing the @debug tags is all that's needed to be able to use newer versions of Aruba (see this previous comment for a more detailed explanation).

Notice that replacing @debug by @announce would also be fine, but the features are actually more readable without that extra output. (@announce is really good for debugging purpose however!)

It ended up being quick and easy : )

[honzajavorek]

This is a tracking issue for porting the changes to each of the hook repositories: apiaryio/dredd-hooks-template#16