Updates for --out=url option

[vincent-psarga]

Various updates discussed during Weekly meeting. The idea I to match more closely what was done in cucumber-jvm and also ease upload to a message service.

• Implement -H and -X curl options
• Use a random port to make tests more reliable
• Abandon chunked encoding (unless -H "Transfer-Encoding: chunked" is specified : we'll add that back if needed, but not for the moment)
• Follow redirects (307)
• Use PUT by default (simplifies S3)

[aslakhellesoy]

Don't keep all the messages in memory :-)

[vincent-psarga]

I'm really facing weird issues right now. When I try to push to a real-server (either a simple Sinatra server or an S3 upload) and I'm getting 40x errors.

I tried do debugging and came with that:

fake_content = Tempfile.new('justXs')
fake_content.write('X' * content.length)

Kernel.puts "Sending real content"
content.rewind
req = build_request(uri, method, headers.merge('Content-Length' => content.size.to_s))
req.body_stream = content
response = @http.request(req)
Kernel.puts "Got response: #{response.code}"

Kernel.puts "Sending fake content"
fake_content.rewind
req = build_request(uri, method, headers.merge('Content-Length' => content.size.to_s))
req.body_stream = fake_content
response = @http.request(req)
Kernel.puts "Got response: #{response.code}"

So this seems to work with fake content:

╰─$ bundle exec cucumber features/docs/getting_started.feature --format message --out="http://localhost:4567/"
WARNING: Nokogiri was built against LibXML version 2.9.10, but has dynamically loaded 2.9.4
Using the default profile...
........

2 scenarios (2 passed)
8 steps (8 passed)
0m1.221s
Sending real content
Got response: 400
Sending fake content
Got response: 200

Note that the tests are passing. I'm wondering if there's anything I missing making the upload failed with NDJson content but working with plain character repeated :/

@aslakhellesoy @mpkorstanje any idea by any chance ?

[vincent-psarga]

I've been able to pinpoint a bit more the messages that are causing issues on upload:

        if envelope.step_definition
          if envelope.step_definition.source_reference.uri.include?('aruba/cucumber/command.rb') ||
            envelope.step_definition.source_reference.uri.include?('aruba/cucumber/file.rb')
              envelope.step_definition.pattern.source = ''
          end
       end

Adding this to the message generator made it work (yes, obviously that ugly :D ). The weird thing is that the expressions used in those files do not seem to have that much "weird" content (encoding or other that could cause any issue).

Here's the list of the step that are removed:

^I run `([^`]*)`$
^I successfully run `(.*?)`(?: for up to ([\d.]+) seconds)?$
^I run the following (?:commands|script)(?: (?:with|in) `([^`]+)`)?:$
^I run `([^`]*)` interactively$
^I run `([^`]*)` in background$
^I type "([^"]*)"$
^I close the stdin stream$
^I pipe in (?:a|the) file(?: named)? "([^"]*)"$
^I (terminate|stop) the command (?:"([^"]*)"|(?:started last))$
^I stop the command(?: started last)? if (output|stdout|stderr) contains:$
^I wait for (?:output|stdout) to contain:$
^I wait for (?:output|stdout) to contain "([^"]*)"$
^the output should be (\d+) bytes long$
^(?:the )?(output|stderr|stdout) should( not)? contain( exactly)? "([^"]*)"$
^(?:the )?(output|stderr|stdout) from "([^"]*)" should( not)? contain( exactly)? "([^"]*)"$
^(?:the )?(output|stderr|stdout) should not contain( exactly)?:$
^(?:the )?(output|stderr|stdout) should contain( exactly)?:$
^(?:the )?(output|stderr|stdout) from "([^"]*)" should not contain( exactly)?:$
^(?:the )?(output|stderr|stdout) from "([^"]*)" should contain( exactly)?:$
^the output should( not)? match /([^/]*)/$
^the output should( not)? match %r<([^>]*)>$
^the output should( not)? match:$
^the exit status should( not)? be (\d+)$
^it should not (pass|fail) with "(.*?)"$
^it should (pass|fail) with "(.*?)"$
^it should not (pass|fail) with:$
^it should (pass|fail) with:$
^it should not (pass|fail) with exactly:$
^it should (pass|fail) with exactly:$
^it should not (pass|fail) (?:with regexp?|matching):$
^it should (pass|fail) (?:with regexp?|matching):$
^(?:the )?(output|stderr|stdout) should not contain anything$
^(?:the )?(output|stdout|stderr) should( not)? contain all of these lines:$
^the (?:default )?aruba io wait timeout is ([\d.]+) seconds?$
^the (?:default )?aruba exit timeout is ([\d.]+) seconds?$
^the (?:default )?aruba stop signal is "([^"]*)"$
^I wait ([\d.]+) seconds? for (?:a|the) command to start up$
^I send the signal "([^"]*)" to the command (?:"([^"]*)"|(?:started last))$
^I look for executables in "(.*)" within the current directory$
^I use (?:a|the) fixture(?: named)? "([^"]*)"$
^I copy (?:a|the) (?:file|directory)(?: (?:named|from))? "([^"]*)" to "([^"]*)"$
^I move (?:a|the) (?:file|directory)(?: (?:named|from))? "([^"]*)" to "([^"]*)"$
^(?:a|the|(?:an empty)) directory(?: named)? "([^"]*)"$
^(?:a|the) directory(?: named)? "([^"]*)" with mode "([^"]*)"$
^(?:a|the) file(?: named)? "([^"]*)" with:$
^(?:an|the) executable(?: named)? "([^"]*)" with:$
^(?:a|the) file(?: named)? "([^"]*)" with "([^"]*)"$
^(?:a|the) file(?: named)? "([^"]*)" with mode "([^"]*)" and with:$
^(?:a|the) (\d+) byte file(?: named)? "([^"]*)"$
^(?:an|the) empty file(?: named)? "([^"]*)"$
^(?:an|the) empty file(?: named)? "([^"]*)" with mode "([^"]*)"$
^I write to "([^"]*)" with:$
^I overwrite(?: (?:a|the) file(?: named)?)? "([^"]*)" with:$
^I append to "([^"]*)" with:$
^I append to "([^"]*)" with "([^"]*)"$
^I remove (?:a|the) (?:file|directory)(?: named)? "([^"]*)"( with full force)?$
^(?:a|the) (?:file|directory)(?: named)? "([^"]*)" does not exist$
^I cd to "([^"]*)"$
^the following files should (not )?exist:$
^(?:a|the) file(?: named)? "([^"]*)" should (not )?exist(?: anymore)?$
^(?:a|the) directory(?: named)? "([^"]*)" should (not )?exist(?: anymore)?$
^(?:a|the) file matching %r<(.*?)> should (not )?exist$
^(?:a|the) (\d+) byte file(?: named)? "([^"]*)" should (not )?exist$
^the following directories should (not )?exist:$
^(?:a|the) file(?: named)? "([^"]*)" should (not )?contain "([^"]*)"$
^(?:a|the) file(?: named)? "([^"]*)" should (not )?contain:$
^(?:a|the) file(?: named)? "([^"]*)" should (not )?contain exactly:$
^(?:a|the) file(?: named)? "([^"]*)" should (not )?match %r<([^>]*)>$
^(?:a|the) file(?: named)? "([^"]*)" should (not )?match /([^/]*)/$
^(?:a|the) file(?: named)? "([^"]*)" should (not )?be equal to file "([^"]*)"
^(?:a|the) (?:file|directory)(?: named)? "([^"]*)" should( not)? have permissions "([^"]*)"$

I don't have a clue exactly which ones* causes the issues and why :/

• one of the certainties I have is that there's multiple one causing issues at least ...

[mpkorstanje]

I guess you are implicitly using url encoding? But not encoding the content? Shouldn't the client handle that?

[vincent-psarga]

I guess you are implicitly using url encoding? But not encoding the content? Shouldn't the client handle that?

I tried to use url encoding on the content (when writing it to the Tempfile) but that still does not fix the issue with S3 upload :/