Enhancing subtest support for TAP::Parser

[foursixnine]

Hi, I have some tests that produce the following output (along with other stuff), I'm planning to extend our ci in openQA to have a bit of better support for console based tests, however, while playing with Test::Harness, I'm struggling to figure out how subtests are being processed... I could find that there are some tests that are trying to parse at least subtests but nothing else.

I could not find any docs regarding Subtests in the TAP docs or related sites (perhaps I didn't look well enough?)

Any hints?

So long story short, I'm loading the file with code like this:

use TAP::Parser;
use feature 'say';
use Carp;

my $parser = TAP::Parser->new({tap => "t/source_tests/tap_format_example_complex.tap"});
confess "Failed " . $parser->parse_errors if $parser->parse_errors;

while ( my $result = $parser->next ) {
    say $result->type . ' - '.$result->as_string;
}

And upon running my simple test, I get:

test - not ok 24 - group filter
test - ok 25 - POST /api/v1/isos
test - ok 26 - 200 OK
test - not ok 27 - 10 new jobs created
test - ok 28 - GET /api/v1/jobs
test - not ok 29 - server_32 is only parent of client1_32
test - not ok 30 - server_32 is only parent of client2_32
test - not ok 31 - server_32 has no parents
test - ok 32 - kde is not created for 32bit machine
test - ok 33 - advanced_kde is not created for 32bit machine
test - not ok 34 - server_64 is only parent of client1_64
test - not ok 35 - server_64 is only parent of client2_64
test - not ok 36 - server_64 has no parents
test - ok 37 - no (unexpected) warnings (via END block)
unknown - Dubious, test returned 255 (wstat 65280, 0xff00)
unknown - Failed 9/37 subtests 
unknown - 
unknown - Test Summary Report
unknown - -------------------
unknown - t/api/02-iso.t (Wstat: 65280 Tests: 37 Failed: 9)
unknown -   Failed tests:  23-24, 27, 29-31, 34-36
unknown -   Non-zero exit status: 255
unknown -   Parse errors: No plan found in TAP output
unknown - Files=1, Tests=37,  9 wallclock secs ( 0.02 usr  0.01 sys +  5.05 cusr  0.18 csys =  5.26 CPU)
unknown - Result: FAIL

Also the following results not being shown is kinda strange... :

• unknown - Failed 9/37 subtests
• unknown - Result: FAIL

File:

t/api/02-iso.t .. 
ok 1 - GET /api/v1/jobs/99927
...
ok 21 - 200 OK
ok 22 - we have no gru download tasks to start with
# Subtest: Multi Machine job dependencies
    ok 1 - POST /api/v1/isos
    not ok 2 - 200 OK
    not ok 3 - only one job created due to group filter
    1..3
not ok 23 - Multi Machine job dependencies
# Subtest: group filter
    ok 1 - POST /api/v1/isos
    ok 2 - 200 OK
    ok 3 - no jobs created if group invalid
    ok 4 - POST /api/v1/isos
    ok 5 - 200 OK
    not ok 6 - only one job created due to group filter
    ok 7 - POST /api/v1/isos
    ok 8 - 200 OK
    not ok 9 - only one job created due to group filter (by ID)
    1..9
not ok 24 - group filter
...
ok 32 - kde is not created for 32bit machine
ok 33 - advanced_kde is not created for 32bit machine
not ok 34 - server_64 is only parent of client1_64
not ok 35 - server_64 is only parent of client2_64
not ok 36 - server_64 has no parents
ok 37 - no (unexpected) warnings (via END block)
Dubious, test returned 255 (wstat 65280, 0xff00)
Failed 9/37 subtests 

Test Summary Report
-------------------
t/api/02-iso.t (Wstat: 65280 Tests: 37 Failed: 9)
  Failed tests:  23-24, 27, 29-31, 34-36
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
Files=1, Tests=37,  9 wallclock secs ( 0.02 usr  0.01 sys +  5.05 cusr  0.18 csys =  5.26 CPU)
Result: FAIL

[Leont]

That looks like the output of a prove -v, which is not quite TAP. As far as I can tell the parse is fine .

As for subtests, they're not actually parsed by TAP::Harness currently (the grammar allows them to be ignored. This is unfortunate from a reporting point of view but not breaking the core functionality (because there's always a summarizing normal test).

[kiwiroy]

Ah, so this is why a passing local $TODO = ...; in a subtest is not reported.