Publishing of Karma-generated JUnit report file does not include classname value

[adriaanthomas]

When running unit tests using Karma, if we publish the results using karma-junit-reporter, VSO does not display the value of the classname, which makes reading the results very hard.

Part of a unit test:

describe('given the contractAccountAuthorizationService', function () {
    describe('initialise', function() {
        it('should return a promise', function(done) {
            // some implementation...
        });
    });

    describe('whenInitialised', function() {
        it('should return a promise', function(done) {
            // some implementation...
        });
    });
});

(for simplicity placed in one file here, but we have it(...) functions with the same description between different spec files too)

Example JUnit report:

<?xml version="1.0"?>
<testsuites>
  <testsuite name="PhantomJS 1.9.8 (Windows 7 0.0.0)" package="" timestamp="2015-08-18T09:26:02" id="0" hostname="C121441" tests="59" errors="0" failures="0" time="0.343">
    <properties>
      <property name="browser.fullName" value="Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.9.8 Safari/534.34"/>
    </properties>
    <testcase name="should return a promise" time="0.020" classname="PhantomJS 1.9.8 (Windows 7 0.0.0).given the contractAccountAuthorizationService initialise"/>
    <testcase name="should return a promise" time="0.021" classname="PhantomJS 1.9.8 (Windows 7 0.0.0).given the contractAccountAuthorizationService whenInitialised"/>
  </testsuite>
</testsuites>

We then publish the test results using the Publish Test Results task.

In VSO, the results are displayed as:

... but the full value of the classname attribute is not displayed anywhere, which makes it very hard to find the context of a test result.

FWIW we also tried exporting results using karma-trx-reporter but that causes a failure in the Publish-TestResults Cmdlet: ##[error]Object reference not set to an instance of an object.

So, how can we provide better unit testing reports for Karma tests?

[adriaanthomas]

FYI for now we have worked around this a bit by using karma-spec-reporter to output all results to console. The relevant parts in our karma.conf.js:

config.set({
  // don't use colors as they don't work in VSO console output
  colors: false,

  reporters: ['spec'],

  specReporter: {
    // Unicode characters don't work in VSO console output so fall back to plain ascii
    prefixes: {
      success: '[v] ',
      failure: '[X] ',
      skipped: '[-] '
    }
  }
});

This means we can see a detailed enough test report when clicking on the relevant Gulp task in the VSO build results, and we don't really look at the test results anymore. There is no timing output in the console this way but we can live with that.

It's a big shame that colors don't work in the Gulp task output, but the result is still readably enough. Also, an email notifying a build failure just contains the stderr output, which is Unexpected exit code 1 returned from tool gulp.cmd, which means that team members have to click on the link to VSO to see what actually went wrong.

[madhurig]

The classname from the JUnit testcase result is set to the "Automated test storage" field on VSO (https://github.com/Microsoft/vso-agent/blob/master/src/agent/testresultreader.ts line 222) but that is not displayed in the VSO Test results columns. The columns are not customizable currently but you can filter on "Automated test storage" field. If we include the classname in the test case title field, would that be a good option for you? Are you using the windows build agent or the build agent for Linux and OSX?

Regarding the 2 issues with publishing trx format results to VSO, I will forward those issues to the owner.

[adriaanthomas]

For this use case including the classname in the test case name would work.

I also checked a Java-JUnit project (I created a build definition for spring-petclinic). If I use the default option on the Maven task Publish to VSO/TFS, the test report contains a test result per test suite, and JUnit has created a test suite for every class:

When I use the tooltip, I can just about read what class this result applies to.

However when I disable the default test result publishing, add a separate Publish Test Results task and tick Merge Test Results, I am left with the same problem as indicated for my Jasmine test above:

But if the field already exists in the VSO data model, I think it would be far better to (give the option to) show it in a column; this would also make filtering more usable (because we would still have both fields there?).

We did consider writing a post-test task that transforms the XML classname to prefix in the name attribute, but decided it was too much work for all applications.

We use the Windows build agent.

[adriaanthomas]

Looks like this might get solved by karma-runner/karma-junit-reporter#62, if they include the full context in the test name.

[adriaanthomas]

I can confirm that karma-runner/karma-junit-reporter#85 fixes this issue. I'm closing this.