outputFile dropped in v0.3.4 - related to Jenkins CI integration

[erkobridee]

Reading the Karma Runner v0.13 doc to integrate with Jenkins CI, there indicates to use outputFile

But in the last version (0.3.4 index.js) I don't found this support in the code, until 0.3.3 in the index.js was loaded in line 10 and used inside writeXmlForBrowser function

for now to generate report to Jenkins CI I'm using v0.2.2 and Jenkins CI config from Karma Runner v0.12

...
    // test results reporter to use
    // available reporters: https://npmjs.org/browse/keyword/karma-reporter
    reporters: ['junit', 'coverage'],

    // https://karma-runner.github.io/0.12/plus/jenkins.html
    // https://github.com/karma-runner/karma-junit-reporter
    junitReporter: {
      outputFile: 'tests_out/junit/test-results.xml'
    },

    // https://github.com/karma-runner/karma-coverage
    // https://github.com/yahoo/istanbul

    // optionally, configure the reporter
    coverageReporter: {
      type : 'lcovonly', // produces an lcov.info file
      dir : 'tests_out/coverage/',
      subdir: '.'
    },
...

[tonyfinn]

Yeah, the support for this seems to have been removed in 0.3.4: https://github.com/karma-runner/karma-junit-reporter/pull/36/files#diff-168726dbe96b3ce427e7fedce31bb0bcR40

(Also, even if you were only using outputDir, the format of the output file name changed, which caused us a surprise this morning)

It is still documented though, so should really either add it back or add a note that to the changelog that it is no longer supported.

[mpellerin42]

+1

[evandavis]

+1

[leigh-mckenzie-pi]

+1

[dignifiedquire]

Yes this was an oversight when I merged #36. This option should be added back in for sure. Very sorry about tha

[dignifiedquire]

Please checkout out #59 if that fixes the issue.

[erkobridee]

hi @dignifiedquire thanks

reading the new code, will fix partially... because of lines 44 and 46. Why we need output the browser name? 😃 (I use this config for CI flow, that will be executed by Jenkins CI)

ps.: in v0.2.2 we can define all output path, so if I define the config:

...
    // https://karma-runner.github.io/0.12/plus/jenkins.html
    // https://github.com/karma-runner/karma-junit-reporter/tree/v0.2.2
    junitReporter: {
      outputFile: 'tests_out/junit/test-results.xml'
    },
...

this will generate the exactly output in the config: tests_out/junit/test-results.xml

but with the proposed change if I config:

...
    // https://karma-runner.github.io/0.13/plus/jenkins.html
    // https://github.com/karma-runner/karma-junit-reporter/tree/fix-output-file
    junitReporter: {
      outputDir: 'tests_out/junit',
      outputFile: 'test-results.xml'
    },
...

and for example choose the PhantomJS browser to run tests, this will generate: tests_out/junit/PHANTOMJS_{version}/test-results.xml

[dignifiedquire]

@erkobridee the issue is that when there are multiple browsers we need a way to distinguish different test reports, and your config file does not know anything about that. So we need to either add the browser name as a directory or to the file name.

[erkobridee]

@dignifiedquire is it possible in somehow have one config to force this kind of output? (like we have in v0.2.2)

Thanks

[evandavis]

@erkobridee agreed; it seems like anything after v0.2.2 breaks Jenkins integration because it won't know where to expect the file. I wonder if we need to fork junit-reporter into jenkins-reporter to restore this. (Actually I think someone has already done that...)

[erkobridee]

@evandavis @dignifiedquire

I'm thinking... we could do like in v0.2.2 where, if we define outputFile this will define all output path and if we define only the outputDir this should output report per browser

well, that's my suggestion o/

What do you think? 😄

[simonvanderveldt]

@erkobridee That makes sense (just ran into this issue as well)
Does that mean that when setting outputFile different browser results will all be present in that single file?

[evandavis]

@simonvanderveldt I think the problem is trying to satisfy two different use cases. On one hand, a jUnit style reporter should be able to generate reports for each browser. However, for Jenkins integration, we (likely) just need the results of a Phantom (or other headless) test where we expect the results to always be in the same place.

[erkobridee]

@simonvanderveldt like @evandavis said, when we thinking and integrating with Jenkins CI we expect only 1 file to be read by Jenkins CI (in CI server normally we have one linux serve without UI and we put tests with karma to run only in PhantomJS, so we are running Unit Tests I don't expect any weird behavior to execute javascript among each browser) 😄

[simonvanderveldt]

Agreed.

Also @erkobridee the help from the Publish JUnit test result report says

To use this feature, first set up your build to run tests, then specify the path to JUnit XML files in the Ant glob syntax, such as */build/test-reports/.xml. Be sure not to include any non-report files into this pattern. You can specify multiple patterns of files separated by commas.

So it should match any file matching that pattern, not only test-results.xml

But I guess you mean that you want to point it to 1 specific file which you know for sure contains the correct (in your case PhantomJS) results?

[erkobridee]

@simonvanderveldt yeap, that was what we could do in v0.2.2 <o/

[simonvanderveldt]

@erkobridee
Wouldn't tests_out/junit/TESTS-PhantomJS*.xml in Jenkins together with outputDir: tests_out/junit in karma.conf.js work in that case?

[erkobridee]

@simonvanderveldt interesting tip... I don't test this yet... o/

[elicwhite]

It looks like there were commits that fixed this issue, what is the status of this? We look forward to a patch release.

[dignifiedquire]

I can understand that you need a predefined location, but the reporter will not know this, and if there are multiple browser reports written and we do not distinguish them they will overwrite each other which is really not a desired behaviour.

One thing that would be possible is to not nest the folders, but instead allow a template string, so you could write:

outputFile: 'results-${browser}.xml'
// or if you want to risk overwriting
outputFile: 'results.xml'

But please first explore if the tip from @simonvanderveldt using a glob in Jenkins solves the issue.

[elicwhite]

suite is option flag as well, perhaps the xml file should be prefixed with suite if suite is specified.

[erkobridee]

@simonvanderveldt I test you suggestion and works fine 😄

well the actual "big" issue today is the Jenkins CI integration doc on Karma Runner v0.13 page is outdated...

so I write one little project to make some tests... and doc how to config Jenkins CI correctly

• [GitHub] erkobridee / jenkins-ci-and-karma-junit-reporter - Example to execute Karma Runner through Jenkinks CI and load xml report file
  • releases
    • v0.1.0 - karma-junit-reporter v0.2.2
      • on post-build action to load test report xmls : tests_out/junit/*.xml
    • v0.2.0 - karma-junit-reporter v0.3.4
      • on post-build action to load test report xmls : tests_out/junit/*.xml
    • v0.3.0-alpha - karma-junit-reporter commit b4583b5 related to #59
      • on post-build action to load test report xmls : tests_out/junit/**/*.xml

one weird thing I saw...

loading xml generate from v0.2.2 show only one entry

and load xml generate from v0.3.4 and the commit from #59 generate 2 entries

with copy: @dignifiedquire

[aindlq]

@simonvanderveldt @dignifiedquire
I found the cause of duplication, see my comments for b4583b5