Collect test run information on CircleCI #169
Conversation
When running the tests on CicleCI, emit XUNIT test reports so that CircleCI call collect and show test statistics. Tests statistics are shown in two places: - each single build gives an overview of the failing tests - the build-insights page at https://circleci.com/build-insights/gh/igrishaev/etaoin/master
|
Note that it says "All checks have failed" because there are still the broken tests of before ... |
| ; When running the tests as `lein test2junit`, emit XUNIT test reports to enable CircleCI | ||
| ; to collect statistics over time | ||
| :plugins [[test2junit "1.1.2"]] | ||
| :test2junit-output-dir "target/test2junit" |
There was a problem hiding this comment.
@marco-m It would be great to put the reports into a special CI folder to collect them afterwards. The folder's path is kept in the CIRCLE_ARTIFACTS env variable. So that value passed to the :test2junit-output-dir parameter would be:
(or (System/getenv "CIRCLE_ARTIFACTS")
(System/getenv "ETAOIN_ARTIFACTS")
"target/test2junit")
There was a problem hiding this comment.
Hello Ivan, I had something very similar to what you are proposing on my first attempt. Then I realized that the documentation mentioning CIRCLE_TEST_REPORTS (see https://github.com/kimh/circleci-build-recipies/tree/clojure-test-metadata-with-test2junit) is outdated, it works with CircleCI 1.0, while we are using 2.0 (and 1.0 will be unsupported end of Aug 2018).
So let me see if I understand correctly: with the PR as-is you are already getting test details as in the 4 links in the first comment of this PR. What you would like is to store the XUNIT test output as an artifact, so that you can consult it offline ? I know that CircleCI supports this, but I wanted to be sure this is what you want.
There was a problem hiding this comment.
@marco-m well, I just want to achieve the following: if the tests are run on CI, the reports should be stored in a special CI's folder that can be referenced with a special env variable. The profit of it is CI stores those artefacts and you may download them any time you want.
There was a problem hiding this comment.
@igrishaev The CIRCLE_ARTIFACTS env variable is available only for CircleCI 1.0. With 2.0, they don't do any automatic handling of artifacts (see https://circleci.com/docs/2.0/artifacts/ and https://circleci.com/docs/2.0/env-vars/, variable CIRCLE_ARTIFACTS is NOT mentioned). It is possible, but it requires more code.
Personally I don't see the added value, since if one month from now I am curious about the test results of one month ago, well, I can go to the CircleCI web page and see all the test results since the project has been added to CircleCI, no need to download any XUNIT file.
To me, given my understanding, this feature request is outside the scope of this PR :-)
When running the tests on CicleCI, emit XUNIT test reports so
that CircleCI call collect and show test statistics.
Tests statistics are shown in two places: