Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Collect test run information on CircleCI #169

Merged
merged 1 commit into from
Aug 10, 2018
Merged

Collect test run information on CircleCI #169

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 10 additions & 6 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,25 +45,29 @@ jobs:
# and everything but firefox to allow to compare them (and see that
# the firefox tests report "ran 0 tests" due to the fixture bug)

- run: lein test etaoin.xpath-test
- run: lein test2junit etaoin.xpath-test

- run:
name: lein test etaoin.api-test (firefox)
name: lein test2junit etaoin.api-test (firefox)
when: always
environment:
ETAOIN_TEST_DRIVERS: "[:firefox]"
command: lein test etaoin.api-test
command: lein test2junit etaoin.api-test

- run:
name: lein test etaoin.api-test (all but firefox)
name: lein test2junit etaoin.api-test (all but firefox)
when: always
environment:
ETAOIN_TEST_DRIVERS: "[:chrome :phantom]"
command: lein test etaoin.api-test
command: lein test2junit etaoin.api-test

- run:
when: always
command: lein test etaoin.api-test2
command: lein test2junit etaoin.api-test2

- store_test_results:
path: target/test2junit


# TODO
#
Expand Down
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,4 @@ node_modules
package-lock.json
TAGS
*.iml
build.xml
5 changes: 5 additions & 0 deletions project.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,11 @@
[org.clojure/tools.logging "0.3.1"]
[org.clojure/data.codec "0.1.0"]]

; When running the tests as `lein test2junit`, emit XUNIT test reports to enable CircleCI
; to collect statistics over time
:plugins [[test2junit "1.1.2"]]
:test2junit-output-dir "target/test2junit"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@marco-m It would be great to put the reports into a special CI folder to collect them afterwards. The folder's path is kept in the CIRCLE_ARTIFACTS env variable. So that value passed to the :test2junit-output-dir parameter would be:

(or (System/getenv "CIRCLE_ARTIFACTS")
    (System/getenv "ETAOIN_ARTIFACTS")
    "target/test2junit")

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello Ivan, I had something very similar to what you are proposing on my first attempt. Then I realized that the documentation mentioning CIRCLE_TEST_REPORTS (see https://github.com/kimh/circleci-build-recipies/tree/clojure-test-metadata-with-test2junit) is outdated, it works with CircleCI 1.0, while we are using 2.0 (and 1.0 will be unsupported end of Aug 2018).

So let me see if I understand correctly: with the PR as-is you are already getting test details as in the 4 links in the first comment of this PR. What you would like is to store the XUNIT test output as an artifact, so that you can consult it offline ? I know that CircleCI supports this, but I wanted to be sure this is what you want.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@marco-m well, I just want to achieve the following: if the tests are run on CI, the reports should be stored in a special CI's folder that can be referenced with a special env variable. The profit of it is CI stores those artefacts and you may download them any time you want.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@igrishaev The CIRCLE_ARTIFACTS env variable is available only for CircleCI 1.0. With 2.0, they don't do any automatic handling of artifacts (see https://circleci.com/docs/2.0/artifacts/ and https://circleci.com/docs/2.0/env-vars/, variable CIRCLE_ARTIFACTS is NOT mentioned). It is possible, but it requires more code.

Personally I don't see the added value, since if one month from now I am curious about the test results of one month ago, well, I can go to the CircleCI web page and see all the test results since the project has been added to CircleCI, no need to download any XUNIT file.

To me, given my understanding, this feature request is outside the scope of this PR :-)


:autodoc {:name "Etaoin"
:page-title "Etaoin API Documentation"
:description "Pure Clojure Webdriver protocol implementation."
Expand Down