`ci`: Run tests with nextest #4740

Joining7943 · 2023-04-13T18:07:15Z

This pr installs cargo-nextest as test runner in ci workflows and jobs previously running tests with cargo test.

nextest comes with a better execution model making it less error prone, so that test binaries exiting with error still report the test which caused the error. It also comes with other features like

test retries and marking intermittently failing tests as flaky
showing per test execution times and particularly slow tests
... see the docs

Note there's no performance improvement in the ci because of the lack of a multi-core test environment. Together with the installation step of nextest the test jobs even take around 5-10 seconds longer.

In case of the android workflow, the nextest binary has to be built from source because none of the pre-built binaries works on x86 android targets. The binary is stored in the emulator image, so that step isn't needed often. The caching strategy for the avd cache changed, so that a successfully built image is cached before running the build and test step or else the upload depends on the build and tests running successfully.

I added a nextest target to the GNUMakefile instead of replacing the existing test target.

Before merging this pr it's needed to clear cached emulator images/snapshots from the github caches or else the android workflow fails due the missing nextest binary in the currently restored images. We could also change the cache key or maybe there's another possibility?

Closes #4737

sylvestre · 2023-04-14T19:04:58Z

test retries and marking intermittently failing tests as flaky

I love this one!
just for that, moving is clearly super interesting :)

Maybe remove d41591f in a new commit

.config/nextest.toml

sylvestre · 2023-04-14T19:07:20Z

please update
https://github.com/uutils/coreutils/blob/main/CONTRIBUTING.md#testing

Joining7943 · 2023-04-14T20:27:33Z

Maybe remove d41591f in a new commit

The change doesn't affect the gnu tests, only the rust tests. It's a replacement for cargo test.

please update
https://github.com/uutils/coreutils/blob/main/CONTRIBUTING.md#testing

Ok, no problem

Joining7943 · 2023-04-15T09:11:00Z

How to proceed with the avd cache in the github ci? New cache key? I'm asking because this image has a size of more than 2GB. Having two images around is unnecessary and it would take half of the space github makes available per default.

Joining7943 · 2023-04-15T13:54:08Z

This pr is ready for review. The problem with the emulator image seems to have solved itself and the current image in the cache is the one with the nextest dependency in it. Note the date fuzzer went nuts and produced around 250000 lines of error messages, but this also happened in other prs, so I'm pretty sure it has nothing to do with this one. The android workflow fails because the test step fails with 3 failing tests and 1 flaky test.

Joining7943 · 2023-04-16T13:23:21Z

Update: I checked the situation around the avd cache in the github cache again, and I think the safest way to make the caching in the android workflow with the nextest binary work is to change the cache key.

sylvestre · 2023-04-18T09:03:14Z

is it ready for review?

Joining7943 · 2023-04-18T13:35:34Z

is it ready for review?

Sure :)

sylvestre · 2023-04-19T06:52:12Z

.config/nextest.toml

@@ -0,0 +1,6 @@
+[profile.ci]
+retries = 3


actually, i prefer 2, sorry. if a test fails twice, in a row, it is a big deal

No problem.

sylvestre · 2023-04-19T06:52:31Z

.config/nextest.toml

@@ -0,0 +1,6 @@
+[profile.ci]
+retries = 3
+status-level = "all"


could you please document these options?

Would placing a link to the official docs above [profile.ci] do the job, too?

sylvestre · 2023-04-19T06:54:22Z

@tertsdiepraam @cakebaker I think we should take it, WDYT ?

cakebaker · 2023-04-19T08:23:19Z

@sylvestre +1 to give it a try. In the worst case we can easily return to cargo test.

tertsdiepraam · 2023-04-19T08:58:49Z

.github/workflows/CICD.yml

      env:
        RUST_BACKTRACE: "1"
+        CARGO_TERM_COLOR: "always"


@sylvestre you previously objected to this change in #4165. Have you changed your mind?

In any case we can specify it just once, like I did in that PR.

yeah, i missed that change
could you please remove it?

In any case we can specify it just once, like I did in that PR.

I activated the colors only for tests, for now. Sorry, I missed the discussion in #4165. Having colors in the ci would be really great, though. @sylvestre Are you looking at the raw logs just locally? Removing the color codes would be a oneliner sed -Ei 's/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g' PATTERN.

i am looking at logs in github action too

tertsdiepraam · 2023-04-19T09:03:37Z

Apart from the comment above I agree that this can be merged.

sylvestre · 2023-04-19T09:05:14Z

.github/workflows/CICD.yml

      env:
        RUST_BACKTRACE: "1"
+        CARGO_TERM_COLOR: "always"


Suggested change

CARGO_TERM_COLOR: "always"

github-actions · 2023-04-19T22:35:24Z

GNU testsuite comparison:

Congrats! The gnu test tests/misc/timeout is no longer failing!
Congrats! The gnu test tests/tail-2/inotify-dir-recreate is no longer failing!

sylvestre · 2023-04-20T07:50:38Z

seems that more people like logs with color than I was expecting. let's do take it then :)

Joining7943 force-pushed the ci-use-nextest-as-test-runner branch 21 times, most recently from 6adc112 to a97f203 Compare April 14, 2023 17:33

Joining7943 marked this pull request as ready for review April 14, 2023 18:10

sylvestre reviewed Apr 14, 2023

View reviewed changes

.config/nextest.toml Show resolved Hide resolved

Joining7943 force-pushed the ci-use-nextest-as-test-runner branch from 484496b to c02099f Compare April 15, 2023 08:47

Joining7943 force-pushed the ci-use-nextest-as-test-runner branch from c02099f to 370d4e9 Compare April 15, 2023 10:13

Joining7943 marked this pull request as draft April 15, 2023 11:25

Joining7943 marked this pull request as ready for review April 15, 2023 13:41

Joining7943 force-pushed the ci-use-nextest-as-test-runner branch from c22c153 to 3f61f33 Compare April 16, 2023 13:17

uutils deleted a comment from github-actions bot Apr 17, 2023

sylvestre force-pushed the ci-use-nextest-as-test-runner branch from 3f61f33 to dc775d9 Compare April 18, 2023 09:06

uutils deleted a comment from github-actions bot Apr 18, 2023

uutils deleted a comment from github-actions bot Apr 19, 2023

sylvestre requested review from cakebaker and tertsdiepraam April 19, 2023 06:50

sylvestre reviewed Apr 19, 2023

View reviewed changes

tertsdiepraam reviewed Apr 19, 2023

View reviewed changes

sylvestre reviewed Apr 19, 2023

View reviewed changes

.github/workflows/CICD.yml

env:

RUST_BACKTRACE: "1"

CARGO_TERM_COLOR: "always"

Copy link

Contributor

sylvestre Apr 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

CARGO_TERM_COLOR: "always"

Joining7943 mentioned this pull request Apr 19, 2023

ci/android: speed up build and test #4745

Merged

Joining7943 added 4 commits April 19, 2023 22:06

ci: Use nextest as test runner instead of cargo test

78ce521

android-commands.sh: Reformat file with shftm

037aaf0

Update CONTRIBUTING.md: Add nextest usage examples

18d5e9a

ci/android: Use different caching strategy

3495b82

Joining7943 force-pushed the ci-use-nextest-as-test-runner branch from dc775d9 to 3495b82 Compare April 19, 2023 21:46

sylvestre merged commit 7445293 into uutils:main Apr 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`ci`: Run tests with nextest #4740

`ci`: Run tests with nextest #4740

Joining7943 commented Apr 13, 2023 •

edited

Loading

sylvestre commented Apr 14, 2023

sylvestre commented Apr 14, 2023

Joining7943 commented Apr 14, 2023

Joining7943 commented Apr 15, 2023

Joining7943 commented Apr 15, 2023

Joining7943 commented Apr 16, 2023

sylvestre commented Apr 18, 2023

Joining7943 commented Apr 18, 2023

sylvestre Apr 19, 2023

Joining7943 Apr 19, 2023

Joining7943 Apr 19, 2023

sylvestre Apr 19, 2023

Joining7943 Apr 19, 2023

sylvestre Apr 20, 2023

sylvestre commented Apr 19, 2023

cakebaker commented Apr 19, 2023

tertsdiepraam Apr 19, 2023 •

edited

Loading

sylvestre Apr 19, 2023

Joining7943 Apr 19, 2023

sylvestre Apr 20, 2023

tertsdiepraam commented Apr 19, 2023 •

edited

Loading

sylvestre Apr 19, 2023

github-actions bot commented Apr 19, 2023

sylvestre commented Apr 20, 2023

ci: Run tests with nextest #4740

ci: Run tests with nextest #4740

Conversation

Joining7943 commented Apr 13, 2023 • edited Loading

sylvestre commented Apr 14, 2023

sylvestre commented Apr 14, 2023

Joining7943 commented Apr 14, 2023

Joining7943 commented Apr 15, 2023

Joining7943 commented Apr 15, 2023

Joining7943 commented Apr 16, 2023

sylvestre commented Apr 18, 2023

Joining7943 commented Apr 18, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sylvestre commented Apr 19, 2023

cakebaker commented Apr 19, 2023

tertsdiepraam Apr 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tertsdiepraam commented Apr 19, 2023 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Apr 19, 2023

sylvestre commented Apr 20, 2023

`ci`: Run tests with nextest #4740

`ci`: Run tests with nextest #4740

Joining7943 commented Apr 13, 2023 •

edited

Loading

tertsdiepraam Apr 19, 2023 •

edited

Loading

tertsdiepraam commented Apr 19, 2023 •

edited

Loading