ci: Use nextest as test runner instead of cargo test

[Joining7943]

nextest is an alternative to the standard cargo test test runner. Some advantages over the standard test runner which I found could be of interest for us:

• It's faster. The total execution time of cargo test --features unix on my linux pc was 1m 35s. The total execution time of cargo nextest run --features unix --no-fail-fast was 1m.
• The output of nextest contains the execution time of each test. This would help to identify bottlenecks and figuring out the correct timeout for UCommand.
• Tests can be configured to be retried if they fail, either for all tests or individually. This would help identify intermittently failing tests. Also, if a test was failing just intermittently, it is marked as flaky but doesn't cause the whole test run to exit with failure.
• Individual tests or subsets can be configured to run not in parallel with other tests. I thought of utils/tests which use the time crate and use methods which aren't safe to be run in parallel and cause tests to fail intermittently.

It's not critical but I also found the output of nextest a little bit clearer than that of cargo test.

Here's also the link to the official docs.

A limitation of nextest is, that it's not usable on all cross build targets. But, as far as I could see there's a pre-built binary for all systems we currently run tests on.

[tertsdiepraam]

I looked into this before and sometimes use it locally. I think the main performance improvement is better parallelism of tests. So, does it still have a performance improvement when parallelism is limited (like in the CI)? Another thing to try is whether the output and logs readable in GH actions.

[Joining7943]

So, does it still have a performance improvement when parallelism is limited (like in the CI)?

I think, that's something we just would have to try. From the docs, I also would guess that most of the performance improvement comes from parallelism. But even if it's not that significant than on my home pc, 10s per full test run makes like 100s in total per ci run and with 1000s of ci runs, it's maybe still worth it in the end. However, I found the other features like showing each test execution time, having retries, tests configuration possibilites etc. more interesting. What's your impression here?

Another thing to try is whether the output and logs readable in GH actions.

What would cause a problem?

[tertsdiepraam]

What would cause a problem?

If I recall correctly, it shows the progress of multiple tests at the same time and overwrites lines in the output. I'm not sure how well the CI supports that.

[Joining7943]

Ok. They must have changed that. The output is changing fast but as far as I can tell it's one line after another. Per default, there's a sticky bottom line showing the current progress, which would cause problems in the ci. But, it can be switched off. A progress bar in the ci doesn't make to much sense anyways.

[tertsdiepraam]

Alright, should be good then! I might have misremembered.

[Joining7943]

No problem :) I'll give it a try then...