Various improvements and features for future development

[LukasKalbertodt]

I think this library could use some improvements in various areas. Unfortunately, I don't actively use it currently. And since my time is short and I have other projects that I'm more interested in, I will not implement all those things now, but just post this issue with a random list of improvements. Note that the library is still totally usable, it just doesn't have the perfect API (IMO) and does not support all of the newest libtest CLI flags/features.

---

Get good integration tests first!

• --include-ignored
• print total time rust-lang/rust@470c059
• get rid of rayon (I think it's overkill and makes the dependency chain longer to compile)
• catch panics and report as failures
• --logfile PATH Write logs to the specified file
• when single thread, run on main thread
• Ignore message rust-lang/rust@bb3b557
• --shuffle ordering rust-lang/rust@fa23d4f
  • --shuffle-seed
  • Alphabetical order otherwise?
  • RUST_TEST_SHUFFLE
• RUST_TEST_THREADS
• Run tests in new process
• multiple filters

Unstable

• --format json/junit
• --ensure-time
• --exclude-should-panic Excludes tests marked as should_panic (UNSTABLE)
• --report-time
• --force-run-in-process Forces tests to run in-process when panic=abort

[jbabyhacker]

Thanks for making this crate, I like it a lot. Since you are making a list of future improvements, I have a suggestion. For situations with a lot of tests, it would be nice to display a progress bar. This can be achieved using the indicatif crate example.

[LukasKalbertodt]

I surprisingly found some time and motivation to work on this again, so some of these boxes are now ticked. I have currently no plans to implement any of the other mentioned things. (PRs regarding those are still welcome)

[LukasKalbertodt]

@jbabyhacker Thanks for the suggestion, but libtest-mimic tries to be very close to the official test harness. And since that doesn't have a progress bar, this library won't have one either.

[jedbrown]

Hi, thanks for creating this. "Run tests in new process" sounds like what I'm looking for to unit test rsmpi, but instead of directly calling the test executable (target/debug/foo_test-83955d27b7ee0d5b for example), I need to splice in a command like mpiexec -n 4 such that the harness executing the sub-process behaves like:

$ mpiexec -n 4 target/debug/foo_test-83955d27b7ee0d5b parallel_test --nocapture --exact --noharness

where --noharness is a way to remove the printing that normally surrounds the test itself.

This actually runs four processes that will communicate amongst themselves using MPI. The test passes if that parallel job produces stdout/stderr and exit code as specified in the test logic (the function I'll provide to Trial::test()). This will pair with using nextest's threads-required to avoid oversubscription.

Does this seem like a reasonable way to proceed? Do you have implementation tips for adding this feature?

[untitaker]

I suggest to provide an option to print panic stacktraces. Right now it seems that RUST_BACKTRACE=1 is not effective in finding the codeline from where a panic occured.