Keep an output directory when a test fails

[lexspoon]

I would like a way to save diagnostic information to the filesystem after a failed test.

There are many ways it could work, but as a concrete strawman: define an environment variable TEST_DIAGNOSTICS_DIR which points to a directory where a test can save additional diagnostic info. This directory tree will then be brought back to somewhere the user can use it, maybe under bazel-out. As a stretch goal, it would be helpful if scala_test can also print out the final directory path if it's non-empty, so that the developer knows to go inspect the directory and see what they see. I can potentially add a README from my test framework to explain to the developer what is going on and give them breadcrumbs.

The immediate reason I want this is because I have some test cases that create DuckDB databases. If the test fails, I would like to tinker with the database manually, so I need a way to export the database out of the failing test and get it back to the developer. I have modified my ScalaTest tests to do the export, but I am hitting a wall on actually getting the snapshot back to the developer. Bazel won't let me save it in $HOME due to sandboxing, and if I save it to a temp directory or to the current working directory, it appears to get cleaned up before I can access it.

I believe I could get something to work using the debug-sandbox option, but that's pretty roundabout and would be much slower for the developer than if this was a built-in behavior of scala_test.

I think this is a general issue for test utilities and not limited to this DuckDB scenario. Here are some other scenarios:

• If a test runs web browser tests, the test may want to save a copy of a screenshot of the browser while in the failed state.
• Any golden files tests may want to save the incorrect output files if they do not match the golden files. The developer can then diff them and can potentially promote them to be the new golden files.
• Any file format implementation, e.g. the implementation of a database commit log, would benefit from keeping a snapshot of any corrupting or non-parsing file so that it can be manually debugged.

As some prior art, I developed this testing tool, and it has a similar feature for its own databases. With that tool, if a test passes, there is no extra output created. If a test fails, then a database snapshot is saved so that the developer can experiment with it. It worked really well in practice. The tool already knows what is going on and is in a really good position to help the developer out.