logictest: improve observability of TeamCity timeouts

[mgartner]

It's possible for logic tests to run longer than the time allotted for them in TC, in which case TC abruptly kills the tests and prints a timeout message to the log:

-- Test timed out at 2022-03-17 20:31:33 UTC --

Example Log

From the logs, it's unclear which test(s) were running when the timeout occurred. We should ensure that go test timeouts first, so we get an informative stack trace that will help track down the issue.

Jira issue: CRDB-14009
Epic CRDB-8349

[mgartner]

Until this is fixed, you can use awk and a database to try to find the running tests at the time of the timeout:

cat test.log | awk '/RUN|done/ { if ($3 == "done:") { n = split($4, a, "/"); split($7, c, ":"); print "DONE" "," c[1] "/" a[n] } else { split($3, b, "/"); print $2 "," b[2] "/" b[3] } }' > running_tests.csv

Import running_tests.csv into a table like:

CREATE TABLE running_tests (status TEXT, name TEXT)

And run the query to find tests that started but did not finish:

SELECT name
FROM running_tests t
WHERE status = 'RUN' AND NOT EXISTS (
  SELECT * FROM running_tests s
  WHERE status = 'DONE' AND t.name = s.name
)

[tbg]

I mislead you a bit - in the above failure, bazel actually timed out the test. So maybe one ask is for bazel to send a signal before killing, and then the signal can dump stacks and print auxiliary information.

[ajwerner]

It'd be amazing to get stack traces.