teamcity: improve UX of diagnosing package timeouts

[tbg]

This is what it takes today:

diagnosing-package-timeout-smaller.mov

I can't imagine many people can do this sequence of clicks. I've had to do it quite a few times over the last couple of days. It keeps coming up. How can we improve the UX around package timeouts? Problems I see:

• test output doesn't indicate when the test has been running for a long time -> maybe we can put something into log.Scope or leaktest to call t.Logf("test has been running for X minutes") for each minute test has been running. This helps distinguish two kinds of problems (test timeout too short for how long the tests in the pkg shard take in aggregate vs single test gets stuck/is verrrry slow). Not distinguishing them can lead to filing issues against tests that are not to blame.
• finding the failing shard requires knowing that you need to search the build log for FAIL:, can we somehow make the shard available in each test output? If it's in the env we could just t.Logf("shard: %d", ...) from leaktest or log.Scope to capture most tests.

Jira issue: CRDB-29997

[jlinder]

This may improve with EngFlow. We will re-evaulate once we've made the switch.