TeamCity + roachtest: When a node disappears, many log files are not present in the build artifacts

[jlinder]

In this example, node 6 logs are missing along with all the *.dmesg.txt, *.journalctl.txt and debug.zip files.

Expected files looks more like this:

Figure out why gathering the rest of the logs are skipped when a node disappears and its logs are inaccessible and fix it.

Fixing issue https://github.com/cockroachdb/dev-inf/issues/73 may help.

Issue #48414 led to this issue being created.

Epic CRDB-10428

Jira issue: CRDB-4323

[blathers-crl]

Hi @jlinder, I've guessed the C-ategory of your issue and suitably labeled it. Please re-label if inaccurate.

While you're here, please consider adding an A- label to help keep our repository tidy.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[jlinder]

Bob had the idea to start a roachtest, ssh to one of the roachprod nodes, and run the OOM producing sled he created for issue https://github.com/cockroachdb/dev-inf/issues/79 to see if the logs don't get fully collected like noted in this issue.

[github-actions]

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!