log: panics during server startup don't make it to the console

[andreimatei]

Panics that happen during server startup go into a black hole. They don't make it to the console even when running with --logtostderr. This has annoyed me several times now. The panic is printed in cockroach-stderr.log, though.
To repro, put a panic at the beginning of sql.Server.PreStart(), for example. I think it has something to do with this stderr capturing business.

Jira issue: CRDB-21389

[knz]

This is working more-or-less as intended.

The contract is this:

1. top priority is that we can retrieve these panics in debug zip for prod clusters. Therefore, they must be written to a file.
2. the go runtime is built such that it cannot write errors to both a file and stderr. (That "capturing business" exists only because the go runtime is being obtuse about this).
3. therefore, we have chosen to redirect them to a file to satisfy (1) at the expense of stderr.

You can work around this with --log='capture-stray-errors: {enable: false}' for your situation, but then debug zip won't see the panic any more.

There's a couple of things we could do about this:

• patch the go runtime (since we now can do this. It wasn't when we designed the log package)
• add some recovers here and there to redirect the panic object through the logging stack, but that would be just playing whack-a-mole, as long as *: ban the go keyword in non-test code #58164 is open.

[andreimatei]

You can work around this with --log='capture-stray-errors: {enable: false}'

Good to know.

add some recovers here and there to redirect the panic object through the logging stack, but that would be just playing whack-a-mole, as long as #58164 is open.

I think a recover around the goroutine doing Server.PreStart() would go a long way towards making me happier :)