logs: setting --log='file-defaults: {format: json} does not impact the format of cockroach-stderr.log

[thtruo]

Observation
When I run a command like cockroach start-single-node --certs-dir=certs --log='file-defaults: {format: json}' to set the logging format to json, I notice that cockroach-stderr.log continues to be in crdb-v1 format.

Expectation
When I run a command like cockroach start-single-node --certs-dir=certs --log='file-defaults: {format: json}', I expect cockroach-stderr.log to be structured as json as well, not crdb-v1.

Repro steps

• Run cockroach start-single-node --certs-dir=certs --log='file-defaults: {format: json}'
• cd into cockroach-data/ and run tail cockroach.log | jq . which outputs JSON logs ✅
• Run tail cockroach-pebble.log | jq . which outputs JSON logs ✅
• Run tail cockroach-stderr.log | jq . which does NOT output JSON logs. These logs are in crdb-v1 format

[thtruo]

cc @piyush-singh

[knz]

cockroach-stderr.log is not an output sink for log events; it's an output file for things that are unrelated to logging really.

It's an oddity due to some unfixable problems in the Go runtime system.

We need to find a way to drive the idea home that it's not a log file like the others. Meanwhile we can't drive attention away from this file entirely because it does contain useful details sometimes.

Maybe we could rename it?

[knz]

In any case, the fact it's different must be called out in documentation properly. @thtruo how do you suggest we pick up the special semantics of this file effectively in explanations, etc?

[thtruo]

Ack. Agreed that this should be called out in documentation cc @taroface FYI a good discussion topic

That's helpful context @knz. I see your perspective around the need to keep that file in some way. I don't have strong opinions around how we name it, but here's one idea. Perhaps removing the cockroach- prefix from cockroach-stderr.log can help illustrate that point for a user? My line of thinking is - since every other log file has that prefix, it implies they are similar in the sense they carry relevant log events. stderr.log on the other hand is basically the catchall for everything else and naming it differently could help drive that distinction home.

Whether we rename or not, at the very least, we'll have this explained in docs.

[thtruo]

Nvm, my suggestion about removing the cockroach- prefix does not make sense. It was helpful for me to review that portion of your RFC again. I wonder if by default calling it cockroach-strayerr.log would make that clearer?

Looking at the cockroach debug check-log-config --log='file-defaults: {format: json}' command again makes it clear what's happening: format:json doesn't get applied anyway to things that get sent into the stderr log file.

Sounds like this issue is not really a functional bug but an area for potential user confusion. I'm okay with closing this if you are cc @knz

[knz]

I think at this point it is mostly a doc project and there's no clear change engineering-wise.

However if we need to make a callout about this in docs, it won't be sufficient to rely on the stream of past release notes. I would recommend creating a specific docs issue instead.

[thtruo]

Ack. Closing this. Tracking the relevant docs issue here cockroachdb/docs#9344