util/log: implement a json log entry parser

[cameronnunez]

The json log format currently does not have a parser. Because of this, tech support tools like debug zip and debug merge-logs fail during the construction of a parser for a log that is json formatted. Customers have expressed a desire to switch from crdb log formats to json formatting to be compatible with third-party tools.

We want to create a json log entry parser to remedy this, one that exists alongside the other parsers.

Jira issue: CRDB-8178

[dhartunian]

I started filing a bug but discovered this one in the process so attaching my details as well. The one thing to note here is that this appears to be more severe than just affecting merge-logs but also prevents the logs from being exported at all.

Describe the problem

Use the following log config in logconfig.yaml (note the use of the json-fluent type!)

file-defaults:
  max-file-size: 10MiB
  max-group-size: 100MiB
  buffered-writes: true
  filter: INFO
  format: json-fluent
  redact: false
  redactable: true
  exit-on-error: true
  auditable: false
sinks:
  file-groups:
    sql-slow:
      channels: [SQL_PERF]
      format: crdb-v2
      redact: false
      redactable: true
    default:
      channels: [DEV, OPS, HEALTH]
      format: json-fluent
      redact: false
      redactable: true

Start a server:

./cockroach-short start-single-node --insecure --log-config-file=./logconfig.yaml

Generate a debug.zip

./cockroach-short debug zip /tmp/jsondebug.zip --insecure

Inspect zip file for default log:

cd /tmp
tar -xzf jsondebug.zip
cd debug/nodes/1/logs
cat cockroachshort.David-Hartunian-MacBook-Pro.david.2021-11-23T18_37_10Z.006761.log.err.txt

returns

rpc error: code = Unknown desc = decoding format: failed to extract log file format from the log

Expected behavior

The log file in the cockroach-data directory should be in the debug.zip file

This is what the local log looks like:

13:41:43 ❯ tail cockroach-data/logs/cockroachshort.David-Hartunian-MacBook-Pro.david.2021-11-23T18_37_10Z.006761.log
{"tag":"cockroachshort.dev","channel_numeric":0,"channel":"DEV","timestamp":"1637692645.753415000","cluster_id":"e3dc94b7-6f13-4c9f-bad8-da197ecc956c","node_id":1,"version":"v22.1.0-alpha.00000000-1480-g8bc5cb51a7","severity_numeric":2,"severity":"WARNING","goroutine":1433,"file":"jobs/registry.go","line":1458,"entry_counter":168,"redactable":1,"tags":{"n":"1"},"message":"failed to record error for job 713104539858632705: non-cancelable: marking migration complete for 21.1-1114: ‹migration-job-mark-job-succeeded›: cannot acquire lease when draining: non-cancelable: marking migration complete for 21.1-1114: ‹migration-job-mark-job-succeeded›: cannot acquire lease when draining"}
{"tag":"cockroachshort.ops","channel_numeric":1,"channel":"OPS","timestamp":"1637692645.792529000","cluster_id":"e3dc94b7-6f13-4c9f-bad8-da197ecc956c","node_id":1,"version":"v22.1.0-alpha.00000000-1480-g8bc5cb51a7","severity_numeric":1,"severity":"INFO","goroutine":1492,"file":"server/drain.go","line":161,"entry_counter":169,"redactable":1,"tags":{"server drain process":""},"message":"drain remaining: 6"}
{"tag":"cockroachshort.ops","channel_numeric":1,"channel":"OPS","timestamp":"1637692645.811880000","cluster_id":"e3dc94b7-6f13-4c9f-bad8-da197ecc956c","node_id":1,"version":"v22.1.0-alpha.00000000-1480-g8bc5cb51a7","severity_numeric":1,"severity":"INFO","goroutine":1492,"file":"server/drain.go","line":163,"entry_counter":170,"redactable":1,"tags":{"server drain process":""},"message":"drain details: descriptor leases: 5, liveness record: 1"}
{"tag":"cockroachshort.ops","channel_numeric":1,"channel":"OPS","timestamp":"1637692646.093153000","cluster_id":"e3dc94b7-6f13-4c9f-bad8-da197ecc956c","node_id":1,"version":"v22.1.0-alpha.00000000-1480-g8bc5cb51a7","severity_numeric":1,"severity":"INFO","goroutine":1492,"file":"server/drain.go","line":161,"entry_counter":171,"redactable":1,"tags":{"server drain process":""},"message":"drain remaining: 0"}
{"tag":"cockroachshort.dev","channel_numeric":0,"channel":"DEV","timestamp":"1637692646.093328000","cluster_id":"e3dc94b7-6f13-4c9f-bad8-da197ecc956c","node_id":1,"version":"v22.1.0-alpha.00000000-1480-g8bc5cb51a7","severity_numeric":2,"severity":"WARNING","goroutine":434,"file":"jobs/registry.go","line":851,"entry_counter":172,"redactable":1,"tags":{"n":"1"},"message":"canceling all adopted jobs due to stopper quiescing"}
{"tag":"cockroachshort.health","channel_numeric":2,"channel":"HEALTH","timestamp":"1637692646.093363000","cluster_id":"e3dc94b7-6f13-4c9f-bad8-da197ecc956c","node_id":1,"version":"v22.1.0-alpha.00000000-1480-g8bc5cb51a7","severity_numeric":1,"severity":"INFO","goroutine":528,"file":"rpc/context.go","line":688,"entry_counter":173,"redactable":1,"tags":{"n":"1"},"message":"closing [{n1: ‹David-Hartunian-MacBook-Pro:26257› (default)} {n0: ‹David-Hartunian-MacBook-Pro:26257› (default)}]"}
{"tag":"cockroachshort.dev","channel_numeric":0,"channel":"DEV","timestamp":"1637692646.093307000","cluster_id":"e3dc94b7-6f13-4c9f-bad8-da197ecc956c","node_id":1,"version":"v22.1.0-alpha.00000000-1480-g8bc5cb51a7","severity_numeric":2,"severity":"WARNING","goroutine":336,"file":"sql/sqlliveness/slinstance/slinstance.go","line":220,"entry_counter":174,"redactable":1,"tags":{"n":"1"},"message":"exiting heartbeat loop"}

Environment:

• CockroachDB version: 22.0 (master branch)

Additional context
This bugfix should be backported to 21.2 and 21.1 if possible to enable smooth log collection for debugging.

[cameronnunez]

Regarding debug zip, it seems it was originally designed to be a best effort utility with no guarantees to work for any configuration. Maybe there needs to be discussion if we want more guarantees. If we collectively decide we do want more guarantees, there's a number of things we'll need to do including ensuring a parser for every log format.

[cameronnunez]

cc @thtruo for triage

[thtruo]

This bubbled up again. Based on a followup with @cameronnunez, the Server team suggests it's best for Obs Infra to take ownership here. This is part of the transition where Obs Infra owns logging - so I'm updating the team label for us to triage cc @dhartunian

[kenliu-crl]

manually reviewed and updated