document redactable logs + logging format

[taroface]

"Unsafe" data can be automatically stripped out of logs using a new flag. Reference

• document new flag --redactable-logs
• document the logging format to allow "users to audit the correctness of our redaction algorithms"
• 20.2 feature. may be backported to 20.1/19.2

PR: cockroachdb/cockroach#48051

From release notes:

The commands cockroach debug zip and cockroach debug merge-logs support a new flag --redact-logs. When specified, it causes log messages that potentially contain confidential data or PII to be redacted away. This feature is experimental. Note: Currently, this redaction is extremely aggressive and may hinder the ability of Cockroach Labs to investigate issues from log files, especially if --redactable-logs is not enabled server-side. Other potentially-confidential data may still be retrieved by cockroach debug zip; the new flag only applies to retrieved log files inside the output zip file. [#48051][#48051] {% comment %}doc{% endcomment %}

The server CLI commands (start, start-single-node, demo) now support a new flag named --redactable-logs, which introduces markers in generated log entries to facilitate redaction of sensitive information by cockroach debug zip and other CockroachDB APIs. This is an experimental feature currently not enabled by default, but is recommended for all deployments and will be enabled by default in a later version. Note: the flag is advertised (in --help) by cockroach start and cockroach start-single-node, where log files are expected as a matter of course; it is also supported, yet hidden (from --help), by every other CLI command because every command can also produce log entries when passed --log-dir or --logtostderr. [#48051][#48051] {% comment %}doc{% endcomment %}

Advanced troubleshooting information (such as internal details from the Go runtime) is now saved in a separate log file named <program>-stderr.xxx.log in the configured log directory. [#48051][#48051] {% comment %}doc{% endcomment %}

Update: Related issue: util/logs: request redactable logs by default #8395

[jseldess]

Related email from @knz:

Hi all,

we're shortly implementing a change that equip error objects with unsafe data redaction, as well as enabling redactable logs by default.

What this means for you:

1. you will start to see special characters introduced in log messages, for example:

I200822 07:47:26.055746 1 server/server.go:1487 ⋮ [n1] starting postgres server at ‹127.0.0.1:28764› (use: ‹127.0.0.1:28764›)
                                                ^^                                 ^^              ^^      ^^              ^^

The first one is called "vertical ellipsis" (⋮) indicates that the remainder of the log line is redactable. The "‹" and "›" are small angle brackets and enclose sensitive information.

We need "⋮" for compatibility when merging logs across versions: entries from past versions will not contain ‹/› markers and yet will be unsafe for reporting or inclusion in redacted debug zips.

(This feature was already available when the flag --redactable-logs was passed; we're now enabling this by default. Can be disabled with --redactable-logs=false.)

2. error objects will also use the redaction logic internally; this will help make errors non-redactable in logs, and hopefully, the information enclosed in Sentry reports will now be more comprehensive and clearer.

3. we'll be starting to present redacted log entries in roachtest failures, to "dog food" the mechanism: if we can't troubleshoot our own issues without the sensitive bits edited out, we have no chance to do it with customers who use log redaction.

If you find yourself missing information, consider perusing the advice from the presentation last month to make more details non-redactable.

4. reminder: you can use cockroach debug merge-logs --redact to edit the sensitive bits out from a log file. This will be also available via cockroach debug zip.

(you can also use cockroach debug merge-logs --redactable-output=false to remove the special characters without redacting information out.)

At this time, the following issues are known, and fixes for them will be added soon:

• the formatting verb %T, to display a go type, breaks when used directly. The symptom is that you see redact.redactableArg instead of your value type in both error messages or logging output instead of the actual type.

This is the highest priority flaw remaining to be fixed, and a patch will follow shortly. This is tracked in cockroachdb/cockroach#53207

[jseldess]

@Amruta-Ranade, is there still more work to be done here, or did #7490 resolve this?