release-21.1: util/log,server/debug: fixes to logging and the logspy api endpoint

[mikeCRL]

cockroachdb/cockroach#66328 --- as well as a security vulnerability (see release note below). Additionally, it was impossible to launch multiple logspy sessions simultaneously on the same node, for example using different grep filters. This commit rectifies the situation. At a technical level, we have two changes: one in the logging Go API and one in the /debug/logspy HTTP API. For the logging changes, this patch replaces the "interceptor" singleton callback function that takes over (*loggerT).outputLogEntry(), by a new interceptor sink that lives alongside the other sinks on every channel. Because the interceptor logic is now a regular sink, log entries continue to be delivered to other sinks while an interceptor is active. Reusing the sink abstraction, with its own sink configuration with no severity filter, clarifies that the interceptor accepts all the log entries regardless of which filters are configured on other sinks. Additionally, the interceptor sink now supports multiple concurrent interception sessions. Each log entry is delivered to all current interceptors. The logspy logic is adapted to use this facility, so that multiple logspy requests can run side-by-side. For the HTTP API change, we are changing the /debug/logspy semantics. This is explained in the release note below. Release note (security update): All the logging output to files or network sinks was previously disabled temporarily while an operator was using the /debug/logspy HTTP API, resulting in lost entries and a breach of auditability guarantees. This behavior has been corrected. Release note (bug fix): Log entries are not lost any more while the /debug/logspy HTTP API is being used. This bug had existed since CockroachDB v1.1. Release note (api change): The /debug/logspy HTTP API has changed. The endpoint now returns JSON data by default. This change is motivated as follows: - the previous format, crdb-v1, cannot be parsed reliably. - using JSON entries guarantees that the text of each entry all fits on a single line of output (newline characters inside the messages are escaped). This makes filtering easier and more reliable. - using JSON enables the user to apply jq on the output, for example via curl -s .../debug/logspy | jq ... If the previous format is desired, the user can pass the query argument &flatten=1 to the logspy URL to obtain the previous flat text format (crdb-v1) instead. Co-authored-by: Yevgeniy Miretskiy [email protected]

[mikeCRL]

Created during a test of an issue-creation script. Issue is not needed. Closing.