telemetry: move the diagnostic protos to structured logging + submit payloads to reg server for backward-compat

[knz]

This was discussed with @vilamurugu and @petermattis

Instead of pushing state changes to a central registration server, we could instead emit telemetry events in an append-only store (e.g. on S3) and mine the telemetry data using event aggregation.

This approach could be called “telemetry logging” to contrast it with the (current) “telemetry updates”.

How would this work?

@petermattis suggests leveraging the new infrastructure for structured logging. We could operate as follows:

• define a new logging channel TELEMETRY.
• create a new structured log event type in eventpb that embeds telemetry data (this could possibly be as simple as a new proto message type that embeds a field of the type defined in the diagnosticspb package)
• change the telemetry update loop to use log.StructuredEvent instead of a custom HTTP PUT/GET client and request.
• for CockroachCloud customers, set up a logging config on each server that redirects TELEMETRY events to our shared log collector.
• for on-prem customers and backward compatibility, implement a new logging sink for a HTTP collector (log: implement a HTTP sink #63980), and ensure that the default config redirects TELEMETRY events to our existing registration HTTP endpoint via the network logging subsystem

This plan depends on the existence of a log collector for CockroachCloud, however we already have implemented such a thing for other security purposes. We can thus extend its usage.

Once this plan is implemented, we would still receive the telemetry for on-prem users, albeit using HTTP requests issued by the logging system (instead of the ad-hoc HTTP client in our current diagnostics package).

Additionally, we would receive the telemetry events from CC clusters through the log collector and these can be redirected to a CRL-internal S3 bucket by a configuration at the level of the log collector.

Another benefit is that it would allow on-prem users with network restrictions to collect telemetry events using their own log collector, and deliver them to us manually upon request. (Currently telemetry from isolated on-prem clusters is simply lost.)

Epic: CC-4303

Jira issue: CRDB-6782

[knz]

Once we have a TELEMETRY log channel configured as described above, we'd have an alternate approach to "time to first query", simpler than that proposed in #63812:

• the init node would log a "cluster initialized" event on the TELEMETRY channel upon cluster initialization.
• every node would log a "first query executed" event on the TELEMETRY channel, the first time it runs a query after starting up
• upon log aggregation, we'd simply scan the log event stream, and see the log timestamp difference between the "cluster init" event and the first "first query exec" event.

A drawback of this approach is that a failure in the network log collector could cause the "cluster init" event or some of the "first query" events to be lost.

[logston]

Looks great @knz!

I have a couple of notes/questions:

we could instead emit telemetry events in an append-only store (e.g. on S3) and mine the telemetry data using event aggregation. / for CockroachCloud customers, set up a logging config on each server that redirects TELEMETRY events to our shared log collector.

Like you mention, for CC this will be a small lift for 21.1+ clusters once the we can direct TELEMETRY events via a logging channel configuration. 20.2 CC clusters will require a lot of care but perhaps we can delay porting them until they've migrated to 21.1, obviating the need for that extra work.

for on-prem customers and backward compatibility, implement a new logging sink for a HTTP collector, and ensure that the default config redirects TELEMETRY events to our existing registration HTTP endpoint via the network logging subsystem

I see this being the case for some time to come. For on prem, it will be tricky to ship logs to our global sink without a proxy given the need for authn/authz when POSTing to the sink. We could provide on-prem customers with a suggested solution/config for relaying telemetry logs such that the isolated vs. non-isolated customers have a very similar setup. Isolated customers would simply have once less sink in their fluentbit/fluentd/logstash config.

and ensure that the default config redirects TELEMETRY events to our existing registration HTTP endpoint via the network logging subsystem

To "close the loop", we could have this endpoint proxy the events to our global sink such that all logs (both CC and on-prem) end up in the same location. Perhaps this was what you were alluding to already and I misunderstood.

Additionally, we would receive the telemetry events from CC clusters through the log collector and these can be redirected to a CRL-internal S3 bucket by a configuration at the level of the log collector.

Nit: Instead, can we dump these logs to a per cluster bucket as well as the global sink? This would require minimal changes in CC.

[logston]

I see now that there is a more in depth doc that precedes this issue. Feel free to ignore my comments here in favor of my comments in the other document.

[knz]

I think the other doc is much more narrow in scope. The "log-based telemetry project", if we start it, will have more stakeholders and a wider impact.

Nit: Instead, can we dump these logs to a per cluster bucket as well as the global sink? This would require minimal changes in CC.

Yes we can do this. The issue at top is a general vision, and the decision of which events gets routed where is a matter of configuration -- i.e. a decision that the consumer of the feature (The SREs and the data team) make together separately.

[andy-kimball]

A downside to changing how telemetry works: we'll stop getting telemetry data for older CRDB versions (we still get lots of reports from versions that are years old), since older versions will only know how to log to the registry service, which presumably will be shut down. Or, if it's not shut down, we'll be maintaining two parallel pipelines, which means even more work and complication.

Also, the biggest problems I've seen with the reporting pipeline are:

1. There is too much data, making it take 10's of minutes to run even simple queries.
2. The data is low quality, because there are many test/dev clusters operating, and so the Product team doesn't trust it and doesn't use it much.

Is it really the telemetry technology that is problematic (i.e. RPC call vs. logging-based)? We could switch to logging-based telemetry, and I think we'd still be in the same bad place, because that solution doesn't address either problem #1 or problem #2 above.

[knz]

Or, if it's not shut down, we'll be maintaining two parallel pipelines, which means even more work and complication.

The way it's being envisioned is this:

• for on-prem users, we will continue HTTP uploads. The implementation thereof inside crdb will move from the diagnostics package to the util/log package via a HTTP sink (log: implement a HTTP sink #63980)
• however, this does not mean we are going to maintain 2 pipelines; instead the HTTP collector will be changed to append the data to S3 (or the storage bucket) instead of storing it inside a crdb cluster. This will unify the aggregation across on-prem and CC clusters.

This way, as a technology end result:

• inside crdb, the emission of telemetry is stillimplemented in 1 place
• inside CRL, the data arrives in 1 place
• we preserve compatibility with previous crdb versions by keeping register.cockroachdb.com available
• the data for old versions can arrive in the S3 bucket so that PMs can see it

the biggest problems I've seen with the reporting pipeline are: [...]

The project discussed here is somewhat orthogonal to these two problems.

FYI:

• the solution to your "problem 1" will be to cull old data from the collected telemetry.
• the solution to "problem 2" is to filter the data upfront, either in register.cockroachdb.com or later in the pipeline, before it gets to the storage buckets.

I think it's going to be useful to call out "problem 2" during the architectural discussion next week.

[thtruo]

cc @JuanLeon1

[knz]

Heads up that the reason why this issue is still open is that we have not yet migrated all the payloads in the diagnostic report to a structured logging solution yet.

[thtruo]

cc @dhartunian @nkodali as this was brought up in the logging knowledge transfer session

[github-actions]

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!