multitenant: timeseries double counts some metrics

[lidorcarmel]

In c2c, source cluster, I see write BW to disk over 1.3GBps, but the GCP console says 0.66GBps. Looks like IOPS have the same issue. If I just look at the system tenant I still see the double BW, but if I look at the app tenant I see the real 0.66GBps number.

Jira issue: CRDB-30702

[abarganier]

Thanks for reporting, let me see what I can find. Will report back.

[abarganier]

This is reproducible using a multitenant cockroach demo cluster.

Procedure:

1. cockroach demo
2. Open DB Console
3. Open the Metrics page, such as "Hardware".
4. Observe a value on one of the charts, such as "Host CPU Percent" with the "Tenant" dropdown set to "System" or "All"
5. Now switch the "Tenant" dropdown to "demoapp" and observe the same chart. Note that its values are 1/2 of when the dropdown is set to "System" or "All".

I believe the problem here exists with the query side. The data in tsdb looks correct to me.

Let's focus on a metric that this occurs with (note: SQL level metrics appear to not be affected). We will choose cr.node.sys.cpu.host.combined.percent-normalized.

The MetricsRecorder records node-level metrics (including our CPU metric) to TSDB segmented by tenant. For example:

$ ./cockroach debug tsdump --certs-dir=/Users/abarganier/.cockroach-demo --format=text --host=localhost:26358
...
cr.node.sys.cpu.host.combined.percent-normalized 1
1692306000000000000 0.31227394648052653
1692306010000000000 0.30643540341695974
1692306020000000000 0.3125720067723935
1692306030000000000 0.3125720067723935
1692306040000000000 0.19457822044461875
cr.node.sys.cpu.host.combined.percent-normalized 1-2
1692306000000000000 0.31227394648052653
1692306010000000000 0.30643540341695974
1692306020000000000 0.3125720067723935
1692306030000000000 0.19457822044461875
1692306040000000000 0.17129056189003986

We can see the metric exists in CRDB twice, one with a source field of 1 (node 1 of the system tenant), and another with a source field of 1-2 (node 1, of the tenant with tenant ID 2, hence 1-2).

Both metrics alone are recorded correctly. However, when you select "All" or "System" from the dropdown on the metrics page, the tsdb query appears to be summing the values together across tenants, which leads to our reported values being 2x what they actually are.

We added the behavior for multitenant timeseries queries in #98077, which was made available in the UI with #92694. There was likely a misunderstanding in the UI code as to how the query API works, or perhaps the query code itself is not intuitive.

We should find the right solution here - does it exist in the query layer? Or in how the UI uses the query endpoint? We will get this scheduled with the team.

[Santamaura]

We can see the metric exists in CRDB twice, one with a source field of 1 (node 1 of the system tenant), and another with a source field of 1-2 (node 1, of the tenant with tenant ID 2, hence 1-2).

The ui queries for metrics based on the selection of that dropdown; selecting "All" sets tenant_id=undefined, "System" sets tenant_id=1, "demoapp" sets tenant_id=2 so there's probably somewhere in the server side code that interprets tenant_id=undefined and tenant_id=1 as the same, will look some more

[Santamaura]

So it seems like the current server side query behaviour is what I said previously; if it's system tenant or no tenant id provided we return the aggregated ts data across all tenants which I suppose is because in the initial implementation the ts data being viewed didn't need to be as granular but now that it does need to be more granular we should update the code to reflect this. Should have a PR up soon to fix this.