db-console: per-store metrics not displayed after importing multi-store tsdump

[kvoli]

Describe the problem

Custom metrics page doesn't display per-node metrics (multi-store) after importing TSDump.

To Reproduce

See TSDump from this issue (internal).

Stage the TSDump using the linked debug zip stores yaml and tsdump.gob.

Expected behavior

Per-node metrics are displayed in the custom metrics page.

Environment:

• CockroachDB version v23.1.13
• Client app TSDump single node import

Jira issue: CRDB-37214

[sumeerbhola]

I would consider this a release blocker -- it seriously hinders our ability to use existing metrics. Thoughts?

[abarganier]

Definitely some unfortunate behavior related to multi-store nodes.

If I run a multi-store cluster and then run a workload:

$ ./cockroach start-single-node --insecure --store=./cockroach-data/store1 --store=./cockroach-data/store2 --store=./cockroach-data/store3
$ ./cockroach workload init tpcc
$ ./cockroach workload run tpcc

If you view a store-level metric such as cr.store.capacity.used, you get something resembling this for the cluster (which, in our case, is also the node, because we only have 1 node):

We see a value exceeding 750MiB. This is because we have 3 stores, and the sum of the usage of each node is what's plotted on the chart. If we take a tsdump we can see the raw metric values for each store. Note that the sum of all 3 matches what's plotted:

cr.store.capacity.used 1
1712690930000000000 3.373435e+08 // 337MiB

cr.store.capacity.used 2
1712690930000000000 2.85115949e+08 // 285MiB

cr.store.capacity.used 3
1712690930000000000 1.83637545e+08 // 183MiB

However, if we tick the "Per Node" box, we see the node-level view, with values around ~340MiB, which we can tell is the value from store1:

This is confusing. First of all, the data here is at the store-level, not the node-level. In the tsdump, the ID attribute associated with the key is either a storeID or a nodeID.

Even so, n1's combined stores are using ~750MiB, but we can only see that in the cluster-level view. As soon as we switch to node-level, suddenly n1 is only reported as using ~340MiB. In reality, what's being reported here is store1.

TL;DR:

1. Store metrics are being recorded into TSDB at the proper store level, so the underlying data is correct. It's how it's being queried/displayed that's problematic.
2. If a node has multiple stores, the cluster-level view will SUM all the stores across each node to derive a value. However, if you switch to the node-level view, each node only reports values from its 1st store instead of SUM. This is an inconsistent and misleading experience.

[abarganier]

Potentially related: link (internal)