Add per-tenant cache metrics

[dannykopping]

What this PR does / why we need it:
In order to improve & troubleshoot our cache performance, we need per-tenant metrics. Right now we track cache value size, hit/miss ratio, request latency, and requested key count on a global level. All of these metrics (except for request latency) are too coarse right now to be very useful. Request latency is irrelevant per tenant because we don't have a way to dedicate a caching backend per tenant, so this would be a very noisy per-tenant metric.

Which issue(s) this PR fixes:
#6318

Special notes for your reviewer:
This setting is defaulted to false due to potential cardinality explosions.

This PR adds a tenant label for the following metrics:

• loki_cache_value_size_bytes
• loki_cache_fetched_keys
• loki_cache_hits

If one has a large number of active tenants, this can result in the creation of many more streams.

Checklist

• Documentation added
• Tests updated
• Is this an important fix or new feature? Add an entry in the CHANGELOG.md.
• Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md

[trevorwhitney]

I like where this is headed :)

[trevorwhitney]

WDYT about just always doing this based on if tenancy is enabled (ie. auth = true)? Curious how much cardinality explosion we're really protecting users from here, and we already have sooooo many configs.

[dannykopping]

You know it's rude to review draft PRs 😛 but thanks for taking a look

Cardinality is something that I plan to address in the description of the PR, but finishing something else up first.

[trevorwhitney]

haha, it's like we're pairing...but asynchronously! yeah, I didn't take too critical an eye, but couldn't resist getting my opinions in early.

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0.3%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0.3%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[owen-d]

I'm not in favor of adding this due to cardinality concerns on large clusters. It will be helpful on smaller clusters, but at the expense of being a toggle away from a cardinality bomb on large ones :(. Instead, can we include tenant information in logs for later extraction via loki?

[dannykopping]

I'm not in favor of adding this due to cardinality concerns on large clusters. It will be helpful on smaller clusters, but at the expense of being a toggle away from a cardinality bomb on large ones :(. Instead, can we include tenant information in logs for later extraction via loki?

I'm fine with parking this for now 👍
#6317 gives us most of what we need for now, except for cache value sizes, but we can factor that in somehow into the logs if need to.