tag_cardinality_limit issues

[NasAmin]

Hi,

We are starting to use tag_cardinality_limit transform to enforce some cardinality controls on our tags.
Vector is setup to receive metrics from dozens of applications. Vector seems to be printing the cardinality metric like this:

{
  "name": "value_limit_reached_total",
  "namespace": "vector",
  "tags": {
    "component_id": "drop_high_cardinality_tags",
    "component_kind": "transform",
    "component_name": "drop_high_cardinality_tags",
    "component_type": "tag_cardinality_limit"
  },
  "timestamp": "2022-11-15T14:32:51.657818181Z",
  "kind": "absolute",
  "counter": {
    "value": 1
  }
},
{
  "name": "tag_value_limit_exceeded_total",
  "namespace": "vector",
  "tags": {
    "component_id": "drop_high_cardinality_tags",
    "component_kind": "transform",
    "component_name": "drop_high_cardinality_tags",
    "component_type": "tag_cardinality_limit"
  },
  "timestamp": "2022-11-15T14:32:51.657818181Z",
  "kind": "absolute",
  "counter": {
    "value": 1080
  }
}

Looking at the internal logs, I am getting entries like this:

{
  "timestamp": "2022-11-15T14:31:46.054273Z",
  "level": "DEBUG",
  "message": "Rejecting tag after hitting configured value_limit.",
  "tag_key": "name",
  "tag_value": "7bfadae4ec1fc4eb7247fedd385641314940085384283b7b45df5d4f6e8a51a1",
  "internal_log_rate_secs": 10,
  "target": "vector::internal_events::tag_cardinality_limit",
  "span": {
    "component_id": "drop_high_cardinality_tags",
    "component_kind": "transform",
    "component_name": "drop_high_cardinality_tags",
    "component_type": "tag_cardinality_limit",
    "name": "transform"
  },
  "spans": [
    {
      "component_id": "drop_high_cardinality_tags",
      "component_kind": "transform",
      "component_name": "drop_high_cardinality_tags",
      "component_type": "tag_cardinality_limit",
      "name": "transform"
    }
  ]
}

As you can see, the above log entry just says tag_key: name which is the most common tag name. How will I know which metric its coming from?

This implies that Vector is just counting cardinality for a label regardless of which metric its coming from. Is this supposed to be for tag cardinality per metric or is it independent of the metric?

If its the latter then it doesn't really seem to be useful. I would appreciate you help in understanding this transform and its correct usage.

What I am really looking for is how will I determine which metric is emitting a high cardinality tag and then talk to the relevant team to fix it.

Kind regards,

Nas

Originally posted by @NasAmin in #15218

[jszwedko]

Hi @NasAmin !

Apologies for the delayed response, I wasn't super familiar with that transform. Looking at it, it does appear to limit regardless of metric name; just looking at the tags. I agree it would be useful to limit per-metric. I can also see that it would be useful to have the metric name in that log output. I created two new issues to track this:

• Include metric name of limited metric in internal logs emitted by tag_cardinality_limit #15742
• Limit tag cardinality by metric in tag_cardinality_limit transform #15743

Given I think those two issues cover this one, I'll close this one, but feel free to leave any additional thoughts here or on those issues!