Tags Not Transmitted with Metrics When Using StatsD Integration with DataDog

[galoniz]

Brief summary

When using k6 to send metrics to Datadog via the StatsD protocol, the metrics arrive in Datadog without any of the tags specified in the k6 script. This issue appears to be isolated to k6's interaction with StatsD, as manually sending metrics with tags via netcat to DogStatsD works as expected, and the tags are correctly processed by Datadog.

Steps to Reproduce:

Set up a Datadog Agent with DogStatsD enabled.
Create a simple k6 script that includes tags in the options block.
Configure k6 to send metrics to DogStatsD via the K6_STATSD_ADDR environment variable.
Run the k6 script.
Inspect the metrics in Datadog.
Example k6 Script:

import http from 'k6/http';
import { Counter } from 'k6/metrics';

const myCounter = new Counter('my_counter');

export const options = {
  stages: [
    { duration: '10s', target: 10 },
    { duration: '20s', target: 20 },
  ],
  tags: {
    test_run_id: 'no-cache-flag',
  },
};

export default function () {
  http.get('https://test-api.example.com');
  myCounter.add(1);  // Increment a counter metric
}

Testrun example: 
apiVersion: k6.io/v1alpha1
kind: TestRun
metadata:
  generateName: testrun-sample-
spec:
  parallelism: 2
  arguments: "--out statsd --verbose"
  script:
    configMap:
      name: no-cache-flag
      file: no-cache-flag.js
    env:
      - name: K6_STATSD_ADDR
        value: <DD_ADDRESS>:8125
      - name: K6_STATSD_ENABLE_TAGS
        value: "true"

Environment:
on Kubernetes cluster
k6 operator: https://github.com/grafana/k6-operator helm chart version 3.7.0 (using k6 runner version ghcr.io/grafana/k6-operator:latest-runner)
Datadog Agent Version: gcr.io/datadoghq/agent:7.53.0
Expected Behavior:
The metrics sent by k6 should arrive in Datadog with the test_run_id tag attached.

Actual Behavior:
The metrics arrive in Datadog, but the test_run_id tag (and any other tags) are missing.

Additional Information:

Manually sending metrics with tags to DogStatsD via netcat works correctly, and the tags appear in Datadog.
No errors or warnings are visible in k6's output that suggest a problem with tag transmission.
Possible Root Cause:
It seems that there may be an issue with how k6 formats or transmits tags when sending metrics through the StatsD protocol. This issue could be related to the way k6 handles the tagging internally before sending data to StatsD.

Workaround:
As of now, there is no known workaround. The issue requires a direct fix or investigation into k6's StatsD implementation.

GitHub Issue References:
#63

k6-operator version or image

ghcr.io/grafana/k6-operator:latest-runner

Helm chart version (if applicable)

3.7.0

TestRun / PrivateLoadZone YAML

apiVersion: k6.io/v1alpha1
kind: TestRun
metadata:
generateName: testrun-sample-
spec:
parallelism: 2
arguments: "--out statsd --verbose"
script:
configMap:
name: no-cache-flag
file: no-cache-flag.js
env:
- name: K6_STATSD_ADDR
value: <DD_ADDRESS>:8125
- name: K6_STATSD_ENABLE_TAGS
value: "true"

Other environment details (if applicable)

No response

Steps to reproduce the problem

Example k6 Script:

import http from 'k6/http';
import { Counter } from 'k6/metrics';

const myCounter = new Counter('my_counter');

export const options = {
  stages: [
    { duration: '10s', target: 10 },
    { duration: '20s', target: 20 },
  ],
  tags: {
    test_run_id: 'no-cache-flag',
  },
};

export default function () {
  http.get('https://test-api.example.com');
  myCounter.add(1);  // Increment a counter metric
}

Testrun example: 
apiVersion: k6.io/v1alpha1
kind: TestRun
metadata:
  generateName: testrun-sample-
spec:
  parallelism: 2
  arguments: "--out statsd --verbose"
  script:
    configMap:
      name: no-cache-flag
      file: no-cache-flag.js
    env:
      - name: K6_STATSD_ADDR
        value: <DD_ADDRESS>:8125
      - name: K6_STATSD_ENABLE_TAGS
        value: "true"

Expected behaviour

tags appear on DD for filtering

Actual behaviour

The metrics arrive in Datadog, but the test_run_id tag (and any other tags) are missing.

[galoniz]

update - when passing tag via command line --tag test_run_id=test_release_0.0.1 it works, but nothing when trying via code, not on global options and not specifically on a request

[yorugac]

Hi @galoniz, this doesn't seem like an issue of k6-operator but that of k6 and / or new StatsD extension. Furthermore, StatsD will soon be removed from k6.

I suggest switching to the new StatsD extension and see if it helps with tags propagation. If it doesn't, it makes sense to narrow down the issue to k6 run test.js --out statsd, i.e. make an example without k6-operator, and open an issue in that extension's repo.

Hope that helps! I'll be closing this issue now.

[galoniz]

@yorugac is there an example of usage of k6 operator with this extension as image? any expectation of supporting that in the future?

[yorugac]

@galoniz No, there isn't. There might be an example with standalone k6 in the relevant repo.

any expectation of supporting that in the future?

Not at the moment. As mentioned above, k6 is going to deprecate StatsD support. If someone is willing to contribute and ideally maintain such an example, we could add that. But even then, perhaps, it makes more sense in the relevant extension repo.