Future of StatsD outputs

[imiric]

The k6-core team recently had an internal meeting to discuss the current status of StatsD outputs in k6, the several issues reported by users, and the best way to move forward that benefits both users, and maintenance of k6.

Background

The statsd output is a generic implementation of the StatsD line protocol. It allows pushing metrics to any StatsD server, and, in theory, any StatsD backend.

Under the hood, it uses the DogStatsD client library, but it doesn't use any Datadog-specific functionality. This is why the same output can be used to send metrics to both Datadog and New Relic.

Issues

The problem is that, on its own, this generic implementation is very basic, and only supports 3 metric types (Counters, Gauges, and Sets). Datadog and New Relic have extended the protocol to add support for more metric types, though these changes, are not standard, and likely wouldn't work across different servers and backends. So we can't use this functionality in the statsd output without risking breaking generic StatsD support, or specifically supporting individual backends, which we want to avoid.

There are several issues reported by users on the forum:

• https://community.k6.io/t/trend-metrics-on-datadog-percentile/3995
• https://community.k6.io/t/http-req-duration-wrong-datadog/440
• https://community.k6.io/t/why-do-k6-http-req-duration-max-and-k6-http-req-duration-95percentile-have-the-same-value-in-datadog/5487
• https://community.k6.io/t/percentiles-calculation-on-new-relic/3690
• https://community.k6.io/t/how-to-get-the-correct-values-for-p-90-on-k6-newrelic/5192

Some of these, particularly those related to percentile values, and summarized in #2819, could be fixed by using the non-standard metric types, such as Distribution. But, as mentioned above, we can't use it directly in the output.

Proposed solutions

We agreed that the way forward consists of several steps:

1. Research the status of alternative protocol support for Datadog and New Relic.

   For example, New Relic supports Prometheus Remote Write natively, which k6 can output via xk6-output-prometheus-remote.

   Datadog only supports pulling metrics from Prometheus endpoints (not Remote Write), but could technically support it via Vector (owned by Datadog), which does have a RW source.

2. Depending on the results of the above research, create separate k6 Datadog and New Relic output extensions. These will initially not solve any of the reported issues, and will essentially be an extraction and duplication of the current statsd output. The benefit of this is that being separate outputs would allow them to use any backend-specific functionality.

   The k6-core team will be in charge of this development and initial maintenance.

   The reason this depends on the research from step 1, is because if we find that both backends work fine with Prometheus RW, then we can just update our documentation instead of creating separate extensions. Or if only New Relic does, then only create the extension for Datadog, etc.

3. Contact the respective Datadog and New Relic teams, explain our situation and existing issues, and politely transfer maintenance of the extensions to them.

   We believe their teams will have better knowledge about the best ways to fix the current issues, and would be better equipped to continue maintaining the integration with their backends.

   We would essentially like to achieve the same situation as with the current Elasticsearch output, which is maintained by Elastic.

4. Decide whether to continue supporting the plain StatsD output in k6, or if it should be deprecated in favor of Prometheus or OpenTelemetry.

   The StatsD project itself is quite old, and hasn't seen updates in years. With all its limitations, it's unlikely anyone is still using it. Especially since nowadays most backends support modern protocols, like Prometheus, OpenTelemetry or OpenMetrics.

I can create sub-issues for each step if it makes more sense. Let me know.

[javaducky]

I'll be discussing option 3 next week with someone from New Relic.

[javaducky]

Relocation is in progress. I've created a new extension from the current implementation in k6 core and will transfer ownership to the future maintainer once a plan has been established to deprecate the existing core functionality.

[tjovicic]

Is there a way to help with this feature?

[javaducky]

Hi @tjovicic! Ownership of the extension has been transferred and is now available at https://github.com/LeonAdato/xk6-output-statsd. Definitely feel free to contribute to follow and contribute there!

This issue remains open to track the deprecation and subsequent removal from the core k6 product.

[ostrovanka]

hello, don't know if this is a correct place to ask, but I was wondering is there a plan to fix metrics sent to DataDog (I assume also via StatsD) from k6 cloud? Right now the metrics in DD are very different from what k6 cloud dashboard shows (I suppose the root cause of the issue is the same as in local k6 runs sending metrics via StatsD)

[papatelst]

@javaducky Will output-statsd be available as part of regular k6 image in future ? Ideally statsd should be replaced by output-statsd in regular k6 image itself. It would be nice if it is available as part of regular K6 image instead of building a custom binary through a docker command everytime we run a test on CI server.

[javaducky]

Will output-statsd be available as part of regular k6 image in future ?

No. Currently, it is in the core k6 image but deprecated. It will be completely removed in a future release, at which point StatsD support will only be provided using the extension and a custom build.

Part of the reason for removing StatsD support is that the major observability vendors (Grafana, DataDog, New Relic, et al) are standardizing on use of Prometheus over StatsD.

[starryknight620]

Hello @javaducky I wanted to check in and see what the current status of this is. I'm on the Support team at Datadog and customers would like to integrate k6, but not the current version which is soon to be deprecated. While we offer an API to submit JSON data (either as a metric or to graphs on dashboards) that can grabbed from k6 test results, it's an added step. Is there any discussion around using Prometheus as the plan going forward, in place of StatsD?

[codebien]

Hey @starryknight620,
if Prometheus means the pull model, then no, there isn't any progress regarding that. And, considering that it doesn't fit very well with the k6 context it is unlikely to happen. Currently, there are more chances regarding the 4th option to have OpenTelemtry supported and to use any system compatible with it.

[nicholaspshaw]

Hello,

I'm currently attempting to run k6 load tests via circleCI and output the results to datadog dashboards. Given the current state of things, it's unclear to me the recommended approach to achieve just that.

Is statsD to be avoided while this is worked out? Is statsD using the extension reliable and likely to remain supported (https://github.com/javaducky/xk6-output-statsd) ? Or is the recommendation to avoid this entirely and use the JSON approach mentioned above by @starryknight620 ?

[javaducky]

Hello @nicholaspshaw ! Ideally, you'll use the built-in Prometheus Remote Write functionality from k6. This is the preferred method going forward. TBH, I'm not 100% sure if DataDog supports this just yet.

We didn't want to completely discard the StatsD protocol support which is why we moved it into an extension and found a sponsor for it going forward (https://github.com/LeonAdato). The official repository for the extension is https://github.com/LeonAdato/xk6-output-statsd. The level of support is unknown...so sadly "use at your own risk" should be applied.

[nicholaspshaw]

Thank you for the clarity @javaducky, this is for enterprise so it seems like relying on an individual and their kindness to keep updated would not be a suitable path going forward for us.

@starryknight620 can you please offer some clarity if Datadog supports the built-in Prometheus Remote Write functionality from k6?