Optional System Tags not passed to Influxdb and Default System Tags disabled

[perrinj3]

Brief summary

When the System Tags vu and iter are specified in a K6 test either from the CLI or in the config.json the tags do not appear in the Influxdb output. Also specifying these tags completely disables all the default system tags.
If the output is a JSON file the optional system tags do appear as expected.

k6 version

v0.39

OS

windows

Docker version and image (if applicable)

No response

Steps to reproduce the problem

Specify system tags vu and iter in the config file like this
"systemTags": ["iter","vu"],

or from CLI like this
--system-tags vu --system-tags iter

run k6 with option
-- out influxdb=http://influxdb host

Expected behaviour

{"type":"Metric","data":{"name":"http_reqs","type":"counter","contains":"default","tainted":null,"thresholds":[],"submetrics":null},"metric":"http_reqs"}
{"type":"Point","data":{"time":"2022-08-30T15:09:27.2544213+10:00","value":1,"tags":{"type":"API Call","iter":"0","vu":"1","group":"Login","name":"name3","scenario":"Smoke Test1"}},"metric":"http_reqs"}
{"type":"Metric","data":{"name":"http_req_duration","type":"trend","contains":"time","tainted":null,"thresholds":[],"submetrics":null},"metric":"http_req_duration"}
{"type":"Point","data":{"time":"2022-08-30T15:09:27.2544213+10:00","value":2.0133,"tags":{"group":"Login","name":"name3","scenario":"Smoke Test1","type":"API Call","iter":"0","vu":"1"}},"metric":"http_req_duration"}
{"type":"Metric","data":{"name":"http_req_blocked","type":"trend","contains":"time","tainted":null,"thresholds":[],"submetrics":null},"metric":"http_req_blocked"}
{"type":"Point","data":{"time":"2022-08-30T15:09:27.2544213+10:00","value":2.9932,"tags":{"group":"Login","name":"name3","scenario":"Smoke Test1","type":"API Call","iter":"0","vu":"1"}},"metric":"http_req_blocked"}

Actual behaviour

optional system tags iter and vu do not appear as tags in the Influxdb output and the default system tags are all suppressed

[codebien]

Hi @perrinj3,
I have tried the latest version and it seems to me that it works as expected. How are you verifying your data points? Note that as the documentation is mentioning vu and iter are submitted by default as InfluxDB's fields.

For me, executing

k6 run  --system-tags vu,iter,url,name -o influxdb samples/http_get.js

returns the expected values on the influxdb server

> select * from http_reqs
name: http_reqs
time                iter name                    url                     value vu
----                ---- ----                    ---                     ----- --
1661856026596699962 0    https://test-api.k6.io/ https://test-api.k6.io/ 1     1

Also specifying these tags completely disables all the default system tags.

Regarding this part, it is the expected behaviour, it does not add the provided items on top instead the option sets the entire list.

I remove the bug label for now and I will wait for your updates.

[na--]

I'll close this, since it's the expected behavior and configurable via K6_INFLUXDB_TAGS_AS_FIELDS.

[perrinj3]

Thanks for this. I thought I had responded but it didn't update for some reason.
I can now see that this is expected behaviour for k6 to Influxdb
However if I output to OpenTelemetry using the InfluxDB receiver the vu and iter system tags are rejected. They seem to be passed as strings like this in the raw InfluxDB line protocol data sent to the collector
http_req_waiting,ip=52.70.134.36,ocsp_status=unknown,test=test iter="0",value=52.795516,vu="1" 1662145520898775240
http_req_receiving,ip=52.70.134.36,ocsp_status=unknown,test=test iter="0",value=0.246273,vu="1" 1662145520898775240

Refer to open-telemetry/opentelemetry-collector-contrib#13771 for more detail.
I'm not clear whether K6_INFLUXDB_TAGS_AS_FIELDS=vu:int,iter:int would help. I tried it but they still get rejected
Any ideas how I can work around this? I'm happy to open a new issue if that's the best thing to do

[na--]

Can you try to send them as tags, not as fields, by leaving the value empty? Like this K6_INFLUXDB_TAGS_AS_FIELDS=""

[perrinj3]

That doesn't seem to make any difference. I can see them as tags in an outputted JSON file
{"type":"Point","data":{"time":"2022-09-08T18:48:44.7400702+10:00","value":3.9884,"tags":{"vu":"2","iter":"4"}},"metric":"iteration_duration"}
But opentel rejects vu and iter. Other system tags like status are passed through OpenTel ok

[na--]

This JSON line is from the k6 json output, right? It is a completely different thing from the influxdb output and it's not affected by K6_INFLUXDB_TAGS_AS_FIELDS. To debug this, I suggest trying to dump the raw data that k6 sends to influx, e.g. by using something like mitmproxy.

[perrinj3]

Thanks for that advice. I did look at the raw influxdb output using Fiddler and could see that the K6_INFLUXDB_TAGS_AS_FIELDS="" option did change the iter and vu tags from a string to a numeric. Opentelemetry then accepted the tags and passed them through.

[perrinj3]

Hello Ivan Thanks for looking into this. I can confirm that this works as expected when outputting directly to InfluxDB. However in our case we are outputting via OpenTelemetry using the InfluxDB receiver. I expected that this would treat the vu and iter system-tags in exactly the same ways as it does other tags but couldn’t see then in the OpenTelemetry logs. So sorry to be misleading. I didn’t have easy access to a influxdb instance. Given your help I reported the issue to Opentelemetry and got the following response. See below or open-telemetry/opentelemetry-collector-contrib#13771 (comment) ====================================================================================== ``` Thanks for that information. After looking into it further, I can reproduce the issue, though I am not sure what to do about it. However, I believe it is either an issue with k6, or the library used to parse the InfluxDB line protocol, not an issue within the receiver itself. Here is what I have found: From what I can tell, k6 is sending the vu and iter system tags as string-typed fields on the metric data points instead of as tags. Here is a couple lines of the raw InfluxDB line protocol data sent to the collector: http_req_waiting,ip=52.70.134.36,ocsp_status=unknown,test=test iter="0",value=52.795516,vu="1" 1662145520898775240 http_req_receiving,ip=52.70.134.36,ocsp_status=unknown,test=test iter="0",value=0.246273,vu="1" 1662145520898775240 The library can't determine the type of the data points, so tries to parse each field as a separate metric. It appends the name of the field to the end of the original metric name from the line protocol as seen here<https://github.com/influxdata/influxdb-observability/blob/main/influx2otel/metrics.go#L213> (which is why in the Collector logs the metric names are all *_value). It checks the type of each field on the original data point, and vu and iter don't have valid values since they are strings, so it prints out an error message<https://github.com/influxdata/influxdb-observability/blob/main/influx2otel/metrics.go#L209> and drops them. Given that InfluxDB appears to handle this case, I believe it is an issue with the pdata conversion library, but I don't have enough context to say for sure. ``` Is this something you can assist with? I’m not clear on where the problem lies but I wonder why iter and vu would be specified as differently typed fields to the other tags and why Influxdb handles this better than the Opentel influxdb receiver. Regards, John From: Ivan ***@***.***> Sent: Tuesday, August 30, 2022 8:48 PM To: grafana/k6 ***@***.***> Cc: John Perrin ***@***.***>; Mention ***@***.***> Subject: Re: [grafana/k6] Optional System Tags not passed to Influxdb and Default System Tags disabled (Issue #2668) Hi @perrinj3<https://github.com/perrinj3>, I have tried the latest version and it seems to me that it works as expected. How are you verifying your data points? For me, executing k6 run --system-tags vu,iter,url,name -o influxdb samples/http_get.js returns the expected values on the influxdb server

select * from http_reqs

name: http_reqs time iter name url value vu

…

---- ---- ---- --- ----- -- 1661856026596699962 0 https://test-api.k6.io/ https://test-api.k6.io/ 1 1 I remove the bug label for now and I will wait for your updates. — Reply to this email directly, view it on GitHub<#2668 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ASE3UA5RGDSHREELCXZXUWTV3XRHPANCNFSM6AAAAAAQABXPNY>. You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>> This e-mail and any attachments to it (the "Communication") is, unless otherwise stated, confidential, may contain copyright material and is for the use only of the intended recipient. If you receive the Communication in error, please notify the sender immediately by return e-mail, delete the Communication and the return e-mail, and do not read, copy, retransmit or otherwise deal with it. Any views expressed in the Communication are those of the individual sender only, unless expressly stated to be those of Australia and New Zealand Banking Group Limited ABN 11 005 357 522, or any of its related entities including ANZ Bank New Zealand Limited (together "ANZ"). ANZ does not accept liability in connection with the integrity of or errors in the Communication, computer virus, data corruption, interference or delay arising from or in respect of the Communication.

[olegbespalov]

At the time of writing, there is an xk6-output-opentelemetry extension is available and soonish we're planning to make it part of the k6 #2557

Closing the issue as not planned