inputs.influxdb_listener with influx_parser_type = "upstream" - superfluous response.WriteHeader #10900
Labels
Comments
|
Hi, Thanks for trying out the new parser and the bug report. I think you are right that the message is coming from the call to Thanks! |
|
here you go: |
|
Ah, that interesting that is actually from here, before we have even got to the parser.
This is copy-paste code from the previous parser logic as well so until after this block there should be no differences, so this surprises me. |
|
I'm sending encoded data ( I there something else I can provide to help out? |
Do you happen to have an example input you could share with me to help reproduce this? Ideally, if we could run something like: to help me reproduce and work through whatever is going on. |
|
While trying to locate what lines were raising the error (by only enabling subsets of my collection) I've found that even just sending the internal monitor data (inputs.internal) will raise an error... therefore you should be able to replicate it without problems, here is the line protocol anyway (exported it by redirecting the result of |
|
@powersj were you able to replicate the issue? |
|
@Trovalo - I've been out of the office so still catching up. I did try to send the lines you provided and was unable to reproduce any issues: [agent]
omit_hostname = false
[[outputs.file]]
[[inputs.influxdb_listener]]
influx_parser_type = "upstream"What I sent: echo 'internal_memstats,company=quantumdatis_demo,host=QDSRVMONITOR,telegraf_instance=telegraf.dev alloc_bytes=5639512i,frees=17209i,heap_alloc_bytes=5639512i,heap_idle_bytes=5505024i,heap_in_use_bytes=6881280i,heap_objects=11381i,heap_released_bytes=5095424i,heap_sys_bytes=12386304i,mallocs=28590i,num_gc=2i,pointer_lookups=0i,sys_bytes=17984104i,total_alloc_bytes=8262392i 1648825166000000000' | gzip | curl -v -i --data-binary @- -H "Content-Encoding: gzip" http://localhost:8186/write
echo 'internal_write,company=quantumdatis_demo,host=QDSRVMONITOR,output=influxdb,telegraf_instance=telegraf.dev,version=1.2.2 buffer_limit=15000i,buffer_size=0i,errors=0i,metrics_added=0i,metrics_dropped=0i,metrics_filtered=0i,metrics_written=0i,write_time_ns=0i 1648825166000000000' | gzip | curl -v -i --data-binary @- -H "Content-Encoding: gzip" http://localhost:8186/write
echo 'internal_process,company=quantumdatis_demo,host=QDSRVMONITOR,processor=converter,telegraf_instance=telegraf.dev,version=1.2.2 errors=0i 1648825166000000000' | gzip | curl -v -i --data-binary @- -H "Content-Encoding: gzip" http://localhost:8186/write
echo 'internal_process,company=quantumdatis_demo,host=QDSRVMONITOR,processor=strings,telegraf_instance=telegraf.dev,version=1.2.2 errors=0i 1648825166000000000' | gzip | curl -v -i --data-binary @- -H "Content-Encoding: gzip" http://localhost:8186/write
echo 'internal_process,company=quantumdatis_demo,host=QDSRVMONITOR,processor=rename,telegraf_instance=telegraf.dev,version=1.2.2 errors=0i 1648825166000000000' | gzip | curl -v -i --data-binary @- -H "Content-Encoding: gzip" http://localhost:8186/write
echo 'internal_agent,company=quantumdatis_demo,go_version=1.17.6,host=QDSRVMONITOR,telegraf_instance=telegraf.dev,version=1.2.2 gather_errors=0i,metrics_dropped=0i,metrics_gathered=1i,metrics_written=0i 1648825166000000000' | gzip | curl -v -i --data-binary @- -H "Content-Encoding: gzip" http://localhost:8186/write
echo 'internal_gather,company=quantumdatis_demo,host=QDSRVMONITOR,input=internal,telegraf_instance=telegraf.dev,version=1.2.2 errors=0i,gather_time_ns=0i,metrics_gathered=1i 1648825166000000000' | gzip | curl -v -i --data-binary @- -H "Content-Encoding: gzip" http://localhost:8186/write2022-04-11T16:06:04Z I! Starting Telegraf 1.23.0-777f8bf7
2022-04-11T16:06:04Z I! Loaded inputs: influxdb_listener
2022-04-11T16:06:04Z I! Loaded aggregators:
2022-04-11T16:06:04Z I! Loaded processors:
2022-04-11T16:06:04Z I! Loaded outputs: file
2022-04-11T16:06:04Z I! Tags enabled: host=ryzen
2022-04-11T16:06:04Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"ryzen", Flush Interval:10s
2022-04-11T16:06:04Z D! [agent] Initializing plugins
2022-04-11T16:06:04Z D! [agent] Connecting outputs
2022-04-11T16:06:04Z D! [agent] Attempting connection to [outputs.file]
2022-04-11T16:06:04Z D! [agent] Successfully connected to outputs.file
2022-04-11T16:06:04Z D! [agent] Starting service inputs
2022-04-11T16:06:04Z I! [inputs.influxdb_listener] Started HTTP listener service on :8186
2022-04-11T16:06:14Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
internal_memstats,company=quantumdatis_demo,host=QDSRVMONITOR,telegraf_instance=telegraf.dev alloc_bytes=5639512i,frees=17209i,heap_alloc_bytes=5639512i,heap_idle_bytes=5505024i,heap_in_use_bytes=6881280i,heap_objects=11381i,heap_released_bytes=5095424i,heap_sys_bytes=12386304i,mallocs=28590i,num_gc=2i,pointer_lookups=0i,sys_bytes=17984104i,total_alloc_bytes=8262392i 1648825166000000000
internal_write,company=quantumdatis_demo,host=QDSRVMONITOR,output=influxdb,telegraf_instance=telegraf.dev,version=1.2.2 buffer_limit=15000i,buffer_size=0i,errors=0i,metrics_added=0i,metrics_dropped=0i,metrics_filtered=0i,metrics_written=0i,write_time_ns=0i 1648825166000000000
internal_process,company=quantumdatis_demo,host=QDSRVMONITOR,processor=converter,telegraf_instance=telegraf.dev,version=1.2.2 errors=0i 1648825166000000000
internal_process,company=quantumdatis_demo,host=QDSRVMONITOR,processor=strings,telegraf_instance=telegraf.dev,version=1.2.2 errors=0i 1648825166000000000
internal_process,company=quantumdatis_demo,host=QDSRVMONITOR,processor=rename,telegraf_instance=telegraf.dev,version=1.2.2 errors=0i 1648825166000000000
internal_agent,company=quantumdatis_demo,go_version=1.17.6,host=QDSRVMONITOR,telegraf_instance=telegraf.dev,version=1.2.2 gather_errors=0i,metrics_dropped=0i,metrics_gathered=1i,metrics_written=0i 1648825166000000000
internal_gather,company=quantumdatis_demo,host=QDSRVMONITOR,input=internal,telegraf_instance=telegraf.dev,version=1.2.2 errors=0i,gather_time_ns=0i,metrics_gathered=1i 1648825166000000000
2022-04-11T16:06:24Z D! [outputs.file] Wrote batch of 7 metrics in 72.03µsI tried setting up a second telegraf with this config: [agent]
omit_hostname = false
[[outputs.http]]
url = "http://localhost:8186/write"
[[inputs.internal]]and have had it sending data without issue. Any other thoughts on what I might need to do to reproduce? Thanks! |
|
I'm seeing this as well. I'm using a number of different listeners to sort imcoming ports. |
|
updating |
|
@andrewbierbaum are you also using |
|
Yes, that is what I was attempting, and it started throwing these messages upon changing it to upstream. |
I'm running on Windows, maybe is something related to that. @andrewbierbaum which OS are you using? |
|
I'm on Linux, Ubuntu 18.04 |
powersj
added a commit
to powersj/telegraf
that referenced
this issue
Apr 12, 2022
Throughout the influx parsing logic there are checks for receiving an EOF body. When received the parsing logic returns. When content is gzip we check for any error and return that error, but do not do any specific chesk for EOF. Fixes: influxdata#10900
|
Given the error is about parsing an EOF, I believe your metrics are correctly getting parsed. Throughout the code we have places where we specifically check for EOF and if we get that error, return or move on. I am thinking that this is yet another case of that, but without reproducing I do not yet feel confident of this. I have put up PR #10976 that does add this check but wanted to see if you two could give the artifacts in that PR a try. Thanks! |
powersj
added a commit
to powersj/telegraf
that referenced
this issue
Apr 12, 2022
Throughout the influx parsing logic there are checks for receiving an EOF body. When received the parsing logic returns. When content is gzip we check for any error and return that error, but do not do any specific chesk for EOF. Fixes: influxdata#10900
powersj
added a commit
to powersj/telegraf
that referenced
this issue
Apr 12, 2022
Throughout the influx parsing logic there are checks for receiving an EOF body. When received the parsing logic returns. When content is gzip we check for any error and return that error, but do not do any specific chesk for EOF. Fixes: influxdata#10900
|
@powersj |
I should be able to try it tomorrow, thanks. |
|
@andrewbierbaum the PR just had a comment added to it by the telegraf-tiger. In the comment are links to download various artifacts that are built using the PR. Included are DEB, RPM, tarball, etc. If you could download one of those and add it to your dev environment that would be awesome. Thanks again! |
|
@powersj I was able to wget and install it cleanly. Restarting telegraf with |
|
@andrewbierbaum thank you for giving it a shot! I just pushed another update and artifacts are already posted that will print some debug messages, can you give that a shot? Sorry for the back and forth :\ |
|
@powersj It's a quick test on my side, so happy to keep testing |
|
Were either of you able to run the latest PR artifacts on #10976? I was hoping to get the output to see what is happening when a failure occurs. Thanks! |
|
@powersj Sorry, I misunderstood your previous message from last week or I could have run it when you updated it last week. Looks like at least a similar output. Let me know if I can help at all. |
|
I'm getting a similar error |
powersj
added a commit
to powersj/telegraf
that referenced
this issue
Apr 21, 2022
|
@andrewbierbaum I have pushed yet another artifact, could you give that a try? Thanks! |
still looks like a similar output |
Can you add |
|
@powersj sure, here is the output. I thought I still had that on from previous, but must have removed it at some point. |
powersj
added a commit
to powersj/telegraf
that referenced
this issue
Apr 25, 2022
powersj
added a commit
to powersj/telegraf
that referenced
this issue
Apr 25, 2022
The influxdb_listener input plugin, when using the upstream parser had bad logic that would write first using the new upstream parser and then attempt to parse the data again with the internal parser right after. This was leading to EOF errors on the internal parser because the data had already been read. Fixes: influxdata#10900
Doh! These log lines reveal the issue: when using the upstream parser, my bad logic was using the upstream parser and then always trying to write the data we just did again with the internal parser. I have pushed a fix and would appreciate another test once artifacts are posted in ~20mins. Thanks very much for the back and forth! edit: artifacts are up! |
|
@powersj I now see a clean output from |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Relevant telegraf.conf
Logs from Telegraf
System info
Telegraf 1.22.0 on WIndows
Docker
No response
Steps to reproduce
Using the old parser does not trigger any error.
From what we've seen, despite this error, the data are still passing and everything seems to work, the main issue is that the log gets flooded by error messages
Expected behavior
not getting errors
Actual behavior
the error superfluous response.WriteHeader call is triggered continuously (every ~1sec)
Additional info
No response
The text was updated successfully, but these errors were encountered: