Tail input with csv-data and csv_skip_rows

[joriws]

My input.tail config is:

  files = ["testlog"]
  from_beginning = true
  pipe = false
  data_format = "csv"
  csv_skip_rows = 1
  csv_header_row_count = 1
  csv_delimiter = ","
  csv_trim_space = true
  csv_tag_columns = ["instance"]
  csv_timestamp_column = "date"
  csv_timestamp_format = "2006-01-02-15:04"

And testlog-file starts as..

version=13.0
date,instance,Incoming Requests,Outgoing Requests,Incoming Answers 2xxx,Outgoing Answers 2xxx,Incoming Answers UTD,Outgoing Answers UTD,Incoming Answers Redirect,Outgoing Answers Redirect,Incoming Answers Other,Outgoing Answers Other,Retransmit Requests,Rejected Requests Filtering,Rejected Requests In-gress Filtering,Rejected Requests Other,Timeout Requests,Discarded Answers

It seems that input is not skipping over line "version=13.0" which it should be according to " csv_skip_rows = 1".

If I comment out first line and csv_skip_rows then telegraf works as expected.

Error is:

2019-01-23T16:02:47Z D! [inputs.tail] tail added for file: testlog
2019-01-23T16:02:47Z E! [inputs.tail]: Error in plugin: E! Malformed log line in testlog: [date,instance,Incoming Requests,Outgoing Requests,Incoming Answers 2xxx,Outgoing Answers 2xxx,Incoming Answers UTD,Outgoing Answers UTD,Incoming Answers Redirect,Outgoing Answers Redirect,Incoming Answers Other,Outgoing Answers Other,Retransmit Requests,Rejected Requests Filtering,Rejected Requests In-gress Filtering,Rejected Requests Other,Timeout Requests,Discarded Answers], Error: timestamp column: date could not be found

[joriws]

Thanks, that was fast..

I am sorry on very quick report yesterday - telegraf version I am using is go get'd yesterday

telegraf --version

Telegraf unknown (git: master efbc83c)

--

I experimented more with input.tail and tried csv_comment:
- added configuration to existing
csv_comment = "#"

- changed input file
#version=13.0
date,instance,Incoming Requests,Outgoing Requests,Incoming Answers 2xxx,Outgoing Answers 2xxx,Incoming Answers UTD,Outgoing Answers UTD,Incoming Answers Redirect,Outgoing Answers Redirec...

output
2019-01-24T07:25:35Z D! [agent] Starting service inputs
2019-01-24T07:25:35Z D! [inputs.tail] tail added for file: testlog
2019-01-24T07:25:35Z E! [inputs.tail]: Error in plugin: E! Malformed log line in testlog: [#version=13.0], Error: EOF
2019-01-24T07:25:35Z E! [inputs.tail]: Error in plugin: E! Malformed log line in testlog: [date,instance,In coming Requests,Outgoing Requests,Incoming Answers 2xxx,Outgoing Answers 2xxx,Incoming Answers UTD,Outgoing Answers UTD,Incoming Answers Redirect,Outgoing Answers Redirect,Incoming Answers Other,Outgoing Answers Other,Retransmit Requests,Rejected Requests Filtering,Rejected Requests In-gress Filtering,Rejected Requests Other,Timeout Requests,Discarded Answers], Error: [parsers.csv] data columns must be specified

I hope I tried to use csv_comment correctly.. I am evaluating TICK stack so I started using telegraf yesterday..

--

I converted input.tail to input.file and ..

• csv_comment works as expected - outputs CSV data starting from line 3
• csv_skip_rows works different, not more parsing error on headers but...

2019-01-24T07:33:35Z D! [agent] Starting service inputs
2019-01-24T07:33:40Z E! [inputs.file]: Error in plugin: timestamp column: date could not be found
2019-01-24T07:33:50Z E! [inputs.file]: Error in plugin: timestamp column: date could not be found