win_perf_counters seems to have broken in latest update

[SimonMcN]

hi
moved from version 1.15.3 to 1.16.1 and I think a plugin update has broken something. This did used to work.

[[inputs.win_perf_counters]]
UseWildcardsExpansion = true

log file now reporting:

2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(java)\Handle Count", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(cmd#4)\Private Bytes", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(cmd#3)\Private Bytes", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(cmd#2)\Private Bytes", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(cmd#1)\Private Bytes", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(java#4)\Private Bytes", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(cmd)\Private Bytes", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(java#3)\Private Bytes", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(java#2)\Private Bytes", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(java#1)\Private Bytes", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(java)\Private Bytes", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(cmd#4)\Thread Count", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(cmd#3)\Thread Count", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(cmd#2)\Thread Count", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(cmd#1)\Thread Count", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(java#4)\Thread Count", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(cmd)\Thread Count", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(java#3)\Thread Count", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(java#2)\Thread Count", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(java#1)\Thread Count", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(java)\Thread Count", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(cmd#4)\Virtual Bytes", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(cmd#3)\Virtual Bytes", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(cmd#2)\Virtual Bytes", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(cmd#1)\Virtual Bytes", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(java#4)\Virtual Bytes", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(cmd)\Virtual Bytes", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(java#3)\Virtual Bytes", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(java#2)\Virtual Bytes", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(java#1)\Virtual Bytes", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(java)\Virtual Bytes", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(cmd#4)\Working Set", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(cmd#3)\Working Set", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(cmd#2)\Working Set", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(cmd#1)\Working Set", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(java#4)\Working Set", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(cmd)\Working Set", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(java#3)\Working Set", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(java#2)\Working Set", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(java#1)\Working Set", will skip metric: The data is not valid.
2020-11-10T12:21:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\mymachine\Process(java)\Working Set", will skip metric: The data is not valid.

[ssoroka]

I don't believe this is a change in functionality, we just started logging these messages instead of quietly discarding them.

Do you think this is because the names aren't escaped properly?

[SimonMcN]

oh, git hub is removing the escaping

[SimonMcN]

[mosyang]

The same problem in Telegraf 1.16.2
[inputs.win_perf_counters] error while getting value for counter "\Process(opcflux)\% Processor Time", will skip metric: The data is not valid.

[barbarajoost]

The problem resists in 1.17.0.

2021-02-04T14:28:00Z W! [inputs.win_perf_counters] error while getting value for counter "\\SYSPCX35183\Process(notepad++#1)\Private Bytes", will skip metric: The data is not valid.
2021-02-04T14:28:00Z W! [inputs.win_perf_counters] error while getting value for counter "\\SYSPCX35183\Process(notepad++)\Private Bytes", will skip metric: The data is not valid.
2021-02-04T14:28:00Z W! [inputs.win_perf_counters] error while getting value for counter "\\SYSPCX35183\Process(notepad++#1)\% Processor Time", will skip metric: The data is not valid.
2021-02-04T14:28:00Z W! [inputs.win_perf_counters] error while getting value for counter "\\SYSPCX35183\Process(notepad++)\% Processor Time", will skip metric: The data is not valid.
2021-02-04T14:29:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\SYSPCX35183\Process(audiodg)\Private Bytes", will skip metric: The data is not valid.
2021-02-04T14:29:01Z W! [inputs.win_perf_counters] error while getting value for counter "\\SYSPCX35183\Process(audiodg)\% Processor Time", will skip metric: The data is not valid.
2021-02-04T14:30:00Z W! [inputs.win_perf_counters] error while getting value for counter "\\SYSPCX35183\Process(svchost#86)\Private Bytes", will skip metric: The data is not valid.

etc.

[barbarajoost]

Same in 1.17.2

2021-02-04T15:05:23Z I! Loaded inputs: mem win_perf_counters
2021-02-04T15:05:23Z I! Loaded aggregators:
2021-02-04T15:05:23Z I! Loaded processors: rename
2021-02-04T15:05:23Z I! Loaded outputs: graphite
2021-02-04T15:05:23Z I! Tags enabled: host=syspcx35183
2021-02-04T15:05:23Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"syspcx35183", Flush Interval:10s
2021-02-04T15:07:00Z W! [inputs.win_perf_counters] error while getting value for counter "\\SYSPCX35183\Process(SearchFilterHost)\Private Bytes", will skip metric: The data is not valid.
2021-02-04T15:07:00Z W! [inputs.win_perf_counters] error while getting value for counter "\\SYSPCX35183\Process(SearchFilterHost)\% Processor Time", will skip metric: The data is not valid.

[barbarajoost]

The values of this counters are usually >100%, f. e. 50% on every processor core => 4 Cores x 50% = 200%. There were similar issues in the past with CPU load performance counters.

Telegraf should not try scale this values on it's own. This can be done in Grafana in the dashboard. Older versions of Telegraf has done this in this way.

[let-thomas]

OK, for CPU, but what about "Private Bytes"? Or others? Could not be reason of not existing process (i.e. killed/finished) in the meantime?
2021-02-23T13:40:15Z W! [inputs.win_perf_counters] error while getting value for counter "\\HUB-30-254\Process(conhost#2)\Handle Count", will skip metric: The data is not valid.
2021-02-23T13:40:15Z W! [inputs.win_perf_counters] error while getting value for counter "\\HUB-30-254\Process(powershell)\Handle Count", will skip metric: The data is not valid.
2021-02-23T13:40:15Z W! [inputs.win_perf_counters] error while getting value for counter "\\HUB-30-254\Process(conhost#2)\Private Bytes", will skip metric: The data is not valid.
2021-02-23T13:40:15Z W! [inputs.win_perf_counters] error while getting value for counter "\\HUB-30-254\Process(powershell)\Private Bytes", will skip metric: The data is not valid.
2021-02-23T13:40:15Z W! [inputs.win_perf_counters] error while getting value for counter "\\HUB-30-254\Process(conhost#2)\Thread Count", will skip metric: The data is not valid.
2021-02-23T13:40:15Z W! [inputs.win_perf_counters] error while getting value for counter "\\HUB-30-254\Process(powershell)\Thread Count", will skip metric: The data is not valid.

[barbarajoost]

For a while I used the last working version I had, 1.15.1. Today I tried to change to 1.20, but the problem resists. After so much time nobody found the reason for this behaviour?

2021-09-22T17:09:00Z W! [inputs.win_perf_counters] error while getting value for counter "\\SYSPCX35183\Process(svchost#87)\Private Bytes", will skip metric: The data is not valid.
2021-09-22T17:09:00Z W! [inputs.win_perf_counters] error while getting value for counter "\\SYSPCX35183\Process(svchost#86)\Private Bytes", will skip metric: The data is not valid.
2021-09-22T17:09:00Z W! [inputs.win_perf_counters] error while getting value for counter "\\SYSPCX35183\Process(svchost#87)\% Processor Time", will skip metric: The data is not valid.
2021-09-22T17:09:00Z W! [inputs.win_perf_counters] error while getting value for counter "\\SYSPCX35183\Process(svchost#86)\% Processor Time", will skip metric: The data is not valid.

[erSitzt]

Same here with 1.20.2

[rgroenewoudt]

I often see the error for processes that have just ended.

[simonwhybrow-cbre]

Problem is still there in v1.22.0
I've tried adding in the IgnoredErrors section, but none of the PDH errors supress this warning.

Is there a way to stop these warnings being logged so Windows event logs don't get clogged up?

[robbyq92]

1.22.3 same

[sspaink]

Recently the log message was updated to include the instance name: #10995 to help identify the culprit behind the log messages. This might help you reduce the warnings, an option is to have this error only show up when running Telegraf with --debug but this could lead to metrics being skipped without you knowing for a while. Maybe having this configurable would be the best option.

I am not a user of win_perf_counters, so interested to hear from the people who are facing problems with this issue. Is the included instance name sufficient or do we need to add an option to ignore these warnings?

[telegraf-tiger]

Hello! I am closing this issue due to inactivity. I hope you were able to resolve your problem, if not please try posting this question in our Community Slack or Community Page. Thank you!