Metricbeat fails to get information for protected processes under Windows

[adriansr]

Issue

For confirmed bugs, please report:

• Version: 7.x
• Operating System: Windows
• Discuss Forum URL:
• Steps to Reproduce:

When running the system metricset, the debug log is filled with messages like:

process/process.go:475  Skip process pid=0: error getting process state for pid=0: getProcName failed: OpenProcess failed for pid=0: The parameter is incorrect.; getProcStatus failed: OpenProcess failed for pid=0: The parameter is incorrect.; getParentPid failed: OpenProcess failed for pid=0: The parameter is incorrect.
2019-11-20T15:38:58.734Z        DEBUG   [processes]     process/process.go:475  Skip process pid=4: error getting process state for pid=4: getProcName failed: GetProcessImageFileName failed for pid=4: GetProcessImageFileName failed: invalid argument
2019-11-20T15:38:58.736Z        DEBUG   [processes]     process/process.go:486  Error getting details for process Registry with pid=120: error getting process mem for pid=120: OpenProcess failed for pid=120: Access is denied.
2019-11-20T15:38:58.736Z        DEBUG   [processes]     process/process.go:486  Error getting details for process smss.exe with pid=464: error getting process mem for pid=464: OpenProcess failed for pid=464: Access is denied.
2019-11-20T15:38:58.737Z        DEBUG   [processes]     process/process.go:486  Error getting details for process csrss.exe with pid=700: error getting process mem for pid=700: OpenProcess failed for pid=700: Access is denied.
2019-11-20T15:38:58.737Z        DEBUG   [processes]     process/process.go:486  Error getting details for process wininit.exe with pid=812: error getting process mem for pid=812: OpenProcess failed for pid=812: Access is denied.
2019-11-20T15:38:58.737Z        DEBUG   [processes]     process/process.go:486  Error getting details for process csrss.exe with pid=820: error getting process mem for pid=820: OpenProcess failed for pid=820: Access is denied.
2019-11-20T15:38:58.737Z        DEBUG   [processes]     process/process.go:486  Error getting details for process services.exe with pid=884: error getting process mem for pid=884: OpenProcess failed for pid=884: Access is denied.
2019-11-20T15:38:58.752Z        DEBUG   [processes]     process/process.go:486  Error getting details for process MemCompression with pid=3648: error getting process mem for pid=3648: OpenProcess failed for pid=3648: Access is denied.
2019-11-20T15:38:58.755Z        DEBUG   [processes]     process/process.go:486  Error getting details for process svchost.exe with pid=4412: error getting process mem for pid=4412: OpenProcess failed for pid=4412: Access is denied.
2019-11-20T15:38:58.764Z        DEBUG   [processes]     process/process.go:486  Error getting details for process MsMpEng.exe with pid=5708: error getting process mem for pid=5708: OpenProcess failed for pid=5708: Access is denied.
2019-11-20T15:38:58.773Z        DEBUG   [processes]     process/process.go:486  Error getting details for process NisSrv.exe with pid=9004: error getting process mem for pid=9004: OpenProcess failed for pid=9004: Access is denied.
2019-11-20T15:38:58.780Z        DEBUG   [processes]     process/process.go:486  Error getting details for process SecurityHealthService.exe with pid=12232: error getting process mem for pid=12232: OpenProcess failed for pid=12232: Access is denied.
2019-11-20T15:38:58.792Z        DEBUG   [processes]     process/process.go:486  Error getting details for process SgrmBroker.exe with pid=17156: error getting process mem for pid=17156: OpenProcess failed for pid=17156: Access is denied.
2019-11-20T15:38:58.793Z        DEBUG   [processes]     process/process.go:486  Error getting details for process svchost.exe with pid=11332: error getting process mem for pid=11332: OpenProcess failed for pid=11332: Access is denied.
2019-11-20T15:38:58.796Z        DEBUG   [processes]     process/process.go:486  Error getting details for process svchost.exe with pid=13536: error getting process mem for pid=13536: OpenProcess failed for pid=13536: Access is denied.
2019-11-20T15:38:58.797Z        DEBUG   [processes]     process/process.go:486  Error getting details for process init with pid=2828: error getting process mem for pid=2828: OpenProcess failed for pid=2828: Access is denied.
2019-11-20T15:38:58.799Z        DEBUG   [processes]     process/process.go:486  Error getting details for process init with pid=16416: error getting process mem for pid=16416: OpenProcess failed for pid=16416: Access is denied.
2019-11-20T15:38:58.799Z        DEBUG   [processes]     process/process.go:486  Error getting details for process bash with pid=15504: error getting process mem for pid=15504: OpenProcess failed for pid=15504: Access is denied.
2019-11-20T15:38:58.814Z        DEBUG   [processes]     process/process.go:434  Filtered top 

We would like to understand if there are some permissions missing for Metricbeat.exe that could allow to fetch information from all or most of the processes that are currently failing. It would be good to compare with Process Explorer which can list this information for all processes.

Definition of done

• Metricbeat gets information for all processors
• Create a followup task to create tests when Windows runner will be available

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[blakerouse]

@adriansr How was metricbeat started on Windows? Is it running as a service or is it being executed directly? If being executed directly, can you provide how you are starting it? Administrator cmd? Administrator powershell?

[willemdh]

We have the same problem. Multiple processes are not getting indexed...

2021-10-26T16:12:17.134+0200	DEBUG	[processes]	process/process.go:577	Error getting details for process Registry with pid=100: error getting process mem for pid=100: OpenProcess failed for pid=100: Access is denied.
2021-10-26T16:12:17.134+0200	DEBUG	[processes]	process/process.go:577	Error getting details for process smss.exe with pid=292: error getting process mem for pid=292: OpenProcess failed for pid=292: Access is denied.
2021-10-26T16:12:17.134+0200	DEBUG	[processes]	process/process.go:577	Error getting details for process csrss.exe with pid=396: error getting process mem for pid=396: OpenProcess failed for pid=396: Access is denied.
2021-10-26T16:12:17.135+0200	DEBUG	[processes]	process/process.go:577	Error getting details for process csrss.exe with pid=504: error getting process mem for pid=504: OpenProcess failed for pid=504: Access is denied.
2021-10-26T16:12:17.135+0200	DEBUG	[processes]	process/process.go:577	Error getting details for process wininit.exe with pid=552: error getting process mem for pid=552: OpenProcess failed for pid=552: Access is denied.
2021-10-26T16:12:17.136+0200	DEBUG	[processes]	process/process.go:577	Error getting details for process services.exe with pid=636: error getting process mem for pid=636: OpenProcess failed for pid=636: Access is denied.
2021-10-26T16:12:17.164+0200	DEBUG	[processes]	process/process.go:577	Error getting details for process cyserver.exe with pid=1604: error getting process mem for pid=1604: OpenProcess failed for pid=1604: Access is denied.
2021-10-26T16:12:17.164+0200	DEBUG	[processes]	process/process.go:577	Error getting details for process svchost.exe with pid=4284: error getting process mem for pid=4284: OpenProcess failed for pid=4284: Access is denied.
2021-10-26T16:12:17.170+0200	DEBUG	[processes]	process/process.go:577	Error getting details for process csrss.exe with pid=6544: error getting process mem for pid=6544: OpenProcess failed for pid=6544: Access is denied.
2021-10-26T16:12:17.182+0200	DEBUG	[processes]	process/process.go:577	Error getting details for process csrss.exe with pid=7148: error getting process mem for pid=7148: OpenProcess failed for pid=7148: Access is denied.

[narph]

In short, we are using standard user-mode Windows 32 APIs to query information on process internals, and we are unable to perform operations on protected processes due to their higher level of security.

Some references below offer detailed information on this, the access rights we use to query information (OpenProcess 32 api more precisely) look to be denied for protected processes.

https://www.microsoftpressstore.com/articles/article.aspx?p=2233328&seqNum=2
https://docs.microsoft.com/en-us/windows/win32/procthread/process-security-and-access-rights#protected-processes
https://docs.microsoft.com/en-us/windows/win32/procthread/process-security-and-access-rights

We are also using the PROCESS_QUERY_LIMITED_INFORMATION flag, that should grant limited access to information on the process but, in this case, the information we require from the process is still not accessible through this option.

It would be good to compare with Process Explorer which can list this information for all processes.

Process Explorer also uses standard user-mode Windows APIs to query information so indeed, it would be interesting if it can collect the exact process information we are not able to.
We will have to investigate this scenario and confirm we are not able to provide certain metrics for protected process.

[willemdh]

Thanks @narph for the info. It would be Nice if we would be able to Get cpu and memory info From protected processen to Get a complete picture imho.

[andrewkroh]

Is Metricbeat using go-sysinfo, gosigar, or some its own code to fetch process info?

Metricbeat can probably be made more robust when encountering protected processes. Ideally it would fallback back to using PROCESS_QUERY_LIMITED_INFORMATION to open the process. And then fetch as much information as is available (based on docs linked by narph I think this is only the path to the executable).

I looked over the go-sysinfo library and made some notes.

1. It will fallback to PROCESS_QUERY_LIMITED_INFORMATION. https://github.com/elastic/go-sysinfo/blob/504d69c91710df28aa7fa7cc5d1a624d9e918126/providers/windows/process_windows.go#L268-L280
2. It ignores PID 0 and 4 (which are never accessible) to limit error logging. https://github.com/elastic/go-sysinfo/blob/504d69c91710df28aa7fa7cc5d1a624d9e918126/providers/windows/process_windows.go#L49-L51
3. It should probably ignore access denied errors when calling GetProcessTimes because PROCESS_QUERY_LIMITED_INFORMATION is not listed to provide this info. https://github.com/elastic/go-sysinfo/blob/504d69c91710df28aa7fa7cc5d1a624d9e918126/providers/windows/process_windows.go#L114-L117

[andrewkroh]

Is Metricbeat using go-sysinfo, gosigar, or some its own code to fetch process info?

Metricbeat is using the libbeat/metric code that uses gosigar.

beats/libbeat/metric/system/process/process.go

Lines 110 to 115 in da8bc7c

	// newProcess creates a new Process object and initializes it with process
	// state information. If the process's command line and environment variables
	// are known they should be passed in to avoid re-fetching the information.
	func newProcess(pid int, cmdline string, env common.MapStr) (*Process, error) {
	state := sigar.ProcState{}
	err := state.Get(pid)

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[willemdh]

+1

[gabriellandau]

Metricbeat is using the libbeat/metric code that uses gosigar.

Thanks for that. I see two things we can fix.

1. ProcMem::Get() is requesting PROCESS_VM_READ, which will definitely fail on PPL Endpoint. It's passed to GetProcessMemoryInfo here. That API only needs PROCESS_QUERY_LIMITED_INFORMATION on OS versions that Endpoint supports, so we can get rid of PROCESS_VM_READ

2. getProcCredName() here is requesting PROCESS_QUERY_INFORMATION when we really only need PROCESS_QUERY_LIMITED_INFORMATION.

ProcArgs::Get() here is requesting PROCESS_VM_READ. That will not succeed against PPL Endpoint, and there's no quick workaround. We need to ensure that such failures are non-fatal, and that MetricBeat still returns the data it can successfully collect for the Endpoint process.

[willemdh]

Thanks @gabriellandau

This week we had an issue with lsass.exe on a system and we missed it completely as Metricbeat cannot capture lsass.exe cpu usage...

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[nfritts]

FYI I've seen another customer come in with questions about why they aren't getting metric data about Endpoint that would have been solved/prevented by resolving this issue.

[rdner]

The error message from the description is not logged anymore due to removal in #30076 (perhaps by mistake). We should put it back while fixing this issue.

[rdner]

It's better to ask @pierrehilbert about estimations.

[belimawr]

After some investigation and discussion with the team, my current plan for this task is:

1. Collect metrics in a different order, grouping all metrics that require the same access right, starting with the lower ones
2. Returning partial metrics in case some of them fail. My understanding is that on Windows we don't have enough permissions to read all metrics from all process.
3. Errors will be added to the event's error field so it's easy to spot why metrics are missing.

[cmacknz]

Reopening, let's not close this until the system metrics dependency is updated in Beats and Agent.