Craspad under Epic's "Easy Anti-Cheat"

[kristjanvalur]

Description

We've been running into problem with our "crashpad" backend under epic's easy anti-cheat client protection software.
The solution places some restrictions on the running process, causing the windows version of crashpad to fail
to initialize.

We have consulted with Epic, and they provided us with a diff to "crashpad" which has been employed by other
users. It basically limits the permission of created processes and promotes a process when necessary.

We would have added this directly to our own fork of sentry-native, or provided a PR, but I noticed that "crashpad" is a submodule maintained by you.

Would you be interested in a PR to "getsentry/crashpad" ?

When does the problem happen

• During build
• During run-time
• When capturing a hard crash

Environment

• OS: Windows

Steps To Reproduce

Run an executable under the "Easy Anti Cheat" supervisor, notice how sentry reports that it "failed to launch crashpad handler."

[Swatinem]

Integrating changes directly into our fork of crashpad is a good idea. Otherwise, I’m surprised that this did never come up upstream, as I believe there must be other games that use upstream crashpad instead of our fork.
Either way, fixing this upstream or in our fork is definitely a good idea.

It basically limits the permission of created processes and promotes a process when necessary.

Do you have any more details here? Does it restrict the operation of crashpad, or the possibility / fidelity of the collected minidump in any way?

[kristjanvalur]

I guess people aren't using crashpad much, but rely on breakpad. This is the default, for example, for your own UE plugin (we are still using our own, which was used as a model for yours, I believe). We want this to get immediate results from CI environments etc.
I'll attach the diff here, you can see for yourself.
crashpad.diff.txt

[kristjanvalur]

Basically, it optains a limited handle to the parent process, until such time as it has to examine its stack, when it can promote the handle. The watchdog process prohibits memory access to the process until and if it crashes, at which time it will produce a short time window where the crash handler can access the process memory.
There is also a provision there to handle a failure to suspend a thread (for some reason which I did not analyze in depth).

More info on EAC and crash handlers is here:
https://dev.epicgames.com/docs/game-services/anti-cheat/using-anti-cheat#external-crash-dumpers

[supervacuus]

I'll attach the diff here, you can see for yourself.
crashpad.diff.txt

Thanks. That is a sensible approach; I could imagine merging something like this. We could be even more defensive and open the process with limited access only after we fail with full access. But I like limiting access as long as we don't need it.

There is also a provision there to handle a failure to suspend a thread (for some reason which I did not analyze in depth).

From the docs you linked, the EAC watchdog/supervisor has already suspended threads in the application. In that case, the suspension from the crashpad_handler would fail. So they bypass the error path for threads they already suspended so that the crashpad_handler can take the context snapshots.

[mq1n]

After upgrading from version 0.7.2 to 0.7.5, minidumps are no longer created when a crash occurs. Could this be due to the changes made in this update? I was only able to access the following log messages:

[51864] [51864:38292:20240531,160028.162:ERROR scoped_process_suspend.cc:31] NtSuspendProcess: {Access Denied} A process has requested access to an object, but has not been granted those access rights. (0xc0000022)

[51864] [51864:38292:20240531,160028.163:ERROR process_info.cc:119] ReadProcessMemory bool __cdecl crashpad::`anonymous-namespace'::ReadStruct<struct crashpad::process_types::PEB<struct crashpad::process_types::internal::Traits32>>(void *,unsigned __int64,struct crashpad::process_types::PEB<struct crashpad::process_types::internal::Traits32> *): Access is denied. (5)

[51864] [51864:38292:20240531,160028.163:ERROR process_info.cc:552] ReadProcessData failed

[supervacuus]

After upgrading from version 0.7.2 to 0.7.5, minidumps are no longer created when a crash occurs. Could this be due to the changes made in this update?

It might. You have missing memory access rights. None of our tests have revealed a regression yet, but we may have ignored a path.

Can you create a separate issue where you can reference this one but provide more details on your setup, especially the Windows + SDK version, how you use the Native SDK, and what kind of crash you are provoking? There may be a dump path that is not covered by the deferred access promotion.

Can you reproduce the failure to crash with the sentry_example?