lldb sos error: You can run the debugger command 'setclrpath <directory>' to control the load of libmscordaccore.dylib.

[bruno-garcia]

Description

I'm trying to use lldb with sos to debug a minidump created with crashpad. Everything (build, crash and debug the app) is happening on the same machine (macOS 10.15.7).

I followed the steps in the docs in this repo to install sos and then to configure it and debug the dump.
Everything went well, including (lldb) loadsymbols (at least didn't error out).

➜  Sentry.Minidump.Sample git:(main) ✗ lldb --core dotnet-minidump.dmp bin/Debug/net5.0/osx-x64/Sentry.Minidump.Sample
Added Microsoft public symbol server
(lldb) target create "bin/Debug/net5.0/osx-x64/Sentry.Minidump.Sample" --core "dotnet-minidump.dmp"
Core file '/Users/bruno/git/sentry-dotnet-minidump/sample/Sentry.Minidump.Sample/dotnet-minidump.dmp' (x86_64) was loaded.

(lldb) setsymbolserver -directory /Users/bruno/git/sentry-dotnet-minidump/sample/Sentry.Minidump.Sample/bin/Debug/net5.0/osx-x64
Added symbol directory path: /Users/bruno/git/sentry-dotnet-minidump/sample/Sentry.Minidump.Sample/bin/Debug/net5.0/osx-x64

But then any other sos command would fail:

(lldb) dumpdomain
Failed to load data access module, 0x80131c4f
You can run the debugger command 'setclrpath <directory>' to control the load of libmscordaccore.dylib.
If that succeeds, the SOS command should work on retry.

For more information see https://go.microsoft.com/fwlink/?linkid=2135652
DumpDomain  failed

Same for clrthreads and others.

I can confirm I have it:

 find /usr/local/share/dotnet/shared/Microsoft.NETCore.App | grep libmscordaccore
/usr/local/share/dotnet/shared/Microsoft.NETCore.App/5.0.0-rc.1.20451.14/libmscordaccore.dylib
/usr/local/share/dotnet/shared/Microsoft.NETCore.App/2.1.22/libmscordaccore.dylib
/usr/local/share/dotnet/shared/Microsoft.NETCore.App/5.0.1/libmscordaccore.dylib
/usr/local/share/dotnet/shared/Microsoft.NETCore.App/2.1.23/libmscordaccore.dylib
/usr/local/share/dotnet/shared/Microsoft.NETCore.App/5.0.0/libmscordaccore.dylib
/usr/local/share/dotnet/shared/Microsoft.NETCore.App/3.1.8/libmscordaccore.dylib
/usr/local/share/dotnet/shared/Microsoft.NETCore.App/3.1.9/libmscordaccore.dylib
/usr/local/share/dotnet/shared/Microsoft.NETCore.App/3.1.10/libmscordaccore.dylib

The minidump was processed by Sentry without problems:

With the stack trace of 11 threads:

i.e:

Repro

The dump:
sentry-dotnet-minidump-issue.dmp.zip

The source code is in this repo: https://github.com/getsentry/sentry-dotnet-minidump

Similar to #414

Configuration

dotnet-sos --version
5.0.160202+5734230e3ee516339a4b0e4729def135027aa255

`dotnet info`

``` ➜ osx-x64 git:(main) ✗ dotnet --info .NET SDK (reflecting any global.json): Version: 5.0.101 Commit: d05174dc5a

Runtime Environment:
OS Name: Mac OS X
OS Version: 10.15
OS Platform: Darwin
RID: osx.10.15-x64
Base Path: /usr/local/share/dotnet/sdk/5.0.101/

Host (useful for support):
Version: 5.0.1
Commit: b02e13abab

.NET SDKs installed:
2.1.810 [/usr/local/share/dotnet/sdk]
3.1.402 [/usr/local/share/dotnet/sdk]
3.1.403 [/usr/local/share/dotnet/sdk]
3.1.404 [/usr/local/share/dotnet/sdk]
5.0.100-rc.1.20452.10 [/usr/local/share/dotnet/sdk]
5.0.100 [/usr/local/share/dotnet/sdk]
5.0.101 [/usr/local/share/dotnet/sdk]

.NET runtimes installed:
Microsoft.AspNetCore.All 2.1.22 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.22 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.8 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.9 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.10 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.0-rc.1.20451.17 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.0 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.1 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.22 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.23 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.8 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.9 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.10 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.0-rc.1.20451.14 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.0 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.1 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET runtimes or SDKs:
https://aka.ms/dotnet-download

</details>

<!--
* Is this related to a specific tool?
* What OS and version, and what distro if applicable?
* What is the architecture (x64, x86, ARM, ARM64)?
* Do you know whether it is specific to that configuration?
* Are you running in any particular type of environment? (e.g. Containers, a cloud scenario, app you are trying to target is a different user)
* Is it a self-contained published application?
* What's the output of `dotnet info`
  -->

[bruno-garcia]

I tried loading it with ClrMD and it also complaied about libmscordaccore.dylib. Passing a path explicitly fails since it can't figure out the CLR version.

Found CLR Version: 0.0.0.0
Unhandled exception. System.InvalidOperationException: Mismatched dac. Dac version: 5.0.20.51904, expected: 0.0.0.0.
   at Microsoft.Diagnostics.Runtime.ClrInfo.CreateRuntime(String dacPath, Boolean ignoreMismatch)

I wonder if crashpad just doesn't include something needed here?

[mikem8361]

Could you run dbgout in lldb before running the SOS command (i.e. dumpdomain, clrthreads, etc.)? It may give us a clue what is missing from the core dump.

This coredump was created by crashpad you linked into the runtime? Does it use the DAC EnumMemoryRegions to add memory to the dump? createdump uses it here. If the dump isn't a full dump it is probably missing the managed state SOS requires. I'm curious why you don't use/enable createdump described here?

[bruno-garcia]

crashpad created the minidump from a separate process (crashpad_handler). I built sentry-native which links crashpad and starts the handler when the app starts. I didn't link sentry-native (or crashpad) with the runtime though, I P/Invoked into the native init method of the native library which linked crashpad.

Does it use the DAC EnumMemoryRegions to add memory to the dump?

I doubt it does, I'll dig into it further to learn how that works.

If the dump isn't a full dump it is probably missing the managed state SOS requires.

Sounds very much like that's the case.

I'm curious why you don't use/enable createdump described here?

My goal was to provide a NuGet package with a managed API to initialize the crash handler and generate the minidump when the process is crashing. And have that minidump get uploaded to a crash reporting service. All of that is built into sentry-native (one of the backends is crashpad which I used here). I wasn't aware that some special memory regions would need to be included and that crashpad wouldn't be doing that (as we suspect to be the issue here).

Would you recommend I use createdump as a library somehow? Is there a simple enough way to trigger a crash dump from outer process (I assume the process can't dump itself in case it's corrupted. Quoting @noahfalk here:

However there would be a different issue in the scenario you described, MiniDumpWriteDump is not designed to take a dump of the same process it is executing within.

(lldb) dbgout
Debug output logging enabled
(lldb) dumpdomain
DataTarget::ReadVirtual FAILED 80004005 address 100ba0c08 size 000002ac
Failed to load data access module, 0x80131c4f
You can run the debugger command 'setclrpath <directory>' to control the load of libmscordaccore.dylib.
If that succeeds, the SOS command should work on retry.

For more information see https://go.microsoft.com/fwlink/?linkid=2135652
DumpDomain  failed

Thanks for the pointers.

[mikem8361]

Yes, it looks like the core dump is missing some memory that SOS requires.

No, createdump can't be used "in-process" as is. It uses ptrace to suspend/resume and get the registers which wouldn't work in-proc without a lot of work.

[bruno-garcia]

@mikem8361 I'm not planning to create a dump from the running process. I mentioned to use as a library so I can write more code in the handler (upload to the backend).

I see the runtime actually runs:

https://github.com/dotnet/runtime/blob/262d44875080ff7421b314f7c6f580a92e471701/src/coreclr/pal/src/thread/process.cpp#L3073-L3155

Might be easier to replicate this than to patch crashpad to include DAC EnumMemoryRegions.

[mikem8361]

exec'ing createdump and passing it the pid of the process to create dump would work. You wouldn't really need crashpad anymore.

[bruno-garcia]

Thanks again @mikem8361