Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix corrupted mini-dumps on Windows that VS can not load #13489

Closed
pr8x opened this issue Sep 26, 2019 · 16 comments
Closed

Fix corrupted mini-dumps on Windows that VS can not load #13489

pr8x opened this issue Sep 26, 2019 · 16 comments
Assignees
Milestone

Comments

@pr8x
Copy link

pr8x commented Sep 26, 2019

I am hosting a dotnet module inside my native C++ application and use google breakpad to generate minidumps. I noticed that only MiniDumpWithFullMemory seems to be useful for getting proper managed stacks. Some of these dumps can get quite big however. I was wondering if you guys have some clue on reducing the size a bit. Exporting "every readable page in the process" seems quite extreme.
http://www.debuginfo.com/tools/clrdump.html claims to reduce the size of .NET dumps while maintaining relevant stack/thread information. It's not open-source AFAICT and seems to be targeting .NET Framework (not .NET Core!). Apparently he's using (according to https://stackoverflow.com/a/8258911/2354690):

MiniDumpNormal
MiniDumpWithProcessThreadData
MiniDumpWithThreadInfo
MiniDumpWithUnloadedModules

They don't seem to work for me. Any ideas?

@scalablecory scalablecory transferred this issue from dotnet/core Sep 26, 2019
@noahfalk
Copy link
Member

@mikem8361, can you help here?

@noahfalk
Copy link
Member

noahfalk commented Oct 1, 2019

Ping @mikem8361 : )

In the meantime here are some things that may help:
env vars to take (small) dumps on crash: https://github.com/dotnet/coreclr/blob/master/Documentation/botr/xplat-minidump-generation.md#configurationpolicy
a tool that can take small dumps: https://github.com/dotnet/diagnostics/blob/master/documentation/dotnet-dump-instructions.md
The source in the dotnet-dump tool which calls MiniDumpWriteDump:
https://github.com/dotnet/diagnostics/blob/master/src/Tools/dotnet-dump/Dumper.Windows.cs#L27

@pr8x
Copy link
Author

pr8x commented Oct 1, 2019

Thanks. Didn't investigate dotnet-dump so far. I am going to try the flags they use.

@mikem8361
Copy link
Member

You may not need all the flags that dotnet-dump collect uses. I need to look into it because windbg/cdb's .dump /mw <dump_name> generates a "heap" dump and it uses just MiniDumpWithTokenInformation + MiniDumpWithPrivateReadWriteMemory (0x40200).

@pr8x
Copy link
Author

pr8x commented Oct 2, 2019

I tried using the flags:

 MiniDumpWithDataSegs |
        MiniDumpWithPrivateReadWriteMemory |
        MiniDumpWithHandleData |
        MiniDumpWithUnloadedModules |
        MiniDumpWithFullMemoryInfo |
        MiniDumpWithThreadInfo |
        MiniDumpWithTokenInformation

But I keep getting a symbol error when trying to "Debug with mixed". Oddly, I don't get this when I generate with full memory.
image

It seems like it's failing to retrieve System.Runtime.dll symbols from the symbol server.

SYMSRV: HTTPGET: /download/symbols/System.Runtime.dll/EB15006C/System.Runtime.dll
SYMSRV: HttpQueryInfo: 80190194 - HTTP_STATUS_NOT_FOUND

Could be an issue with VS, but it seems to be somehow related to the minidump flags. How come it successfully resolves symbols for fully memory dumps?

Edit: Okay I narrowed down the issue a bit. For most flag combinations (except MiniDumpWithFullMemory) it seems to be segfaulting (in mscordaccore.dll) during the call to MiniDumpWriteDump. Not sure if this is a breakpad issue or some other problem. Either way this is probably the cause why it fails to debug the dump.

mscordaccore.dll!TypeDesc::GetName(SString & ssBuf) Line 234 C++
mscordaccore.dll!TypeHandle::GetName(SString & result) Line 756 C++
mscordaccore.dll!SigFormat::AddType(TypeHandle th) Line 570 C++
mscordaccore.dll!SigFormat::AddTypeString(Module * pModule, SigPointer sig, const SigTypeContext * pTypeContext) Line 347 C++
mscordaccore.dll!SigFormat::AddTypeString(Module * pModule, SigPointer sig, const SigTypeContext * pTypeContext) Line 277 C++
mscordaccore.dll!SigFormat::FormatSig(MetaSig & sig, const char * szMemberName, const char *) Line 367 C++
mscordaccore.dll!SigFormat::SigFormat(MethodDesc * pMeth, TypeHandle owner, int) Line 51 C++
mscordaccore.dll!TypeString::AppendMethodImpl(SString & ss, MethodDesc * pMD, Instantiation typeInstantiation, const unsigned long format) Line 1099 C++
[Inline Frame] mscordaccore.dll!TypeString::AppendMethodInternal(SString &) Line 1046 C++
mscordaccore.dll!MethodDesc::EnumMemoryRegions(CLRDataEnumMemoryFlags flags) Line 5452 C++
mscordaccore.dll!Module::EnumMemoryRegions(CLRDataEnumMemoryFlags flags, bool enumThis) Line 13929 C++
mscordaccore.dll!EditAndContinueModule::EnumMemoryRegions(CLRDataEnumMemoryFlags flags, bool enumThis) Line 1583 C++
mscordaccore.dll!EEClass::EnumMemoryRegions(CLRDataEnumMemoryFlags flags, MethodTable * pMT) Line 3272 C++
mscordaccore.dll!MethodTable::EnumMemoryRegions(CLRDataEnumMemoryFlags flags) Line 9606 C++
mscordaccore.dll!MethodTable::EnumMemoryRegions(CLRDataEnumMemoryFlags flags) Line 9610 C++
mscordaccore.dll!MethodDescChunk::EnumMemoryRegions(CLRDataEnumMemoryFlags flags) Line 5528 C++
mscordaccore.dll!MethodDesc::EnumMemoryRegions(CLRDataEnumMemoryFlags flags) Line 5436 C++
[Inline Frame] mscordaccore.dll!InstMethodHashTable::EnumMemoryRegionsForEntry(InstMethodHashEntry * pEntry, CLRDataEnumMemoryFlags) Line 446 C++
mscordaccore.dll!NgenHashTable<InstMethodHashTable,InstMethodHashEntry,4>::BaseEnumMemoryRegions(CLRDataEnumMemoryFlags flags) Line 1095 C++
[Inline Frame] mscordaccore.dll!InstMethodHashTable::EnumMemoryRegions(CLRDataEnumMemoryFlags) Line 438 C++
mscordaccore.dll!Module::EnumMemoryRegions(CLRDataEnumMemoryFlags flags, bool enumThis) Line 13875 C++
mscordaccore.dll!EditAndContinueModule::EnumMemoryRegions(CLRDataEnumMemoryFlags flags, bool enumThis) Line 1583 C++
mscordaccore.dll!EEClass::EnumMemoryRegions(CLRDataEnumMemoryFlags flags, MethodTable * pMT) Line 3272 C++
mscordaccore.dll!MethodTable::EnumMemoryRegions(CLRDataEnumMemoryFlags flags) Line 9606 C++
mscordaccore.dll!MethodTable::EnumMemoryRegions(CLRDataEnumMemoryFlags flags) Line 9610 C++
mscordaccore.dll!Module::EnumMemoryRegions(CLRDataEnumMemoryFlags flags, bool enumThis) Line 13910 C++
mscordaccore.dll!EditAndContinueModule::EnumMemoryRegions(CLRDataEnumMemoryFlags flags, bool enumThis) Line 1583 C++
mscordaccore.dll!EEClass::EnumMemoryRegions(CLRDataEnumMemoryFlags flags, MethodTable * pMT) Line 3272 C++
mscordaccore.dll!MethodTable::EnumMemoryRegions(CLRDataEnumMemoryFlags flags) Line 9606 C++
mscordaccore.dll!TypeHandle::EnumMemoryRegions(CLRDataEnumMemoryFlags flags) Line 1469 C++
mscordaccore.dll!Module::EnumMemoryRegions(CLRDataEnumMemoryFlags flags, bool enumThis) Line 13920 C++
mscordaccore.dll!EditAndContinueModule::EnumMemoryRegions(CLRDataEnumMemoryFlags flags, bool enumThis) Line 1583 C++
mscordaccore.dll!ClassLoader::EnumMemoryRegions(CLRDataEnumMemoryFlags flags) Line 5641 C++
mscordaccore.dll!Assembly::EnumMemoryRegions(CLRDataEnumMemoryFlags flags) Line 2365 C++
mscordaccore.dll!DomainAssembly::EnumMemoryRegions(CLRDataEnumMemoryFlags flags) Line 2476 C++
mscordaccore.dll!AppDomain::EnumMemoryRegions(CLRDataEnumMemoryFlags flags, bool enumThis) Line 6878 C++
mscordaccore.dll!Assembly::EnumMemoryRegions(CLRDataEnumMemoryFlags flags) Line 2361 C++
mscordaccore.dll!DomainAssembly::EnumMemoryRegions(CLRDataEnumMemoryFlags flags) Line 2476 C++
mscordaccore.dll!DomainLocalModule::EnumMemoryRegions(CLRDataEnumMemoryFlags flags) Line 6820 C++
mscordaccore.dll!Module::EnumMemoryRegions(CLRDataEnumMemoryFlags flags, bool enumThis) Line 13849 C++
mscordaccore.dll!EEClass::EnumMemoryRegions(CLRDataEnumMemoryFlags flags, MethodTable * pMT) Line 3272 C++
mscordaccore.dll!MethodTable::EnumMemoryRegions(CLRDataEnumMemoryFlags flags) Line 9606 C++
mscordaccore.dll!ParamTypeDesc::EnumMemoryRegions(CLRDataEnumMemoryFlags flags) Line 2321 C++
mscordaccore.dll!ClrDataAccess::EnumMemCLRStatic(CLRDataEnumMemoryFlags flags) Line 288 C++
mscordaccore.dll!ClrDataAccess::EnumMemoryRegionsWorkerHeap(CLRDataEnumMemoryFlags flags) Line 325 C++
mscordaccore.dll!ClrDataAccess::EnumMemoryRegionsWrapper(CLRDataEnumMemoryFlags flags) Line 1831 C++
mscordaccore.dll!ClrDataAccess::EnumMemoryRegions(ICLRDataEnumMemoryRegionsCallback * callback, unsigned int miniDumpFlags, CLRDataEnumMemoryFlags flags) Line 1935 C++
dbgcore.dll!GenGetProcessInfo(unsigned long,struct _MINIDUMP_STATE *,struct _INTERNAL_PROCESS * *,struct _LIST_ENTRY *) Unknown
dbgcore.dll!MiniDumpProvideDump�() Unknown
dbgcore.dll!MiniDumpWriteDump�() Unknown

@mikem8361
Copy link
Member

I'm currently investigating this problem. Sorry for the delay.

@mikem8361
Copy link
Member

mikem8361 commented Oct 8, 2019 via email

@mikem8361 mikem8361 changed the title Reducing the size of crash dumps (Windows) Fixed corrupted mini-dumps on Windows that VS can not load Oct 9, 2019
@pr8x pr8x changed the title Fixed corrupted mini-dumps on Windows that VS can not load Fix corrupted mini-dumps on Windows that VS can not load Oct 10, 2019
@pr8x
Copy link
Author

pr8x commented Oct 10, 2019

I changed the title. It sounded like I fixed them 😄

@pr8x
Copy link
Author

pr8x commented Oct 30, 2019

Hey @mikem8361, did you figure out what is causing the issue?

@mikem8361
Copy link
Member

mikem8361 commented Oct 30, 2019 via email

mikem8361 referenced this issue in mikem8361/coreclr Oct 30, 2019
Fix a function that was ifdef'ed needed for the metadata locator callbacks to work.

Fix some not properly DAC'ized code in the type desc and server GC code. Caused an exception during dump generation.

Issue: https://github.com/dotnet/coreclr/issues/26907
mikem8361 referenced this issue in dotnet/coreclr Oct 30, 2019
Fix a function that was ifdef'ed needed for the metadata locator callbacks to work.

Fix some not properly DAC'ized code in the type desc and server GC code. Caused an exception during dump generation.

Issue: https://github.com/dotnet/coreclr/issues/26907
@pr8x
Copy link
Author

pr8x commented Oct 31, 2019

That's great news! Do you have any tracking ticket for the Visual Studio update?

@pr8x
Copy link
Author

pr8x commented Nov 7, 2019

Hi, Sorry for the bump, but do you happen to know if there's a release candidate for Visual Studio that fixes the issue? I am currently running VS2019 v16.3.7 and the problem still exists. I created a small repro here: https://github.com/pr8x/netcore31_bug_minidump

@mikem8361
Copy link
Member

It looks like the VS fixes are going into 16.5 preview 1. @chuckries can answer for sure. You will also need .NET Core 3.1 coming out in December for minidumps to work end-to-end.

@mikem8361
Copy link
Member

This issue is just waiting for @chuckries to confirm the VS version.

@pr8x
Copy link
Author

pr8x commented Dec 13, 2019

Hey, sorry to revive this issue. But it seems like that with .NET Core 3.1 and Visual Studio 16.4.0 it is still crashing while generating the dump. This time it's a couple of Microsoft C++ HRException thrown within the DAC (Flags used: MiniDumpWithTokenInformation | MiniDumpWithPrivateReadWriteMemory):

KernelBase.dll!RaiseException�()
mscordaccore.dll!_CxxThrowException(void * pExceptionObject, const _s__ThrowInfo * pThrowInfo) Line 133
at d:\agent_work\2\s\src\vctools\crt\vcruntime\src\eh\throw.cpp(133)
mscordaccore.dll!DacError(HRESULT err) Line 118
at f:\workspace_work\1\s\src\debug\daccess\dacfn.cpp(118)
mscordaccore.dll!DacGetThreadContext(Thread * thread, _CONTEXT * context) Line 95
at f:\workspace_work\1\s\src\debug\daccess\reimpl.cpp(95)
mscordaccore.dll!Thread::EnumMemoryRegionsWorker(CLRDataEnumMemoryFlags flags) Line 8780
at f:\workspace_work\1\s\src\vm\threads.cpp(8780)
mscordaccore.dll!Thread::EnumMemoryRegions(CLRDataEnumMemoryFlags flags) Line 8743
at f:\workspace_work\1\s\src\vm\threads.cpp(8743)
mscordaccore.dll!ThreadStore::EnumMemoryRegions(CLRDataEnumMemoryFlags flags) Line 8915
at f:\workspace_work\1\s\src\vm\threads.cpp(8915)
mscordaccore.dll!ClrDataAccess::EnumMemDumpAllThreadsStack(CLRDataEnumMemoryFlags flags) Line 1173
at f:\workspace_work\1\s\src\debug\daccess\enummem.cpp(1173)
mscordaccore.dll!ClrDataAccess::EnumMemoryRegionsWorkerHeap(CLRDataEnumMemoryFlags flags) Line 337
at f:\workspace_work\1\s\src\debug\daccess\enummem.cpp(337)
mscordaccore.dll!ClrDataAccess::EnumMemoryRegionsWrapper(CLRDataEnumMemoryFlags flags) Line 1831
at f:\workspace_work\1\s\src\debug\daccess\enummem.cpp(1831)
mscordaccore.dll!ClrDataAccess::EnumMemoryRegions(ICLRDataEnumMemoryRegionsCallback * callback, unsigned int miniDumpFlags, CLRDataEnumMemoryFlags flags) Line 1935
at f:\workspace_work\1\s\src\debug\daccess\enummem.cpp(1935)
dbgcore.dll!GenGetProcessInfo(unsigned long,struct _MINIDUMP_STATE *,struct _INTERNAL_PROCESS * *,struct _LIST_ENTRY *)
dbgcore.dll!MiniDumpProvideDump�()
dbgcore.dll!MiniDumpWriteDump�()

@chuckries Do you already know which version of VS is shipping the fixes?

@danmoseley
Copy link
Member

@pr8x it might be better to open a new issue with a link back to this closed one

@msftgits msftgits transferred this issue from dotnet/coreclr Jan 31, 2020
@msftgits msftgits added this to the 3.1 milestone Jan 31, 2020
@ghost ghost locked as resolved and limited conversation to collaborators Dec 12, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

5 participants