Create Breakpad receiving server

[hhyyrylainen]

So I had a look at tools for processing Breakpad minidumps (and the debug symbols) and it unfortunately looks like we can't use any of them...

• https://github.com/mozilla-services/socorro (this is very feature complete but requires extensive setup and S3 storage, which costs money)
• https://github.com/wk8/sentry_breakpad (tool for transferring processed breakpad reports into Sentry tool, we can use this once we have something processing the minidumps...)
• https://github.com/acrisci/simple-breakpad-server (this would be good, but there is no permission things so anyone could break it if it is exposed to the internet)

So we need to write our own software to receive minidumps and debug info (we need also a script for creating the debug info, this is a bit difficult on windows as locating the pdb files is not simple).

We can do some of these tasks manually so this isn't critical for 0.4.0.1

[hhyyrylainen]

This needs some more work.

[hhyyrylainen]

This is basically working but logging in for developers is not implemented so I'm ATM the only one who can upload debug symbols and thus make releases.

[hhyyrylainen]

Probably the only thing really missing is some functionality to make other accounts admins and generate API tokens for your own account.

[hhyyrylainen]

I'm going to consider this done enough to close this issue.

[bruno-garcia]

Curious: Why didn't you guys use Sentry? Since you can self host and it supports minidumps already?

[hhyyrylainen]

How is it free? From what I could tell when I looked at it, that service was only paid.

Also in the end we need to use both a breakpad normal stackwalk executable and one forked to work with mingw (as Godot is built with mingw for Windows). And from what I saw Sentry uses just breakpad as it is the only C++ crash reporting option that is maintained. So we might not even be able to use Sentry at all unless they were open to adding a mingw variant of the crash dump processing tool.

[bruno-garcia]

Sentry builds docker images to make it easier for you to self host: https://github.com/getsentry/self-hosted/ It's free.
There's no paid self hosting model. Sentry only makes money out of SaaS (sentry.io). And sponsors open source projects with a free account. (Sentry + Open Source)

I'd imagine folks would be open to supporting more platforms. It's pretty much in the company's DNA.
The Native SDK, which supports a series of platforms with different backends (including breakpad and crashpad) is here: https://github.com/getsentry/sentry-native

The only issue referring mingw I found was this but refers to building on Windows.

I suggest opening an issue to discuss it with the team. Specially if you're willing to open PRs to contribute, I imagine folks would be thrilled.

The server-side symbolication stuff is also open source:
Service: https://github.com/getsentry/symbolicator
Library behind it: https://github.com/getsentry/symbolic

[bruno-garcia]

Oh it's right on the README:

MinGW:

64-bits is the only platform supported for now. LLVM + Clang are mandatory here : they are required to generate .pdb files, used by Crashpad for the report generation.

For your application to generate the appropriate .pdb output, you need to activate CodeView file format generation on your application target. To do so, update your own CMakeLists.txt with something like target_compile_options(${yourApplicationTarget} PRIVATE -gcodeview).

If you use a MSYS2 environement to compile with MinGW, make sure to :

Create an environement variable MINGW_ROOT (ex : C:/msys64/mingw64)
Run from mingw64.exe : pacman -S --needed - < ./toolchains/msys2-mingw64-pkglist.txt
Build as :

Configure with Ninja as generator and use the MSYS2 toolchain file

$ cmake -GNinja -Bbuild -H. -DCMAKE_TOOLCHAIN_FILE=toolchains/msys2.cmake

build with Ninja

$ ninja -C build

[hhyyrylainen]

Sentry builds docker images to make it easier for you to self host: https://github.com/getsentry/self-hosted/ It's free.
There's no paid self hosting model. Sentry only makes money out of SaaS (sentry.io). And sponsors open source projects with a free account. (Sentry + Open Source)

Ah I see it's one of those cases where you hide the self hosted option so well on your pricing page / main website that people can't find it. Sadly I see that often, it takes a lot of conviction to put the "free if you run on your server" on the page where you are trying to convert visitors to paid customers. I can kind of understand it but then you constantly need to always correct people and say the self hosted option technically exists, but is so hard to find that you can't find it unless you are told about it specifically.

64-bits is the only platform supported for now. LLVM + Clang are mandatory here : they are required to generate .pdb files, used by Crashpad for the report generation.

Yeah, that's not how the Godot build process works (at least for 3.x), it uses gcc. And like I said the only way I got that working was by using that special fork of breakpad that supports mingw that way I can extract symbols from the Windows build of Godot export templates and by using the special stackwalk executable I can decode crash dumps correctly.

[bruno-garcia]

Ah I see it's one of those cases where you hide the self hosted option so well on your pricing page / main website that people can't find it. Sadly I see that often, it takes a lot of conviction to put the "free if you run on your server" on the page where you are trying to convert visitors to paid customers. I can kind of understand it but then you constantly need to always correct people and say the self hosted option technically exists, but is so hard to find that you can't find it unless you are told about it specifically.

Sentry has been available on GitHub since 2008, it only became a company a lot later. There's a lot about Sentry and open source out there:

https://open.sentry.io/

Including staffing people whose sole purpose is to make Sentry easy for folks to run them for free. So if it doesn't seem easy to find out, perhaps something they might be interested in looking in.

Now if you end up on SaaS pricing, I wouldn't even expect call outs to Self Hosted. Most people dont' want to run servers themselves and are happy with Sentry's free tier, or the 29 USD paid version. So I don't believe it's "malice" that there's no banner there pointing people to docker images. But granted, this software being available for folks to run for free is funded by the SaaS, so IMHO it wouldn't be all that weird to focus on pricing details for SaaS instead of trying to convince people to run servers themselves