Android NDK Segfault crash related to sentry_value_set_by_key_n

[aaronjosephsonos]

Description

Our Android project has seen crashes related to a Segfault in Sentry native code. There are no places in our code that are directly calling nativeAddBreadcrumb, so thinking something under the hood in Sentry library is making the call. Still, there seems to be a Segfault bug in sentry native code sentry_value.c's file. Thanks in advance for taking a look, and please let me know if there is additional information I can help provide.

When does the problem happen

• During build
• During run-time
• When capturing a hard crash

Environment

• OS: Have seen crashes on Android 10,12,13,14
• Compiler:
• CMake version and config:
• Sentry version - 7.4.0
• Sentry compose version - 7.4.0
• Sentry gradle plugin version - 4.3.0

Subset of Sentry options enabled

            options.enableAllAutoBreadcrumbs(true)
            options.isAnrEnabled = true
            options.isEnableNdk = true
            options.isAttachStacktrace = true
            options.isSendDefaultPii = false
            options.isEnableScopeSync = true
            options.maxBreadcrumbs = 500
            options.isEnableAutoSessionTracking = true
            options.isAttachThreads = true
            options.anrTimeoutIntervalMillis = 10000
            options.isAttachScreenshot = false
            options.tracesSampleRate = null
            options.profilesSampleRate = null
            when (configProvider.buildEnvironment) {
                Environment.Development -> {
                    options.maxBreadcrumbs = 700
                    options.isSendDefaultPii = true
                    options.isAttachScreenshot = true
                    options.tracesSampleRate = 1.0
                    options.profilesSampleRate = 1.0
                }
                Environment.Alpha, Environment.Beta -> {
                    options.tracesSampleRate = 0.4
                    options.profilesSampleRate = 0.4
                }
                Environment.Production -> {
                    // use Sonos (and Sentry) default values
                }
            }

Steps To Reproduce
I myself have not run into this issue and have not found a reliable way to reproduce. Our project has seen the crash approximately 43 times in the past 30 days across 32 users.
Log output

Exception Type: Unknown (SIGSEGV)

Application Specific Information:
Segfault

Thread 0 Crashed:
0   base.apk                        0x71ef138ce0        sentry_value_set_by_key_n (sentry_value.c:482)
1   base.apk                        0x71ef13ae30        [inlined] sentry_value_set_by_key (sentry_value.c:494)
2   base.apk                        0x71ef13ae30        [inlined] timestamp_value (sentry_value.c:1163)
3   base.apk                        0x71ef13ae30        sentry_value_new_breadcrumb_n (sentry_value.c:1173)
4   base.apk                        0x71ef0b01f4        Java_io_sentry_android_ndk_NativeScope_nativeAddBreadcrumb (sentry.c:174)
5   base.odex                       0x7160a890a0        <unknown> + 486952964256

[supervacuus]

Hi @aaronjosephsonos, thanks for the report.

There are no places in our code that are directly calling nativeAddBreadcrumb, so thinking something under the hood in Sentry library is making the call.

Whenever you add breadcrumbs to your Android application, a call will be made to nativeAddBreadcrumb() because breadcrumbs will be synced to the Native SDK.

You might do this directly via any Interface providing an addBreadcrumb() method or indirectly via an integration that automatically adds them, for instance, during an activity state or navigation change.

Still, there seems to be a Segfault bug in sentry native code sentry_value.c's file. Thanks in advance for taking a look, and please let me know if there is additional information I can help provide.

I understand you currently cannot reproduce this locally, but any additional context (like a logcat output or a tombstone) will be helpful for me. Do you get these reports via Sentry? If so, could you provide a link to one of our employees, [email protected], for one such event?

This is the second report from Android, in which a freshly created object has either uninitialized members or refers to memory it doesn't own. AFAICR, this also appeared while adding a timestamp to a breadcrumb object (i.e., the first key to be added). Back then, I thought we might hit an issue with the page_allocator because it happened in the signal handler, but this seems to be happening outside any raised signal.

In any case, I will investigate.

Our project has seen the crash approximately 43 times in the past 30 days across 32 users.

Does this issue appear only with recent updates?

[supervacuus]

Cc @kahest: While my hopes are low that we will be able to repro this quickly, this is another concrete driver for #962

[aaronjosephsonos]

Thanks for the response @supervacuus!

Whenever you add breadcrumbs to your Android application, a call will be made to nativeAddBreadcrumb() because breadcrumbs will be synced to the Native SDK.

You might do this directly via any Interface providing an addBreadcrumb() method or indirectly via an integration that automatically adds them, for instance, during an activity state or navigation change.

Ahh thats good to know. We are calling addBreadcumbs pretty often in our code, basically anytime we log something we are calling addBreadcrumbs. So potentially some log message that occasionally occurs may be causing this bug to pop up.

I understand you currently cannot reproduce this locally, but any additional context (like a logcat output or a tombstone) will be helpful for me. Do you get these reports via Sentry? If so, could you provide a link to one of our employees, [email protected], for one such event?

Yes, we get these reports in Sentry. My workplace has a Slack channel with Sentry reps. I will share the events in that channel with them, and hopefully they will be able to forward them to you. I'll keep you posted with that correspondance.

Does this issue appear only with recent updates?

The first crash was reported on February 16, at that time we were on Sentry Android version 6.29.0 and Sentry android gradle version 3.12.0

[supervacuus]

So potentially some log message that occasionally occurs may be causing this bug to pop up.

I hope it is not a particular log message (although that would make the case easy). The breadcrumbs are a pretty beaten path, especially on mobile, and if the error were in that code, I would be surprised that we're not getting many more reports. But it is hard to say at this point.

I will share the events in that channel with them, and hopefully they will be able to forward them to you. I'll keep you posted with that correspondence.

Thanks!

[kahest]

Our internal crash reporting indicates that this was related/caused by a specific app release and most likely not by an SDK issue - we're keeping this open for a bit for further input

[ionoy]

We also see this issue reported in Google Play. There aren't that many cases, and they seem random. The stack trace points to addBreadcrumb just as described above.

  #00  pc 0x00000000000550d0  /apex/com.android.runtime/lib64/bionic/libc.so (__strlen_aarch64+16)
  #01  pc 0x0000000000069040  /data/app/~~MxMknbSw2B1InjMS8lHCIg==/com.springfin.docutrack-NIPgtRgS3PujIDchTt8h8A==/split_config.arm64_v8a.apk!libsentry.so (BuildId: b8159197a216f654a35b91c96e5c254da1061566)
  #02  pc 0x000000000006cc10  /data/app/~~MxMknbSw2B1InjMS8lHCIg==/com.springfin.docutrack-NIPgtRgS3PujIDchTt8h8A==/split_config.arm64_v8a.apk!libsentry.so (sentry_value_set_by_key_n+92) (BuildId: b8159197a216f654a35b91c96e5c254da1061566)
  #03  pc 0x0000000000002368  /data/app/~~MxMknbSw2B1InjMS8lHCIg==/com.springfin.docutrack-NIPgtRgS3PujIDchTt8h8A==/split_config.arm64_v8a.apk!libsentry-android.so (Java_io_sentry_android_ndk_NativeScope_nativeAddBreadcrumb+560) (BuildId: b145cbf70adb9cf3532c318f02caa53f22523296)
  #04  pc 0x0000000000074eb4  /data/app/~~MxMknbSw2B1InjMS8lHCIg==/com.springfin.docutrack-NIPgtRgS3PujIDchTt8h8A==/oat/arm64/base.odex (art_jni_trampoline+196)
  #05  pc 0x00000000001854e0  /data/app/~~MxMknbSw2B1InjMS8lHCIg==/com.springfin.docutrack-NIPgtRgS3PujIDchTt8h8A==/oat/arm64/base.odex (io.sentry.android.ndk.NativeScope.addBreadcrumb+96)
  #06  pc 0x00000000001a3cf0  /data/app/~~MxMknbSw2B1InjMS8lHCIg==/com.springfin.docutrack-NIPgtRgS3PujIDchTt8h8A==/oat/arm64/base.odex (io.sentry.android.ndk.NdkScopeObserver.addBreadcrumb+800)
  #07  pc 0x0000000000159f14  /data/app/~~MxMknbSw2B1InjMS8lHCIg==/com.springfin.docutrack-NIPgtRgS3PujIDchTt8h8A==/oat/arm64/base.odex (io.sentry.Scope.addBreadcrumb+612)
  #08  pc 0x000000000014c578  /data/app/~~MxMknbSw2B1InjMS8lHCIg==/com.springfin.docutrack-NIPgtRgS3PujIDchTt8h8A==/oat/arm64/base.odex (io.sentry.Hub.addBreadcrumb+568)
  #09  pc 0x000000000014c308  /data/app/~~MxMknbSw2B1InjMS8lHCIg==/com.springfin.docutrack-NIPgtRgS3PujIDchTt8h8A==/oat/arm64/base.odex (io.sentry.Hub.addBreadcrumb+232)
  #10  pc 0x0000000000103b7c  /data/app/~~MxMknbSw2B1InjMS8lHCIg==/com.springfin.docutrack-NIPgtRgS3PujIDchTt8h8A==/oat/arm64/base.odex (io.sentry.Sentry.addBreadcrumb+332)
  #11  pc 0x000000000033b680  /apex/com.android.art/lib64/libart.so (art_quick_invoke_static_stub+640)
  #12  pc 0x00000000003815d0  /apex/com.android.art/lib64/libart.so (art::JValue art::InvokeWithJValues<art::ArtMethod*>(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, art::ArtMethod*, jvalue const*)+652)
  #13  pc 0x000000000056bb64  /apex/com.android.art/lib64/libart.so (art::JNI<false>::CallStaticVoidMethodA(_JNIEnv*, _jclass*, _jmethodID*, jvalue const*)+164)
  #14  pc 0x000000000002e680  /data/app/~~MxMknbSw2B1InjMS8lHCIg==/com.springfin.docutrack-NIPgtRgS3PujIDchTt8h8A==/split_config.arm64_v8a.apk

[kodebach]

We're also seeing this crash in our app. The first crash was on Feb 5 with io.sentry.android.gradle:4.2.0. The latest crash currently uses io.sentry.android.gradle:4.10.0. Since then it happend sporadically every few days on random devices (43 events in the last 90 days on 11 devices). We're not calling addBreadcrumb directly, but we have enabled all the automatic integrations including Timber and have increased the maxBreadcrumbs option to 500 (default 100).

OS Version: Android 13 (TP1A.220624.014.G525FXXSACXCA)
Report Version: 104

Exception Type: Unknown (SIGSEGV)

Application Specific Information:
Segfault

Thread 0 Crashed:
0   split_config.arm64_v8a.apk      0x7234e7a04c        sentry_value_set_by_key_n
1   split_config.arm64_v8a.apk      0x7234e7c270        sentry_value_new_breadcrumb_n
2   split_config.arm64_v8a.apk      0x724e2231f4        Java_io_sentry_android_ndk_NativeScope_nativeAddBreadcrumb
3   libart.so                       0x7253351230        <unknown> + 491022258736
4   libart.so                       0x72535b8868        <unknown> + 491024779368
5   <unknown>                       0x71e05aa92c        <unknown>
6   <unknown>                       0x9d75f464          <unknown>

[JoshuaMoelans]

In a recent investigation concerning this issue, we stress-tested the bread-crumb and set_by_key paths using sanitizers via NDK. However, we could still not reproduce the problem or show obvious corruption along the mentioned paths.

We will continue to monitor the issue, but the currently available stack traces, while pointing to a common execution path, do not provide any insight into the causes. Users with stable reproduction are welcome to contact us at [email protected].

[kodebach]

We have not seen this issue in our apps since upgrading to io.sentry.android.gradle:4.11.0, i.e. io.sentry:sentry-android:7.13.0. AFAICT the version of sentry-native is unchanged since io.sentry:sentry-android:7.11.0, i.e. io.sentry.android.gradle:4.9.0 (for which we have seen crashes), but maybe the calling side in io.sentry:sentry-android:7.13.0 changed. In any case the issue seems to be gone for now.