You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We have disabled a number of tests on iOS and Android.
In cases where these are flaky both on CI and on a local dev machine, these are disabled with an #if directive:
#if __IOS__
Skip.If(true,"Flaky on iOS");
#endif
When they're only flaky when run in CI, we check for an environment variable instead:
Skip.If(TestEnvironment.IsGitHubActions,"This test is flaky in CI");
Ultimately we need to either work out why the tests fail and fix the code/environment/tests or simply remove the tests. There's no point in having tests that we don't trust in the repository. Preferably we'd fix the tests though.
The error messages is similar for both of the above tests:
<test name="Sentry.Maui.Tests.SentryMauiScreenshotTests.CaptureException_RemoveScreenshot_NotContainsScreenshotAttachmentAsync" type="Sentry.Maui.Tests.SentryMauiScreenshotTests" method="CaptureException_RemoveScreenshot_NotContainsScreenshotAttachmentAsync" time="0.0890636" result="Fail">
<failure exception-type="Xunit.Sdk.XunitException">
<message><![CDATA[Expected envelope not to be <null>.]]></message>
<stack-trace><![CDATA[ at FluentAssertions.Execution.XUnit2TestFramework.Throw(String message)
at FluentAssertions.Execution.TestFrameworkProvider.Throw(String message)
at FluentAssertions.Execution.DefaultAssertionStrategy.HandleFailure(String message)
at FluentAssertions.Execution.AssertionScope.FailWith(Func`1 failReasonFunc)
at FluentAssertions.Execution.AssertionScope.FailWith(Func`1 failReasonFunc)
at FluentAssertions.Execution.AssertionScope.FailWith(String message)
at FluentAssertions.Primitives.ReferenceTypeAssertions`2[[Sentry.Protocol.Envelopes.Envelope, Sentry, Version=4.11.0.0, Culture=neutral, PublicKeyToken=fba2ec45388e2af0],[Sentry.Testing.EnvelopeAssertions, Sentry.Testing, Version=4.11.0.0, Culture=neutral, PublicKeyToken=fba2ec45388e2af0]].NotBeNull(String because, Object[] becauseArgs)
at Sentry.Maui.Tests.SentryMauiScreenshotTests.CaptureException_RemoveScreenshot_NotContainsScreenshotAttachmentAsync()
--- End of stack trace from previous location ---]]></stack-trace>
</failure>
</test>
Flaky Tests on Android
We're not getting much useful information back about why the tests are failing on Android. Xharness simply says that the tests timed out after 900 seconds.
XHarness exit code: 70 (TIMED_OUT)
There are some adb logs but it's very hard to get any useful information out of these.
On Android, it appears the reliability of tests is impacted by the API Level that we're targeting.
We used to test against API levels 27 -31, for example, but the tests would almost invariably fail in one or more of those API levels, so we dropped back to testing against just 27 and 31 in this PR.
In PR #3628 the tests were often passing against API level 27 but failing against API level 33.
We have disabled a number of tests on iOS and Android.
In cases where these are flaky both on CI and on a local dev machine, these are disabled with an
#if
directive:When they're only flaky when run in CI, we check for an environment variable instead:
Ultimately we need to either work out why the tests fail and fix the code/environment/tests or simply remove the tests. There's no point in having tests that we don't trust in the repository. Preferably we'd fix the tests though.
Flaky Tests on iOS
Logging Tests
sentry-dotnet/test/Sentry.Extensions.Logging.Tests/LoggingTests.cs
Lines 15 to 19 in 0acaff1
sentry-dotnet/test/Sentry.Extensions.Logging.Tests/LoggingTests.cs
Lines 61 to 65 in 0acaff1
The error message for both of the above tests is similar:
Maui Screenshot Tests
sentry-dotnet/test/Sentry.Maui.Tests/SentryMauiScreenshotTests.cs
Lines 36 to 42 in 0acaff1
sentry-dotnet/test/Sentry.Maui.Tests/SentryMauiScreenshotTests.cs
Lines 73 to 78 in 0acaff1
The error messages is similar for both of the above tests:
Flaky Tests on Android
We're not getting much useful information back about why the tests are failing on Android. Xharness simply says that the tests timed out after 900 seconds.
There are some adb logs but it's very hard to get any useful information out of these.
Logging Tests
sentry-dotnet/test/Sentry.Extensions.Logging.Tests/LoggingTests.cs
Lines 15 to 20 in 0acaff1
Hub Tests
sentry-dotnet/test/Sentry.Tests/HubTests.cs
Lines 1137 to 1139 in 0acaff1
SDK Tests
https://github.com/getsentry/sentry-dotnet/blob/0acaff17253c1d1e3c572f0e28c779ced5987f5f/test/Sentry.Tests/SentrySdkTests.cs#L430-L432
Android API Levels
On Android, it appears the reliability of tests is impacted by the API Level that we're targeting.
We used to test against API levels 27 -31, for example, but the tests would almost invariably fail in one or more of those API levels, so we dropped back to testing against just 27 and 31 in this PR.
In PR #3628 the tests were often passing against API level 27 but failing against API level 33.
This may be related to Restrictions on non-SDK interfaces but, if so, I don't understand how.
The text was updated successfully, but these errors were encountered: