android JS scheduling test failure - likely timing related / fails for some machines / not on others

[mikehardy]

          Hmmm 🤔 I'm getting a persistent error in local testing on the release-2.18 branch and on main (`./gradlew jacocoAndroidTestReport`) on the JS scheduling test:

> Task :AnkiDroid:connectedPlayDebugAndroidTest

com.ichi2.anki.ReviewerTest > testCustomSchedulerWithCustomData[TestingAVD(AVD) - 14] FAILED 
        java.lang.AssertionError:
        Expected: <3000>

That seems suspiciously like the sort of thing that may break with changes in this area. It appears to work in CI though

Local environment is macOS 14.5 (current) with this debug info from emulator:

AnkiDroid Version = 2.19alpha0-debug (e605223cf0f22cf9eb0ee48b6d4e68f47205cc41)

Backend Version = 0.1.38-anki24.04.1 (24.04 429bc9e14cefb597646a0e1beac6ef140f226b6f)

Android Version = 14 (SDK 34)

ProductFlavor = play

Manufacturer = Google

Model = sdk_gphone64_arm64

Hardware = ranchu

Webview User Agent = Mozilla/5.0 (Linux; Android 14; sdk_gphone64_arm64 Build/UE1A.230829.036.A2; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/113.0.5672.136 Mobile Safari/537.36

ACRA UUID = e3ff48c8-1d2f-4017-b648-30585f787ac1

FSRS Enabled = false

Crash Reports Enabled = false

I'm going to hold off on the 2.18.1 release as I'd like to get this in, and maybe it's something trivial ?
If it looks like it will take a while to investigate and fix please just say so - it's fine - and I'll do 2.18.1 without it as that still represents incremental improvement for users, and get a fix in for 2.18.2

Originally posted by @mikehardy in #16457 (comment)

[david-allison]

This test is ignored in CI

It fails on 464cf29 (v2.18.0)

I don't think this is a blocker for release

[mikehardy]

oooo - interesting. So it's an earlier regression?

For me locally git co v2.18.0 && ./gradlew jacocoAndroidTestReport succeeded twice in a row

git co release-2.18 && ./gradlew jacocoAndroidTestReport fails every time 🤔

Further, if I just drop the related commits from release-2.18 it passes - are you certain v2.18.0 is failing for you? Because it is working for me / also works if I just drop the commits so I question your repro methodology but am (most of all) prepared for an environment difference as this seems timing-related

[david-allison]

Confirmed on my M1:

./gradlew jacocoAndroidTestReport fails

at 464cf29

java.lang.AssertionError:
Expected: <3000>
but: was <0>
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
at com.ichi2.anki.ReviewerTest.testCustomSchedulerWithCustomData(ReviewerTest.kt:93)

[mikehardy]

David and I worked on this together for a bit, and David reproduces failure on the tag v2.18.0 whereas on my MacBook Air M2 it works for me on the v2.18.0 tag and when I revert the commits but it doesn't work with the commits.

So we have different results reproducible, which leads even more to the idea the timing on this one is flaky, but this is likely not a regression here.

So I'm going to release 2.18.1 with the changes and hopefully we can fix this forward so it's stable

[BrayanDSO]

Gonna need the backend upgrade after all. postBaseUrl got undefined.

ankidroid/Anki-Android-Backend#382

or #16468. Both work

[BrayanDSO]

For the future, think of a less flaky method of testing the Reviewer's webview