Fix part of #973: Stabilize state fragment tests #1630
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This piggybacks off of the solution introduced in #1276 for Robolectric. That PR allows Robolectric tests in app module to share dependencies with production components by creating a test application & telling Robolectric to use it instead of OppiaApplication via a @config annotation. This PR achieves the same thing by using a custom test runner that reads the same annotation and forces Espresso & instrumentation to use the test application instead of the default OppiaApplication. This setup will be easier once #59 is finished since we can specify the application in per-test manifests that both Robolectric & Espresso will respect. Note that while this lets the same test coroutine dispatchers work in both production & test code for Espresso, some additional work was needed to ensure the coroutines behave correctly. In particular, the coroutines use a fake system clock in Robolectric that requires explicit synchronization points in the tests to allow the clock to move forward & properly coordinate execution between background & main thread operations. However, in Espresso, since everything is using a real clock an idling resource is the preferred way to synchronize execution: it allows the background coroutines to operate in real-time much like they would in production, and then notify Espresso when completed. The test dispatchers API is now setup to support both synchronization mechanisms for both Robolectric & Espresso (the idling resource does nothing on Robolectric and the force synchronization effectively does nothing on Espresso). The first test being demonstrated as now stable is SplashActivityTest (as part of downstream work in #1397.
Espresso. Note that some issues were found during this: #1612 (#1611 was found a few weeks ago, but it also affects these tests). To ensure the tests can still be run, a @runon annotation was added to allow tests to target specific test platforms. The tests that currently fail on Robolectric due to #1611 and #1612 are disabled for that platform. The test suite as a whole has been verified to pass in its current state on both Robolectric and Espresso (on a Pixel XL). The aim of this PR is to actually enable critical state fragment tests in CI, so both StateFragmentTest and StateFragmentLocalTest are being enabled in GitHub actions.
…abilize-state-fragment-test Conflicts: app/src/sharedTest/java/org/oppia/app/player/state/StateFragmentTest.kt
…abilize-state-fragment-test
24 tasks
|
Thanks all! |
Base automatically changed from
introduce-test-dispatchers-support-in-espresso
to
develop
August 12, 2020 08:11
BenHenning
changed the base branch from
develop
to
introduce-coroutine-executor-service
|
Ran This PR should get a follow-up review before being merged (& of course after its blocking PRs are also merged). |
|
This will require another pass. @rt4914 & @anandwana001 PTAL. |
anandwana001
reviewed
Sep 2, 2020
anandwana001
approved these changes
Sep 2, 2020
There was a problem hiding this comment.
LGTM, Thanks @BenHenning
I test running all test cases, below are the result
Espresso Result
Robolectric Result
…troduce-coroutine-executor-service
…ate-fragment-test
|
Thanks for verifying that the tests are passing for you @anandwana001! I'm assuming your comment is optional since it's not suggesting changes. |
8 tasks
Base automatically changed from
introduce-coroutine-executor-service
to
develop
September 3, 2020 03:55
BenHenning
added a commit
that referenced
this pull request
Sep 3, 2020
* Introduce new executor service which allows interop with Kotlin coroutines, plus a test to verify that it fundamentally follows one interpretation of ExecutorService's API. * Fix flaky timeout tests by improving cancellation cooperation for invokeAny() and provide longer timeouts for tests that are CPU-sensitive. * Add documentation & clean up unused code. * Lint fixes. * Significantly reorganize invokeAll() to try and make it more cooperative for cancellation, and increase timeout times in tests to reduce flakiness for time-sensitive tests. Some tests are remaining flaky, so ignoring those. Re-add maybeWithTimeoutOrNull since it actually was needed. * Lint fixes. * Revert "Fixes part of #40 & #42: Generalisation Highfi Mobile Portrait + Landscape - Buttons (#1653)" This reverts commit 1bb1ffa. * Ensure terminated tasks do not interfere with one another (timeouts should happen individually for each task during termination). This fixes a failure observed in StateFragmentLocalTest in #1630. * Fix awaitTermination & improve test. Improve stack trace for test dispatcher timeouts. * Address reviewer comments.
BenHenning
changed the title
BenHenning
added a commit
that referenced
this pull request
Sep 3, 2020
* Add support for showing concept cards in feedback, and add a concept card as one of the remediation pathways for 'the meaning of equal parts' lesson. * Introduce test coroutine dispatchers support in Espresso. This piggybacks off of the solution introduced in #1276 for Robolectric. That PR allows Robolectric tests in app module to share dependencies with production components by creating a test application & telling Robolectric to use it instead of OppiaApplication via a @config annotation. This PR achieves the same thing by using a custom test runner that reads the same annotation and forces Espresso & instrumentation to use the test application instead of the default OppiaApplication. This setup will be easier once #59 is finished since we can specify the application in per-test manifests that both Robolectric & Espresso will respect. Note that while this lets the same test coroutine dispatchers work in both production & test code for Espresso, some additional work was needed to ensure the coroutines behave correctly. In particular, the coroutines use a fake system clock in Robolectric that requires explicit synchronization points in the tests to allow the clock to move forward & properly coordinate execution between background & main thread operations. However, in Espresso, since everything is using a real clock an idling resource is the preferred way to synchronize execution: it allows the background coroutines to operate in real-time much like they would in production, and then notify Espresso when completed. The test dispatchers API is now setup to support both synchronization mechanisms for both Robolectric & Espresso (the idling resource does nothing on Robolectric and the force synchronization effectively does nothing on Espresso). The first test being demonstrated as now stable is SplashActivityTest (as part of downstream work in #1397. * Revert "Fixes #941: Add radar effect in Hints and solution (#1475)" This reverts commit 41eb10b. * Stabilize StateFragmentTest such that it passes on both Robolectric and Espresso. Note that some issues were found during this: #1612 (#1611 was found a few weeks ago, but it also affects these tests). To ensure the tests can still be run, a @runon annotation was added to allow tests to target specific test platforms. The tests that currently fail on Robolectric due to #1611 and #1612 are disabled for that platform. The test suite as a whole has been verified to pass in its current state on both Robolectric and Espresso (on a Pixel XL). The aim of this PR is to actually enable critical state fragment tests in CI, so both StateFragmentTest and StateFragmentLocalTest are being enabled in GitHub actions. * Enable StateFragmentTest (Robolectric) & StateFragmentLocalTest for CI. * Add thorough documentation for new dispatchers. * Clean up comments & add additional documentation. * Fix lint errors. * Fix broken test after changes to FakeSystemClock. * Fix linter errors. * Update test lesson to include references to concept cards. * Lint fixes & use HtmlCompat instead of Html. * Add support for the newer & finalized tag format. * Lint fixes. * Use a custom executor service for Glide requests that coordinates with Oppia's test dispatchers. Note that this does not actually introduce the service--that will happen in a new branch. * Introduce new executor service which allows interop with Kotlin coroutines, plus a test to verify that it fundamentally follows one interpretation of ExecutorService's API. * Fix flaky timeout tests by improving cancellation cooperation for invokeAny() and provide longer timeouts for tests that are CPU-sensitive. * Add documentation & clean up unused code. * Lint fixes. * Significantly reorganize invokeAll() to try and make it more cooperative for cancellation, and increase timeout times in tests to reduce flakiness for time-sensitive tests. Some tests are remaining flaky, so ignoring those. Re-add maybeWithTimeoutOrNull since it actually was needed. * Lint fixes. * Post-merge module fixes. * Post-merge fixes with ratio input & add a TODO to improve speed of the new coroutine executor service. * Revert "Fixes part of #40 & #42: Generalisation Highfi Mobile Portrait + Landscape - Buttons (#1653)" This reverts commit 1bb1ffa. * Ensure terminated tasks do not interfere with one another (timeouts should happen individually for each task during termination). This fixes a failure observed in StateFragmentLocalTest in #1630. * Ignore failing tests until #1769 is resolved. * Fix awaitTermination & improve test. Improve stack trace for test dispatcher timeouts. * Fix slow & broken tests in Robolectric for StateFragmentLocalTest. * Add missing deps for StateFragmentLocalTest. * Address TODOs (including adding support for list tags which replaces the old handler & adds nested custom tag support), and add tests. * Lint fixes. * Address reviewer comments. * Address review comments. Fix new concept card tests on Espresso & add landscape versions (required configuring hints to show quickly to avoid delaying the test, and fixing a bug in the espresso test dispatcher). Add support for disabling concept cards of they aren't enabled for parsing particular HTML (the default behavior is to ignore the custom tag). * Add support for concept cards in questions. Note that it's not clear how to test verifying that pressing the exit button closes the concept card since the exit button is part of the dialog's toolbar. * Fix image-breaking duplicated code in HtmlParser, fix a paragraph parsing issue in BulletTagHandler, and add tests for CustomHtmlContentHandler. * Lint fixes.
prayutsu
pushed a commit
to prayutsu/oppia-android
that referenced
this pull request
Sep 3, 2020
* Introduce new executor service which allows interop with Kotlin coroutines, plus a test to verify that it fundamentally follows one interpretation of ExecutorService's API. * Fix flaky timeout tests by improving cancellation cooperation for invokeAny() and provide longer timeouts for tests that are CPU-sensitive. * Add documentation & clean up unused code. * Lint fixes. * Significantly reorganize invokeAll() to try and make it more cooperative for cancellation, and increase timeout times in tests to reduce flakiness for time-sensitive tests. Some tests are remaining flaky, so ignoring those. Re-add maybeWithTimeoutOrNull since it actually was needed. * Lint fixes. * Revert "Fixes part of oppia#40 & oppia#42: Generalisation Highfi Mobile Portrait + Landscape - Buttons (oppia#1653)" This reverts commit 1bb1ffa. * Ensure terminated tasks do not interfere with one another (timeouts should happen individually for each task during termination). This fixes a failure observed in StateFragmentLocalTest in oppia#1630. * Fix awaitTermination & improve test. Improve stack trace for test dispatcher timeouts. * Address reviewer comments.
prayutsu
pushed a commit
to prayutsu/oppia-android
that referenced
this pull request
Sep 3, 2020
…#1764] (oppia#1630) * Introduce test coroutine dispatchers support in Espresso. This piggybacks off of the solution introduced in oppia#1276 for Robolectric. That PR allows Robolectric tests in app module to share dependencies with production components by creating a test application & telling Robolectric to use it instead of OppiaApplication via a @config annotation. This PR achieves the same thing by using a custom test runner that reads the same annotation and forces Espresso & instrumentation to use the test application instead of the default OppiaApplication. This setup will be easier once oppia#59 is finished since we can specify the application in per-test manifests that both Robolectric & Espresso will respect. Note that while this lets the same test coroutine dispatchers work in both production & test code for Espresso, some additional work was needed to ensure the coroutines behave correctly. In particular, the coroutines use a fake system clock in Robolectric that requires explicit synchronization points in the tests to allow the clock to move forward & properly coordinate execution between background & main thread operations. However, in Espresso, since everything is using a real clock an idling resource is the preferred way to synchronize execution: it allows the background coroutines to operate in real-time much like they would in production, and then notify Espresso when completed. The test dispatchers API is now setup to support both synchronization mechanisms for both Robolectric & Espresso (the idling resource does nothing on Robolectric and the force synchronization effectively does nothing on Espresso). The first test being demonstrated as now stable is SplashActivityTest (as part of downstream work in oppia#1397. * Revert "Fixes oppia#941: Add radar effect in Hints and solution (oppia#1475)" This reverts commit 41eb10b. * Stabilize StateFragmentTest such that it passes on both Robolectric and Espresso. Note that some issues were found during this: oppia#1612 (oppia#1611 was found a few weeks ago, but it also affects these tests). To ensure the tests can still be run, a @runon annotation was added to allow tests to target specific test platforms. The tests that currently fail on Robolectric due to oppia#1611 and oppia#1612 are disabled for that platform. The test suite as a whole has been verified to pass in its current state on both Robolectric and Espresso (on a Pixel XL). The aim of this PR is to actually enable critical state fragment tests in CI, so both StateFragmentTest and StateFragmentLocalTest are being enabled in GitHub actions. * Enable StateFragmentTest (Robolectric) & StateFragmentLocalTest for CI. * Add thorough documentation for new dispatchers. * Clean up comments & add additional documentation. * Fix lint errors. * Fix broken test after changes to FakeSystemClock. * Fix linter errors. * Use a custom executor service for Glide requests that coordinates with Oppia's test dispatchers. Note that this does not actually introduce the service--that will happen in a new branch. * Introduce new executor service which allows interop with Kotlin coroutines, plus a test to verify that it fundamentally follows one interpretation of ExecutorService's API. * Fix flaky timeout tests by improving cancellation cooperation for invokeAny() and provide longer timeouts for tests that are CPU-sensitive. * Add documentation & clean up unused code. * Lint fixes. * Significantly reorganize invokeAll() to try and make it more cooperative for cancellation, and increase timeout times in tests to reduce flakiness for time-sensitive tests. Some tests are remaining flaky, so ignoring those. Re-add maybeWithTimeoutOrNull since it actually was needed. * Lint fixes. * Post-merge module fixes. * Post-merge fixes with ratio input & add a TODO to improve speed of the new coroutine executor service. * Revert "Fixes part of oppia#40 & oppia#42: Generalisation Highfi Mobile Portrait + Landscape - Buttons (oppia#1653)" This reverts commit 1bb1ffa. * Ensure terminated tasks do not interfere with one another (timeouts should happen individually for each task during termination). This fixes a failure observed in StateFragmentLocalTest in oppia#1630. * Ignore failing tests until oppia#1769 is resolved. * Fix awaitTermination & improve test. Improve stack trace for test dispatcher timeouts. * Fix slow & broken tests in Robolectric for StateFragmentLocalTest. * Add missing deps for StateFragmentLocalTest. * Address reviewer comments.
prayutsu
pushed a commit
to prayutsu/oppia-android
that referenced
this pull request
Sep 3, 2020
* Add support for showing concept cards in feedback, and add a concept card as one of the remediation pathways for 'the meaning of equal parts' lesson. * Introduce test coroutine dispatchers support in Espresso. This piggybacks off of the solution introduced in oppia#1276 for Robolectric. That PR allows Robolectric tests in app module to share dependencies with production components by creating a test application & telling Robolectric to use it instead of OppiaApplication via a @config annotation. This PR achieves the same thing by using a custom test runner that reads the same annotation and forces Espresso & instrumentation to use the test application instead of the default OppiaApplication. This setup will be easier once oppia#59 is finished since we can specify the application in per-test manifests that both Robolectric & Espresso will respect. Note that while this lets the same test coroutine dispatchers work in both production & test code for Espresso, some additional work was needed to ensure the coroutines behave correctly. In particular, the coroutines use a fake system clock in Robolectric that requires explicit synchronization points in the tests to allow the clock to move forward & properly coordinate execution between background & main thread operations. However, in Espresso, since everything is using a real clock an idling resource is the preferred way to synchronize execution: it allows the background coroutines to operate in real-time much like they would in production, and then notify Espresso when completed. The test dispatchers API is now setup to support both synchronization mechanisms for both Robolectric & Espresso (the idling resource does nothing on Robolectric and the force synchronization effectively does nothing on Espresso). The first test being demonstrated as now stable is SplashActivityTest (as part of downstream work in oppia#1397. * Revert "Fixes oppia#941: Add radar effect in Hints and solution (oppia#1475)" This reverts commit 41eb10b. * Stabilize StateFragmentTest such that it passes on both Robolectric and Espresso. Note that some issues were found during this: oppia#1612 (oppia#1611 was found a few weeks ago, but it also affects these tests). To ensure the tests can still be run, a @runon annotation was added to allow tests to target specific test platforms. The tests that currently fail on Robolectric due to oppia#1611 and oppia#1612 are disabled for that platform. The test suite as a whole has been verified to pass in its current state on both Robolectric and Espresso (on a Pixel XL). The aim of this PR is to actually enable critical state fragment tests in CI, so both StateFragmentTest and StateFragmentLocalTest are being enabled in GitHub actions. * Enable StateFragmentTest (Robolectric) & StateFragmentLocalTest for CI. * Add thorough documentation for new dispatchers. * Clean up comments & add additional documentation. * Fix lint errors. * Fix broken test after changes to FakeSystemClock. * Fix linter errors. * Update test lesson to include references to concept cards. * Lint fixes & use HtmlCompat instead of Html. * Add support for the newer & finalized tag format. * Lint fixes. * Use a custom executor service for Glide requests that coordinates with Oppia's test dispatchers. Note that this does not actually introduce the service--that will happen in a new branch. * Introduce new executor service which allows interop with Kotlin coroutines, plus a test to verify that it fundamentally follows one interpretation of ExecutorService's API. * Fix flaky timeout tests by improving cancellation cooperation for invokeAny() and provide longer timeouts for tests that are CPU-sensitive. * Add documentation & clean up unused code. * Lint fixes. * Significantly reorganize invokeAll() to try and make it more cooperative for cancellation, and increase timeout times in tests to reduce flakiness for time-sensitive tests. Some tests are remaining flaky, so ignoring those. Re-add maybeWithTimeoutOrNull since it actually was needed. * Lint fixes. * Post-merge module fixes. * Post-merge fixes with ratio input & add a TODO to improve speed of the new coroutine executor service. * Revert "Fixes part of oppia#40 & oppia#42: Generalisation Highfi Mobile Portrait + Landscape - Buttons (oppia#1653)" This reverts commit 1bb1ffa. * Ensure terminated tasks do not interfere with one another (timeouts should happen individually for each task during termination). This fixes a failure observed in StateFragmentLocalTest in oppia#1630. * Ignore failing tests until oppia#1769 is resolved. * Fix awaitTermination & improve test. Improve stack trace for test dispatcher timeouts. * Fix slow & broken tests in Robolectric for StateFragmentLocalTest. * Add missing deps for StateFragmentLocalTest. * Address TODOs (including adding support for list tags which replaces the old handler & adds nested custom tag support), and add tests. * Lint fixes. * Address reviewer comments. * Address review comments. Fix new concept card tests on Espresso & add landscape versions (required configuring hints to show quickly to avoid delaying the test, and fixing a bug in the espresso test dispatcher). Add support for disabling concept cards of they aren't enabled for parsing particular HTML (the default behavior is to ignore the custom tag). * Add support for concept cards in questions. Note that it's not clear how to test verifying that pressing the exit button closes the concept card since the exit button is part of the dialog's toolbar. * Fix image-breaking duplicated code in HtmlParser, fix a paragraph parsing issue in BulletTagHandler, and add tests for CustomHtmlContentHandler. * Lint fixes.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix part of #973. Make StateFragmentTest work on both Robolectric & Espresso, and move it & StateFragmentLocalTest to the non-flaky CI workflow.
These tests are especially important to run on CI since they have caught both of the recent regressions that led to PR reverts. Presumably, they will catch other similar issues in the future & in CI will prevent them from being merged.
This work is now possible due to #1623 which allows tests to deterministically run on both Robolectric & Espresso with a consistent testing API. The changes include a large number of changes to StateFragmentTest to generally simplify it & make it more readable, and also to hide the additional synchronization points needed for the test to run properly on Robolectric.
Note that this PR also includes test infrastructure to allow tests to select which platform they run on (new
@RunOnannotation which requires a correspondingOppiaTestRuleto be bound in the test). Some of StateFragmentTest's tests cannot yet run on Espresso due to #1611 and #1612 both of which require changes inside of Robolectric to unblock. Since our CI only runs Robolectric currently, it's not ideal to disable any tests. However, there's no obvious immediate workaround.