Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
…de coverage usage and limitations (oppia#5483) ## Explanation Fixes part of oppia#5343 Fixes oppia#5486 ### Project [PR 2.6 of Project 4.1] ### Changes Made - This PR introduces 2 new wiki pages: - Oppia Android Code Coverage - Writing tests with Good Behavioural Coverage - Fix to the comment upload feature permission issues with PRs created from the forked branches. - Split the code coverage workflow into 2 1. Core code coverage workflow to handle - Collection of changed files - Bucket partitioning - Run coverage in matrices 2. Comment upload workflow to handle - evaluation of reports - generation of md reports - uploading comments - Coverage Status Checks (as the later required `pull_request_target`) - Fix to oppia#5486 - The issue should have arisen as the pr got merged with the branch being deleted while the publish comment job still running, finding it hard to fetch the pr-issue number to publish a comment. - Now the Coverage Check Status was made to be dependent on the comment uploader ie. the final Coverage check job occurs only after the Evaluation and Comment jobs are done, so it will always have the pr reference) ``` check_coverage_results: name: Check Code Coverage Results needs: [ evaluate-code-coverage-reports, comment_coverage_report ] ``` # ### Reasons for splitting the code_coverage workflow The single code_coverage workflow was split into 1. **code_coverage** (to run coverages) 2. **coverage_report** (to generate and publish reports) ### Separating the comment upload job - The primary reason is the need to have ability to upload comments from PRs opened from a fork branch. - [`pull_request_target`](https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows#pull_request_target): For workflows that are triggered by the pull_request_target event, the GITHUB_TOKEN is granted read/write repository permission, even when it is triggered from a fork. - Workflows triggered by `pull_request_target` events are run in the context of the base branch. Since the base branch is considered trusted, workflows triggered by these events will always run, regardless of approval settings. [[GitHub Docs](https://docs.github.com/en/actions/managing-workflow-runs-and-deployments/managing-workflow-runs/approving-workflow-runs-from-public-forks#about-workflow-runs-from-public-forks)] ### Separating the evaluation / generation of report job - While initially it was split to help with 'skip files' - no files changed conditional check, with the introduction to 'SKIP' status check, it doesn't continue to serve the mentioned purpose. - But if we still have it as one workflow then the flow will work as such: Workflow 1: **code_coverage** - compute changed files - run coverage (needs compute changed files) - evaluate / generate md - code coverage check result Workflow 2: **coverage_report** - comment publication If no `.kt` files changes are detected - compute changed files - skips run coverage - skips evaluate / generate md - pass code coverage check result As the workflow was concluded as success, the 2nd workflow runs as, - failed comment publication (as no report is generation due to skip) But expectation is to still produce a pass check for the comment publication. (either to at least skip or upload a skip status as coverage comment report) - With moving it to a separate workflow allows us to not make the evaluation / generation jobs rely on the Run coverage job making it independently behave once the code_coverage workflows are completed successfully. - And it checks if pb files are generated and based on that it decides whether to generate PASS, FAIL or SKIP status checks. ### Separating the coverage status check result job There are 2 main reasons to moving it to new workflow. While it would still make sense to have it with the 1st workflow itself after Run coverage, the drawbacks are, - If the check coverage status result was left with the 1st workflow, then when the Run coverage job completes in the 1st workflow the coverage status check result is set to true on success even before the sibling part of upload comment is done, making it an incomplete status result. - oppia#5486 occurred as it lost its reference to the pr number, so making the coverage check status result dependent on (needs) the comment upload job should resolve this by only allowing the PR to be closed once the comment is uploaded. # **Todo:** - **[Done]** Add a new wiki page for "Writing effective tests / Writing test with good behavioural coverage" ## Essential Checklist - [x] The PR title and explanation each start with "Fix #bugnum: " (If this PR fixes part of an issue, prefix the title with "Fix part of #bugnum: ...".) - [x] Any changes to [scripts/assets](https://github.com/oppia/oppia-android/tree/develop/scripts/assets) files have their rationale included in the PR explanation. - [x] The PR follows the [style guide](https://github.com/oppia/oppia-android/wiki/Coding-style-guide). - [x] The PR does not contain any unnecessary code changes from Android Studio ([reference](https://github.com/oppia/oppia-android/wiki/Guidance-on-submitting-a-PR#undo-unnecessary-changes)). - [x] The PR is made from a branch that's **not** called "develop" and is up-to-date with "develop". - [x] The PR is **assigned** to the appropriate reviewers ([reference](https://github.com/oppia/oppia-android/wiki/Guidance-on-submitting-a-PR#clarification-regarding-assignees-and-reviewers-section)). ## For UI-specific PRs only If your PR includes UI-related changes, then: - Add screenshots for portrait/landscape for both a tablet & phone of the before & after UI changes - For the screenshots above, include both English and pseudo-localized (RTL) screenshots (see [RTL guide](https://github.com/oppia/oppia-android/wiki/RTL-Guidelines)) - Add a video showing the full UX flow with a screen reader enabled (see [accessibility guide](https://github.com/oppia/oppia-android/wiki/Accessibility-A11y-Guide)) - For PRs introducing new UI elements or color changes, both light and dark mode screenshots must be included - Add a screenshot demonstrating that you ran affected Espresso tests locally & that they're passing --------- Co-authored-by: Ben Henning <[email protected]>
- Loading branch information
1 parent
01e02ec
commit 1a68c9f