ci: skip template check for flaky tests issues #23857

gauthierpetetin · 2024-04-04T19:06:41Z

Description

Flaky test issues do not need to match issues template as their description just includes a link to circleci. Example here: Fix "Test Snap Name Lookup tests name-lookup functionalit..." flaky tests #22702
By skipping the template check for issues labeled with flaky tests, we avoid them being automatically labeled with INVALID-ISSUE-TEMPLATE.

Related issues

Fixes: None

Manual testing steps

Go to this test repo which inlcudes the same changes as this PR
Create a new issue with empty description
Wait for 30s until the template check Github action is executed
Issue shall automatically labeled with INVALID-ISSUE-TEMPLATE
Label the issue with flaky tests label
Edit the issue's description (for example by adding a random character in it), to re-trigger the template check Github action
Wait for 30s until the template check Github action is executed
INVALID-ISSUE-TEMPLATE label shall automatically be removed from issue

Screenshots/Recordings

None

Pre-merge author checklist

I’ve followed MetaMask Coding Standards.
I've completed the PR template to the best of my ability
I’ve included tests if applicable
I’ve documented my code using JSDoc format if applicable
I’ve applied the right labels on the PR (see labeling guidelines). Not required for external contributors.

Pre-merge reviewer checklist

I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

Flaky tests issues do not match our issues templates. Their description just includes a link to cricleci.

github-actions · 2024-04-04T19:06:54Z

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

codecov · 2024-04-04T19:16:23Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 69.02%. Comparing base (9e21127) to head (dbf1ffb).
Report is 143 commits behind head on develop.

❗ Current head dbf1ffb differs from pull request most recent head 723ca33. Consider uploading reports for the commit 723ca33 to get more accurate results

Additional details and impacted files

@@           Coverage Diff            @@
##           develop   #23857   +/-   ##
========================================
  Coverage    69.02%   69.02%           
========================================
  Files         1164     1164           
  Lines        44513    44513           
  Branches     11890    11890           
========================================
  Hits         30721    30721           
  Misses       13792    13792

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

metamaskbot · 2024-04-04T19:19:43Z

Builds ready [dbf1ffb]

builds: chrome, firefox
builds (beta): chrome
builds (flask): chrome, firefox
builds (MMI): chrome, firefox
builds (test): chrome, firefox
builds (test-flask): chrome, firefox
build viz: Build System
mv3: Background Module Init Stats
mv3: UI Init Stats
mv3: Module Load Stats
mv3: Bundle Size Stats
mv2: E2e Actions Stats
code coverage: Report
storybook: Storybook
typescript migration: Dashboard
all artifacts
bundle viz:
- background: 0, 1, 2, 3, 4, 5
- common: 0, 1, 2, 3, 4, 5, 6
- content-script: 0
- ui: 0, 1, 10, 11, 12, 2, 3, 4, 5, 6, 7, 8, 9

Page Load Metrics (872 ± 517 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	71	387	140	78	38
		domContentLoaded	10	58	27	15	7
		load	56	2742	872	1077	517
		domInteractive	10	58	27	15	7

Bundle size diffs

background: 0 Bytes (0.00%)
ui: 0 Bytes (0.00%)
common: 0 Bytes (0.00%)

chloeYue · 2024-04-17T14:41:27Z

Test OK with gauthierpetetin-test/repo_test#177

chloeYue

Test OK

.github/guidelines/LABELING_GUIDELINES.md

davidmurdoch · 2024-04-17T19:44:44Z

.github/scripts/check-template-and-add-labels.ts

+    // If labelable is a flaky test report, no template is needed (we just add a link to circle.ci in the description), we skip the template checks
+    const flakyTestsLabelFound = findLabel(labelable, flakyTestsLabel);
+    if (flakyTestsLabelFound?.id) {
+      console.log(`Issue ${labelable?.number} was created to report a flaky test. No template is needed in that case: we just add a link to circle.ci in the description. Skip template checks.`);


where does the link get added?

You made me realise this sentence wasn't very clear. I updated it in this commit.
The idea is that flaky test issue's description is just a link redirecting to circle.ci which is why we don't expect issue's description to match issue template. Example flaky test issue: #22702

.github/scripts/shared/labelable.ts

metamaskbot · 2024-04-18T11:26:48Z

Builds ready [723ca33]

builds: chrome, firefox
builds (beta): chrome
builds (flask): chrome, firefox
builds (MMI): chrome, firefox
builds (test): chrome, firefox
builds (test-flask): chrome, firefox
build viz: Build System
mv3: Background Module Init Stats
mv3: UI Init Stats
mv3: Module Load Stats
mv3: Bundle Size Stats
mv2: E2e Actions Stats
code coverage: Report
storybook: Storybook
typescript migration: Dashboard
all artifacts
bundle viz:
- background: 0, 1, 2, 3, 4, 5
- common: 0, 1, 2, 3, 4, 5, 6
- content-script: 0
- ui: 0, 1, 10, 11, 12, 2, 3, 4, 5, 6, 7, 8, 9

Page Load Metrics (414 ± 382 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	72	390	108	67	32
		domContentLoaded	10	52	22	10	5
		load	59	2423	414	796	382
		domInteractive	10	52	22	10	5

Bundle size diffs

background: 0 Bytes (0.00%)
ui: 0 Bytes (0.00%)
common: 0 Bytes (0.00%)

chloeYue

Test OK !

seaona · 2024-05-24T09:50:12Z

hi @gauthierpetetin , could you add the flaky-tests label to this issue?
This will help me finishing with the testing flow of the PR 🙇‍♀️ thank you v much!!

gauthierpetetin-test/repo_test#178

seaona

thank you @gauthierpetetin the changes look good to me and the manual testing works as expected 👍

gauthierpetetin-test/repo_test#178

metamaskbot · 2024-05-24T13:49:44Z

Builds ready [7683d9b]

builds: chrome, firefox
builds (beta): chrome
builds (flask): chrome, firefox
builds (MMI): chrome, firefox
builds (test): chrome, firefox
builds (test-flask): chrome, firefox
build viz: Build System
mv3: Background Module Init Stats
mv3: UI Init Stats
mv3: Module Load Stats
mv3: Bundle Size Stats
mv2: E2e Actions Stats
code coverage: Report
storybook: Storybook
typescript migration: Dashboard
all artifacts
bundle viz:
- background: 0, 1, 2, 3, 4, 5, 6
- common: 0, 1, 2, 3, 4, 5, 6, 7, 8
- content-script: 0
- ui: 0, 1, 10, 11, 2, 3, 4, 5, 6, 7, 8, 9

Page Load Metrics (911 ± 559 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	61	141	86	20	9
		domContentLoaded	9	42	15	8	4
		load	50	2979	911	1165	559
		domInteractive	9	42	15	8	4

Bundle size diffs

background: 0 Bytes (0.00%)
ui: 0 Bytes (0.00%)
common: 0 Bytes (0.00%)

gauthierpetetin added 2 commits April 4, 2024 14:50

fix(labeling guidelines): Add definition of regression-develop label

08c910c

fix(action): skip template check for flaky tests issues

dbf1ffb

Flaky tests issues do not match our issues templates. Their description just includes a link to cricleci.

gauthierpetetin added the team-extension-platform label Apr 4, 2024

gauthierpetetin self-assigned this Apr 4, 2024

gauthierpetetin requested a review from a team as a code owner April 4, 2024 19:06

metamaskbot added the needs-dev-review PR needs reviews from other engineers (in order to receive required approvals) label Apr 4, 2024

chloeYue previously approved these changes Apr 17, 2024

View reviewed changes

davidmurdoch changed the title ~~Fix/flaky tests issues template skip~~ ci: skip template check for flaky tests issues Apr 17, 2024

davidmurdoch reviewed Apr 17, 2024

View reviewed changes

.github/guidelines/LABELING_GUIDELINES.md Outdated Show resolved Hide resolved

davidmurdoch reviewed Apr 17, 2024

View reviewed changes

.github/scripts/shared/labelable.ts Outdated Show resolved Hide resolved

davidmurdoch reviewed Apr 17, 2024

View reviewed changes

.github/scripts/shared/labelable.ts Outdated Show resolved Hide resolved

fix(action): small adjustments to resolve PR commments

723ca33

gauthierpetetin dismissed chloeYue’s stale review via 723ca33 April 17, 2024 21:54

gauthierpetetin requested review from davidmurdoch and chloeYue April 22, 2024 10:34

gauthierpetetin mentioned this pull request May 1, 2024

Fix/flaky tests issues template skip 2 MetaMask/metamask-mobile#9301

Merged

7 tasks

gauthierpetetin added 2 commits May 15, 2024 11:19

Merge branch 'develop' into fix/flaky-tests-issues-template-skip

90c07cb

Merge branch 'develop' into fix/flaky-tests-issues-template-skip

8d6571e

chloeYue approved these changes May 22, 2024

View reviewed changes

seaona approved these changes May 24, 2024

View reviewed changes

Merge branch 'develop' into fix/flaky-tests-issues-template-skip

7683d9b

gauthierpetetin merged commit 9b06ab4 into develop May 24, 2024
71 of 72 checks passed

gauthierpetetin deleted the fix/flaky-tests-issues-template-skip branch May 24, 2024 13:55

github-actions bot locked and limited conversation to collaborators May 24, 2024

github-actions bot removed the needs-dev-review PR needs reviews from other engineers (in order to receive required approvals) label May 24, 2024

metamaskbot added the release-11.18.0 label May 24, 2024

gauthierpetetin added the release-12.0.0 Issue or pull request that will be included in release 12.0.0 label Jun 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: skip template check for flaky tests issues #23857

ci: skip template check for flaky tests issues #23857

gauthierpetetin commented Apr 4, 2024 •

edited

Loading

github-actions bot commented Apr 4, 2024

codecov bot commented Apr 4, 2024 •

edited

Loading

metamaskbot commented Apr 4, 2024

chloeYue commented Apr 17, 2024

chloeYue left a comment

davidmurdoch Apr 17, 2024

gauthierpetetin Apr 17, 2024 •

edited

Loading

metamaskbot commented Apr 18, 2024

chloeYue left a comment

seaona commented May 24, 2024

seaona left a comment

metamaskbot commented May 24, 2024

ci: skip template check for flaky tests issues #23857

ci: skip template check for flaky tests issues #23857

Conversation

gauthierpetetin commented Apr 4, 2024 • edited Loading

Description

Related issues

Manual testing steps

Screenshots/Recordings

Pre-merge author checklist

Pre-merge reviewer checklist

github-actions bot commented Apr 4, 2024

codecov bot commented Apr 4, 2024 • edited Loading

Codecov Report

metamaskbot commented Apr 4, 2024

chloeYue commented Apr 17, 2024

chloeYue left a comment

Choose a reason for hiding this comment

davidmurdoch Apr 17, 2024

Choose a reason for hiding this comment

gauthierpetetin Apr 17, 2024 • edited Loading

Choose a reason for hiding this comment

metamaskbot commented Apr 18, 2024

chloeYue left a comment

Choose a reason for hiding this comment

seaona commented May 24, 2024

seaona left a comment

Choose a reason for hiding this comment

metamaskbot commented May 24, 2024

gauthierpetetin commented Apr 4, 2024 •

edited

Loading

codecov bot commented Apr 4, 2024 •

edited

Loading

gauthierpetetin Apr 17, 2024 •

edited

Loading