[e2e]: log exceptions when tests fail #16453

PeterYinusa · 2022-11-10T18:00:59Z

Explanation

Problem: Many engineers have run into issues where the e2e tests are failing on the pipeline and its not immediately obvious as to why. Usually it occurs when theres a Lavamoat error thrown on the background, preventing the login screen from loading

Solution: This PR surfaces exceptions thrown on the ui/background when a test fails

Notes: The webdriver method used is only available on Chrome, this is consistent with the existing error capturing logic we have

Screenshots/Screencaps

Before

yarn run v1.22.11
$ SELENIUM_BROWSER=chrome node test/e2e/run-all.js
$ /Users/peteryinusa/Documents/GitHub/metamask-extension-e2e/node_modules/.bin/mocha --no-config --timeout 60000 /Users/peteryinusa/Documents/GitHub/metamask-extension-e2e/test/e2e/tests/account-details.spec.js --exit


  Show account details
Starting ChromeDriver 107.0.5304.62 (1eec40d3a5764881c92085aaee66d25075c159aa-refs/branch-heads/5304@{#942}) on port 63365
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
    1) should show the QR code for the account


  0 passing (25s)
  1 failing

  1) Show account details
       should show the QR code for the account:
     TimeoutError: Waiting for element to be located By(css selector, #password)
Wait timed out after 10029ms
      at /Users/peteryinusa/Documents/GitHub/metamask-extension-e2e/node_modules/selenium-webdriver/lib/webdriver.js:906:17
      at processTicksAndRejections (node:internal/process/task_queues:96:5)



info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

After

yarn run v1.22.11
$ node test/e2e/run-e2e-test.js test/e2e/tests/account-details.spec.js --browser chrome
$ /Users/peteryinusa/Documents/GitHub/metamask-extension-e2e/node_modules/.bin/mocha --no-config --timeout 60000 test/e2e/tests/account-details.spec.js --exit


  Show account details
Starting ChromeDriver 107.0.5304.62 (1eec40d3a5764881c92085aaee66d25075c159aa-refs/branch-heads/5304@{#942}) on port 54610
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
    1) should show the QR code for the account


  0 passing (25s)
  1 failing

  1) Show account details
       should show the QR code for the account:
     Error: Errors found in browser console:
Error: LavaMoat - required package not in allowlist: package "@metamask/controllers>eth-keyring-controller" requested "browserify>events" as "events"
  at requireRelative (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:11029:15)
  at requireRelativeWithContext (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:10978:18)
  at Object.<anonymous> (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/common-2.js:1:205594)
  at internalRequire (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:10966:27)
  at requireRelative (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:11009:27)
  at requireRelativeWithContext (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:10978:18)
  at Object.<anonymous> (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/common-2.js:1:60750)
  at internalRequire (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:10966:27)
  at requireRelative (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:11009:27)
  at requireRelativeWithContext (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:10978:18)
  at Object.<anonymous> (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/common-2.js:1:56697)
  at internalRequire (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:10966:27)
  at requireRelative (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:11009:27)
  at requireRelativeWithContext (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:10978:18)
  at Object.<anonymous> (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/ui-7.js:1:70478)
  at internalRequire (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:10966:27)
  at requireRelative (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:11009:27)
  at requireRelativeWithContext (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:10978:18)
  at Object.<anonymous> (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/ui-8.js:1:545545)
  at internalRequire (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:10966:27)
  at requireRelative (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:11009:27)
  at requireRelativeWithContext (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:10978:18)
  at Object.<anonymous> (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/ui-8.js:1:560923)
  at internalRequire (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:10966:27)
  at requireRelative (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:11009:27)
  at requireRelativeWithContext (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:10978:18)
  at Object.<anonymous> (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/ui-9.js:1:18764)
  at internalRequire (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:10966:27)
  at requireRelative (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:11009:27)
  at requireRelativeWithContext (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:10978:18)
  at Object.<anonymous> (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/ui-8.js:1:270248)
  at internalRequire (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:10966:27)
  at requireRelative (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:11009:27)
  at requireRelativeWithContext (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:10978:18)
  at Object.<anonymous> (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/ui-8.js:1:280767)
  at internalRequire (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:10966:27)
  at requireRelative (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:11009:27)
  at requireRelativeWithContext (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:10978:18)
  at Object.<anonymous> (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/ui-8.js:1:263482)
  at internalRequire (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:10966:27)
  at requireRelative (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:11009:27)
  at requireRelativeWithContext (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:10978:18)
  at Object.<anonymous> (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/ui-7.js:1:507185)
  at internalRequire (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:10966:27)
  at requireRelative (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:11009:27)
  at requireRelativeWithContext (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:10978:18)
  at Object.<anonymous> (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/ui-7.js:1:87850)
  at internalRequire (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:10966:27)
  at requireRelative (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:11009:27)
  at requireRelativeWithContext (chrome-extension://eemkccoomdelmfifgfcnkfbjhhfenbpk/runtime-lavamoat.js:10978:18)
      at withFixtures (test/e2e/helpers.js:158:15)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async Context.<anonymous> (test/e2e/tests/account-details.spec.js:16:5)



info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Manual Testing Steps

checkout this branch
throw and error in the background here

async function initialize(remotePort) {
  const initState = await loadStateFromPersistence();
  const initLangCode = await getFirstPreferredLangCode();
  throw new Error('Something went wrong');
  setupController(initState, initLangCode, remotePort);
  if (!isManifestV3) {
    await loadPhishingWarningPage();
  }
  log.info('MetaMask initialization complete.');
}

yarn setup
yarn build test
yarn test:e2e:single test/e2e/tests/account-details.spec.js --browser chrome
check the exception is shown (previously the test would just log that it timed out waiting for the password field on the lock screen)

Pre-merge author checklist

I've clearly explained:
- What problem this PR is solving
- How this problem was solved
- How reviewers can test my changes
Sufficient automated test coverage has been added

Pre-merge reviewer checklist

Manual testing (e.g. pull and build branch, run in browser, test code being changed)
PR is linked to the appropriate GitHub issue
IF this PR fixes a bug in the release milestone, add this PR to the release milestone

If further QA is required (e.g. new feature, complex testing steps, large refactor), add the Extension QA Board label.

In this case, a QA Engineer approval will be be required.

github-actions · 2022-11-10T18:01:10Z

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

metamaskbot · 2022-11-10T18:20:17Z

Builds ready [475bc91]

builds: chrome, firefox
builds (beta): chrome, firefox
builds (flask): chrome, firefox
build viz: Build System
mv3: Background Module Init Stats
mv3: UI Init Stats
mv3: Module Load Stats
mv3: Bundle Size Stats
mv2: E2e Actions Stats
code coverage: Report
storybook: Storybook
typescript migration: Dashboard
all artifacts
bundle viz:
- background: 0, 1, 2, 3, 4
- common: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9
- content-script: 0
- contentscript: contentscript
- disable-console: disable-console
- inpage: inpage
- policy-load: policy-load
- sentry-install: sentry-install
- ui: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9

Page Load Metrics (2281 ± 115 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	95	1841	206	376	180
		domContentLoaded	1709	2805	2258	230	111
		load	1709	2886	2281	240	115
		domInteractive	1709	2805	2258	230	111

Bundle size diffs

background: 0 bytes
ui: 0 bytes
common: 0 bytes

metamaskbot · 2022-11-11T11:20:33Z

Builds ready [f0a1eb8]

builds: chrome, firefox
builds (beta): chrome, firefox
builds (flask): chrome, firefox
build viz: Build System
mv3: Background Module Init Stats
mv3: UI Init Stats
mv3: Module Load Stats
mv3: Bundle Size Stats
mv2: E2e Actions Stats
code coverage: Report
storybook: Storybook
typescript migration: Dashboard
all artifacts
bundle viz:
- background: 0, 1, 2, 3, 4
- common: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9
- content-script: 0
- contentscript: contentscript
- disable-console: disable-console
- inpage: inpage
- policy-load: policy-load
- sentry-install: sentry-install
- ui: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9

Page Load Metrics (2461 ± 156 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	99	261	133	38	18
		domContentLoaded	1858	3039	2443	326	156
		load	1862	3039	2461	325	156
		domInteractive	1857	3039	2443	326	156

Bundle size diffs

background: 0 bytes
ui: 0 bytes
common: 0 bytes

metamaskbot · 2022-11-11T12:28:17Z

Builds ready [9d14fc2]

builds: chrome, firefox
builds (beta): chrome, firefox
builds (flask): chrome, firefox
build viz: Build System
mv3: Background Module Init Stats
mv3: UI Init Stats
mv3: Module Load Stats
mv3: Bundle Size Stats
mv2: E2e Actions Stats
code coverage: Report
storybook: Storybook
typescript migration: Dashboard
all artifacts
bundle viz:
- background: 0, 1, 2, 3, 4
- common: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9
- content-script: 0
- contentscript: contentscript
- disable-console: disable-console
- inpage: inpage
- policy-load: policy-load
- sentry-install: sentry-install
- ui: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9

Page Load Metrics (2230 ± 98 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	91	167	116	20	9
		domContentLoaded	1837	2746	2209	193	93
		load	1837	2845	2230	204	98
		domInteractive	1837	2746	2209	193	93

Bundle size diffs

background: 0 bytes
ui: 0 bytes
common: 0 bytes

Gudahtt

LGTM!

log exceptions inc background

475bc91

PeterYinusa marked this pull request as ready for review November 10, 2022 18:21

PeterYinusa requested a review from a team as a code owner November 10, 2022 18:21

PeterYinusa requested a review from danjm November 10, 2022 18:21

PeterYinusa changed the title ~~log exceptions inc background~~ [e2e]: log background exceptions Nov 10, 2022

show full stacktrace

f0a1eb8

PeterYinusa changed the title ~~[e2e]: log background exceptions~~ [e2e]: log exceptions when tests fail Nov 11, 2022

PeterYinusa added 2 commits November 11, 2022 11:48

use driver api

a76177a

lint

9d14fc2

brad-decker approved these changes Nov 11, 2022

View reviewed changes

Gudahtt approved these changes Nov 11, 2022

View reviewed changes

PeterYinusa merged commit b2293e6 into develop Nov 14, 2022

PeterYinusa deleted the e2e-log-exceptions branch November 14, 2022 14:35

github-actions bot locked and limited conversation to collaborators Nov 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[e2e]: log exceptions when tests fail #16453

[e2e]: log exceptions when tests fail #16453

PeterYinusa commented Nov 10, 2022 •

edited

Loading

github-actions bot commented Nov 10, 2022

metamaskbot commented Nov 10, 2022

metamaskbot commented Nov 11, 2022

metamaskbot commented Nov 11, 2022

Gudahtt left a comment

[e2e]: log exceptions when tests fail #16453

[e2e]: log exceptions when tests fail #16453

Conversation

PeterYinusa commented Nov 10, 2022 • edited Loading

Explanation

Screenshots/Screencaps

Before

After

Manual Testing Steps

Pre-merge author checklist

Pre-merge reviewer checklist

github-actions bot commented Nov 10, 2022

metamaskbot commented Nov 10, 2022

metamaskbot commented Nov 11, 2022

metamaskbot commented Nov 11, 2022

Gudahtt left a comment

Choose a reason for hiding this comment

PeterYinusa commented Nov 10, 2022 •

edited

Loading