Fix Cascading test failures and reported as Ganache errors

[hjetpoluru]

CI failure reported on Dec 6 (MultiRpc: should migrate to multi rpc @no-mmi)

test-e2e-chrome
Test path - /test/e2e/tests/network/multi-rpc.spec.ts
Test scenario

• MultiRpc: should migrate to multi rpc @no-mmi
• MultiRpc: should select rpc from modal
• MultiRpc: should select rpc from edit menu
• MultiRpc: should select rpc from settings @no-mmi

CI failure reported on Dec 5 (Sign Typed Data Signature Request can queue multiple...)

test-e2e-chrome-webpack
Test Path -/test/e2e/tests/signature/signature-request.spec.js
Test scenario

• Sign Typed Data Signature Request can queue multiple Signature Requests of Sign Typed Data and confirm
• Sign Typed Data Signature Request can queue multiple Signature Requests of Sign Typed Data V3 and confirm
• Sign Typed Data Signature Request can queue multiple Signature Requests of Sign Typed Data V4 and confirm
• Sign Typed Data Signature Request can initiate and reject a Signature Request of Sign Typed Data
• Sign Typed Data Signature Request can initiate and reject a Signature Request of Sign Typed Data V3
• Sign Typed Data Signature Request can initiate and reject a Signature Request of Sign Typed Data V4
• Sign Typed Data Signature Request can queue multiple Signature Requests of Sign Typed Data and reject
• Sign Typed Data Signature Request can queue multiple Signature Requests of Sign Typed Data V3 and reject
• Sign Typed Data Signature Request can queue multiple Signature Requests of Sign Typed Data V4 and reject

CI failure reported on Dec 3 (Encrypt Decrypt should decrypt an encrypted message)

test-e2e-chrome-webpack
Test path - /test/e2e/tests/dapp-interactions/encrypt-decrypt.spec.js
Test scenario

• Encrypt Decrypt should decrypt an encrypted message
• Encrypt Decrypt should encrypt and decrypt multiple messages
• Encrypt Decrypt should show balance correctly as ETH
• Encrypt Decrypt should show balance correctly in native tokens

CI failure reported on Dec 2 (Encrypt Decrypt should show balance correctly as ETH)

test-e2e-chrome
Test path - /test/e2e/tests/dapp-interactions/encrypt-decrypt.spec.js
Test scenario

• Encrypt Decrypt should show balance correctly as ETH
• Encrypt Decrypt should show balance correctly in native tokens

CI failure reported on Dec 2 (Settings - general tab @no-mmi validate "العربية" ...)

test-e2e-chrome
Test path - /test/e2e/tests/settings/change-language.spec.ts
Test scenario

• Settings - general tab @no-mmi validate "Deutsch" language on error messages
• Settings - general tab @no-mmi validate "मानक हिन्दी" language on tooltips
• Settings - general tab @no-mmi validate "Magyar" language change on hypertext
• Settings - general tab @no-mmi validate "العربية" language change on page indent

CI failure reported on Nov 22(Send ETH from inside MetaMask finds the transaction ...)

Build - Regular
Browser - Chrome
Test path - /test/e2e/tests/transaction/send-eth.spec.js
Test scenario

• Send ETH from inside MetaMask finds the transaction in the transactions list using advanced gas modal
• Send ETH from inside MetaMask finds the transaction in the transactions list when sending to a Multisig Address
• Send ETH from inside MetaMask shows no error when cancel transaction when sending via QR code
• Send ETH from inside MetaMask from dapp using advanced gas controls should display the correct gas price on the legacy transaction
• Send ETH from inside MetaMask from dapp using advanced gas controls should display correct gas values for EIP-1559 transaction
• Send ETH from inside MetaMask to non-contract address with data that matches ERC20 transfer data signature renders the correct recipient on the confirmation screen

CI failure reported on Nov 20th (Reveal SRP through settings completes quiz and revea...)

Build - Regular
Browser - Chrome
Test Path - /test/e2e/tests/settings/settings-security-reveal-srp.spec.ts
Test Scenario

• Reveal SRP through settings completes quiz and reveals SRP text
• Reveal SRP through settings completes quiz and reveals SRP QR after wrong answers in quiz

CI failure reported on Nov 12th (Marketing cookieId should not be send to segment whe...)

Build - Regular
Browser - Chrome
Test Path - /test/e2e/tests/metrics/marketing-cookieid.spec.ts
Test Scenario

• Marketing cookieId should not be send to segment when dataCollectionForMarketing is never toggled on
• Marketing cookieId should not be send to segment when participateInMetaMetrics is never toggled on
• Marketing cookieId should updates marketingCampaignCookieId to null when dataCollectionForMarketing is toggled off

CI failure reported on Nov 12th (Encrypt Decrypt should show balance correctly as ETH)

Build - Regular
Browser - Chrome
Test Path - /test/e2e/tests/dapp-interactions/encrypt-decrypt.spec.js
Test Scenario

• Encrypt Decrypt should show balance correctly as ETH
• Encrypt Decrypt should show balance correctly in native tokens

CI failure reported on Nov 12th(Confirmation Redesign Token Send @no-mmi ERC721 dApp...)

Build - Regular
Browser - Chrome
Test Path - /test/e2e/tests/confirmations/transactions/nft-token-send-redesign.spec.ts
Test Scenario

• Confirmation Redesign Token Send @no-mmi ERC721 dApp initiated Sends a type 0 transaction (Legacy)
• Confirmation Redesign Token Send @no-mmi ERC721 dApp initiated Sends a type 2 transaction (EIP1559)
• Confirmation Redesign Token Send @no-mmi ERC1155 Wallet initiated Sends a type 0 transaction (Legacy)
• Confirmation Redesign Token Send @no-mmi ERC1155 Wallet initiated Sends a type 2 transaction (EIP1559)

Note - This above flaky test scenario has happened on the second attempt too.

CI failure reported on Nov 11th (Sign Typed Data Signature Request can initiate and r...)

Build - mmi
Browser - Chrome
Test Path - /test/e2e/tests/signature/signature-request.spec.js
Test scenario

• Sign Typed Data Signature Request can initiate and reject a Signature Request of Sign Typed Data V4
• Sign Typed Data Signature Request can queue multiple Signature Requests of Sign Typed Data and reject
• Sign Typed Data Signature Request can queue multiple Signature Requests of Sign Typed Data V3 and reject
• Sign Typed Data Signature Request can queue multiple Signature Requests of Sign Typed Data V4 and reject

CI failure reported on Nov 1st(ERC1155 NFTs testdapp interaction should batch trans...)

Build - mmi
Browser - Chrome
Test Path - /test/e2e/tests/tokens/nft/erc1155-interaction.spec.js
Test scenario

• ERC1155 NFTs testdapp interaction should batch transfers ERC1155 token
• ERC1155 NFTs testdapp interaction should enable approval for a third party address to manage all ERC1155 token
• ERC1155 NFTs testdapp interaction should revoke approval for a third party address to manage all ERC1155 token

CI failure reported on Oct 29th (Request-queue UI changes handles three confirmations...)

Build - Webpack
Browser - Chrome
Test Path -/test/e2e/tests/request-queuing/ui.spec.js
Test scenario

• Request-queue UI changes should show network specific to domain @no-mmi
• Request-queue UI changes handles three confirmations on three confirmations concurrently @no-mmi
• Request-queue UI changes should gracefully handle deleted network @no-mmi
• Request-queue UI changes handles three confirmations on three confirmations concurrently @no-mmi
• Request-queue UI changes should gracefully handle deleted network @no-mmi
• Request-queue UI changes should signal from UI to dapp the network change @no-mmi
• Request-queue UI changes should autoswitch networks to the last used network for domain
• Request-queue UI changes should autoswitch networks when last confirmation from another network is rejected
• Request-queue UI changes should gracefully handle network connectivity failure for signatures @no-mmi
• Request-queue UI changes should gracefully handle network connectivity failure for confirmations @no-mmi

CI failure reported on Oct 23rd (ERC721 NFTs testdapp interaction should prompt users...)

Build - Webpack
Browser - Chrome
Test Path - /test/e2e/tests/tokens/nft/erc721-interaction.spec.js
Test Scenario

• ERC721 NFTs testdapp interaction should prompt users to add their NFTs to their wallet (all at once)
• ERC721 NFTs testdapp interaction should transfer a single ERC721 NFT from one account to another
• ERC721 NFTs testdapp interaction should approve an address to transfer a single ERC721 NFT
• ERC721 NFTs testdapp interaction should enable approval for a third party address to manage all ERC721 NFTs @no-mmi
• ERC721 NFTs testdapp interaction should disable approval for a third party address to manage all ERC721 NFTs @no-mmi

CI failure reported on Oct 23rd (Snap Account Signatures @no-mmi can sign with approv...)

Build - Webpack
Browser - Chrome
Test Path - /test/e2e/tests/account/snap-account-signatures.spec.ts
Test Scenario -

• Snap Account Signatures @no-mmi can sign with approve flow
• Snap Account Signatures @no-mmi can sign with reject flow

CI failure reported on Oct 23rd (Unlock wallet should send first three Page metric ev...)

Build - Webpack
Browser - Chrome
Test Path - /test/e2e/tests/metrics/unlock-wallet.spec.js
Test Scenario -

• Unlock wallet should send first three Page metric events upon fullscreen page load

CI failure reported on Oct 21st (Request-queue UI changes handles three confirmations...)

Build - Webpack
Browser - Chrome
Test Path -/test/e2e/tests/request-queuing/ui.spec.js
Test Scenario -

• Request-queue UI changes handles three confirmations on three confirmations concurrently @no-mmi
• Request-queue UI changes should gracefully handle deleted network @no-mmi
• Request-queue UI changes should signal from UI to dapp the network change @no-mmi
• Request-queue UI changes should autoswitch networks to the last used network for domain
• Request-queue UI changes should autoswitch networks when last confirmation from another network is rejected
• Request-queue UI changes should gracefully handle network connectivity failure for signatures @no-mmi
• Request-queue UI changes should gracefully handle network connectivity failure for confirmations @no-mmi

CI failure reported on Oct 21st(Review Permissions page should disconnect when click...)

Build - mmi
Browser - Chrome
Test Path -/test/e2e/tests/connections/review-permissions-page.spec.js
Test Scenario -

• Review Permissions page should disconnect when click on Disconnect button in connections page

Executing testcase: 'Review Permissions page should disconnect when click on Disconnect button in connections page'

2024-10-18T16:07:12.682Z [driver] Called 'navigate' with arguments []
ServerMochaToBackground got a client connection
manifest-flag-mocha-hooks.ts -- afterEach hook

CI failure reported on Oct 18th (Add account should display correct new account name ...)

Build - Regular
Browser - Chrome
Test Path - /test/e2e/tests/account/add-account.spec.js
Test Scenario -

• Add account should display correct new account name after create
• Add account should not affect public address when using secret recovery phrase to recover account with non-zero balance @no-mmi
• Add account should be possible to remove an account imported with a private key, but should not be possible to remove an account generated from the SRP imported in onboarding @no-mmi

CI failure reported on Oct 18th (Snap Account Signatures @no-mmi can sign with reject...)

Build - Webpack
Browser - Chrome
Test Path -/test/e2e/tests/account/snap-account-signatures.spec.ts
Test Scenario -

• Snap Account Signatures @no-mmi can sign with approve flow
• Snap Account Signatures @no-mmi can sign with reject flow

CI failure reported on Oct 17th (Gas estimates generated by MetaMask Send on a networ...)

Build - Regular
Browser - Chrome
Test Path -/test/e2e/tests/transaction/gas-estimates.spec.js
Test Scenario -

• Gas estimates generated by MetaMask Send on a network that is not EIP-1559 compatible show expected gas defaults on a network supported by legacy gas API when that API is down
• Gas estimates generated by MetaMask Send on a network that is not EIP-1559 compatible show expected gas defaults on a network not supported by legacy gas API

Build - rpc-mmi
Browser - Chrome
Test Path -/test/e2e/json-rpc/switchEthereumChain.spec.js
Test Scenario -

CI failure reported on Oct 17th (Switch Ethereum Chain for two dapps switches the cha...)

Build - rpc-mmi
Browser - Chrome
Test Path -/test/e2e/json-rpc/switchEthereumChain.spec.js
Test Scenario -

• Switch Ethereum Chain for two dapps switches the chainId of two dapps when switchEthereumChain of one dapp is confirmed
• Switch Ethereum Chain for two dapps queues switchEthereumChain request from second dapp after send tx request
• Switch Ethereum Chain for two dapps queues send tx after switchEthereum request with a warning, confirming removes pending tx
• Switch Ethereum Chain for two dapps queues send tx after switchEthereum request with a warning, if switchEthereum request is cancelled should show pending tx

CI failure reported on Oct 17th (Snap Account Signatures @no-mmi can sign with approv...)

Build - Webpack
Browser - Chrome
Test Path -/test/e2e/tests/account/snap-account-signatures.spec.ts
Test Scenario -

• Snap Account Signatures @no-mmi can sign with approve flow
• Snap Account Signatures @no-mmi can sign with reject flow

CI failure reported on Oct 16th (Request-queue UI changes should autoswitch networks ...)

CI failure reported on Oct 16th (Request-queue UI changes handles three confirmations...)

Build - Webpack
Browser - Chrome
Test Path -/test/e2e/tests/request-queuing/ui.spec.js
Test Scenario -

• Request-queue UI changes handles three confirmations on three confirmations concurrently @no-mmi
• Request-queue UI changes should gracefully handle deleted network @no-mmi
• Request-queue UI changes should signal from UI to dapp the network change @no-mmi
• Request-queue UI changes should autoswitch networks to the last used network for domain

CI failure reported on Oct 14th (Create Snap Account @no-mmi create Snap account canc...)

Build - Webpack
Browser - Chrome
Test Path -/test/e2e/tests/account/create-snap-account.spec.ts
Test Scenario -

• Create Snap Account @no-mmi create Snap account canceling on confirmation screen results in error on Snap
• Create Snap Account @no-mmi create Snap account canceling on fill name screen results in error on Snap

CI failure reported on Oct 14th(Confirmation Redesign ERC20 Approve Component Submit...)

Build - Regular
Browser - Chrome
Test Path - /test/e2e/tests/confirmations/transactions/revoke-allowance-redesign.spec.ts
Test Scenario -

• Confirmation Redesign ERC20 Approve Component Submit an Approve transaction @no-mmi Sends a type 0 transaction (Legacy)
• Confirmation Redesign ERC20 Approve Component Submit an Approve transaction @no-mmi Sends a type 2 transaction (EIP1559)
• Confirmation Redesign ERC20 Increase Allowance Submit an increase allowance transaction @no-mmi Sends a type 0 transaction (Legacy) with a small spending cap
• Confirmation Redesign ERC20 Increase Allowance Submit an increase allowance transaction @no-mmi Sends a type 2 transaction (EIP1559) with a small spending cap
• Confirmation Redesign ERC20 Increase Allowance Submit an increase allowance transaction @no-mmi Sends a type 0 transaction (Legacy) with a large spending cap
• Confirmation Redesign ERC20 Increase Allowance Submit an increase allowance transaction @no-mmi Sends a type 2 transaction (EIP1559) with a large spending cap
• Confirmation Redesign ERC20 Revoke Allowance Submit an revoke transaction @no-mmi Sends a type 0 transaction (Legacy)
• Confirmation Redesign ERC20 Revoke Allowance Submit an revoke transaction @no-mmi Sends a type 2 transaction (EIP1559)

CI failure reported on Oct 11th (Snap Account Signatures can sign with approve flow)

Build - Webpack
Browser - Chrome
Test Path - /test/e2e/accounts/snap-account-signatures.spec.ts
Test Scenario -

• Snap Account Signatures can sign with approve flow
• Snap Account Signatures can sign with reject flow

CI failure reported on Oct 9th (Sentry errors before initialization, after opting ou...)

Build - Regular
Browser - Chrome
Test Path - /test/e2e/tests/metrics/errors.spec.js
Test Scenario -

• Sentry errors before initialization, after opting out of metrics @no-mmi should NOT send error events in the background
• Sentry errors before initialization, after opting out of metrics @no-mmi should NOT send error events in the UI
• Sentry errors before initialization, after opting into metrics @no-mmi should send error events in background
• Sentry errors before initialization, after opting into metrics @no-mmi should capture background application state
• Sentry errors before initialization, after opting into metrics @no-mmi should send error events in UI
• Sentry errors before initialization, after opting into metrics @no-mmi should capture UI application state
• Sentry errors after initialization, after opting out of metrics @no-mmi should NOT send error events in the background
• Sentry errors after initialization, after opting out of metrics @no-mmi should NOT send error events in the UI
• Sentry errors after initialization, after opting into metrics @no-mmi should send error events in background
• Sentry errors after initialization, after opting into metrics @no-mmi should capture background application state
• Sentry errors after initialization, after opting into metrics @no-mmi should send error events in UI
• Sentry errors after initialization, after opting into metrics @no-mmi should capture UI application state

CI failure reported on Oct 9th (Malicious Confirmation Signature - Bad Domain @no-mm...)

Build - Webpack
Browser - Chrome
Test Path - /test/e2e/tests/confirmations/signatures/malicious-signatures.spec.ts
Test Scenario -

• Malicious Confirmation Signature - Bad Domain @no-mmi initiates and rejects from confirmation screen
• Malicious Confirmation Signature - Bad Domain @no-mmi initiates and rejects from alert friction modal

CI failure reported on Oct 8th (Request-queue UI changes handles three confirmations...)

Build - Webpack
Browser - Chrome
Test Path - /test/e2e/tests/request-queuing/ui.spec.js
Test Scenario -

• Request-queue UI changes should show network specific to domain @no-mmi
• Request-queue UI changes handles three confirmations on three confirmations concurrently @no-mmi
• Request-queue UI changes should gracefully handle deleted network @no-mmi
• Request-queue UI changes should signal from UI to dapp the network change @no-mmi
• Request-queue UI changes should autoswitch networks to the last used network for domain
• Request-queue UI changes should autoswitch networks when last confirmation from another network is rejected
• Request-queue UI changes should gracefully handle network connectivity failure for signatures @no-mmi
• Request-queue UI changes should gracefully handle network connectivity failure for confirmations @no-mmi

CI failure reported on Sept 27(Malicious Confirmation Signature - Bad Domain @no-mm...)

Build - Webpack
Browser - Chrome
Test Path - /test/e2e/tests/confirmations/signatures/malicious-signatures.spec.ts
Test Scenario -

• Malicious Confirmation Signature - Bad Domain @no-mmi initiates and rejects from confirmation screen
• Malicious Confirmation Signature - Bad Domain @no-mmi initiates and rejects from alert friction modal

CI failure reported on Sept 27(Snap Account Signatures can sign with sync flow)

Build - Webpack
Browser - Chrome
Test Path - /test/e2e/accounts/snap-account-signatures.spec.ts
Test Scenario -

• Snap Account Signatures can sign with sync flow
• Snap Account Signatures can sign with approve flow
• Snap Account Signatures can sign with reject flow

Error: Timeout of 120000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/home/circleci/project/test/e2e/accounts/snap-account-signatures.spec.ts)
    at listOnTimeout (node:internal/timers:581:17)
    at process.processTimers (node:internal/timers:519:7)

CI failure reported on Sept 26(Create token, approve token and approve token withou...

Build - Webpack
Browser - Chrome
Test Path - /test/e2e/tests/tokens/custom-token-add-approve.spec.js
Test Scenario -

• Create token, approve token and approve token without gas approves an already created token and displays the token approval data @no-mmi
• Create token, approve token and approve token without gas set custom spending cap, customizes gas, edit spending cap and checks transaction in transaction list @no-mmi
• Create token, approve token and approve token without gas set maximum spending cap, submits the transaction and finds the transaction in the transactions list @no-mmi
• Create token, approve token and approve token without gas approves token without gas, set site suggested spending cap, submits the transaction and finds the transaction in the transactions list @no-mmi

CI failure reported on Sept 20
https://app.circleci.com/pipelines/github/MetaMask/metamask-extension/101484/workflows/02584d3e-9e9d-49d9-9953-0944167dace6/jobs/3778719/tests

Build - Webpack
Browser - Chrome
Test Path - /test/e2e/tests/transaction/send-eth.spec.js
Test Scenario -

• Send ETH from inside MetaMask finds the transaction in the transactions list using advanced gas modal
• Send ETH from inside MetaMask finds the transaction in the transactions list when sending to a Multisig Address
• Send ETH from inside MetaMask shows no error when cancel transaction when sending via QR code
• Send ETH from inside MetaMask from dapp using advanced gas controls should display the correct gas price on the legacy transaction
• Send ETH from inside MetaMask from dapp using advanced gas controls should display correct gas values for EIP-1559 transaction
• Send ETH from inside MetaMask to non-contract address with data that matches ERC20 transfer data signature renders the correct recipient on the confirmation screen

CI failure reported on Sept 20
https://app.circleci.com/pipelines/github/MetaMask/metamask-extension/101369/workflows/6c8551cb-1b8c-4ff5-a912-889d65e289b2/jobs/3774012/tests

Build - mmi
Browser - Chrome
Test Path - /test/e2e/tests/signature/signature-request.spec.js
Test Scenario -

• Sign Typed Data Signature Request can queue multiple Signature Requests of Sign Typed Data V3 and reject
• Sign Typed Data Signature Request can queue multiple Signature Requests of Sign Typed Data V4 and reject

CI failure reported on Sept 19
https://app.circleci.com/pipelines/github/MetaMask/metamask-extension/101259/workflows/506f3637-17c1-42dd-82b0-8e62ee9e6ff3/jobs/3770240/tests

Build - Regular
Browser - Chrome
Test Path - /test/e2e/tests/metrics/dapp-viewed.spec.js
Test Scenario -

• Dapp viewed Event @no-mmi is sent when refreshing dapp with one account connected
• Dapp viewed Event @no-mmi is sent when navigating to a connected dapp
• Dapp viewed Event @no-mmi is sent when connecting dapp with two accounts
• Dapp viewed Event @no-mmi is sent when reconnect to a dapp that has been connected before

CI failure reported on Sept 19
https://app.circleci.com/pipelines/github/MetaMask/metamask-extension/101117/workflows/42636518-13bc-42c7-8624-9334fe1ca94f/jobs/3764726/tests

Build - mmi
Browser - Chrome
Test Path - /test/e2e/tests/settings/settings-search.spec.js
Test Scenario -

• Settings Search should find element inside the "Security & privacy" tab
• Settings Search should find element inside the Alerts tab
• Settings Search should find element inside the Networks tab
• Settings Search should find element inside the Experimental tab
• Settings Search should find element inside the About tab
• Settings Search should display "Element not found" for a non-existing element

CI failure reported on Sept 17
https://app.circleci.com/pipelines/github/MetaMask/metamask-extension/100638/workflows/39f21f3a-0ac3-454b-a34a-3abe8ec618ac/jobs/3747023/tests

Build - webpack
Browser - Chrome
Test Path - /test/e2e/tests/simulation-details/simulation-details.spec.ts
Test Scenarios -

• Simulation Details renders send eth transaction
• Simulation Details renders buy ERC20 transaction
• Simulation Details renders buy ERC721 transaction
• Simulation Details renders buy ERC1155 transaction
• Simulation Details renders no changes transaction
• Simulation Details displays error message if transaction will fail or revert
• Simulation Details does not display if chain is not supported
• Simulation Details displays generic error message

Another CI failure reported on Sept 17
https://app.circleci.com/pipelines/github/MetaMask/metamask-extension/100523/workflows/8e4175f9-0b34-416f-815d-2524eb92820e/jobs/3742442/tests

Build - webpack
Browser - Chrome
Test Path - /test/e2e/tests/transaction/send-eth.spec.js
Test Scenario -

• Send ETH from inside MetaMask finds the transaction in the transactions list using advanced gas modal
• Send ETH from inside MetaMask finds the transaction in the transactions list when sending to a Multisig Address
• Send ETH from inside MetaMask shows no error when cancel transaction when sending via QR code
• Send ETH from inside MetaMask from dapp using advanced gas controls should display the correct gas price on the legacy transaction

CI failure reported on Sept 13
https://app.circleci.com/pipelines/github/MetaMask/metamask-extension/100198/workflows/7d530039-f9aa-486c-ab61-17e60c5647ca/jobs/3730323/tests

Build - mmi
Browser - Chrome
Test Path - /test/e2e/tests/network/add-custom-network.spec.js
Test scenarios -

• Custom network JSON-RPC API don't validate bad rpc custom network when toggle is off
• Custom network JSON-RPC API don't add unreachable custom network
• Custom network Popular Networks List add custom network and switch the network
• Custom network Popular Networks List add custom network and not switch the network
• Custom network Popular Networks List delete the Arbitrum network
• Custom network Popular Networks List when the network details validation toggle is turned on, validate user inserted details against data from 'chainid.network'
• Custom network Popular Networks List when the network details validation toggle is turned off, don't validate user inserted details
• Custom network customNetwork should add mainnet network
• Custom network customNetwork should check symbol and show warnings
• Custom network customNetwork should add collision network

Initial error message for the first test scenario

Error: Timeout of 80000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/home/circleci/project/test/e2e/tests/network/add-custom-network.spec.js)
    at listOnTimeout (node:internal/timers:581:17)
    at process.processTimers (node:internal/timers:519:7)

Error: listen EADDRINUSE: address already in use 127.0.0.1:8545.
  (Ran on CircleCI Node 1 of 12, Job test-e2e-chrome-mmi)
    at /home/circleci/project/node_modules/ganache/dist/node/1.js:2:5133
    at async Promise.allSettled (index 1)
    at async Ganache.start (test/e2e/seeder/ganache.ts:2:1285)
    at async withFixtures (test/e2e/helpers.js:105:7)
    at async Context.<anonymous> (test/e2e/tests/network/add-custom-network.spec.js:393:7)

[HowardBraham]

@hjetpoluru I found a very simple reproduction for this error.

In any suite that has multiple tests, add this.timeout(5000); right under the describe( line.

The first test will fail with Error: Timeout of 5000ms exceeded....
The subsequent tests will fail with Error: listen EADDRINUSE: address already in use 127.0.0.1:8545

I did a little digging, and what's happening is that if mocha causes the test to quit (rather than the test internally throwing an error), the blocks in withFixtures() for catch and finally don't run. So the teardown doesn't happen properly, and the next tests fail.

This is not an easy thing to fix, it's a huge funnel cake.

[hjetpoluru]

Discussion about this flaky test in the slack channel

[hjetpoluru]

PR that @davidmurdoch helped to organize.

Gist of it is that withFixtures must be called in a describe block, which will:

• install a afterEach handler that will be called after each tests completes (pass/failure/timeout)
• return a start function that a test can use

If a test uses the start function it will set up an end function that will always be called after the test finishes (pass/failure/timeout). The end function does all the clean up that withFixtures did previously, but it now does it even if the tests time out.

Additionally, this changes how we run tests "with fixtures". Instead of passing a callback we now await the call to start, which will give us our fixtures in it's return value (like the driver).
This is how it is used:

describe("tests", function() {
  const fixtures = withFixtures();

  it("times out", async function() {
    const { driver } = fixtures.start({ ... options ... });
    // the test goes here, NOT in a callback
    await new Promise(resolve => setTimeout(resolve. 100000));
    // even though this test times out, the next function will still pass because `withFixtures` ensures mocha _always_ runs the clean up functions
  });

  it("does not time out", async function() {
    const { driver } = fixtures.start({ ... options ... });
    // do things
  });
});

Open issue:
Still the cascading error happen and the result appears in the other test that is executed.

[gauthierpetetin]

Comments from technical refinement:

• 2 separate issues here
  • When a test times out, we have an error that bubbles up to the top, but we keep running the next tests. Since Ganache is still running from the first test, it results in the next test to fail as well. => Will be resolved in this PR: test: this was experiment PR and will be deleted #27220
  • When an async function is running, there's no way we can say "stop executing the rest of that function". When a test times out, we can't stop the Mocha function.
• Suggested solutions:
  • Every test could have its own spec file?
    • This would increase our CI by ~50x
  • We can kinda solve it by designing our APIs to return Promises for any asynchronous work, which the test can await during teardown. Not applicable to all cases though, particularly more intengration-like tests.
  • Every time before we run a test, we check the status of a cancellation token
  • Possible mitigation: when you have multiple cascading unit test failures, look at the first one
  • Try out TestNG?
  • Ask Devin.ai for a solution?

---

Comment in next technical refinement:

• We got a bit carried away there with imagining a perfect solution I think.
• David raised a good point in the previous discussion, which is that for a test timeout there is no perfect way to prevent failure from cascading to other tests in the same process. But we can put that case aside, and prevent cascading failures in other scenarios
• Next steps:
  • Replicate @hjetpoluru ’s withFixtures change in all tests: fix: flaky test after with fixtures #27431
  • We would need to split up into several PRs
  • Create a label to bypass the 5x tests re-run?