feat: support test retries (#3968)

* add retries e2e test * restore runner/test/eslintrc * use mocha pass event, move runner.spec to runner package * fix .eslintignore * remove npmInstall logic in helper/e2e script, force custom reporters to use our mocha * temp 04/09/20 [skip ci] * add retries output to server/reporter, fix mocha pass event order, cleanup * e2e tests - dont run electron tests in other browsers * Update readme to reflect how to start server for cypress tests * fix after merge * fix .coffee -> .js after merge * fix attempt.tsx * fix runnable titles emitted to terminal reporters * fix more tests: update snapshots, fix 7_record_spec, 8_reporters_spec * remove styling for 'attempt-error-region' so it's not indented - This was the older styling before error improvements and is no longer necessary. * try 2: fix rerun before/after hooks * fix runner with only, runner snapshots, lint fixes * temp 04/29/20 [skip ci] * backport changes from test-retries * change logic to rerun before hooks after top navigation * fix windowSize for browser e2e test * fix windowSize for xvfb chrome in e2e test * ok fine, just disable screenshots * fix after merge: decaffed navigation.js * update server/unit test snapshots * fix after merge: decaffed aliases.js * fix usage of cypress --parallel flag in circle.yml * fix circle.yml integration-tests jobs * fix decaf related typo * fix circle.yml separate command for runner-integration-tests * update runner/integration tests + snapshot after error improvements * fix runner/integration snapshots for chrome/ff stacktrace differences * rerun ci * fix passing --parallel to runner-integration tests * perf: faster lookup for hooks without runnables * fix afterAll hook switch logic * simplify mocha prototype patches * fix decaf utils.coffee after merge * backport to before/after fix * backport to before/after fix 2 * cleanup from decaf, fix ui/package.json * update helpers, simplify runner.spec * fix lint-types errors, flaky spec * fix noExit passed to e2e test inline options * cleanup snapshot utility - refactor to use util file * remove before/after changes * make cy obj a class instance * cleanup/unmerge before/after fixes PR... * more cleanup * add comment * fix runner.spec * cleanup snapshot utility more, cleanup reporter.spec * fix after merge * minor rename variable * fix after merge: decaffed files * fix specName in reporterHeader, spec_helper require * replace reporter specPath usages with spec object from config * cleanup, fix specs, fix types tests * fix config spec paths in isolated runner, fix snapshot plugin button * combine runner.spec.js and runner_spec.js * fix incorrect merge * minor minor cleanup * rename driver/test/cypress to driver/test * use yarn workspace over lerna for individual package commands * add error message to driver start * remove usage of wait-on * update <reference types/>, import string * fix driver/readme * fix readmes after regex replace * revert wait-on changes * Revert "revert wait-on changes" This reverts commit 6de684c. * update yarn.lock * fix broken path in spec * fix broken paths in specs with @packages/driver * move runner/test/cypress into runner/cypress * start server in pluginsFile in runner/cypress tests * fix more broken spec paths * fix broken paths after runner/cypress folder move * move type definition loading for driver/cypress into dedicated file * move internal-types to "types" folder, fix driver/index.d.ts * fix type-check in packages/runner. not exactly sure why * fix runner type-check by excluding test folder in tsconfig * bump timeout on e2e/8_error_ui_spec * update snapshot utility, rename tests in runner/runner.spec, fix README yarn commands * delete old spec * fix snapshot naming, remove redundant test in reporter_spec * fix file renames after merge * rename runner/ snapshot * update server/unit/reporter_spec snapshot * update runner/runner_spec snapshot * rename runner snapshot file * address feedback: move server reporter snapshot specs out * address feedback: add comment about exposing globals * fix test-retries after merging isolated-runner * fix runner/test helper, update snapshot * address feedback: split out runner/retries spec, move reporter/ui tests to runner/ui spec (mostly done), various cleanup * fix scrolling, attempt opening, update snapshots * fix e2e support file * fix 5_spec_isolation * fix mislabeling attempt screenshots * only add test results prevAttempts if exists * fix reporter/unit tests, server/unit tests * remove dead code, fix test isOpen * update snapshots for retries.mochaEvents, fix snapshot error in state hydration test, remove dead snapshots * new moduleAPI schema using attempts array, fix wrapping errors from hook retries, update snapshots * add displayError, null out fields in moduleAPI schema * change default retries to {runMode:2, openMode:0} * fix reporter type-check * upgrade json-schemas, update snapshots * reformat error.stack to be only stacktrace, update snapshots * fix stacktrace replacing in 5_spec_isolation * fix navigation error causing infinite reloading, bump timeout on e2e/8_error_ui * fix server/unit tests for new schema * fix reporter/unit tests * fix reporting duplicate screenshots using cy.screenshot during test retry * update snapshot for 6_uncaught_support_file_spec * bump x-route-version: 3 * fix test.tsx collapsible content, css, fix e2e/8_error_ui, e2e projects excluding retries * fix css, fix padding in runnable-instruments, fix runner/integration tests * fixup after merge * fix reporter/runner to work with split hooks * update api tests, runner/cypress tests, reporter * fix 5_spec_isolation snapshots, fix runner/cypress errors.spec, fix null reference in test.tsx * fix e2e/non_root spec, fix type_check, fix reporter/unit tests * setup percy snapshots in runner/cypress, fix driver/runner test:after:run event, add tests for only,skip in runner/cypress, fix retried css * add customPercySnapshot * fix circle.yml * fix circle.yml 2 * fix circle.yml 3 * add warning for incompatible retries plugin * add more percy snapshots * fix firefox screenshot resolution in e2e test * Fix testConfigOverrides not affecting viewport (#8006) * finish adding percy snapshots to runner/cypress retries spec, update error msgs, add tests to be fixed * remove .only * fixing missing repo argument * fix testConfigOverrides usage with retries, fix test * fix issues from previous merge * add script that can query CircleCI workflow status * add circleci job to poll * add retries * try yarn lock * retry, percy finalize * check for current running job * do not swallow request error * better print * use job name from circle environment * use debug instead * renamed circle polling script * refactor circle to conditionally run percy-finalize when env var is available - pass job-names to wait on as an argument * use multi-line strings and quote --job-names - rename —circle-jobs to —job-names * add comment * only poll until the jobs to wait for are blocked or running * fix running hooks at correct depth after attempt fails from hook and will retry, update e2e snapshots * fix reporter/unit tests, remove unused toggleOpen code * move custom percy command into @packages/ui-components and apply them to desktop-gui * halt percy finalize job if env variable is not set * if only I could code * update runner/cypress mochaEvent snapshots, fix e2e firefox resolution * fix css for attempt border-left, fix attempt-tag open/close icon, add color to attempt collapsible dot * try percy set viewport width * set default retries back to {runMode:0, openMode:0} * formatting: add backticks to warning message * write explicit test for screenshot overwriting behavior, fix snapshots after changing retries defaults * fix e2e.it.only` * cleanup whitespace * update snapshots * fix cypress module API types for new result schema * build and upload binary for test-retries branch too (linux) * add pre-release PR comment * fix pre-release commit comment * rename runner/cypress test * update retries.ui.spec test titles * fix after merge: use most recent attempt for before/after hooks * add suite title to hook error in runner/cypress tests Co-authored-by: Jennifer Shehane <[email protected]> Co-authored-by: Brian Mann <[email protected]> Co-authored-by: Gleb Bahmutov <[email protected]>
cypress-io · Aug 10, 2020 · 860a20a · 860a20a
1 parent 01cc5e9
commit 860a20a
Show file tree

Hide file tree

Showing 120 changed files with 12,037 additions and 1,813 deletions.
diff --git a/.eslintrc.json b/.eslintrc.json
@@ -6,7 +6,8 @@
     "plugin:@cypress/dev/general"
   ],
   "rules": {
-    "prefer-spread": "off"
+    "prefer-spread": "off",
+    "prefer-rest-params": "off"
   },
   "settings": {
     "react": {

diff --git a/circle.yml b/circle.yml
@@ -129,14 +129,22 @@ commands:
       browser:
         description: browser shortname to target
         type: string
+      percy:
+        description: enable percy
+        type: boolean
+        default: false
     steps:
       - attach_workspace:
           at: ~/
       - run:
           command: |
+            cmd=$([[ <<parameters.percy>> == 'true' ]] && echo 'yarn percy exec --') || true
+
             CYPRESS_KONFIG_ENV=production \
             CYPRESS_RECORD_KEY=$PACKAGES_RECORD_KEY \
-            yarn workspace @packages/runner cypress:run --record --parallel --group runner-integration-<<parameters.browser>> --browser <<parameters.browser>>
+            PERCY_PARALLEL_NONCE=$CIRCLE_WORKFLOW_ID \
+            PERCY_PARALLEL_TOTAL=-1 \
+            $cmd yarn workspace @packages/runner cypress:run --record --parallel --group runner-integration-<<parameters.browser>> --browser <<parameters.browser>>
       - store_test_results:
           path: /tmp/cypress
       - store_artifacts:
@@ -211,6 +219,7 @@ commands:
           command: |
             git clone --depth 1 --no-single-branch https://github.com/cypress-io/<<parameters.repo>>.git /tmp/<<parameters.repo>>
             cd /tmp/<<parameters.repo>> && (git checkout $NEXT_DEV_VERSION || true)
+
   test-binary-against-repo:
     description: |
       Takes the built binary and NPM package, clones given example repo
@@ -337,6 +346,17 @@ commands:
                 path: /tmp/<<parameters.repo>>/cypress/videos
       - store-npm-logs
 
+  wait-on-circle-jobs:
+    description: Polls certain Circle CI jobs until they finish
+    parameters:
+      job-names:
+        description: comma separated list of circle ci job names to wait for
+        type: string
+    steps:
+      - run:
+          name: "Waiting on Circle CI jobs: <<parameters.job-names>>"
+          command: node ./scripts/wait-on-circle-jobs.js --job-names="<<parameters.job-names>>"
+
 jobs:
   ## code checkout and yarn installs
   build:
@@ -400,6 +420,37 @@ jobs:
           command: node cli/bin/cypress info --dev
       - store-npm-logs
 
+  # a special job that keeps polling Circle and when all
+  # individual jobs are finished, it closes the Percy build
+  percy-finalize:
+    <<: *defaults
+    executor: cy-doc
+    parameters:
+      required_env_var:
+        type: env_var_name
+    steps:
+      - attach_workspace:
+          at: ~/
+      - run:
+          # if this is an external pull request, the environment variables
+          # are NOT set for security reasons, thus no need to poll -
+          # and no need to finalize Percy, since there will be no visual tests
+          name: Check if <<parameters.required_env_var>> is set
+          command: |
+            if [[ -v <<parameters.required_env_var>> ]]; then
+              echo "Internal PR, good to go"
+            else
+              echo "This is an external PR, cannot access other services"
+              circleci-agent step halt
+            fi
+      - wait-on-circle-jobs:
+          job-names: >
+            desktop-gui-integration-tests-2x,
+            desktop-gui-component-tests,
+            cli-visual-tests,
+            runner-integration-tests-chrome,
+      - run: npx percy finalize --all
+
   cli-visual-tests:
     <<: *defaults
     parallelism: 1
@@ -701,6 +752,7 @@ jobs:
     steps:
       - run-runner-integration-tests:
           browser: chrome
+          percy: true
 
   runner-integration-tests-firefox:
     <<: *defaults
@@ -879,13 +931,6 @@ jobs:
           command: node index.js
           working_directory: packages/launcher
 
-  percy-finalize:
-    <<: *defaults
-    steps:
-      - run:
-          name: "finalizes percy builds"
-          command: npx percy finalize --all
-
   build-binary:
     <<: *defaults
     shell: /bin/bash --login
@@ -1369,6 +1414,11 @@ jobs:
           command: npm run test:ci
           pull_request_id: 515
           folder: examples/fundamentals__typescript
+      - test-binary-against-repo:
+          repo: cypress-example-recipes
+          command: npm test
+          pull_request_id: 513
+          folder: examples/fundamentals__module-api-wrap
 
   "test-binary-against-kitchensink":
     <<: *defaults
@@ -1493,6 +1543,11 @@ linux-workflow: &linux-workflow
         name: Linux lint
         requires:
           - build
+    - percy-finalize:
+        context: test-runner:poll-circle-workflow
+        required_env_var: PERCY_TOKEN # skips job if not defined (external PR)
+        requires:
+          - build
     - lint-types:
         requires:
           - build
@@ -1624,12 +1679,6 @@ linux-workflow: &linux-workflow
         requires:
           - build
 
-    - percy-finalize:
-        requires:
-          - desktop-gui-integration-tests-2x
-          - desktop-gui-component-tests
-          - cli-visual-tests
-
     # various testing scenarios, like building full binary
     # and testing it on a real project
     - test-against-staging:
@@ -1666,6 +1715,7 @@ linux-workflow: &linux-workflow
           branches:
             only:
               - develop
+              - test-retries
         requires:
           - build-npm-package
     - build-binary:
@@ -1677,6 +1727,7 @@ linux-workflow: &linux-workflow
           branches:
             only:
               - develop
+              - test-retries
         requires:
           - build-binary
     - test-npm-module-on-minimum-node-version:
@@ -1733,6 +1784,7 @@ linux-workflow: &linux-workflow
           branches:
             only:
               - develop
+              - test-retries
         requires:
           - upload-npm-package
           - upload-binary

diff --git a/cli/schema/cypress.schema.json b/cli/schema/cypress.schema.json
@@ -243,6 +243,18 @@
       "type": "boolean",
       "default": false,
       "description": "Polyfills `window.fetch` to enable Network spying and stubbing"
+    },
+    "retries": {
+      "type": [
+        "object",
+        "number",
+        "null"
+      ],
+      "default": {
+        "runMode": 0,
+        "openMode": 0
+      },
+      "description": "The number of times to retry a failing. Can be configured to apply only in runMode or openMode"
     }
   }
 }
diff --git a/cli/types/cypress-npm-api.d.ts b/cli/types/cypress-npm-api.d.ts
@@ -7,6 +7,11 @@
 // but for now describe it as an ambient module
 
 declare namespace CypressCommandLine {
+  interface TestError {
+    name: string
+    message: string
+    stack: string
+  }
   /**
    * All options that one can pass to "cypress.run"
    * @see https://on.cypress.io/module-api#cypress-run
@@ -166,14 +171,16 @@ declare namespace CypressCommandLine {
     title: string[]
     state: string
     body: string
-    /**
-     * Error stack string if there is an error
-     */
-    stack: string | null
-    /**
-     * Error message if there is an error
+     /**
+     * Error string as it's presented in console if the test fails
      */
-    error: string | null
+    displayError: string | null
+    attempts: AttemptResult[]
+  }
+
+  interface AttemptResult {
+    state: string
+    error: TestError | null
     timings: any
     failedFromHookId: hookId | null
     wallClockStartedAt: dateTimeISO
@@ -199,6 +206,7 @@ declare namespace CypressCommandLine {
     name: string
     testId: testId
     takenAt: dateTimeISO
+    testAttemptIndex: number
     /**
      * Absolute path to the saved image
      */

diff --git a/cli/types/cypress.d.ts b/cli/types/cypress.d.ts
@@ -334,6 +334,11 @@ declare namespace Cypress {
      */
     getFirefoxGcInterval(): number | null | undefined
 
+    /**
+     * @returns the number of test retries currently enabled for the run
+     */
+    getTestRetries(): number | null
+
     /**
      * Checks if a variable is a valid instance of `cy` or a `cy` chainable.
      *
@@ -2563,6 +2568,13 @@ declare namespace Cypress {
      * the `includeShadowDom` option to some DOM commands.
      */
     experimentalShadowDomSupport: boolean
+    /**
+     * Number of times to retry a failed test.
+     * If a number is set, tests will retry in both runMode and openMode.
+     * To enable test retries only in runMode, set e.g. `{ openMode: null, runMode: 2 }`
+     * @default null
+     */
+    retries: Nullable<number | {runMode: Nullable<number>, openMode: Nullable<number>}>
   }
 
   interface TestConfigOverrides extends Partial<Pick<ConfigOptions, 'baseUrl' | 'defaultCommandTimeout' | 'taskTimeout' | 'animationDistanceThreshold' | 'waitForAnimations' | 'viewportHeight' | 'viewportWidth' | 'requestTimeout' | 'execTimeout' | 'env' | 'responseTimeout'>> {

diff --git a/package.json b/package.json
@@ -136,6 +136,7 @@
     "fs-extra": "8.1.0",
     "gift": "0.10.2",
     "globby": "10.0.1",
+    "got": "11.5.1",
     "gulp": "4.0.2",
     "gulp-awspublish": "4.0.0",
     "gulp-debug": "4.0.0",

diff --git a/packages/desktop-gui/cypress/support/index.js b/packages/desktop-gui/cypress/support/index.js
@@ -1,4 +1,4 @@
-require('@percy/cypress')
+require('@packages/ui-components/cypress/support/customPercyCommand')
 require('cypress-react-unit-test/dist/hooks')
 
 const BluebirdPromise = require('bluebird')

diff --git a/packages/driver/cypress/integration/commands/screenshot_spec.js b/packages/driver/cypress/integration/commands/screenshot_spec.js
@@ -19,6 +19,7 @@ describe('src/cy/commands/screenshot', () => {
       takenAt: new Date().toISOString(),
       name: 'name',
       blackout: ['.foo'],
+      testAttemptIndex: 0,
       duration: 100,
     }
 
@@ -49,7 +50,7 @@ describe('src/cy/commands/screenshot', () => {
 
       Cypress.action('runner:runnable:after:run:async', test, runnable)
       .then(() => {
-        expect(Cypress.action).not.to.be.calledWith('cy:test:set:state')
+        expect(Cypress.action).not.to.be.calledWith('test:set:state')
         expect(Cypress.automation).not.to.be.called
       })
       .finally(() => {
@@ -68,7 +69,7 @@ describe('src/cy/commands/screenshot', () => {
 
       Cypress.action('runner:runnable:after:run:async', test, runnable)
       .then(() => {
-        expect(Cypress.action).not.to.be.calledWith('cy:test:set:state')
+        expect(Cypress.action).not.to.be.calledWith('test:set:state')
         expect(Cypress.automation).not.to.be.called
       })
     })
@@ -89,7 +90,7 @@ describe('src/cy/commands/screenshot', () => {
 
       Cypress.action('runner:runnable:after:run:async', test, runnable)
       .then(() => {
-        expect(Cypress.action).not.to.be.calledWith('cy:test:set:state')
+        expect(Cypress.action).not.to.be.calledWith('test:set:state')
         expect(Cypress.automation).not.to.be.called
       })
     })
@@ -137,6 +138,7 @@ describe('src/cy/commands/screenshot', () => {
           waitForCommandSynchronization: true,
           disableTimersAndAnimations: true,
           blackout: [],
+          testAttemptIndex: 0,
         })
 
         expect(Cypress.action).to.be.calledWith('cy:after:screenshot', {
@@ -147,6 +149,7 @@ describe('src/cy/commands/screenshot', () => {
           waitForCommandSynchronization: true,
           disableTimersAndAnimations: true,
           blackout: [],
+          testAttemptIndex: 0,
         })
       })
     })
@@ -183,6 +186,7 @@ describe('src/cy/commands/screenshot', () => {
           testFailure: true,
           blackout: [],
           scaled: true,
+          testAttemptIndex: 0,
         })
       })
     })
@@ -225,6 +229,7 @@ describe('src/cy/commands/screenshot', () => {
             simple: false,
             scaled: true,
             blackout: [],
+            testAttemptIndex: 0,
           })
         })
       })
@@ -264,6 +269,7 @@ describe('src/cy/commands/screenshot', () => {
           testFailure: true,
           scaled: true,
           blackout: [],
+          testAttemptIndex: 0,
         })
       })
     })
@@ -406,6 +412,7 @@ describe('src/cy/commands/screenshot', () => {
             waitForCommandSynchronization: false,
             disableTimersAndAnimations: true,
             blackout: ['.foo'],
+            testAttemptIndex: 0,
           })
         })
       })
@@ -425,6 +432,7 @@ describe('src/cy/commands/screenshot', () => {
             waitForCommandSynchronization: false,
             disableTimersAndAnimations: true,
             blackout: ['.foo'],
+            testAttemptIndex: 0,
           })
         })
       })
@@ -446,6 +454,7 @@ describe('src/cy/commands/screenshot', () => {
             waitForCommandSynchronization: true,
             disableTimersAndAnimations: true,
             blackout: [],
+            testAttemptIndex: 0,
           })
         })
       })
@@ -466,6 +475,7 @@ describe('src/cy/commands/screenshot', () => {
             waitForCommandSynchronization: false,
             disableTimersAndAnimations: true,
             blackout: ['.foo'],
+            testAttemptIndex: 0,
           })
         })
       })