feat: test config overrides

[kuceb]

• Closes Pass options to specific tests/suites in order to overwrite configuration throughout test/suite #7167
• Closes Expand Cypress.isBrowser() to allow for bool/array checks #7168

User facing changelog

• You can now apply a certain configuration values to a suite or test by passing a options object testConfigOverrides to the test or suite as the second argument.
• Cypress.isBrowser() has been expanded to allow for new matcher arguments to assist in filtering matching browsers.

Additional details

• could be used to declare baseUrl and be able to preemptively switch domains without executing before[each] hooks Code in hooks and test are rerun when changing to a different superdomain #1987 (does not fix)
• will allow test retries feat: support test retries #3968 to specify retry count on per-test basis
• NOT a breaking change since config will only be restored after a test if using the new syntax

How has the user experience changed?

Test/Suite Options

• users can add test configuration in object during test/suite declaration:

it('can test with config', {
    defaultCommandTimeout: 200
  }, () => {
    // test body
  })

• run test only in certain browsers:

it('only runs in ff', {
    browser: 'firefox'
  }, () => {
  })

• api for excluding browsers:

it('does not run in ff', {
    browser: '!firefox'
  }, () => {})

• can apply same options for the whole suite:

describe('suite with config', {
    defaultCommandTimeout: 40
  }, () => {
	// all tests declared here will inherit the config
  })

isBrowser changes

Cypress.isBrowser has been extended to allow for more filter matchers including !browser and passing an array or Object for matching.

Before

Cypress.isBrowser('firefox')
Cypress.isBrowser({ family: 'chromium', channel: 'stable' })

After (can also check these things)

Cypress.isBrowser('!chrome')
Cypress.isBrowser(['electron', 'chrome'])
Cypress.isBrowser(['!electron', '!chrome'])
Cypress.isBrowser([
  { family: 'chromium', channel: 'canary' },
  { family: 'firefox', channel: 'dev' }
])

Tasks

• extend mocha types to allow specifying config properties in test/suite declarations
• further restrict / extend options we accept for test/suites (for example, watchForFileChanges should be removed from inline config)
• support in code
• implement for suites
• fix specs failing due to removing beforeEach config reset
• update isBrowser to support ! syntax for excluding a browser
• change config value 'browsers' to 'browser'
• update browser skipped test title to capture originalTitle in server/reporter
• rename TestConfigOptions to TestConfigOverride

Review task

• patch 'specify' similar to 'it'
• patch 'xit' style skip declarations

PR dependencies

• bugfix for after/before hooks fix rerun before/after hooks on top navigation #7035

PR Tasks

• Have tests been added/updated?
• Has a PR for user-facing changes been opened in cypress-documentation? Test/Suit configuration options + Cypress.isBrowser updates cypress-documentation#2697
• Have API changes been updated in the type definitions?

[cypress-bot]

Thanks for the contribution! Below are some guidelines Cypress uses when doing PR reviews.

• Please write [WIP] in the title of your Pull Request if your PR is not ready for review - someone will review your PR as soon as the [WIP] is removed.
• Please familiarize yourself with the PR Review Checklist and feel free to make updates on your PR based on these guidelines.

PR Review Checklist

If any of the following requirements can't be met, leave a comment in the review selecting 'Request changes', otherwise 'Approve'.

User Experience

• The feature/bugfix is self-documenting from within the product.
• The change provides the end user with a way to fix their problem (no dead ends).

Functionality

• The code works and performs its intended function with the correct logic.
• Performance has been factored in (for example, the code cleans up after itself to not cause memory leaks).
• The code guards against edge cases and invalid input and has tests to cover it.

Maintainability

• The code is readable (too many nested 'if's are a bad sign).
• Names used for variables, methods, etc, clearly describe their function.
• The code is easy to understood and there are relevant comments explaining.
• New algorithms are documented in the code with link(s) to external docs (flowcharts, w3c, chrome, firefox).
• There are comments containing link(s) to the addressed issue (in tests and code).

Quality

• The change does not reimplement code.
• There's not a module from the ecosystem that should be used instead.
• There is no redundant or duplicate code.
• There are no irrelevant comments left in the code.
• Tests are testing the code’s intended functionality in the best way possible.

Internal

• The original issue has been tagged with a release in ZenHub.

[cypress]

---

Test summary

7519 • 0 • 116 • 0

---

Run details

Project	cypress
Status	Passed
Commit	bd4e943
Started	Jun 4, 2020 3:08 PM
Ended	Jun 4, 2020 3:16 PM
Duration	08:12 💡
OS	Linux Debian - 10.1
Browser	Multiple

View run in Cypress Dashboard ➡️

---

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

[mellis481]

Either I'm confused or this is not working as described. Based off the discussion in this thread as well as in the Cypress docs here, I believe I could update a test suite by adding the following for the second parameter:

describe('Users index page', { env: { includedTests: '!platform' } }, () => { 
  ...
}

I'm expecting this to skip a test suite if the cypress.json file contains the following:

{
  "env": {
    "includedTests": "platform"
  }
}

This does not exclude the test suite, however, and I'm not clear why...

[bahmutov]

@mellis481 I am not sure how the confusion sets here:

• only browser parameter filters tests
• the other parameters set the config or environment values for the group of tests

Thus in your example

describe('Users index page', { env: { includedTests: '!platform' } }, () => { 
  it('set the env variable', () => {
     Cypress.env('includedTests') // will be "!platform"
  })
}

Tip: if you want to skip a test using the value of an environment variable, check out https://github.com/cypress-io/cypress-skip-test

[mellis481]

@bahmutov When I read the Cypress docs starting at Test Configuration, it describes allowed config values, any of which I believed could be used to exclude a test. I had assumed that the example used in the "Suite configuration" section was just one example that happened to use the browser config value for suite exclusion. I'm unclear why the only config value that can be used to exclude a test is browser, but thanks for clarifying.

[bahmutov]

Opened cypress-io/cypress-documentation#3200 to clarify it in the docs

[paleite]

@mellis481 I encountered the same issue as you.
It's great that Cypress wants to clarify the docs, but I think it would be even greater if we could improve the interface altogether.
I've posted a suggestion on it as well: cypress-io/cypress-documentation#3200 (comment)
Please let me know what you guys think!