Config groups

[LarsDenBakker]

When trying to think of solutions for multiple new features we want to add, I've thought of a way to combine them all in one new capability. This would provide a solution for:

• [test-runner-*] support for browser context properties #365
• [test-runner-*] Run tests only on a specific browser launcher #310
• [test-runner-*] mocking/stubbing es modules #273
• [test-runner] Advanced Concurrency Settings #238
• A feature request from @daKmoR to allow running only a specific set of tests.

My idea is to allow defining "groups" inside the config, which inherit configuration from the top level config but allow specifying extra options. To run a set of tests with a specific configuration, or to run all tests with an extra configuration.

Example use cases

Run some extra tests using a touch device

View example

export default {
  files: 'test/**/*.test.js',
  groups: [
    { 
      name: 'touch',
      files: 'test/touch/**/*.test.js',
      browerContext: {
        hasTouch: true
      }
    }
  ]
}

Run all tests twice, once regularly and once using touch

View example

export default {
  files: 'test/**/*.test.js',
  groups: [
    { 
      name: 'touch',
      browerContext: {
        hasTouch: true
      }
    }
  ]
}

Mock es modules in some tests using import map

View example

export default {
  groups: [
    { 
      name: 'mocked-foo-a',
      files: 'test/x/**/*.test.js',
      importMap: {
        'foo': './mocks/foo-a.js'
      }
    },
    { 
      name: 'mocked-foo-b',
      files: 'test/y/**/*.test.js',
      importMap: {
        'foo': './mocks/foo-b.js'
      }
    },
  ]
}

Run only some tests on a specific browser

View example

import { playwrightLauncher } from '@web/test-runner-playwright';

export default {
  files: 'test/!(firefox-only)/**/*.test.js',
  groups: [
    { 
      name: 'firefox-only',
      files: 'test/firefox-only/**/*.test.js',
      browsers: [playwrightLauncher({ product: 'firefox' })]
    },
  ]
}

Run tests on firefox sequentially

View example

import { playwrightLauncher } from '@web/test-runner-playwright';

export default {
  files: 'test/**/*.test.js',
  browsers: [
    playwrightLauncher({ product: 'chromium' }),
    playwrightLauncher({ product: 'webkit' }),
  ],
  groups: [
    { 
      name: 'firefox-only',
      files: 'test/**/*.test.js',
      sequential: true,
      browsers: [playwrightLauncher({ product: 'firefox' })]
    },
  ]
}

Since groups are named, we can provide an extra cli flag to run only tests from a specific group using wtr --group foo.

This way you could chunk your monorepo tests, and only run tests from a specific package.

View example

import fs from 'fs';

const packages = fs.readdirSync('packages');

export default {
  groups: packages.map(pkg => ({
    files: `test/${pkg}/*.test.js`,
  }))
}

The name group is totally up for discussion. If anyone has a good idea, let me know :)

[LarsDenBakker]

I think we can also support a string value for groups:

export default {
  files: 'test/**/*.test.js',
  groups: 'test/**/*.config.js'
}

This will resolve the glob pattern, and real all these files as groups. This way you can create per-folder or per-file configs. This would be similar to things like .mocharc and .babelrc, except that you'd need to define the test files still inside the config.

[daKmoR]

this has been implemented docs are here https://modern-web.dev/docs/test-runner/cli-and-configuration/#test-groups