--grep and .only result in 0 tests running when using async describe with await

[tommedema]

Prerequisites

• Checked that your issue isn't already filed by cross referencing issues with the common mistake label
• Checked next-gen ES issues and syntax problems by using the same environment and/or transpiler configuration without Mocha to ensure it isn't just a feature that actually isn't supported in the environment in question or a bug in your code.
• 'Smoke tested' the code to be tested by running it outside the real test suite to get a better sense of whether the problem is in the code under test, your usage of Mocha, or Mocha itself
• Ensured that there is no discrepancy between the locally and globally installed versions of Mocha. You can find them with:
  node node_modules/.bin/mocha --version(Local) and mocha --version(Global). We recommend avoiding the use of globally installed Mocha.

Description

When you have a describe function that is async, and use an await statement within that function, tests run fine, except when using --grep or only.

Steps to Reproduce

Create this simple test suite to reproduce the issue:

/* eslint-disable no-unused-vars */
/* global describe, it, beforeEach, afterEach  */
require('should')

describe('one', async function () {
  await new Promise((resolve, reject) => resolve())
  
  it('should work', function () {
    return 'works'
  })
})

describe('two', function () {
  it('should also work', function () {
    return 'works too'
  })
})

This works: node_modules/mocha/bin/mocha (2 passing)

This does NOT work: node_modules/mocha/bin/mocha --grep="one" (0 passing)

Adding .only also does NOT work.

Expected behavior: [What you expect to happen]

Expect it to run the function matching grep or .only

Actual behavior: [What actually happens]

0 tests are passing/failing/pending

Reproduces how often: [What percentage of the time does it reproduce?]

100%

Versions

Mocha 4.0.1
Node 9.2.1
OSX Sierra

Additional Information

I need to use await inside describe because the inner tests use a port that is found using an async library to discover unused ports.

[tommedema]

Related to this is that you must use an async describe function if you want to dynamically create tests based on some asynchronous operation. This is not possible to do in the before function:

/* eslint-disable no-unused-vars */
/* global describe, it, before */

describe('e2e', async () => {
  
  before(() => {
    console.log('before...')
    
    urls = await getTestUrls()
    
    urls.forEach(url => {
      console.log(url)
      it(`should pass for ${url}`, () => {
        return 'ok'
      })
    })
  })
  
  it('should run a test', () => {
    return 'ok'
  })
  
})

async function getTestUrls() {
  return ['google.com']
}

This results in:

node_modules/mocha/bin/mocha e2e


  e2e
before...
google.com
    ✓ should run a test


  1 passing (5ms)

When you would expect 2 tests to run. If you then change this to a more direct approach:

describe('e2e', async () => {
  
  urls = await getTestUrls()
  
  urls.forEach(url => {
    console.log(url)
    it(`should pass for ${url}`, () => {
      return 'ok'
    })
  })
  
  it('should run a test', () => {
    return 'ok'
  })
  
})

async function getTestUrls() {
  return ['google.com']
}

It results in:

node_modules/mocha/bin/mocha e2e


  0 passing (2ms)

google.com

Expected behavior would be 2 tests running.

Note that this seems to be due to some kind of race condition. If you add a synchronous describe with an asynchronous it, it works:

describe('e2e', () => it('should initialize tests', cb => cb()))

output:

node_modules/mocha/bin/mocha e2e


google.com
  ✓ should pass for google.com
  ✓ should run a test
  e2e
    ✓ should initialize tests


  3 passing (6ms)

[tommedema]

I've resolved this with the --delay flag

Would be good to add this as a hint to the docs in the section on asynchronously adding tests

;(async () => {
  const urls = await getTestUrls()
  console.log(`running suite with ${urls.length} urls`)
  
  describe('e2e', function () {
    this.timeout(TIMEOUT_MS)
    
    for (let url of urls) {
      it(`should pass for ${url}`, () => {
        return 'ok'
      })
    }
  })
  
  run()
})()