Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Skipped tests reported incorrectly under Mocha 8.0+ #116

Closed
Krinkle opened this issue Sep 20, 2020 · 2 comments · Fixed by #131
Closed

Skipped tests reported incorrectly under Mocha 8.0+ #116

Krinkle opened this issue Sep 20, 2020 · 2 comments · Fixed by #131
Assignees
@Krinkle
Labels
Category: npm Package Type: Problem
Milestone
Package 2.0

Comments

@Krinkle
Copy link
Member

Krinkle commented Sep 20, 2020

The result of npm run test-integration with Mocha 8.0 and later is as follows:

log/failed--mocha--8.0.0.log 
not ok 106 Adapters integration > Mocha adapter > Event "testEnd" runtime property
  ---
  message: "failed"
  severity: failed
  actual  : "undefined"
  expected: "number"
  stack:     at collectedData.forEach.value (/js-reporters/test/integration/adapters.js:185:20)
    at Array.forEach (<anonymous>)
    at Object.assert (/js-reporters/test/integration/adapters.js:183:23)
  ...
  ---
  message: "failed"
  severity: failed
  actual  : "undefined"
  expected: "number"
  stack:     at collectedData.forEach.value (/js-reporters/test/integration/adapters.js:185:20)
    at Array.forEach (<anonymous>)
    at Object.assert (/js-reporters/test/integration/adapters.js:183:23)
  ...
ok 107 Adapters integration > Mocha adapter > Event "testEnd" errors property
ok 108 Adapters integration > Mocha adapter > Event "testEnd" assertions property
ok 109 Adapters integration > Mocha adapter > global suite starts
ok 110 Adapters integration > Mocha adapter > global test starts
ok 111 Adapters integration > Mocha adapter > global test ends
ok 112 Adapters integration > Mocha adapter > suite with one passing test starts
ok 113 Adapters integration > Mocha adapter > passing test starts
ok 114 Adapters integration > Mocha adapter > passing test ends
ok 115 Adapters integration > Mocha adapter > suite with one passing test ends
ok 116 Adapters integration > Mocha adapter > suite with one skipped test starts
ok 117 Adapters integration > Mocha adapter > skipped test starts
not ok 118 Adapters integration > Mocha adapter > skipped test ends
  ---
  message: "failed"
  severity: failed
  actual  : {
  "name": "should skip",
  "suiteName": "suite with skipped test",
  "fullName": [
    "suite with skipped test",
    "should skip"
  ],
  "status": "pending",
  "runtime": 0,
  "errors": [],
  "assertions": []
}
  expected: {
  "name": "should skip",
  "suiteName": "suite with skipped test",
  "fullName": [
    "suite with skipped test",
    "should skip"
  ],
  "status": "skipped",
  "errors": [],
  "assertions": []
}
  stack:     at Object.assert (/js-reporters/test/integration/adapters.js:315:18)
  ...
not ok 119 Adapters integration > Mocha adapter > suite with one skipped test ends
  ---
  message: "failed"
  severity: failed
  actual  : "failed"
  expected: "skipped"
  stack:     at Object.assert (/js-reporters/test/integration/adapters.js:295:20)
  ...
  ---
  message: "failed"
  severity: failed
  actual  : {
  "passed": 0,
  "failed": 0,
  "skipped": 0,
  "todo": 0,
  "total": 1
}
  expected: {
  "passed": 0,
  "failed": 0,
  "skipped": 1,
  "todo": 0,
  "total": 1
}
  stack:     at Object.assert (/js-reporters/test/integration/adapters.js:305:20)
  ...
  ---
  message: "failed"
  severity: failed
  actual  : {
  "name": "suite with skipped test",
  "fullName": [
    "suite with skipped test"
  ],
  "tests": [
    {
      "name": "should skip",
      "suiteName": "suite with skipped test",
      "fullName": [
        "suite with skipped test",
        "should skip"
      ],
      "status": "pending",
      "runtime": 0,
      "errors": [],
      "assertions": []
    }
  ],
  "childSuites": [],
  "status": "failed",
  "testCounts": {
    "passed": 0,
    "failed": 0,
    "skipped": 0,
    "todo": 0,
    "total": 1
  },
  "runtime": 0
}
  expected: {
  "name": "suite with skipped test",
  "fullName": [
    "suite with skipped test"
  ],
  "tests": [
    {
      "name": "should skip",
      "suiteName": "suite with skipped test",
      "fullName": [
        "suite with skipped test",
        "should skip"
      ],
      "status": "skipped",
      "errors": [],
      "assertions": []
    }
  ],
  "childSuites": [],
  "status": "skipped",
  "testCounts": {
    "passed": 0,
    "failed": 0,
    "skipped": 1,
    "todo": 0,
    "total": 1
  }
}
  stack:     at Object.assert (/js-reporters/test/integration/adapters.js:315:18)
  ...
ok 120 Adapters integration > Mocha adapter > suite with one failing tests
ok 121 Adapters integration > Mocha adapter > failing test starts
ok 122 Adapters integration > Mocha adapter > failing test ends
ok 123 Adapters integration > Mocha adapter > suite with one failing test ends
ok 124 Adapters integration > Mocha adapter > suite with multiple tests starts
ok 125 Adapters integration > Mocha adapter > passing test starts
ok 126 Adapters integration > Mocha adapter > passing test ends
ok 127 Adapters integration > Mocha adapter > skipped test starts
not ok 128 Adapters integration > Mocha adapter > skipped test ends
  ---
  message: "failed"
  severity: failed
  actual  : {
  "name": "should skip",
  "suiteName": "suite with tests",
  "fullName": [
    "suite with tests",
    "should skip"
  ],
  "status": "pending",
  "runtime": 0,
  "errors": [],
  "assertions": []
}
  expected: {
  "name": "should skip",
  "suiteName": "suite with tests",
  "fullName": [
    "suite with tests",
    "should skip"
  ],
  "status": "skipped",
  "errors": [],
  "assertions": []
}
  stack:     at Object.assert (/js-reporters/test/integration/adapters.js:315:18)
  ...
ok 129 Adapters integration > Mocha adapter > failing test starts
ok 130 Adapters integration > Mocha adapter > failing test ends
not ok 131 Adapters integration > Mocha adapter > suite with multiple tests ends
  ---
  message: "failed"
  severity: failed
  actual  : {
  "passed": 1,
  "failed": 1,
  "skipped": 0,
  "todo": 0,
  "total": 3
}
  expected: {
  "passed": 1,
  "failed": 1,
  "skipped": 1,
  "todo": 0,
  "total": 3
}
  stack:     at Object.assert (/js-reporters/test/integration/adapters.js:305:20)
  ...
  ---
  message: "failed"
  severity: failed
  actual  : {
  "name": "suite with tests",
  "fullName": [
    "suite with tests"
  ],
  "tests": [
    {
      "name": "should pass",
      "suiteName": "suite with tests",
      "fullName": [
        "suite with tests",
        "should pass"
      ],
      "status": "passed",
      "runtime": 0,
      "errors": [],
      "assertions": [
        {
          "passed": true
        }
      ]
    },
    {
      "name": "should skip",
      "suiteName": "suite with tests",
      "fullName": [
        "suite with tests",
        "should skip"
      ],
      "status": "pending",
      "runtime": 0,
      "errors": [],
      "assertions": []
    },
    {
      "name": "should fail",
      "suiteName": "suite with tests",
      "fullName": [
        "suite with tests",
        "should fail"
      ],
      "status": "failed",
      "runtime": 0,
      "errors": [
        {
          "passed": false
        }
      ],
      "assertions": [
        {
          "passed": false
        }
      ]
    }
  ],
  "childSuites": [],
  "status": "failed",
  "testCounts": {
    "passed": 1,
    "failed": 1,
    "skipped": 0,
    "todo": 0,
    "total": 3
  },
  "runtime": 0
}
  expected: {
  "name": "suite with tests",
  "fullName": [
    "suite with tests"
  ],
  "tests": [
    {
      "name": "should pass",
      "suiteName": "suite with tests",
      "fullName": [
        "suite with tests",
        "should pass"
      ],
      "status": "passed",
      "runtime": 0,
      "errors": [],
      "assertions": [
        {
          "passed": true
        }
      ]
    },
    {
      "name": "should skip",
      "suiteName": "suite with tests",
      "fullName": [
        "suite with tests",
        "should skip"
      ],
      "status": "skipped",
      "errors": [],
      "assertions": []
    },
    {
      "name": "should fail",
      "suiteName": "suite with tests",
      "fullName": [
        "suite with tests",
        "should fail"
      ],
      "status": "failed",
      "runtime": 0,
      "errors": [
        {
          "passed": false
        }
      ],
      "assertions": [
        {
          "passed": false
        }
      ]
    }
  ],
  "childSuites": [],
  "status": "failed",
  "testCounts": {
    "passed": 1,
    "failed": 1,
    "skipped": 1,
    "todo": 0,
    "total": 3
  },
  "runtime": 0
}
  stack:     at Object.assert (/js-reporters/test/integration/adapters.js:315:18)
  ...
ok 132 Adapters integration > Mocha adapter > outter suite starts
ok 133 Adapters integration > Mocha adapter > outter test starts
ok 134 Adapters integration > Mocha adapter > outter test ends
ok 135 Adapters integration > Mocha adapter > inner suite starts
ok 136 Adapters integration > Mocha adapter > inner test starts
ok 137 Adapters integration > Mocha adapter > inner test ends
ok 138 Adapters integration > Mocha adapter > inner suite ends
ok 139 Adapters integration > Mocha adapter > outter suite ends
not ok 140 Adapters integration > Mocha adapter > global suite ends
  ---
  message: "failed"
  severity: failed
  actual  : {
  "passed": 5,
  "failed": 2,
  "skipped": 0,
  "todo": 0,
  "total": 9
}
  expected: {
  "passed": 5,
  "failed": 2,
  "skipped": 2,
  "todo": 0,
  "total": 9
}
  stack:     at Object.assert (/js-reporters/test/integration/adapters.js:305:20)
  ...
  ---
  message: "failed"
  severity: failed
  actual  : {
  "fullName": [],
  "tests": [
    {
      "name": "global test",
      "fullName": [
        "global test"
      ],
      "status": "passed",
      "runtime": 0,
      "errors": [],
      "assertions": [
        {
          "passed": true
        }
      ]
    }
  ],
  "childSuites": [
    {
      "name": "suite with passing test",
      "fullName": [
        "suite with passing test"
      ],
      "tests": [
        {
          "name": "should pass",
          "suiteName": "suite with passing test",
          "fullName": [
            "suite with passing test",
            "should pass"
          ],
          "status": "passed",
          "runtime": 0,
          "errors": [],
          "assertions": [
            {
              "passed": true
            }
          ]
        }
      ],
      "childSuites": [],
      "status": "passed",
      "testCounts": {
        "passed": 1,
        "failed": 0,
        "skipped": 0,
        "todo": 0,
        "total": 1
      },
      "runtime": 0
    },
    {
      "name": "suite with skipped test",
      "fullName": [
        "suite with skipped test"
      ],
      "tests": [
        {
          "name": "should skip",
          "suiteName": "suite with skipped test",
          "fullName": [
            "suite with skipped test",
            "should skip"
          ],
          "status": "pending",
          "runtime": 0,
          "errors": [],
          "assertions": []
        }
      ],
      "childSuites": [],
      "status": "failed",
      "testCounts": {
        "passed": 0,
        "failed": 0,
        "skipped": 0,
        "todo": 0,
        "total": 1
      },
      "runtime": 0
    },
    {
      "name": "suite with failing test",
      "fullName": [
        "suite with failing test"
      ],
      "tests": [
        {
          "name": "should fail",
          "suiteName": "suite with failing test",
          "fullName": [
            "suite with failing test",
            "should fail"
          ],
          "status": "failed",
          "runtime": 0,
          "errors": [
            {
              "passed": false
            }
          ],
          "assertions": [
            {
              "passed": false
            }
          ]
        }
      ],
      "childSuites": [],
      "status": "failed",
      "testCounts": {
        "passed": 0,
        "failed": 1,
        "skipped": 0,
        "todo": 0,
        "total": 1
      },
      "runtime": 0
    },
    {
      "name": "suite with tests",
      "fullName": [
        "suite with tests"
      ],
      "tests": [
        {
          "name": "should pass",
          "suiteName": "suite with tests",
          "fullName": [
            "suite with tests",
            "should pass"
          ],
          "status": "passed",
          "runtime": 0,
          "errors": [],
          "assertions": [
            {
              "passed": true
            }
          ]
        },
        {
          "name": "should skip",
          "suiteName": "suite with tests",
          "fullName": [
            "suite with tests",
            "should skip"
          ],
          "status": "pending",
          "runtime": 0,
          "errors": [],
          "assertions": []
        },
        {
          "name": "should fail",
          "suiteName": "suite with tests",
          "fullName": [
            "suite with tests",
            "should fail"
          ],
          "status": "failed",
          "runtime": 0,
          "errors": [
            {
              "passed": false
            }
          ],
          "assertions": [
            {
              "passed": false
            }
          ]
        }
      ],
      "childSuites": [],
      "status": "failed",
      "testCounts": {
        "passed": 1,
        "failed": 1,
        "skipped": 0,
        "todo": 0,
        "total": 3
      },
      "runtime": 0
    },
    {
      "name": "outter suite",
      "fullName": [
        "outter suite"
      ],
      "tests": [
        {
          "name": "outter test",
          "suiteName": "outter suite",
          "fullName": [
            "outter suite",
            "outter test"
          ],
          "status": "passed",
          "runtime": 0,
          "errors": [],
          "assertions": [
            {
              "passed": true
            }
          ]
        }
      ],
      "childSuites": [
        {
          "name": "inner suite",
          "fullName": [
            "outter suite",
            "inner suite"
          ],
          "tests": [
            {
              "name": "inner test",
              "suiteName": "inner suite",
              "fullName": [
                "outter suite",
                "inner suite",
                "inner test"
              ],
              "status": "passed",
              "runtime": 0,
              "errors": [],
              "assertions": [
                {
                  "passed": true
                }
              ]
            }
          ],
          "childSuites": [],
          "status": "passed",
          "testCounts": {
            "passed": 1,
            "failed": 0,
            "skipped": 0,
            "todo": 0,
            "total": 1
          },
          "runtime": 0
        }
      ],
      "status": "passed",
      "testCounts": {
        "passed": 2,
        "failed": 0,
        "skipped": 0,
        "todo": 0,
        "total": 2
      },
      "runtime": 0
    }
  ],
  "status": "failed",
  "testCounts": {
    "passed": 5,
    "failed": 2,
    "skipped": 0,
    "todo": 0,
    "total": 9
  },
  "runtime": 0
}
  expected: {
  "fullName": [],
  "tests": [
    {
      "name": "global test",
      "fullName": [
        "global test"
      ],
      "status": "passed",
      "runtime": 0,
      "errors": [],
      "assertions": [
        {
          "passed": true
        }
      ]
    }
  ],
  "childSuites": [
    {
      "name": "suite with passing test",
      "fullName": [
        "suite with passing test"
      ],
      "tests": [
        {
          "name": "should pass",
          "suiteName": "suite with passing test",
          "fullName": [
            "suite with passing test",
            "should pass"
          ],
          "status": "passed",
          "runtime": 0,
          "errors": [],
          "assertions": [
            {
              "passed": true
            }
          ]
        }
      ],
      "childSuites": [],
      "status": "passed",
      "testCounts": {
        "passed": 1,
        "failed": 0,
        "skipped": 0,
        "todo": 0,
        "total": 1
      },
      "runtime": 0
    },
    {
      "name": "suite with skipped test",
      "fullName": [
        "suite with skipped test"
      ],
      "tests": [
        {
          "name": "should skip",
          "suiteName": "suite with skipped test",
          "fullName": [
            "suite with skipped test",
            "should skip"
          ],
          "status": "skipped",
          "errors": [],
          "assertions": []
        }
      ],
      "childSuites": [],
      "status": "skipped",
      "testCounts": {
        "passed": 0,
        "failed": 0,
        "skipped": 1,
        "todo": 0,
        "total": 1
      }
    },
    {
      "name": "suite with failing test",
      "fullName": [
        "suite with failing test"
      ],
      "tests": [
        {
          "name": "should fail",
          "suiteName": "suite with failing test",
          "fullName": [
            "suite with failing test",
            "should fail"
          ],
          "status": "failed",
          "runtime": 0,
          "errors": [
            {
              "passed": false
            }
          ],
          "assertions": [
            {
              "passed": false
            }
          ]
        }
      ],
      "childSuites": [],
      "status": "failed",
      "testCounts": {
        "passed": 0,
        "failed": 1,
        "skipped": 0,
        "todo": 0,
        "total": 1
      },
      "runtime": 0
    },
    {
      "name": "suite with tests",
      "fullName": [
        "suite with tests"
      ],
      "tests": [
        {
          "name": "should pass",
          "suiteName": "suite with tests",
          "fullName": [
            "suite with tests",
            "should pass"
          ],
          "status": "passed",
          "runtime": 0,
          "errors": [],
          "assertions": [
            {
              "passed": true
            }
          ]
        },
        {
          "name": "should skip",
          "suiteName": "suite with tests",
          "fullName": [
            "suite with tests",
            "should skip"
          ],
          "status": "skipped",
          "errors": [],
          "assertions": []
        },
        {
          "name": "should fail",
          "suiteName": "suite with tests",
          "fullName": [
            "suite with tests",
            "should fail"
          ],
          "status": "failed",
          "runtime": 0,
          "errors": [
            {
              "passed": false
            }
          ],
          "assertions": [
            {
              "passed": false
            }
          ]
        }
      ],
      "childSuites": [],
      "status": "failed",
      "testCounts": {
        "passed": 1,
        "failed": 1,
        "skipped": 1,
        "todo": 0,
        "total": 3
      },
      "runtime": 0
    },
    {
      "name": "outter suite",
      "fullName": [
        "outter suite"
      ],
      "tests": [
        {
          "name": "outter test",
          "suiteName": "outter suite",
          "fullName": [
            "outter suite",
            "outter test"
          ],
          "status": "passed",
          "runtime": 0,
          "errors": [],
          "assertions": [
            {
              "passed": true
            }
          ]
        }
      ],
      "childSuites": [
        {
          "name": "inner suite",
          "fullName": [
            "outter suite",
            "inner suite"
          ],
          "tests": [
            {
              "name": "inner test",
              "suiteName": "inner suite",
              "fullName": [
                "outter suite",
                "inner suite",
                "inner test"
              ],
              "status": "passed",
              "runtime": 0,
              "errors": [],
              "assertions": [
                {
                  "passed": true
                }
              ]
            }
          ],
          "childSuites": [],
          "status": "passed",
          "testCounts": {
            "passed": 1,
            "failed": 0,
            "skipped": 0,
            "todo": 0,
            "total": 1
          },
          "runtime": 0
        }
      ],
      "status": "passed",
      "testCounts": {
        "passed": 2,
        "failed": 0,
        "skipped": 0,
        "todo": 0,
        "total": 2
      },
      "runtime": 0
    }
  ],
  "status": "failed",
  "testCounts": {
    "passed": 5,
    "failed": 2,
    "skipped": 2,
    "todo": 0,
    "total": 9
  },
  "runtime": 0
}
  stack:     at Object.assert (/js-reporters/test/integration/adapters.js:315:18)
  ...
1..140
# pass 134
# skip 0
# todo 0
# fail 6
log/failed--mocha--8.1.3.log 
not ok 106 Adapters integration > Mocha adapter > Event "testEnd" runtime property
  ---
  message: "failed"
  severity: failed
  actual  : "undefined"
  expected: "number"
  stack:     at collectedData.forEach.value (/js-reporters/test/integration/adapters.js:185:20)
    at Array.forEach (<anonymous>)
    at Object.assert (/js-reporters/test/integration/adapters.js:183:23)
  ...
  ---
  message: "failed"
  severity: failed
  actual  : "undefined"
  expected: "number"
  stack:     at collectedData.forEach.value (/js-reporters/test/integration/adapters.js:185:20)
    at Array.forEach (<anonymous>)
    at Object.assert (/js-reporters/test/integration/adapters.js:183:23)
  ...
ok 107 Adapters integration > Mocha adapter > Event "testEnd" errors property
ok 108 Adapters integration > Mocha adapter > Event "testEnd" assertions property
ok 109 Adapters integration > Mocha adapter > global suite starts
ok 110 Adapters integration > Mocha adapter > global test starts
ok 111 Adapters integration > Mocha adapter > global test ends
ok 112 Adapters integration > Mocha adapter > suite with one passing test starts
ok 113 Adapters integration > Mocha adapter > passing test starts
ok 114 Adapters integration > Mocha adapter > passing test ends
ok 115 Adapters integration > Mocha adapter > suite with one passing test ends
ok 116 Adapters integration > Mocha adapter > suite with one skipped test starts
ok 117 Adapters integration > Mocha adapter > skipped test starts
not ok 118 Adapters integration > Mocha adapter > skipped test ends
  ---
  message: "failed"
  severity: failed
  actual  : {
  "name": "should skip",
  "suiteName": "suite with skipped test",
  "fullName": [
    "suite with skipped test",
    "should skip"
  ],
  "status": "pending",
  "runtime": 0,
  "errors": [],
  "assertions": []
}
  expected: {
  "name": "should skip",
  "suiteName": "suite with skipped test",
  "fullName": [
    "suite with skipped test",
    "should skip"
  ],
  "status": "skipped",
  "errors": [],
  "assertions": []
}
  stack:     at Object.assert (/js-reporters/test/integration/adapters.js:315:18)
  ...
not ok 119 Adapters integration > Mocha adapter > suite with one skipped test ends
  ---
  message: "failed"
  severity: failed
  actual  : "failed"
  expected: "skipped"
  stack:     at Object.assert (/js-reporters/test/integration/adapters.js:295:20)
  ...
  ---
  message: "failed"
  severity: failed
  actual  : {
  "passed": 0,
  "failed": 0,
  "skipped": 0,
  "todo": 0,
  "total": 1
}
  expected: {
  "passed": 0,
  "failed": 0,
  "skipped": 1,
  "todo": 0,
  "total": 1
}
  stack:     at Object.assert (/js-reporters/test/integration/adapters.js:305:20)
  ...
  ---
  message: "failed"
  severity: failed
  actual  : {
  "name": "suite with skipped test",
  "fullName": [
    "suite with skipped test"
  ],
  "tests": [
    {
      "name": "should skip",
      "suiteName": "suite with skipped test",
      "fullName": [
        "suite with skipped test",
        "should skip"
      ],
      "status": "pending",
      "runtime": 0,
      "errors": [],
      "assertions": []
    }
  ],
  "childSuites": [],
  "status": "failed",
  "testCounts": {
    "passed": 0,
    "failed": 0,
    "skipped": 0,
    "todo": 0,
    "total": 1
  },
  "runtime": 0
}
  expected: {
  "name": "suite with skipped test",
  "fullName": [
    "suite with skipped test"
  ],
  "tests": [
    {
      "name": "should skip",
      "suiteName": "suite with skipped test",
      "fullName": [
        "suite with skipped test",
        "should skip"
      ],
      "status": "skipped",
      "errors": [],
      "assertions": []
    }
  ],
  "childSuites": [],
  "status": "skipped",
  "testCounts": {
    "passed": 0,
    "failed": 0,
    "skipped": 1,
    "todo": 0,
    "total": 1
  }
}
  stack:     at Object.assert (/js-reporters/test/integration/adapters.js:315:18)
  ...
ok 120 Adapters integration > Mocha adapter > suite with one failing tests
ok 121 Adapters integration > Mocha adapter > failing test starts
ok 122 Adapters integration > Mocha adapter > failing test ends
ok 123 Adapters integration > Mocha adapter > suite with one failing test ends
ok 124 Adapters integration > Mocha adapter > suite with multiple tests starts
ok 125 Adapters integration > Mocha adapter > passing test starts
ok 126 Adapters integration > Mocha adapter > passing test ends
ok 127 Adapters integration > Mocha adapter > skipped test starts
not ok 128 Adapters integration > Mocha adapter > skipped test ends
  ---
  message: "failed"
  severity: failed
  actual  : {
  "name": "should skip",
  "suiteName": "suite with tests",
  "fullName": [
    "suite with tests",
    "should skip"
  ],
  "status": "pending",
  "runtime": 0,
  "errors": [],
  "assertions": []
}
  expected: {
  "name": "should skip",
  "suiteName": "suite with tests",
  "fullName": [
    "suite with tests",
    "should skip"
  ],
  "status": "skipped",
  "errors": [],
  "assertions": []
}
  stack:     at Object.assert (/js-reporters/test/integration/adapters.js:315:18)
  ...
ok 129 Adapters integration > Mocha adapter > failing test starts
ok 130 Adapters integration > Mocha adapter > failing test ends
not ok 131 Adapters integration > Mocha adapter > suite with multiple tests ends
  ---
  message: "failed"
  severity: failed
  actual  : {
  "passed": 1,
  "failed": 1,
  "skipped": 0,
  "todo": 0,
  "total": 3
}
  expected: {
  "passed": 1,
  "failed": 1,
  "skipped": 1,
  "todo": 0,
  "total": 3
}
  stack:     at Object.assert (/js-reporters/test/integration/adapters.js:305:20)
  ...
  ---
  message: "failed"
  severity: failed
  actual  : {
  "name": "suite with tests",
  "fullName": [
    "suite with tests"
  ],
  "tests": [
    {
      "name": "should pass",
      "suiteName": "suite with tests",
      "fullName": [
        "suite with tests",
        "should pass"
      ],
      "status": "passed",
      "runtime": 0,
      "errors": [],
      "assertions": [
        {
          "passed": true
        }
      ]
    },
    {
      "name": "should skip",
      "suiteName": "suite with tests",
      "fullName": [
        "suite with tests",
        "should skip"
      ],
      "status": "pending",
      "runtime": 0,
      "errors": [],
      "assertions": []
    },
    {
      "name": "should fail",
      "suiteName": "suite with tests",
      "fullName": [
        "suite with tests",
        "should fail"
      ],
      "status": "failed",
      "runtime": 0,
      "errors": [
        {
          "passed": false
        }
      ],
      "assertions": [
        {
          "passed": false
        }
      ]
    }
  ],
  "childSuites": [],
  "status": "failed",
  "testCounts": {
    "passed": 1,
    "failed": 1,
    "skipped": 0,
    "todo": 0,
    "total": 3
  },
  "runtime": 0
}
  expected: {
  "name": "suite with tests",
  "fullName": [
    "suite with tests"
  ],
  "tests": [
    {
      "name": "should pass",
      "suiteName": "suite with tests",
      "fullName": [
        "suite with tests",
        "should pass"
      ],
      "status": "passed",
      "runtime": 0,
      "errors": [],
      "assertions": [
        {
          "passed": true
        }
      ]
    },
    {
      "name": "should skip",
      "suiteName": "suite with tests",
      "fullName": [
        "suite with tests",
        "should skip"
      ],
      "status": "skipped",
      "errors": [],
      "assertions": []
    },
    {
      "name": "should fail",
      "suiteName": "suite with tests",
      "fullName": [
        "suite with tests",
        "should fail"
      ],
      "status": "failed",
      "runtime": 0,
      "errors": [
        {
          "passed": false
        }
      ],
      "assertions": [
        {
          "passed": false
        }
      ]
    }
  ],
  "childSuites": [],
  "status": "failed",
  "testCounts": {
    "passed": 1,
    "failed": 1,
    "skipped": 1,
    "todo": 0,
    "total": 3
  },
  "runtime": 0
}
  stack:     at Object.assert (/js-reporters/test/integration/adapters.js:315:18)
  ...
ok 132 Adapters integration > Mocha adapter > outter suite starts
ok 133 Adapters integration > Mocha adapter > outter test starts
ok 134 Adapters integration > Mocha adapter > outter test ends
ok 135 Adapters integration > Mocha adapter > inner suite starts
ok 136 Adapters integration > Mocha adapter > inner test starts
ok 137 Adapters integration > Mocha adapter > inner test ends
ok 138 Adapters integration > Mocha adapter > inner suite ends
ok 139 Adapters integration > Mocha adapter > outter suite ends
not ok 140 Adapters integration > Mocha adapter > global suite ends
  ---
  message: "failed"
  severity: failed
  actual  : {
  "passed": 5,
  "failed": 2,
  "skipped": 0,
  "todo": 0,
  "total": 9
}
  expected: {
  "passed": 5,
  "failed": 2,
  "skipped": 2,
  "todo": 0,
  "total": 9
}
  stack:     at Object.assert (/js-reporters/test/integration/adapters.js:305:20)
  ...
  ---
  message: "failed"
  severity: failed
  actual  : {
  "fullName": [],
  "tests": [
    {
      "name": "global test",
      "fullName": [
        "global test"
      ],
      "status": "passed",
      "runtime": 0,
      "errors": [],
      "assertions": [
        {
          "passed": true
        }
      ]
    }
  ],
  "childSuites": [
    {
      "name": "suite with passing test",
      "fullName": [
        "suite with passing test"
      ],
      "tests": [
        {
          "name": "should pass",
          "suiteName": "suite with passing test",
          "fullName": [
            "suite with passing test",
            "should pass"
          ],
          "status": "passed",
          "runtime": 0,
          "errors": [],
          "assertions": [
            {
              "passed": true
            }
          ]
        }
      ],
      "childSuites": [],
      "status": "passed",
      "testCounts": {
        "passed": 1,
        "failed": 0,
        "skipped": 0,
        "todo": 0,
        "total": 1
      },
      "runtime": 0
    },
    {
      "name": "suite with skipped test",
      "fullName": [
        "suite with skipped test"
      ],
      "tests": [
        {
          "name": "should skip",
          "suiteName": "suite with skipped test",
          "fullName": [
            "suite with skipped test",
            "should skip"
          ],
          "status": "pending",
          "runtime": 0,
          "errors": [],
          "assertions": []
        }
      ],
      "childSuites": [],
      "status": "failed",
      "testCounts": {
        "passed": 0,
        "failed": 0,
        "skipped": 0,
        "todo": 0,
        "total": 1
      },
      "runtime": 0
    },
    {
      "name": "suite with failing test",
      "fullName": [
        "suite with failing test"
      ],
      "tests": [
        {
          "name": "should fail",
          "suiteName": "suite with failing test",
          "fullName": [
            "suite with failing test",
            "should fail"
          ],
          "status": "failed",
          "runtime": 0,
          "errors": [
            {
              "passed": false
            }
          ],
          "assertions": [
            {
              "passed": false
            }
          ]
        }
      ],
      "childSuites": [],
      "status": "failed",
      "testCounts": {
        "passed": 0,
        "failed": 1,
        "skipped": 0,
        "todo": 0,
        "total": 1
      },
      "runtime": 0
    },
    {
      "name": "suite with tests",
      "fullName": [
        "suite with tests"
      ],
      "tests": [
        {
          "name": "should pass",
          "suiteName": "suite with tests",
          "fullName": [
            "suite with tests",
            "should pass"
          ],
          "status": "passed",
          "runtime": 0,
          "errors": [],
          "assertions": [
            {
              "passed": true
            }
          ]
        },
        {
          "name": "should skip",
          "suiteName": "suite with tests",
          "fullName": [
            "suite with tests",
            "should skip"
          ],
          "status": "pending",
          "runtime": 0,
          "errors": [],
          "assertions": []
        },
        {
          "name": "should fail",
          "suiteName": "suite with tests",
          "fullName": [
            "suite with tests",
            "should fail"
          ],
          "status": "failed",
          "runtime": 0,
          "errors": [
            {
              "passed": false
            }
          ],
          "assertions": [
            {
              "passed": false
            }
          ]
        }
      ],
      "childSuites": [],
      "status": "failed",
      "testCounts": {
        "passed": 1,
        "failed": 1,
        "skipped": 0,
        "todo": 0,
        "total": 3
      },
      "runtime": 0
    },
    {
      "name": "outter suite",
      "fullName": [
        "outter suite"
      ],
      "tests": [
        {
          "name": "outter test",
          "suiteName": "outter suite",
          "fullName": [
            "outter suite",
            "outter test"
          ],
          "status": "passed",
          "runtime": 0,
          "errors": [],
          "assertions": [
            {
              "passed": true
            }
          ]
        }
      ],
      "childSuites": [
        {
          "name": "inner suite",
          "fullName": [
            "outter suite",
            "inner suite"
          ],
          "tests": [
            {
              "name": "inner test",
              "suiteName": "inner suite",
              "fullName": [
                "outter suite",
                "inner suite",
                "inner test"
              ],
              "status": "passed",
              "runtime": 0,
              "errors": [],
              "assertions": [
                {
                  "passed": true
                }
              ]
            }
          ],
          "childSuites": [],
          "status": "passed",
          "testCounts": {
            "passed": 1,
            "failed": 0,
            "skipped": 0,
            "todo": 0,
            "total": 1
          },
          "runtime": 0
        }
      ],
      "status": "passed",
      "testCounts": {
        "passed": 2,
        "failed": 0,
        "skipped": 0,
        "todo": 0,
        "total": 2
      },
      "runtime": 0
    }
  ],
  "status": "failed",
  "testCounts": {
    "passed": 5,
    "failed": 2,
    "skipped": 0,
    "todo": 0,
    "total": 9
  },
  "runtime": 0
}
  expected: {
  "fullName": [],
  "tests": [
    {
      "name": "global test",
      "fullName": [
        "global test"
      ],
      "status": "passed",
      "runtime": 0,
      "errors": [],
      "assertions": [
        {
          "passed": true
        }
      ]
    }
  ],
  "childSuites": [
    {
      "name": "suite with passing test",
      "fullName": [
        "suite with passing test"
      ],
      "tests": [
        {
          "name": "should pass",
          "suiteName": "suite with passing test",
          "fullName": [
            "suite with passing test",
            "should pass"
          ],
          "status": "passed",
          "runtime": 0,
          "errors": [],
          "assertions": [
            {
              "passed": true
            }
          ]
        }
      ],
      "childSuites": [],
      "status": "passed",
      "testCounts": {
        "passed": 1,
        "failed": 0,
        "skipped": 0,
        "todo": 0,
        "total": 1
      },
      "runtime": 0
    },
    {
      "name": "suite with skipped test",
      "fullName": [
        "suite with skipped test"
      ],
      "tests": [
        {
          "name": "should skip",
          "suiteName": "suite with skipped test",
          "fullName": [
            "suite with skipped test",
            "should skip"
          ],
          "status": "skipped",
          "errors": [],
          "assertions": []
        }
      ],
      "childSuites": [],
      "status": "skipped",
      "testCounts": {
        "passed": 0,
        "failed": 0,
        "skipped": 1,
        "todo": 0,
        "total": 1
      }
    },
    {
      "name": "suite with failing test",
      "fullName": [
        "suite with failing test"
      ],
      "tests": [
        {
          "name": "should fail",
          "suiteName": "suite with failing test",
          "fullName": [
            "suite with failing test",
            "should fail"
          ],
          "status": "failed",
          "runtime": 0,
          "errors": [
            {
              "passed": false
            }
          ],
          "assertions": [
            {
              "passed": false
            }
          ]
        }
      ],
      "childSuites": [],
      "status": "failed",
      "testCounts": {
        "passed": 0,
        "failed": 1,
        "skipped": 0,
        "todo": 0,
        "total": 1
      },
      "runtime": 0
    },
    {
      "name": "suite with tests",
      "fullName": [
        "suite with tests"
      ],
      "tests": [
        {
          "name": "should pass",
          "suiteName": "suite with tests",
          "fullName": [
            "suite with tests",
            "should pass"
          ],
          "status": "passed",
          "runtime": 0,
          "errors": [],
          "assertions": [
            {
              "passed": true
            }
          ]
        },
        {
          "name": "should skip",
          "suiteName": "suite with tests",
          "fullName": [
            "suite with tests",
            "should skip"
          ],
          "status": "skipped",
          "errors": [],
          "assertions": []
        },
        {
          "name": "should fail",
          "suiteName": "suite with tests",
          "fullName": [
            "suite with tests",
            "should fail"
          ],
          "status": "failed",
          "runtime": 0,
          "errors": [
            {
              "passed": false
            }
          ],
          "assertions": [
            {
              "passed": false
            }
          ]
        }
      ],
      "childSuites": [],
      "status": "failed",
      "testCounts": {
        "passed": 1,
        "failed": 1,
        "skipped": 1,
        "todo": 0,
        "total": 3
      },
      "runtime": 0
    },
    {
      "name": "outter suite",
      "fullName": [
        "outter suite"
      ],
      "tests": [
        {
          "name": "outter test",
          "suiteName": "outter suite",
          "fullName": [
            "outter suite",
            "outter test"
          ],
          "status": "passed",
          "runtime": 0,
          "errors": [],
          "assertions": [
            {
              "passed": true
            }
          ]
        }
      ],
      "childSuites": [
        {
          "name": "inner suite",
          "fullName": [
            "outter suite",
            "inner suite"
          ],
          "tests": [
            {
              "name": "inner test",
              "suiteName": "inner suite",
              "fullName": [
                "outter suite",
                "inner suite",
                "inner test"
              ],
              "status": "passed",
              "runtime": 0,
              "errors": [],
              "assertions": [
                {
                  "passed": true
                }
              ]
            }
          ],
          "childSuites": [],
          "status": "passed",
          "testCounts": {
            "passed": 1,
            "failed": 0,
            "skipped": 0,
            "todo": 0,
            "total": 1
          },
          "runtime": 0
        }
      ],
      "status": "passed",
      "testCounts": {
        "passed": 2,
        "failed": 0,
        "skipped": 0,
        "todo": 0,
        "total": 2
      },
      "runtime": 0
    }
  ],
  "status": "failed",
  "testCounts": {
    "passed": 5,
    "failed": 2,
    "skipped": 2,
    "todo": 0,
    "total": 9
  },
  "runtime": 0
}
  stack:     at Object.assert (/js-reporters/test/integration/adapters.js:315:18)
  ...
1..140
# pass 134
# skip 0
# todo 0
# fail 6

See also:
Krinkle added a commit that referenced this issue Sep 20, 2020
@Krinkle
Test: Install integration versions in a tmpdir and add logging
9445c6c 
Make the "npm run test-versions" script (integration/adapters-run.js)
more robust by installing packages in a temporary directory
entirely separate from the repository's working directory.

Also capture the output of the test runs in a file so that one
can easily inspect and find out why a version is failing. This
can then be used to report issues upstream (if it's a new version),
or to fix something on our end of it is genuine, or to document
it in failing-versions.js if it is a known issue with a past release.

- qunitjs:
  Make the list of old versions that are failing complete
  and document why they fail.

- jasmine:
  Version 2.5.2 and later failed due to a default reporter killing
  killing the process (again). This needs to be fixed on our side
  by clearing the default reporters. Fixed as part of this commit
  since it didn't need a change to the adapter.

  Version 3.0 and later failed due to randomized order.
  Disable in our test.

- mocha:
  Version 8.0 and later are currently failing due to skipped
  tests not being recognised correctly. I've filed
  #116 to follow
  up on this in a later commit as it looks like that would require
  a source change to the adapter.
Krinkle added a commit that referenced this issue Sep 20, 2020
@Krinkle
Test: Install integration versions in a tmpdir and add logging
66c99f4 
Make the "npm run test-versions" script (integration/adapters-run.js)
more robust by installing packages in a temporary directory
entirely separate from the repository's working directory.

Also capture the output of the test runs in a file so that one
can easily inspect and find out why a version is failing. This
can then be used to report issues upstream (if it's a new version),
or to fix something on our end of it is genuine, or to document
it in failing-versions.js if it is a known issue with a past release.

- qunitjs:
  Make the list of old versions that are failing complete
  and document why they fail.

- jasmine:
  Version 2.5.2 and later failed due to a default reporter killing
  killing the process (again). This needs to be fixed on our side
  by clearing the default reporters. Fixed as part of this commit
  since it didn't need a change to the adapter.

  Version 3.0 and later failed due to randomized order.
  Disable in our test.

- mocha:
  Version 8.0 and later are currently failing due to skipped
  tests not being recognised correctly. I've filed
  #116 to follow
  up on this in a later commit as it looks like that would require
  a source change to the adapter.
@Krinkle
Copy link
Member Author

Krinkle commented Sep 23, 2020

Skipped tests used to have an undefined state, but as of Mocha 8.0 they have the "pending" state. Perhaps we can safely map that to "skipped"? Maybe not, since it sounds like it would be used for tests that will run but haven't yet?

/cc @js-reporters/mocha Would welcome your recomendation on this matter :)

@Krinkle Krinkle added this to the Package 2.0 milestone Jan 24, 2021
@Krinkle
Copy link
Member Author

Krinkle commented Feb 14, 2021

Tracked down to mochajs/mocha@8236ffd7494 (mochajs/mocha#4223), which introduced a new state STATE_PENDING ("pending") and assigned it to the .state property which was previously left undefined for this case.

This wasn't mentioned in the changelog for Mocha 8.0.0, but I guess that's fair since the public API has offered isPending() for many years which still works the same as before and actually uses a separate property .pending = true and has for many years, so it didn't even need updating for this change.

Krinkle added a commit that referenced this issue Feb 14, 2021
@Krinkle
MochaAdapter: Fix for Mocha 8.0+ use of STATE_PENDING
a0d40ba 
Re-ran `npm run test-versions` and confirmed compatibility with 8.0
and all versions since.

Fixes #116.
@Krinkle Krinkle self-assigned this Feb 14, 2021
@Krinkle Krinkle mentioned this issue Feb 14, 2021
Merged
Krinkle added a commit that referenced this issue Feb 14, 2021
@Krinkle
MochaAdapter: Fix for Mocha 8.0+ use of STATE_PENDING
4d81c00 
Re-ran `npm run test-versions` and confirmed compatibility with 8.0
and all versions since.

Fixes #116.
@Krinkle Krinkle mentioned this issue Jun 6, 2021
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Krinkle Krinkle

Labels
Category: npm Package Type: Problem
Milestone
Package 2.0
Development

Successfully merging a pull request may close this issue.

MochaAdapter: Fix for Mocha 8.0+ use of STATE_PENDING qunitjs/js-reporters
1 participant
@Krinkle