Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Maintenance: provide expected failure metadata in junit reports #19775

Merged
merged 3 commits into from
Nov 8, 2022

Conversation

yannbf
Copy link
Member

@yannbf yannbf commented Nov 8, 2022

Issue: N/A

What I did

  • Fixed a bug where junit.xml files were getting overwritten when they shouldn't
  • The e2e tests will now attach metadata in case we use test.fail to Junit reports, for expected failures:
<!-- Example with an expected failure -->
<testsuite name="addon-interactions.spec.ts" timestamp="1667901141467" hostname="" tests="1" failures="0" skipped="0" time="6.584" errors="0">
  <testcase name="addon-interactions should have interactions" classname="[chromium] › addon-interactions.spec.ts › addon-interactions › should have interactions" time="6.584">
    <properties>
      <property name="fail" value="React Vite (JS) is failing, however it is expected">
      </property>
    </properties>
  </testcase>
</testsuite>

<!-- Example with no expected failure -->
<testsuite name="addon-viewport.spec.ts" timestamp="1667901141467" hostname="" tests="1" failures="0" skipped="0" time="1.724" errors="0">
  <testcase name="addon-viewport should have viewport button in the toolbar" classname="[chromium] › addon-viewport.spec.ts › addon-viewport › should have viewport button in the toolbar" time="1.724">
  </testcase>
</testsuite>

This way, if we have an e2e test that we know is failing, but we don't want the CI to be red all the time, however we want to know that failure so we can process it somewhere else, we can.

Here's an example of test.fail usage:

test('should have interactions', async ({ page }) => {
    test.fail(
      // eslint-disable-next-line jest/valid-title
      /^(Lit)/i.test(`${templateName}`),
      `${templateName} is failing, however it is expected`
    );
});

How to test

  • Is this testable with Jest or Chromatic screenshots?
  • Does this need a new example in the kitchen sink apps?
  • Does this need an update to the documentation?

If your answer is yes to any of these, please make sure to include it in your PR.

@yannbf yannbf added the build Internal-facing build tooling & test updates label Nov 8, 2022
@yannbf yannbf marked this pull request as ready for review November 8, 2022 11:43
@yannbf yannbf requested a review from kasperpeulen November 8, 2022 11:47
scripts/task.ts Outdated Show resolved Hide resolved
@yannbf yannbf force-pushed the chore/add-e2e-test-metadata branch from fb80cf4 to 79cfb85 Compare November 8, 2022 17:28
@yannbf yannbf changed the title provide expected failure metadata in junit reports Maintenance: provide expected failure metadata in junit reports Nov 8, 2022
@yannbf yannbf merged commit 6a9ecf0 into next Nov 8, 2022
@yannbf yannbf deleted the chore/add-e2e-test-metadata branch November 8, 2022 17:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build Internal-facing build tooling & test updates
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants