[BUG] Playwright sometimes report failed test even when there are no failed assertions or no assertions at all

[anthonyma94]

I have a page that has a selection box where users can check which widgets they want to see. When they check one, the page calls my API for the relevant info and displays a widget on screen. The test passes, but sometimes it would say it failed, even when there are no errors in the errors log.

System info

• Playwright Version: v1.40.1
• Operating System: WSL2 w/ Ubuntu 22.04
• Browser: Chromium
• Other info:

Source code

import { test, expect } from "@playwright/test";

test.beforeEach(async ({ page }) => {
    await page.goto("/rfid");
});

test.setTimeout(5000);

test("Select cell", async ({ page }) => {
    const rfidRequestPromise = page.waitForRequest("/api/rfid/**");
    const selectStationWidget = page
        .locator("div.MuiStack-root")
        .filter({ hasText: "Select stations" });

    await selectStationWidget.click();

    await selectStationWidget
        .getByRole("checkbox", { name: "Select All" })
        .first()
        .check();

    try {
        await rfidRequestPromise;
    } catch (e) {
        console.log("e", e);
    }

    const checkedStatus = await Promise.all(
        (
            await selectStationWidget.getByRole("checkbox").all()
        ).map(async (el) => await el.isChecked())
    );

    expect(checkedStatus.every((status) => status)).toBe(true);

    const selectStationTexts = await selectStationWidget
        .locator("label")
        .filter({ hasNotText: "Select All" })
        .locator("span.MuiTypography-root")
        .allInnerTexts();

    const rfidWidgetEl = page.locator("div.MuiMasonry-root > .MuiBox-root");
    await Promise.allSettled(
        (await rfidWidgetEl.all()).map(async (el) => await el.waitFor())
    );
    const rfidTitles = await rfidWidgetEl.locator("a").allInnerTexts();

    expect(selectStationTexts.sort()).toEqual(rfidTitles.sort());
});

Screenshot of Playwright test UI: https://prnt.sc/pcwUwwXrNa6e

Steps

• Run the test a bunch of times

Expected

It should pass every time

Actual

Sometimes it fails

[yury-s]

Can you attach a self contained example that we could run locally or at least a trace file? Can you also try if it reproduces with 1.41?

[anthonyma94]

Unfortuantely I can't produce an example, it's an internal site. Can you tell me how to create a trace file?

[yury-s]

Please follow our guide. You basically need to set trace: 'on' in the config. You can attach html report then, it contains the traces too.

[joshkay]

I'm having the same issue on 1.41. I randomly encountered it while adding more tests & am I just switching over from cypress to playwright. All of my tests pass without ui mode, but in ui mode some of them randomly report as failed with no visible errors.

I'm not sure how to produce an example either, because it's a relatively large private codebase.

[joshkay]

Here's an example:

[yury-s]

I'm not sure how to produce an example either, because it's a relatively large private codebase.

@joshkay if providing a minimal repro is problematic, can you at least share a trace to help us debug this issue?

[joshkay]

trace.zip

Here's an example of a test that randomly reports as Failed.

[yury-s]

Here's an example:

Are these two screenshots for the same test suite and it fails under UI mode while passing when run from terminal?

[joshkay]

Here's an example:

Are these two screenshots for the same test suite and it fails under UI mode while passing when run from terminal?

Yes exactly, my tests are passing 100% of the time in the terminal. However, in UI mode it's completely random which tests are reporting as failed. There are also no visible errors when they report as failed.