Test is not finished when expect().toPass() exceed test timeout

[danielsobon]

Test setup:

├── @playwright/[email protected]
├── @reportportal/[email protected]

How to reproduce

1. Set up test timeout to 20 sec

const config: PlaywrightTestConfig = {
  globalSetup: require.resolve('./global-setup'),
  /* Maximum time one test can run for. */
  timeout: 20 * 1000,
}

3. Create a test with any of repeatable check for 30 sec

• expect.poll()
• expect().toPass()

import { expect, test } from '@playwright/test';

test.describe('Expect', () => {
  test(`Expect pool @desktop`, async () => {
    await expect
      .poll(
        () => {
          return 1;
        },
        { timeout: 30_000 }
      )
      .toBe(2);
  });
  test('Expect toPass @desktop', async () => {
    await expect(() => {
      expect(1).toBe(2);
    }).toPass({ timeout: 30_000 });
  });
});

4. Configure RP to includeTestSteps: true

Actual result

Test & test step (expect) are still in progress, so the entire run is also not finished

[danielsobon]

hey @AmsterGet, any update regarding the issue? I'm asking because we realized the problem is more serious than we thought.
Because of fact the assertion cannot be finished we don't receive report, nor valid status of entire playwright run.

• None of standard playwright report (json/html/github) is generated when we have integrated reportPortal
• Github job does not catch the failures (says he run is green even if we have failed tests)

[AmsterGet]

Hey @danielsobon!
Thanks for pointing this out!
Our team has begun to investigate this issue.
We will keep you updated on this thread.

[Bam6ycha]

Hey @danielsobon.
This issue has been successfully fixed and will be available in the next version of the agent.
But I have several questions.
Why are you using timeout inside the test greater than your global timeout?
Does your provided example only reproduce the bug, or is it the real case?

If it is only to reproduce, and you have such tests which run as long as the global timeout exceeds we recommend splitting it into small pieces and running them in parallel.

[danielsobon]

@Bam6ycha yep :) We have tests that consist of many steps. Each of them can take some time. In case of bad performance of application, all combined steps can reach the test timeout, so this is a real case.

Second thing, what you can also reproduce is, when we reach expect timeout during preforming operation, example API request.
Example code (I recommend to set 2-3 retries because it might be reproduced in random run):

import test, { expect, request } from '@playwright/test';

test.describe('Expect @desktop @smoke', () => {
  test.only('Expect toPass @desktop @smoke', async () => {
    await expect(async () => {
      const status = await sendRequestStep();
      expect(status).toBe(400); //status returns 200
    }).toPass({ timeout: 5_000 });
  });
});

async function sendRequestStep() {
  return await test.step(`I get available pets status response`, async () => {
    const context = await request.newContext({
      baseURL: 'https://petstore.swagger.io',
    });
    const status = await context
      .get('/v2/pet/findByStatus?status=available', {})
      .then((response) => response.status())
      .finally(async () => await context.dispose());
    return status;
  });
}

RP view

[danielsobon]

hey @AmsterGet do you know when I could expect new release?

[AmsterGet]

Hi @danielsobon !
I suppose in a few days.
Stay tuned, we'll notify you here.

[AmsterGet]

Hi @danielsobon !
Please check the version 5.1.3.

[danielsobon]

@AmsterGet , @Bam6ycha I've already retested it. Problem does not occur anymore, thx for fix! 😎