Reportportal run doesn't end until timeout since version 5.0.3

[StephaneColson]

Since version 5.0.3, even when our Jenkins runs are finished, they appear still pending in the reportportal until the defined timeout.
I just downgraded to version 5.0.2 and we don't have the issue anymore.

Is there something specific to configure since 5.0.3 on our side? I know that RP should receive FinishLaunchRQ, in order to finish launch on UI. but we didn't change anything.

[AmsterGet]

Hello @StephaneColson !
At first glance it looks like a bug, no special configuration is required.
We will address this issue in the next sprint.

[renkyoji]

Hello @StephaneColson ,
Could you run your tests directly from the develop branch? We made several fixes and maybe your problem has already been solved.

[StephaneColson]

Hi @oranje322 I will check when the next version will be released. Currently using a workaround: downgrade to version 5.0.2 😉

[renkyoji]

@StephaneColson ,
Could you share your configs and test examples and some error logs if they exist?
Do you use same file names and retries?

[StephaneColson]

@oranje322

Here's playwrigth.config.ts

import { PlaywrightTestConfig } from '@playwright/test';
const RPconfig = {
  'token': 'DISABLED',
  'endpoint': 'https://<RP-url>/api/v1',
  'project': 'pipelinedesigner',
  'launch': 'Playwright test',
  'description': 'UI tests processed by Playwright',
  "skippedIssue": false,
  'attributes': [
    {
      'key': 'RP_LAUNCH',
      'value': 'TEST-UI tests',
    },
    {
      'value': 'INT',
    },
    {
      'value': 'MANUAL-LAUNCH',
    },
    {
      'value': 'UI-PLAYWRIGHT',
    },
    {
      'value': 'build_number_00',
    }
  ],
};

const config: PlaywrightTestConfig = {
  globalTeardown: require.resolve('./global-teardown'),
  globalSetup: require.resolve('./global-setup'),
  timeout: 3*60*1000, // Timeout for each test, includes test, hooks and fixtures
  globalTimeout: 60*60*1000, // Global timeout for the whole test run
  workers: 1,
  retries: 1,
  reporter: [
    ['list'],
    ['html', { outputFolder: 'html-report', open: 'never' }],
    ['junit', { outputFile: 'junit.xml' }],
    ['@reportportal/agent-js-playwright', RPconfig]
  ],
 use: {
    baseURL: 'https://tpd.int.cloud.talend.com/',
    //headless: false, // true by default
    viewport: { width: 1280, height: 720 },
    ignoreHTTPSErrors: true,
    video: 'retain-on-failure',
    screenshot: 'only-on-failure',
    trace: 'off',
    actionTimeout: 1*60*1000, // Timeout for each action
  },

  projects: [
    {
      name: 'Firefox',
      use: { browserName: 'firefox' },
      expect: {
        timeout: 30*1000 // Timeout for each assertion
      }
    },
    {
      name: 'Chromium',
      use: { browserName: 'chromium' },
      expect: {
        timeout: 30*1000 // Timeout for each assertion
      }
    },
    {
      name: 'WebKit',
      use: { browserName: 'webkit' },
      expect: {
        timeout: 30*1000 // Timeout for each assertion
      }
    }
  ]
};
export default config;

I didn't find any error log and I'm not sure to understand what you mean with Do you use same file names and retries?....
Yes we use retries, and no file names duplication...

[renkyoji]

@StephaneColson ,
You got it right.

We have released version 5.0.4, can you recheck?
Let me know about the results.

[StephaneColson]

Hi @oranje322 I upgraded to 5.0.4 and it is still not working (only working with version before 5.0.2).

Jenkins build is finished:

But still in progress on the Report Portal side more than 10 minutes after the end of the Jenkins build:

[StephaneColson]

I let it run...and timeout (compared to the one above #137 with version 5.0.2)

[renkyoji]

@StephaneColson , @marverix
Could you share your test case which causes this behavior?

[marverix]

@oranje322
It's really simple test case (.ts):

import { expect, Page, test } from "@playwright/test";

test("Sign In Button should be visible on the login page", async ({ page }) => {
  await page.goto('/');
  const signInButton = page.locator("button", { hasText: "Sign in" });
  await expect(signInButton).toBeEnabled();
});

The config

{
  "use": {
    "baseURL": "https://**********************/",
    "headless": true,
    "ignoreHTTPSErrors": true,
    "actionTimeout": 10000,
    "navigationTimeout": 10000,
    "screenshot": "only-on-failure",
    "trace": "on-first-retry"
  },
  "expect": {
    "timeout": 5000
  },
  "timeout": 30000,
  "forbidOnly": false,
  "retries": 0,
  "reporter": [
    [
      "list"
    ],
    [
      "@reportportal/agent-js-playwright",
      {
        "token": "**********************",
        "endpoint": "**********************",
        "project": "**********************",
        "launch": "Playwright Tests",
        "description": "Integration Tests written with Playwright",
        "includeTestSteps": true,
        "debug": false
      }
    ]
  ],
  "projects": [
    {
      "name": "chromium",
      "use": {
        "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.15 Safari/537.36",
        "screen": {
          "width": 1920,
          "height": 1080
        },
        "viewport": {
          "width": 1280,
          "height": 720
        },
        "deviceScaleFactor": 1,
        "isMobile": false,
        "hasTouch": false,
        "defaultBrowserType": "chromium"
      }
    },
    {
      "name": "firefox",
      "use": {
        "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:98.0.2) Gecko/20100101 Firefox/98.0.2",
        "screen": {
          "width": 1920,
          "height": 1080
        },
        "viewport": {
          "width": 1280,
          "height": 720
        },
        "deviceScaleFactor": 1,
        "isMobile": false,
        "hasTouch": false,
        "defaultBrowserType": "firefox"
      }
    }
  ],
  "testDir": "**********************/end-to-end-tests/tests"
}

This executes with 5.0.2 but does not show steps:

Where >=5.0.3 shows nicely steps, but does not end.

[marverix]

This is the diff between 5.0.2 and 5.0.3. From my understanding previously method this.finishSuites() was called always in onEnd. Not it has been moved to onTestEnd, which is ok, but the problem is that it has been put into condition. Now it will call it only if testsLength == 0. Which I guess must be failing? I didn't have time to go deeper into the code, but for me this condition is fishy. I understand that this is some kind of optimization maybe (?) but it seems to fail. Could you please try to call this.finishSuites() always?

[AmsterGet]

Hello @marverix !
Thanks for your comment.
This change was made to finish suites gradually, right after their children, rather than at the end of the test run.
But it looks like the bug is hidden here, we are working on this fix and plan to release the patch version next week.

[renkyoji]

@marverix @StephaneColson
Could you tell me what OS are you using? What version of node js?

[StephaneColson]

@oranje322 The Jenkins/CI is using Ubuntu and node 16

[marverix]

@oranje322 Locally Ubuntu 20.04 and NodeJS 16.
On Jenkins nodes, Debian 11 Bullseye and NodeJS 14.

[AmsterGet]

Hello guys @StephaneColson @marverix !
I've just released the version 5.0.5 with bugfixes.
Could you please try this one?
I've reproduced and fixed the issue similar to yours when includeTestSteps: true, but I'm not sure it is relevant to @StephaneColson notes.
It would be nice if you could create a constantly reproducible sample in case the same failure occurs with a new version.
Looking forward for your reply.

[marverix]

Hello @AmsterGet ,
Yes, I can confirm that 5.0.5 fixes the issue :)
Thank you!

[StephaneColson]

Hello @AmsterGet it seems ok now. Thank you

[AmsterGet]

@StephaneColson @marverix
It's good to hear that guys.
So I'll close the issue.
Feel free to open a new one in case of any questions or issues.

[StephaneColson]

Hi @AmsterGet I thought it was ok but it's not. This morning I found a running process (started 4 hours ago, the playwright tests are ended since a long time ago). All other runs ended after more than 4 hours. Not all, some runs that are completely successful ended on the Report Portal side when the run was finished in Jenkins (30 minutes). Could the issue be only for runs with failing tests?

[AmsterGet]

Hi @StephaneColson !
At first glance, maybe, but I'm not sure, since I could not reproduce anything like that.
Could you create a reproducible sample with a similar environment and configuration to help us investigate the issue?

[aysimpson]

I'm seeing some similar behaviour to this. I downgraded to 5.0.2 and no longer have the issue so it might be related.

I'm finding that when I use test.skip() in my test and have retries specified that my run never completes.

Very simple example for this:

test.describe.only('Test', () => {
  test('example', async ({}) => {
    test.skip();
  });
});

In my playwright config I have set retries: 1

If I run on version 5.03 or 5.05 then the run never completes.

If I run on version 5.02 then the run completes.

Where retries isn't specified then the run completes.

[tr1ble]

I'm seeing some similar behaviour to this. I downgraded to 5.0.2 and no longer have the issue so it might be related.

I'm finding that when I use test.skip() in my test and have retries specified that my run never completes.

Very simple example for this:

test.describe.only('Test', () => {
  test('example', async ({}) => {
    test.skip();
  });
});

In my playwright config I have set retries: 1

If I run on version 5.03 or 5.05 then the run never completes.

If I run on version 5.02 then the run completes.

Where retries isn't specified then the run completes.

Thanks a lot, this helped to investigate a bug

[AmsterGet]

Hello guys!
We have released a version 5.0.7 with a fixes.
Could you confirm that this fixes the issue?

[StephaneColson]

Hi @AmsterGet I'm not working anymore on the project but we could ask to @mockersf 👋

[AmsterGet]

Hi @mockersf!
Could you please recheck the issue with latest agent version?

[pgalmes]

Hello,

I work with @mockersf and I take back the ticket opened by @StephaneColson

I have upgraded the agent to 5.0.7, but we still have the issue 😢
The job fully success on Jenkins, but never ends on ReportPortal (falls on timeout after 3h34)

Other problem, ReportPortal shows us 2 runs with one have no test evaluation 🤔

We didn't have that with 5.0.2

Any idea ?

[AmsterGet]

Hello @pgalmes !
Looks like I missed something 😔
Could you please share the logs from Jenkins?
It would also be very helpful to have reproducible sample to debug this issue.
Thank you.

[pgalmes]

Maybe I've found in which contidions ReportPortal run don't end properly
It's only when there is :

• at least one fail during the BeforeAll step
• AND at least one retry

I reproduce it with this small test

import { expect } from '@playwright/test';
import { test } from '../helpers/fixture';

test.beforeAll(async ({ browser, browserName }) => {
  expect(1).toBeGreaterThan(2);
});

test.describe.only('Test', () => {
    test('example skip', async ({}) => {
      test.skip();
    });

    test('example fail', async ({}) => {
      expect(1).toBeGreaterThan(2);
    });

    test('example success', async ({}) => {
        expect(1).toBeGreaterThan(0);
    });
  });

If i run this test with no retry, it ended in Jenkins with FAILURE status => noRetry.txt

If i run this test with one retry, it ended in Jenkins with UNSTABLE status => oneRetry.txt

If i run this test with one retry but without beforeAll step, it ended in Jenkins with FAILURE status => withoutBeforeAll.txt

In Jenkins :

We can see it on ReportPortal

If we focus on the Unstable test with one retry, we noticed 2 runs (instead of just one)

The first one, which is finished, show me the three tests with the good status

The second one, which didn't stopped properly, show me only the second and the third thest which haven't been evaluated (and have been skipped). I think it waits a return code which never arrives and that's why it falls in timeout.

I think now you have a better idea of what is going wrong 🤞
Let me know if you need more precisions 😃

[AmsterGet]

@pgalmes Thank you!!
That was one of my thoughts about what's going on. It seems that the Playwright has retried the whole suite.
I don't fully understand why it only retries some of them. I will look deeper into this example.
Thanks again for your help!

[AmsterGet]

Hello @pgalmes !
The issue was related to the playwright's re-running the skipped tests.
I'm not sure if they need to be retried, even if they ended after a hook error.
I have plans to create an issue on the playwright's repository to discuss this matter.
Anyway, I've created a fix and now the tests that were skipped will also be correctly reported.
Is there any way for you to test run with unreleased version with fixes? Just to confirm that this solves the right issue.

[AmsterGet]

@pgalmes
I've released the version 5.0.8. Please try it.

[pgalmes]

Hi @AmsterGet ,
I've tried version 5.0.8, and it seems to be good !

Thanks a lot !

[AmsterGet]

@pgalmes
Let's now try with your real tests.
Just to make sure everything is ok :)

[pgalmes]

It seems ok with global run
No failures, but there are some tests which needs retries after failing first on Before Hooks.
But ReportPortal ended successfully :)
I will check on next days that our daily runs won't have issue anymore, but i think the problem is solved !
Thanks you for your help

[AmsterGet]

Awesome!
@pgalmes Thank you!
I'll close the issue, finally :)