Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests are hanging in an in progress state when retries is greater than 0 #85

Open
costa-collibra opened this issue Mar 7, 2023 · 21 comments
Assignees
Labels
bug Something isn't working workaround The workaround available to avoid the bug

Comments

@costa-collibra
Copy link

costa-collibra commented Mar 7, 2023

Here is the config:

const testTags = process.env.TAGS ? process.env.TAGS : 'All';

const RPconfig = {
  token: process.env.REPORT_PORTAL_API_KEY,
  endpoint: process.env.REPORT_PORTAL_URL,
  project: process.env.PROJECT_NAME,
  launch: `${process.env.BRANCH_NAME} - ${process.env.BUILD_NUMBER}`,
  attributes: [
    {
      key: 'branchName',
      value: process.env.BRANCH_NAME,
    },
    {
      key: 'testTags',
      value: testTags,
    },
  ],
  description: 'E2E Playwright test launch',
};

const reportList: ReporterDescription[] = [
  ['list'],
  ['html', { open: 'never' }],
  ['line'],
];

if (process.env.CI === 'true') {
  reportList.push(['./utils/CiReporter.ts']);
  reportList.push(['@reportportal/agent-js-playwright', RPconfig]);
}

const config: PlaywrightTestConfig = {
  testDir: './tests',
  timeout: 2 * 60 * 1000,
  expect: {
    timeout: 5000,
  },
  fullyParallel: true,
  retries: process.env.CI === 'true' ? 1 : 0, // report portal issue when retries is > 0
  workers: process.env.CI === 'true' ? 5 : 1,
  forbidOnly: process.env.CI === 'true',
  reporter: reportList,
  outputDir: 'test-assets/',
  use: {
    browserName: 'chromium',
    headless: process.env.CI === 'true',
    screenshot: {
      mode: 'only-on-failure',
      fullPage: true,
    },
    video: 'retain-on-failure',
    actionTimeout: 5000,
    navigationTimeout: 30000,
    baseURL: process.env.BASE_URL,
    trace: 'retain-on-failure',
    launchOptions: {
      slowMo: 0,
    },
  },
};

export default config;

Report portal results just hang in progress and have to be force stopped.

However when retries is 0 it works.

Please can someone investigate?

@AmsterGet
Copy link
Member

Hello @costa-collibra !
Could you please specify the versions of Playwright and agent-js-playwright you are using?
I'll try to take a look.

@costa-collibra
Copy link
Author

costa-collibra commented Mar 7, 2023

Hi @AmsterGet

Yes of course I am using:

"@playwright/test": "1.31.0",
"@reportportal/agent-js-playwright": "5.0.8",

Thanks for investigating

@costa-collibra
Copy link
Author

image

The 3 dots show the in progress state which is stuck permanently

@AmsterGet
Copy link
Member

@costa-collibra Did you have any errors in the console?

@costa-collibra
Copy link
Author

I dont see any errors

image

in this screenshot you can see 2 runs 1 with retries and 1 without. 1 completes and the other is stuck

Here is the console output

Running 6 tests using 1 worker

  -  1 browseComponent.test.ts:13:8 › Browse panel › @p0e2e User opens Browse panel and searches for community
  ✓  2 browseComponent.test.ts:27:7 › Browse panel › @p0e2e User opens Browse panel and searches for views (20.1s)
  ✓  3 browseComponent.test.ts:41:7 › Browse panel › @p0e2e User opens Browse panel and searches for dashboard (20.1s)
  ✓  4 homepage.test.ts:4:7 › HomePage › @smoke HomePage (19.7s)
  -  5 topNav.test.ts:9:8 › Top Navigation Bar › @smoke Top Navigation should be visible (for new Homepage)
  ✘  6 topNav.test.ts:17:7 › Top Navigation Bar › @p0e2e User navigates to glossary page using Products menu (36.9s)
  1) topNav.test.ts:17:7 › Top Navigation Bar › @p0e2e User navigates to glossary page using Products menu 

    page.goto: Timeout 30000ms exceeded.
    =========================== logs ===========================
    navigating to "https://infra-integration-costa.collibra.com/apps/", waiting until "load"
    ============================================================

       at ../pages/BasePage.ts:18

      16 |   async visit(path: string): Promise<void> {
      17 |     await Promise.all([
    > 18 |       this.page.goto(path),
         |                 ^
      19 |       expect(this.page).toHaveURL(path),
      20 |     ]);
      21 |     await expect(this.topNavComponent.parentContainer).toBeVisible();

        at HomePage.visit (/Users/costa.giannakopoulos/Workspace/e2e-tests/playwright/pages/BasePage.ts:18:17)
        at /Users/costa.giannakopoulos/Workspace/e2e-tests/playwright/pages/HomePage.ts:18:19
        at HomePage.visit (/Users/costa.giannakopoulos/Workspace/e2e-tests/playwright/pages/HomePage.ts:17:16)
        at /Users/costa.giannakopoulos/Workspace/e2e-tests/playwright/tests/topNav.test.ts:5:18

    attachment #1: Console Errors (text/plain) ─────────────────────────────────────────────────────
    test-assets/topNav-Top-Navigation-Bar-p0e2e-User-navigates-to-glossary-page-using-Products-menu/logs.txt
    ────────────────────────────────────────────────────────────────────────────────────────────────

    attachment #2: video (video/webm) ──────────────────────────────────────────────────────────────
    test-assets/topNav-Top-Navigation-Bar-p0e2e-User-navigates-to-glossary-page-using-Products-menu/video.webm
    ────────────────────────────────────────────────────────────────────────────────────────────────

    attachment #3: trace (application/zip) ─────────────────────────────────────────────────────────
    test-assets/topNav-Top-Navigation-Bar-p0e2e-User-navigates-to-glossary-page-using-Products-menu/trace.zip
    Usage:

        npx playwright show-trace test-assets/topNav-Top-Navigation-Bar-p0e2e-User-navigates-to-glossary-page-using-Products-menu/trace.zip

    ────────────────────────────────────────────────────────────────────────────────────────────────

    attachment #4: trace (application/zip) ─────────────────────────────────────────────────────────
    test-assets/topNav-Top-Navigation-Bar-p0e2e-User-navigates-to-glossary-page-using-Products-menu/trace-1.zip
    Usage:

        npx playwright show-trace test-assets/topNav-Top-Navigation-Bar-p0e2e-User-navigates-to-glossary-page-using-Products-menu/trace-1.zip

    ────────────────────────────────────────────────────────────────────────────────────────────────

    attachment #5: screenshot (image/png) ──────────────────────────────────────────────────────────
    test-assets/topNav-Top-Navigation-Bar-p0e2e-User-navigates-to-glossary-page-using-Products-menu/test-failed-1.png
    ────────────────────────────────────────────────────────────────────────────────────────────────




  1) topNav.test.ts:17:7 › Top Navigation Bar › @p0e2e User navigates to glossary page using Products menu 

    page.goto: Timeout 30000ms exceeded.
    =========================== logs ===========================
    navigating to "https://infra-integration-costa.collibra.com/apps/", waiting until "load"
    ============================================================

       at ../pages/BasePage.ts:18

      16 |   async visit(path: string): Promise<void> {
      17 |     await Promise.all([
    > 18 |       this.page.goto(path),
         |                 ^
      19 |       expect(this.page).toHaveURL(path),
      20 |     ]);
      21 |     await expect(this.topNavComponent.parentContainer).toBeVisible();

        at HomePage.visit (/Users/costa.giannakopoulos/Workspace/e2e-tests/playwright/pages/BasePage.ts:18:17)
        at /Users/costa.giannakopoulos/Workspace/e2e-tests/playwright/pages/HomePage.ts:18:19
        at HomePage.visit (/Users/costa.giannakopoulos/Workspace/e2e-tests/playwright/pages/HomePage.ts:17:16)
        at /Users/costa.giannakopoulos/Workspace/e2e-tests/playwright/tests/topNav.test.ts:5:18

    attachment #1: Console Errors (text/plain) ─────────────────────────────────────────────────────
    test-assets/topNav-Top-Navigation-Bar-p0e2e-User-navigates-to-glossary-page-using-Products-menu/logs.txt
    ────────────────────────────────────────────────────────────────────────────────────────────────

    attachment #2: video (video/webm) ──────────────────────────────────────────────────────────────
    test-assets/topNav-Top-Navigation-Bar-p0e2e-User-navigates-to-glossary-page-using-Products-menu/video.webm
    ────────────────────────────────────────────────────────────────────────────────────────────────

    attachment #3: trace (application/zip) ─────────────────────────────────────────────────────────
    test-assets/topNav-Top-Navigation-Bar-p0e2e-User-navigates-to-glossary-page-using-Products-menu/trace.zip
    Usage:

        npx playwright show-trace test-assets/topNav-Top-Navigation-Bar-p0e2e-User-navigates-to-glossary-page-using-Products-menu/trace.zip

    ────────────────────────────────────────────────────────────────────────────────────────────────

    attachment #4: trace (application/zip) ─────────────────────────────────────────────────────────
    test-assets/topNav-Top-Navigation-Bar-p0e2e-User-navigates-to-glossary-page-using-Products-menu/trace-1.zip
    Usage:

        npx playwright show-trace test-assets/topNav-Top-Navigation-Bar-p0e2e-User-navigates-to-glossary-page-using-Products-menu/trace-1.zip

    ────────────────────────────────────────────────────────────────────────────────────────────────

    attachment #5: screenshot (image/png) ──────────────────────────────────────────────────────────
    test-assets/topNav-Top-Navigation-Bar-p0e2e-User-navigates-to-glossary-page-using-Products-menu/test-failed-1.png
    ────────────────────────────────────────────────────────────────────────────────────────────────

  1 flaky
    topNav.test.ts:17:7 › Top Navigation Bar › @p0e2e User navigates to glossary page using Products menu 
  2 skipped
  1 flaky
    topNav.test.ts:17:7 › Top Navigation Bar › @p0e2e User navigates to glossary page using Products menu 
  2 skipped
  3 passed (2.0m)

@costa-collibra
Copy link
Author

@AmsterGet can you reproduce this error on your side?

@AmsterGet
Copy link
Member

Hello @costa-collibra !
Seems like I've reproduced the issue.
In my case the issue occurs (without any errors in console) while using static .skip() annotations in tests and retries > 0.
Do you have static .skip() annotations in your tests?

@costa-collibra
Copy link
Author

costa-collibra commented Mar 13, 2023 via email

@costa-collibra
Copy link
Author

Here is one of the tests

test.fixme(
    '@smoke Top Navigation should be visible (for new Homepage)',
    async ({ homePage }) => {
      const { topNavComponent } = homePage;
      await topNavComponent.verifyTopNavIsVisible();
    },
  );

@AmsterGet AmsterGet added the bug Something isn't working label Mar 14, 2023
@AmsterGet
Copy link
Member

AmsterGet commented Mar 14, 2023

@costa-collibra
I'm trying to find possible workarounds for this case before a fix is available.
So far, I haven't seen In Progress infinite launches when using non-static annotations like:

test('@smoke Top Navigation should be visible (for new Homepage)',
    async ({ homePage }) => {
      test.fixme();
      const { topNavComponent } = homePage;
      await topNavComponent.verifyTopNavIsVisible();
    },
  );

Also I cannot reproduce the issue with playwright/test version lower than 1.31.0.

Could you please recheck these two options on your side and confirm/decline them?

@costa-collibra
Copy link
Author

costa-collibra commented Mar 14, 2023

@AmsterGet Interesting.

So you this bug is only reproducible when:

  • using playwright version 1.31.0
  • test.skip() or test.fixme() is used like this:
test.fixme(
    '@smoke Top Navigation should be visible (for new Homepage)',
    async ({ homePage }) => {
      const { topNavComponent } = homePage;
      await topNavComponent.verifyTopNavIsVisible();
    },
  );

If you just add test.fixme() inside the test it works?

My current deployment of report portal is currently down whilst we figure which team is going to take responsibility for it and that I have a meeting this afternoon.

All being well we can have it back tomorrow and I can verify your questions, please bare with me till then :)

@AmsterGet AmsterGet self-assigned this Mar 14, 2023
@costa-collibra
Copy link
Author

@AmsterGet Hi

The results are no longer hanging when i changed the syntax to this:

test('@smoke Top Navigation should be visible (for new Homepage)', async ({
    homePage,
  }) => {
    test.fixme();
    const { topNavComponent } = homePage;
    await topNavComponent.verifyTopNavIsVisible();
  });

This is with retries set to 1 in the Playwright config.

Most people do however use the other syntax.

Will you be looking to fix this? In the meantime at least we have a workaround

image

@AmsterGet
Copy link
Member

Hello @costa-collibra !
I've made some changes to minimize this issue occurrence, but with current agent implementation we cannot fully avoid this issue without providing additional data from Playwright runner to the reporter.
I plan to post an issue to the Playwright repo with my investigations and suggestions, I'll refer it here once it be ready.
For now the issue reproduced only in case of the hook error and retries enabled that leads to retrying skipped tests marked with .skip() or .fixme() static annotations in the same suite (file or describe block).

I've also added the issue troubleshooting section to the readme file with found workaround and link to this issue.

Feel free to add your feedback on this :)

@AmsterGet AmsterGet added the workaround The workaround available to avoid the bug label Mar 16, 2023
@AmsterGet
Copy link
Member

AmsterGet commented Mar 21, 2023

@costa-collibra I've just released the new version 5.0.9 with some fixes that I've mentioned above.
Could you please try and confirm that the issue no longer exists?

@costa-collibra
Copy link
Author

@AmsterGet thanks for this update. I will update the library today and feedback.

If you can also tag me or send me a link to the issue when you log it with Playwright that would be great. If i can help with any feedback with them I would certainly be happy to do so

@costa-collibra
Copy link
Author

costa-collibra commented Mar 21, 2023

@AmsterGet with your fix in can I go back to this code

test.describe('Top Navigation Bar', () => {
  test.fixme(
    '@smoke Top Navigation should be visible (for new Homepage)',
    async ({ homePage }) => {
      const { topNavComponent } = homePage;
      await topNavComponent.verifyTopNavIsVisible();
    },
  );

or do i need to keep it inside the test

@AmsterGet
Copy link
Member

@costa-collibra yes, you can try your previous solution.
The issue will be reproduced only for case described in Issues troubleshooting section.

@costa-collibra
Copy link
Author

@AmsterGet it worked Thanks alot :)

@AmsterGet AmsterGet pinned this issue Mar 21, 2023
@sumit-gupta91
Copy link

@AmsterGet I am seeing a similar behaviour. Some of it was fixed in latest version 5.0.11 but I think one case is missed and that is still causing Report portal to hang. If the test has test.fail() in it then tests seem to hang on report portal.

@sumit-gupta91
Copy link

sumit-gupta91 commented May 3, 2023

@AmsterGet In this line - https://github.com/razorpay/frontend-website/pull/2044
Should we also not account for test.fail() static annotations ?

@AmsterGet
Copy link
Member

@sumit-gupta91
The page you shared is no longer available.
Could you please provide more details?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working workaround The workaround available to avoid the bug
Projects
None yet
Development

No branches or pull requests

3 participants