Bug: failed test output points to wrong line in code

[UncleGus]

Given

Config:

import "dotenv/config";
import environments from "./config/environments.json";
import { defineConfig, expect } from "@playwright/test";
import { defineBddConfig } from "playwright-bdd";

const validEnvironments = Object.getOwnPropertyNames(environments);
const environmentErrorMessage = `Environment name not found. Edit .env in the root directory to set GALAXY=<environment name>, using one of the following options:
${validEnvironments}`;
expect(validEnvironments, environmentErrorMessage).toContain(
  process.env.GALAXY
);
console.log(`Using environment ${process.env.GALAXY}`);

const testDir = defineBddConfig({
  features: "tests/features/**/*.feature",
  steps: "tests/**/*.ts",
});

export default defineConfig({
  testDir,
  reporter: "html",
  use: { browserName: "chromium", viewport: { width: 1280, height: 880 } },
});

Feature file:

Feature: Automatic renewal of policy

Background:
  Given I am logged in to PolicyCenter as Super user
  And I am using an existing completed account
  And I have a policy
  And the policy type is Renewing
  And the term type is Annual
  And the policy expiration date is 10 days in the future
  And the policy has a coverage of type Milk

@PC_DF_014
Scenario: Policy renewal for Buildings and Contents
  Given the policy defined in the background
  When I start the batch renew process and return to the account summary

Folders:

When

I added a line to throw an error in the steps definition file, and ran the test from the playwright test runner in VSCode, after running bddgen.

Then

The error was thrown and ended the test but the test-results reports the error in the wrong position:

But I expect

This is actually reasonably close, but in most non-artificially caused errors, it's out by 30-40 lines. Probably proportional to the line number. I expect it to point to where it actually failed. This offset also comes through in the playwright HTML report.

Isolated demo

https://github.com/vitalets/playwright-bdd-example/pull/21

In this case, the results point to a line past where it should, to a line that doesn't even exist:

Environment

Playwright-bdd environment info:

platform: win32
node: v20.12.1 
playwright-bdd: v7.4.2
@playwright/test: v1.47.2
@cucumber/cucumber: none
Playwright config file: playwright.config.ts

[vitalets]

Interesting..
I've run your example and it shows correct location:

Possibly some Win vs OSX issue. Could you try to run pure Playwright test that throws such error?

[UncleGus]

Sure, I'll look into it further at my end.

[UncleGus]

When I open a different repo that does not use bddgen, just Playwright on its own, the error report points to the right place. So I don't think it's environmental/system, it's at the repo level. I'll carry on trying to isolate it.

[UncleGus]

I get the same result in both VSCode and IntelliJ IDEA.

[UncleGus]

Reinstalling node modules made no difference.

[UncleGus]

Setting up a new repo via npm init playwright@latest and inserting an error reports it at the correct place.

[UncleGus]

Adding bdd-playwright to the same repo and generating a test spec file with it and adding the error points to the wrong place. Error occurs with both typescript and javascript.

[UncleGus]

I'm really at a loss now. Apart from sharing my actual system via a teams call or something, I'm not sure how to resolve this issue locally or reproduce this issue elsewhere.

[UncleGus]

Disabled WSL and Hyper-V, no change.

[vitalets]

@UncleGus I've simplified playwright-bdd-example - kept only one step, removed cucumber reporter and added non-bdd project. Could you try to clone and run incorrect-fail-line branch? Here is the PR to see the diff.

When I run npm t I'm getting the following 2 errors:

    Error: Fail in bdd test

       at features/steps/index.ts:4

      2 |
      3 | Given("step", async ({}) => {
    > 4 |   throw new Error("Fail in bdd test");
        |         ^
      5 | });
      6 |

...

    Error: Fail in non bdd test

      2 |
      3 | test("non bdd test", async ({}) => {
    > 4 |   throw new Error("Fail in non bdd test");
        |         ^
      5 | });
      6 |

2. Also could you run only non-bdd-project npx playwright test --project non-bdd-project.
3. And if it still produce incorrect line, comment all BDD-related stuff in playwright.config.ts and run npx playwright test --project non-bdd-project again.

Thank you for your investigation!

[UncleGus]

I'll have to get to that tomorrow, but I will say that I've added non-bdd tests to the actual project that I'm working in, and it reports errors at the correct line.

[UncleGus]

Okay, that didn't take as long as I though :P

Here's my output, and it's looking promising:

  1) [bdd-project] › .features-gen\features\homepage.feature.spec.js:6:7 › Playwright Home Page › Check title › Given step 

    Error: Fail in bdd test

       at features\steps\index.ts:4

      2 |
      3 | Given("step", async ({}) => {
    > 4 |   throw new Error("Fail in bdd test");
        |         ^
      5 | });
      6 |

        at Object.<anonymous> (C:\dev\playwright-bdd-example\features\steps\index.ts:4:9)
        at C:\dev\playwright-bdd-example\.features-gen\features\homepage.feature.spec.js:7:5

  2) [non-bdd-project] › non-bdd\index.spec.ts:3:5 › non bdd test ──────────────────────────────────

    Error: Fail in non bdd test

      2 |
      3 | test("non bdd test", async ({}) => {
    > 4 |   throw new Error("Fail in non bdd test");
        |         ^
      5 | });
      6 |

        at C:\dev\playwright-bdd-example\non-bdd\index.spec.ts:4:9

  2 failed
    [bdd-project] › .features-gen\features\homepage.feature.spec.js:6:7 › Playwright Home Page › Check title
    [non-bdd-project] › non-bdd\index.spec.ts:3:5 › non bdd test ───────────────────────────────────

[UncleGus]

Running the non-bdd project on its own got the same results for that test.

[UncleGus]

A colleague cloned the repo and had the exact same issue, so whatever the cause is, it's not specific to my laptop.

[vitalets]

A colleague cloned the repo and had the exact same issue, so whatever the cause is, it's not specific to my laptop.

I will think how to debug that deeper.
Is he also on win?