all:true not working with Next application

[AlessandroVol23]

Versions

• What is this plugin's version? 3.9.12
• What is Cypress version? 8.2.0
• What is your operating system? Mac OS 12.1
• What is the shell? ZSH
• What is the Node version? 14.18.2
• What is the NPM version? 8.3.2
• How do you instrument your application? Cypress does not instrument web application code, so you need to do it yourself.
  Like mentioned in the installation guide here
• When running tests, if you open the web application in regular browser, and open DevTools, do you see window.__coverage__ object? Can you paste a screenshot?

I don't think I do but I get coverage in the end.

• Is there .nyc_output folder? Is there .nyc_output/out.json file. Is it empty? Can you paste at least part of it so we can see the keys and file paths?

This is generated, but no empty objects for empty files.

• Do you have any custom NYC settings in package.json (nyc object) or in other NYC config files

  "nyc": {
    "all": true
  }

• Do you run Cypress tests in a Docker container?
  No

Describe the bug
I use the option all:true but I don't see all files. For example I have some files in pages/.. which are not displayed at all. It seems that just loaded files are used.

The NYC options from the package.json are used because I fiddled around with the include parameter and it is filtering, so that seems to work. I don't see any difference in the output if I use true or false for the all option.

[rbong]

This package generates a placeholder for files that aren't included in tests if you set all.

This looks like this:

  return {
    path: fullPath,
    statementMap: {},
    fnMap: {},
    branchMap: {},
    s: {},
    f: {},
    b: {}
  }

This format gets into the final output, if you check .nyc_output/out.json it looks similar:

  "/home/roger/app/src/pages/api/hello.js": {
    "path": "/home/roger/app/src/pages/api/hello.js",
    "statementMap": {},
    "fnMap": {},
    "branchMap": {},
    "s": {},
    "f": {},
    "b": {}
  },

However if you look at your instrumented code in the .next directory you will find properly instrumented code like this:

  var coverageData = {
    path: "/home/roger/app/src/pages/api/hello.js",
    statementMap: {
      "0": {
        start: {
          line: 7,
          column: 14
        },
        end: {
          line: 7,
          column: 54
        }
      },
      "1": {
        start: {
          line: 7,
          column: 28
        },
        end: {
          line: 7,
          column: 54
        }
      }
    },
    fnMap: {
      "0": {
        name: "(anonymous_0)",
        decl: {
          start: {
            line: 7,
            column: 14
          },
          end: {
            line: 7,
            column: 15
          }
        },
        loc: {
          start: {
            line: 7,
            column: 28
          },
          end: {
            line: 7,
            column: 54
          }
        },
        line: 7
      }
    },
    branchMap: {},
    s: {
      "0": 0,
      "1": 0
    },
    f: {
      "0": 0
    },
    b: {},
    _coverageSchema: "1a1c01bbd47fc00a2c39e90264f33305004495a9",
    hash: "c5a1a371d41e9074c2c6df39a66c2c07ae1141cf"
  };

This looks like all of the other objects of files that were actually imported in out.json.

You can format this as JSOn and paste it in back in out.json:

  "/home/roger/app/src/pages/api/hello.js": {
    "path": "/home/roger/app/src/pages/api/hello.js",
    "statementMap": {
      "0": {
        "start": {
          "line": 7,
          "column": 14
        },
    ...(etc.)
    "f": {
      "0": 0
    },
    "b": {},
    "_coverageSchema": "1a1c01bbd47fc00a2c39e90264f33305004495a9",
    "hash": "c5a1a371d41e9074c2c6df39a66c2c07ae1141cf"
  },

Here's the coverage report before making this change:

----------------------------|---------|----------|---------|---------|-------------------
File                        | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
----------------------------|---------|----------|---------|---------|-------------------    
  hello.js                  |       0 |        0 |       0 |       0 |           

Here's the coverage report after:

----------------------------|---------|----------|---------|---------|-------------------
File                        | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
----------------------------|---------|----------|---------|---------|-------------------           
  hello.js                  |       0 |      100 |       0 |       0 | 7                 

Not pictured: in the first case the result is white (nothing to cover) and in the second case it is red (not covered).

Note also that the branches are 100% because there are no branches in the file, reinforcing the fact that the placeholder should not be in the final output:

    branchMap: {},

Note however it correctly reports 7 uncovered lines in the file.

In other words: the way this plugin is formatting the output makes NYC think the file is empty. The coverage information should be loaded from the transpiled code instead. I am guessing there must be an Istanbul API for this, though the original files also have to be mapped to the transpiled files when using babel-plugin-istanbul.

[rbong]

I've looked into it more - files that are never used don't have any coverage data generated, so just adding more accurate placeholders using existing coverage data still won't fix the problem.

I have src/components/Button.jsx that's currently unused, and it has no coverage data anywhere. It's not enough to rely on babel-plugin-istanbul here, it never touches the file.

[rbong]

I'm working on a PR, this commit fixes the issue

It uses Babel and Istanbul APIs to get the coverage data object if it's not already present instead of using placeholders.

This is a pretty big change, so I'm open to feedback. Not sure how much longer I can work on this either so I'm leaving this here in case I abandon this.

[rbong]

For my purposes I'm able to merge coverage from jest --coverage --coverageReporters json with the Cypress coverage output using istanbul-lib-coverage so I'm abandoning this PR. Sorry.

[rbong]

Standalone workaround fix you can run as a script. Make sure that if you use an environment variable to add babel-plugin-istanbul that you set it.

const fs = require("fs");

const babel = require("@babel/core");
const libInstrument = require("istanbul-lib-instrument");

const nycOutPath = "./.nyc_output/out.json"

const nycOut = JSON.parse(fs.readFileSync(nycOutPath));

Object.entries(nycOut).forEach(([filename, coverage]) => {
  if (coverage.hash) {
    // Not a placeholder
    return;
  }

  let code;

  try {
    // Make sure you set any environment errors you need here to build with babel-plugin-istanbul
    ({ code } = babel.transformFileSync(filename));
  } catch (error) {
    return;
  }

  const initialCoverage = libInstrument.readInitialCoverage(code);

  if (initialCoverage && initialCoverage.coverageData) {
    nycOut[filename] = initialCoverage.coverageData;
  }
});

fs.writeFileSync(nycOutPath, JSON.stringify(nycOut));

[daniel-koudouna]

Adding to the answer above, I created a dummy jest test to generate coverage for all my files:

/// jest.config.js

/** @type {import('@jest/types').Config.InitialOptions} */
const config = {
  collectCoverageFrom: ["./src/**/*"],
  collectCoverage: true,
  coverageReporters: ["lcov", "cobertura", "json"],
};

module.exports = config;

/// cypress/.phony.spec.js

test("adds 1 + 1 to equal 2", () => {
  expect(1 + 1).toBe(2);
});

/// package.json
"scripts": {
    ...,
    "cy:baseline": "jest cypress/.phony.spec.js",
}

We're using codecov in our CI, which merges the results automatically, so I upload the resulting file in the coverage directory as a parallel github action to the actual cypress tests. You can also merge them manually by using the subdir option in the jest config file.

I was searching for ways to achieve this without resorting to jest, but I think this way is preferable since it avoids dealing with additional webpack or other configuration. A proper handling of empty files would be preferable to avoid this, however.

[rbong]

You may run into problems merging the Jest and Cypress coverage. It worked in simple tests but I have run into more issues.

Cypress coverage seems to have anonymous function names, and Jest coverage has named function names.

I am not sure it is possible just to drop the Istanbul Babel function onto the end of the Babel config and have it automatically work. I am still researching the exact cause of the issues merging.

[shandav]

I'm also running into this issue, and I've tried searching everywhere for a solution. I'm using Cypress 10 + a monorepo React based application being bundled using webpack + babel (the package with all the tests is a sibling directory to the actual webapp).

My directory structure:

- root/
       - app/
            - src/
            - package.json
            - .babelrc
            - ...
       - cypress-tests/
            - e2e/
            - package.json
            - ...

I've tried both approaches above (using Jest as a workaround to generate empty coverage with accurate line count, and the standalone script posted above, put inside the cypress-tests directory), however, I've been having difficulty getting either of them to work due to configurations (babel configs/plugins, Jest transformers / babel libraries, issues with monorepo configs for all the aforementioned).

Not sure if Cypress supports this natively or not - but it would probably be good if maybe there was an option to fill empty placeholder code coverage objects with accurate statistics, despite the file(s) not being loaded during tests.

[kevingorry]

+1 we are having the exact same problem. @rbong any update on this ?

[rbong]

The standalone script I posted above should work for you if you're using Cypress only.

As far as merging output with Jest goes, I don't have any updates, next steps are to experiment with modifying babel.config.js until Next.JS produces the same Babel output as Jest. Anyone is free to try.