feature-request: first-class test-coverage support

[bcoe]

I find code-coverage is an extremely valuable tool for ensuring that a codebase is well tested. It's also a great way to figure out performance bottlenecks.

newer versions of v8 have built in code coverage, through the inspector API:

https://v8project.blogspot.com/2017/12/javascript-code-coverage.html

It would be really nice if we could figure out a way to expose this functionality out of the gate in deno, with source-map support; something I've been trying to do retroactively in Node.js.

[bcoe]

CC: @schuay, @TimothyGu, @addaleax

[kt3k]

other references:

• Code Coverage directly from V8
• c8
• Rethinking JavaScript Test Coverage

From nodejs/node

• coverage: expose native V8 coverage coverage: expose native V8 coverage nodejs/node#22527
• coverage: make coverage work for Node.js coverage: make coverage work for Node.js nodejs/node#23941

[kitsonk]

This is really interesting, and bothered me how we would try to accomplish this. Great reminder of the capabilities of V8.

We would need to take the coverage information and for transpiled modules, remap via the source maps. We have the logic in Rust to deal with the source maps and so it should be workable. I assume we would want to output a coverage.json ala Istanbul.js and then we could benefit from the nyc tooling to transform to other formats of coverage analysis.

[bcoe]

@kitsonk see https://github.com/bcoe/c8, which can turn raw v8 output into Istanbul reports.

[kitsonk]

Thinking about this after a while. We need a few things:

• Access to the inspector: APIs required to implement inspector rusty_v8#205
• Use/port: v8-to-istanbul
• Integration into the testing framework
• Ensure that reporters and the like for Istanbul.js work under Deno.

[kt3k]

FYI Rome seems doing something similar on their own. https://github.com/facebookexperimental/rome/blob/master/packages/%40romejs/v8/CoverageCollector.ts

[bartlomieju]

I tentatively started working on this issue - managed to get coverage data when running deno test.

First #4780 must be fixed; secondly data from V8 must be parsed into human readable form.

I like approach that rome took which is completely self-contained solution with text report to console. Parsing coverage data to Rust structs means JSON output is pretty much free (thanks serde); so I suggest starting with our own solution and thinking about integration with other tools after we have working MVP.

[kitsonk]

data from V8 must be parsed into human readable form

Istanbul/nyc would have the most wide support. It is the defacto standard, and it would mean not having to reinvent the wheel. Output of a istanbul format would integrate into a lot of existing toolchains. Even if we incorporate an integrated output (again I would just leverage nyc's logic) we should still output the raw JSON. Why invent our "own" solution?

[Alhadis]

Partly unrelated, but people in this thread might be interested to know d8 has an undocumented --lcov= switch for generating coverage data in LCOV tracefile format:

λ d8 --lcov=coverage.info Ankara/coverage-test.mjs
λ cat coverage.info
SF:/Users/john/Desktop/Ankara/coverage-test.mjs
FN:1,
FNDA:1,
DA:1,1
DA:2,1
end_of_record
SF:/Users/john/Desktop/Ankara/foo.mjs
FN:1,
FNDA:1,
FN:1,foo
FNDA:1,foo
DA:1,1
DA:2,1
DA:3,1
⋮

This is a feature of V8's debugging shell (and not the engine itself), though it probably wouldn't be difficult to integrate into the runtime (v8/src/d8/d8.cc:3220 can help you get started).

In any case, this isn't documented in --help output anywhere, so I figure it might be of interest to you folks. :)

[kt3k]

A recent attempt by Bartek https://github.com/denoland/deno/pull/4802/files

[aricart]

@kitsonk wondering if there are any updates on coverage functionality exposed from Deno? Thanks!

[kitsonk]

No. There are big reactors in the internals at the moment which make this something that would need to come afterwards.

[aricart]

Thanks @kitsonk anxiously awaiting...