test: move heapdump tests to pummel #25181
Conversation
|
pummel tests are not (yet) run in CI, so lite CI seems sufficient. |
|
Hm … just for context, the original motivation for these tests was increasing native code coverage. They do take 20 seconds for me as well, but only when not run in parallel, so things could be worse… Does it make sense to maybe create a directory for tests that should be run locally only on request, and in CI always? We have a lot of tests that would qualify for that, imo, and that might be a good path to speeding up |
The temptation there would be to put all the slow tests in the CI-only directory. My sense is that it would be better to split out unit tests from end-to-end tests. The unit tests will run fast and you can run them very frequently. The end-to-end tests will definitely run on CI but optionally locally as well. (People can choose whether they are that kind of completist or not.) Then we can collect coverage stats for unit tests only and coverage stats for integration tests. The goal would be to get good coverage with both types of tests. If we split out the tests based on speed only, then it's far more subjective. (What is the cut-off and on what type of machine? What of tests that are slow only on, say, Windows?) There's subjectivity with unit-test vs. integration test too, but I would expect it to be less frequent. All that said, as resistant as I've been to the idea (e.g., @BridgeAR proposed something similar at some point, I'm pretty sure, many months ago), separating the slow tests may be a decent first step. I'm just concerned it will also be the last step... |
|
Also: Arguably, we already have a place for slow tests: |
|
Lastly: I don't intend for this to be merged until |
The heapdump tests take a lot more time to run than our other tests in parallel. They are also a bit of an internal test that perhaps does not need to be run on every commit on every platform. This change moves them to the pummel directory where they will be run on a single platform once a day in CI. This shaves more than 20 seconds off `make test` on my laptop, FWIW.
|
The pummel tests will now run nightly in node-daily-master, so removing |
|
Lite CI: https://ci.nodejs.org/job/node-test-pull-request-lite-pipeline/2278/ ✅ Nightly job that includes pummel tests run with these changes: https://ci.nodejs.org/job/node-test-commit-custom-suites/830/ ✅ |
Trott
added
the
author ready
The heapdump tests take a lot more time to run than our other tests in parallel. They are also a bit of an internal test that perhaps does not need to be run on every commit on every platform. This change moves them to the pummel directory where they will be run on a single platform once a day in CI. This shaves more than 20 seconds off `make test` on my laptop, FWIW. PR-URL: nodejs#25181 Reviewed-By: Joyee Cheung <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Matheus Marchini <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: James M Snell <[email protected]>
|
Landed in 87f6804. I'll take a try at creating a |
|
Oh, there already is a test-all... guess I ought to see what that does.... |
The heapdump tests take a lot more time to run than our other tests in parallel. They are also a bit of an internal test that perhaps does not need to be run on every commit on every platform. This change moves them to the pummel directory where they will be run on a single platform once a day in CI. This shaves more than 20 seconds off `make test` on my laptop, FWIW. PR-URL: #25181 Reviewed-By: Joyee Cheung <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Matheus Marchini <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: James M Snell <[email protected]>
|
@Trott C++ coverage on https://coverage.nodejs.org/ dropped a full percent after this landed … do you think you could add |
Yeah, very much something I want done. ( |
OK, I added pummel and benchmark tests. Once #25799 lands, we can change it to run test-all-suites and it will always run all tests no matter where they are under the |
|
(And I kicked off a job to make sure it works. https://ci.nodejs.org/view/All/job/node-test-commit-linux-coverage-daily/21/) |
|
Adding pummel and benchmark tests increased C++ coverage back to 91%. Oddly, JS coverage is down to 92.75% which seems wrong unless some big chunk of largely-untested code landed today, which I don't think happened.... Have to look into that... |
The heapdump tests take a lot more time to run than our other tests in
parallel. They are also a bit of an internal test that perhaps does not
need to be run on every commit on every platform. This change moves them
to the pummel directory where they will be run on a single platform once
a day in CI.
This shaves more than 20 seconds off
make teston my laptop, FWIW.@addaleax @Fishrock123
Checklist
make -j4 test(UNIX), orvcbuild test(Windows) passes