Stryker returns only timeouts (mostly when using Jest) #613
Comments
|
Thanks for reporting this! I have three questions so far:
|
|
|
I have the same issue (every mutant is TimedOut) but I don't have any errors when I run it with trace. My config: My info level output: Any trace/debug output just looks like this: Results are a score of 100%, as every mutation times out. I would really like to use this library, so any help is much appreciated 😄 |
|
@ronderksen @automeow could you share your code in any way? If not, could you create a small repo that reproduces the problem and publish that somewhere? That would help us a lot debugging this issue. Thanks in advance! |
|
FYI I have now fixed this by setting the timeoutMs to 60000. My tests take 26 seconds to run so maybe this was the issue (5000ms not enough)? This might be different to @ronderksen's issue as I had no errors. New output: Hope this can help others Cheers! |
|
Glad to hear it @automeow. 26 seconds is a lot of quite a lot of time :)
Well.. the timeout used by stryker is not 5000ms. Rather, it is the result of this calculation:
Here the @Archcry could there be a problem with the |
|
@nicojs Could it be that jest parallelises test runs? Given the initial test run will be able to use all the cores, but subsequent runs can only use 1, the subsequent runs should take far longer? EDIT: running jest with |
We use the runInBand option to mitigate that problem. That should force the maxWorkers to 1 by default, but apparently not? An other issue might be that our formula was originally designed for pretty fasts tests, in that the 5 second offset is already pretty long. But on a max time of 26 second, 5 seconds is pretty much an after thought. Combine that with the max concurrent test runners and you can get a lot of timeouts potentially. @automeow could you slice the maxConcurrentTestRunners in half and see if that solves the problem for you? (leave the |
|
@nicojs I have tried with 3 concurrent runners and 1, but I have the same issue: |
|
@automeow thanks for your reply. For now, you can increase the |
RuninbandAlias: -i. Run all tests serially in the current process, rather than creating a worker pool of child processes that run tests. This can be useful for debugging. (https://facebook.github.io/jest/docs/en/cli.html#runinband). It should only use one worker when using runinband.
The Jest runner now returns the duration of each individual test as seen here. |
|
Ok, so we're missing the overhead of the start and end of a test run. Right now, we don't really have a way of factoring in that overhead into the calculation. Our assumption is that the overhead should be caught with the timeoutMs. @simondel do you think we should change our calculation? |
|
@nicojs I don't think we should change our calculation just yet. We don't change the Jest config in any way right now I recall it correctly. So if someone turns on code coverage using Jest, it will always run with code coverage on, making it slow on every run and increasing the chance of having this issue. @automeow @ronderksen Could you two create reproduction repositories for the issue to help us figure out if we could perhaps speed up the Jest runner? Perhaps there's also something else that we can do. |
|
👍 for similar issue. will follow the thread 💯 I get timeout results when the report finishes, but I don't get any timeouts on console when I am running in jest config: module.exports = {
rootDir: './',
verbose: true,
collectCoverage: true,
testEnvironment: 'node',
collectCoverageFrom: [
'**/*.js',
'!build/**',
'!coverage/**',
'!jest-config.js',
'!rollup.config.js',
'!.styker-tmp/**',
'!reports/**'
],
coverageDirectory: '<rootDir>/coverage',
testResultsProcessor: './node_modules/jest-junit'
};stryker config module.exports = function (config) {
config.set({
files: [
{
pattern: '**/!(*.test).js',
mutated: true,
included: false
},
{
pattern: '**/*.test.js',
mutated: false,
included: true
},
'!node_modules/**/*.js',
'node_modules/jest-junit/**/*',
'!node_modules/jest-junit/node_modules/**/*',
'!coverage/**/*',
'!build/**/*',
'!reports/**/*',
],
testRunner: 'jest',
mutator: 'javascript',
transpilers: ['babel'],
reporter: ['html', 'clear-text', 'progress'],
coverageAnalysis: 'off',
babelrcFile: '.babelrc',
jest: {
config: require('./jest-config.js')
},
logLevel: "all"
});
};.babelrc {
"presets": [
[
"env",
{
"modules": false,
"targets": {
"node": "8"
}
}
]
],
"plugins": ["transform-object-rest-spread"],
"env": {
"test": {
"presets": [
[
"env",
{
"modules": "commonjs",
"targets": {
"node": "8"
}
}
]
]
}
}
} |
|
@lunafi-dorian Thanks for posting your setup! How long does a regular test run take? (With coverage and without coverage)
Is the jest-junit node_modules folder in your Stryker config because of the |
|
14 test suites and 107 tests:
For stryker (Found 39 of 61 file(s) to be mutated and 914 mutations with 3 test runners, just using dots and no loglevel. Intitial test run finds all tests in 15 seconds): Yes the Note: I have tried increasing timeoutMS and it doesn't help. Hope this information helps! |
|
Awesome, so your regular test run takes 10 seconds. Since Stryker generates 900 mutants and runs with 3 test runners, we can say that each test runner handles 300 mutants. If all test runners can run without slowing down your system, stryker would be done after 50 minutes (300 mutants * 10 seconds / 60). If each test run is 3 seconds (coverage and verbose off), stryker would be done after about 15 minutes (300 mutants * 3 seconds / 60). If stryker is done on your system in 10 and 40 minutes, that's not too bad given this use case. What is the final report by stryker? How many timeouts do you get? |
|
@simondel - only 1 did not timeout. All the rest did. I thought I would add that I do get a few of the |
|
The warning If anyone can provide a small repo which reproduces the problem we would be very grateful |
|
@nicojs I'm facing the same issue when running Stryker on Windows. You can to reproduce this using this repo: https://github.com/khujo/stryker-on-mac. I'm having another issue on my MacBook (hence the name of the repo), see #291. Setting timeoutMs to 60000 solves the issue for me. |
|
I've got an idea. During initial test run we can calculate the overhead the test runner took ( This can be solved by stryker and should not be implemented in test runners. I've discussed this with @Archcry and @simondel and we're taking this into sprint. |
nicojs
changed the title
ghost
assigned 
Take test runner overhead into account when calculating the timeout of a single mutant. The formula is now: ``` timeoutForTestRunMs = netTimeMs * timeoutFactor + timeoutMs + overheadMs ``` Check readme for more information. Fixes #613
|
What's the current situation with this as we are running 2.0.2 and still getting this issue. |
|
Would also love an update - I am running into the same issue |
|
I'd also love an update. Running into the same issue 😢 |
|
@jimmyandrade I solved it on my side - I think the issue was I wasn't using the "smart" jest runner but just running jest directly as a command, which caused all kind of issues. |
|
@andreadellacorte thanks for your answer. Sorry for the delay on answering it. Because Stryker 3.0 was released (#1983 (comment)), I updated my project dependencies and now timeouts are gone 🎉 on https://github.com/multei/openalpr/issues/4 |
|
I think the option in stryker.conf.js was renamed to |
Summary
When I run stryker on Jest tests, I get only timeout results.
Running with logLevel: 'trace' gives this output:
Stryker config
Stryker environment
Your Environment
The text was updated successfully, but these errors were encountered: