azure win10 vs2019 machines have been timing out in debugger tests #3014
Comments
|
The daily master build on the commit prior to the doc commit was also green on those machines: nodejs/node@5e57d24 |
|
I do not know how to log into those machines, but I suspect that there might be some stale Node.js processes running there that block the port 9229. That could cause these tests to timeout. |
|
I couldn't see any stale Node.js processes still running on test-azure_msft-win10_vs2019-x64-1 or test-azure_msft-win10_vs2019-x64-2 but I've just scheduled a reboot of both machines. |
|
The failures still showed up with nodejs/node#44342, so there must be something else that is causing the time out. Also I noticed that this only happes with test-azure_msft-win10_vs2019-x64-1 and -2, the tests don't fail on -3. |
|
Looking at the build history, this seems to happen more with test-azure_msft-win10_vs2019-x64-1, and somewhat less frequently with -2. -3 and -4 seem to be immune to that. |
|
FWIW -1 and -2 did get rebooted (#3014 (comment)) so whatever the problem is it has persisted, or recreated itself, across the reboot. |
|
Another observation: it seems sometimes these failures are accompanied by a few random failures from other tests that involve timeouts, e.g. sequential/test-heap-prof, parallel/test-child-process-exec-timeout, see nodejs/reliability#354 |
|
It's been failing almost every time, making it difficult to land PRs:
|
1 task
nodejs-github-bot
pushed a commit
to nodejs/node
that referenced
this issue
Aug 25, 2022
sequential/test-child-process-execsync and parallel/test-child-process-spawnsync-timeout are both flaky on azure Windows machines, where it may take longer for Node.js to launch and receive output from child processes. These tests work by spawning a child processes that is supposed to sleep for a long time, but the option is configured so that Node.js would terminate them early when a shorter timeout is reached. Then the tests assert that the time taken for the whole thing is shorter than the specified sleep time (meaning the process don't actually get to sleep for that long). To make the tests less brittle on azure Windows, this patch raises the sleep times in those tests on Windows platform, so that the overhead can be taken into account there. PR-URL: #44375 Refs: nodejs/build#3014 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Nitzan Uziely <[email protected]>
sidwebworks
pushed a commit
to sidwebworks/node
that referenced
this issue
Aug 26, 2022
sequential/test-child-process-execsync and parallel/test-child-process-spawnsync-timeout are both flaky on azure Windows machines, where it may take longer for Node.js to launch and receive output from child processes. These tests work by spawning a child processes that is supposed to sleep for a long time, but the option is configured so that Node.js would terminate them early when a shorter timeout is reached. Then the tests assert that the time taken for the whole thing is shorter than the specified sleep time (meaning the process don't actually get to sleep for that long). To make the tests less brittle on azure Windows, this patch raises the sleep times in those tests on Windows platform, so that the overhead can be taken into account there. PR-URL: nodejs#44375 Refs: nodejs/build#3014 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Nitzan Uziely <[email protected]>
|
Looks like the flakes are now gone, the only recent repros in the reliability issues came from resumed builds that weren't rebased. I'll close this issue. |
|
Thank you for working on this @joyeecheung :) |
joyeecheung
added a commit
to joyeecheung/node
that referenced
this issue
Sep 1, 2022
sequential/test-child-process-execsync and parallel/test-child-process-spawnsync-timeout are both flaky on azure Windows machines, where it may take longer for Node.js to launch and receive output from child processes. These tests work by spawning a child processes that is supposed to sleep for a long time, but the option is configured so that Node.js would terminate them early when a shorter timeout is reached. Then the tests assert that the time taken for the whole thing is shorter than the specified sleep time (meaning the process don't actually get to sleep for that long). To make the tests less brittle on azure Windows, this patch raises the sleep times in those tests on Windows platform, so that the overhead can be taken into account there. PR-URL: nodejs#44375 Refs: nodejs/build#3014 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Nitzan Uziely <[email protected]>
joyeecheung
added a commit
to joyeecheung/node
that referenced
this issue
Sep 1, 2022
sequential/test-child-process-execsync and parallel/test-child-process-spawnsync-timeout are both flaky on azure Windows machines, where it may take longer for Node.js to launch and receive output from child processes. These tests work by spawning a child processes that is supposed to sleep for a long time, but the option is configured so that Node.js would terminate them early when a shorter timeout is reached. Then the tests assert that the time taken for the whole thing is shorter than the specified sleep time (meaning the process don't actually get to sleep for that long). To make the tests less brittle on azure Windows, this patch raises the sleep times in those tests on Windows platform, so that the overhead can be taken into account there. PR-URL: nodejs#44375 Refs: nodejs/build#3014 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Nitzan Uziely <[email protected]>
RafaelGSS
pushed a commit
to nodejs/node
that referenced
this issue
Sep 5, 2022
sequential/test-child-process-execsync and parallel/test-child-process-spawnsync-timeout are both flaky on azure Windows machines, where it may take longer for Node.js to launch and receive output from child processes. These tests work by spawning a child processes that is supposed to sleep for a long time, but the option is configured so that Node.js would terminate them early when a shorter timeout is reached. Then the tests assert that the time taken for the whole thing is shorter than the specified sleep time (meaning the process don't actually get to sleep for that long). To make the tests less brittle on azure Windows, this patch raises the sleep times in those tests on Windows platform, so that the overhead can be taken into account there. PR-URL: #44375 Refs: nodejs/build#3014 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Nitzan Uziely <[email protected]>
joyeecheung
added a commit
to joyeecheung/node
that referenced
this issue
Sep 6, 2022
sequential/test-child-process-execsync and parallel/test-child-process-spawnsync-timeout are both flaky on azure Windows machines, where it may take longer for Node.js to launch and receive output from child processes. These tests work by spawning a child processes that is supposed to sleep for a long time, but the option is configured so that Node.js would terminate them early when a shorter timeout is reached. Then the tests assert that the time taken for the whole thing is shorter than the specified sleep time (meaning the process don't actually get to sleep for that long). To make the tests less brittle on azure Windows, this patch raises the sleep times in those tests on Windows platform, so that the overhead can be taken into account there. PR-URL: nodejs#44375 Refs: nodejs/build#3014 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Nitzan Uziely <[email protected]>
Fyko
pushed a commit
to Fyko/node
that referenced
this issue
Sep 15, 2022
sequential/test-child-process-execsync and parallel/test-child-process-spawnsync-timeout are both flaky on azure Windows machines, where it may take longer for Node.js to launch and receive output from child processes. These tests work by spawning a child processes that is supposed to sleep for a long time, but the option is configured so that Node.js would terminate them early when a shorter timeout is reached. Then the tests assert that the time taken for the whole thing is shorter than the specified sleep time (meaning the process don't actually get to sleep for that long). To make the tests less brittle on azure Windows, this patch raises the sleep times in those tests on Windows platform, so that the overhead can be taken into account there. PR-URL: nodejs#44375 Refs: nodejs/build#3014 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Nitzan Uziely <[email protected]>
targos
pushed a commit
to nodejs/node
that referenced
this issue
Sep 16, 2022
sequential/test-child-process-execsync and parallel/test-child-process-spawnsync-timeout are both flaky on azure Windows machines, where it may take longer for Node.js to launch and receive output from child processes. These tests work by spawning a child processes that is supposed to sleep for a long time, but the option is configured so that Node.js would terminate them early when a shorter timeout is reached. Then the tests assert that the time taken for the whole thing is shorter than the specified sleep time (meaning the process don't actually get to sleep for that long). To make the tests less brittle on azure Windows, this patch raises the sleep times in those tests on Windows platform, so that the overhead can be taken into account there. PR-URL: #44375 Refs: nodejs/build#3014 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Nitzan Uziely <[email protected]>
juanarbol
pushed a commit
to nodejs/node
that referenced
this issue
Oct 10, 2022
sequential/test-child-process-execsync and parallel/test-child-process-spawnsync-timeout are both flaky on azure Windows machines, where it may take longer for Node.js to launch and receive output from child processes. These tests work by spawning a child processes that is supposed to sleep for a long time, but the option is configured so that Node.js would terminate them early when a shorter timeout is reached. Then the tests assert that the time taken for the whole thing is shorter than the specified sleep time (meaning the process don't actually get to sleep for that long). To make the tests less brittle on azure Windows, this patch raises the sleep times in those tests on Windows platform, so that the overhead can be taken into account there. PR-URL: #44375 Refs: nodejs/build#3014 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Nitzan Uziely <[email protected]>
juanarbol
pushed a commit
to nodejs/node
that referenced
this issue
Oct 11, 2022
sequential/test-child-process-execsync and parallel/test-child-process-spawnsync-timeout are both flaky on azure Windows machines, where it may take longer for Node.js to launch and receive output from child processes. These tests work by spawning a child processes that is supposed to sleep for a long time, but the option is configured so that Node.js would terminate them early when a shorter timeout is reached. Then the tests assert that the time taken for the whole thing is shorter than the specified sleep time (meaning the process don't actually get to sleep for that long). To make the tests less brittle on azure Windows, this patch raises the sleep times in those tests on Windows platform, so that the overhead can be taken into account there. PR-URL: #44375 Refs: nodejs/build#3014 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Nitzan Uziely <[email protected]>
guangwong
pushed a commit
to noslate-project/node
that referenced
this issue
Jan 3, 2023
sequential/test-child-process-execsync and parallel/test-child-process-spawnsync-timeout are both flaky on azure Windows machines, where it may take longer for Node.js to launch and receive output from child processes. These tests work by spawning a child processes that is supposed to sleep for a long time, but the option is configured so that Node.js would terminate them early when a shorter timeout is reached. Then the tests assert that the time taken for the whole thing is shorter than the specified sleep time (meaning the process don't actually get to sleep for that long). To make the tests less brittle on azure Windows, this patch raises the sleep times in those tests on Windows platform, so that the overhead can be taken into account there. PR-URL: nodejs/node#44375 Refs: nodejs/build#3014 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Nitzan Uziely <[email protected]>
guangwong
pushed a commit
to noslate-project/node
that referenced
this issue
Jan 3, 2023
sequential/test-child-process-execsync and parallel/test-child-process-spawnsync-timeout are both flaky on azure Windows machines, where it may take longer for Node.js to launch and receive output from child processes. These tests work by spawning a child processes that is supposed to sleep for a long time, but the option is configured so that Node.js would terminate them early when a shorter timeout is reached. Then the tests assert that the time taken for the whole thing is shorter than the specified sleep time (meaning the process don't actually get to sleep for that long). To make the tests less brittle on azure Windows, this patch raises the sleep times in those tests on Windows platform, so that the overhead can be taken into account there. PR-URL: nodejs/node#44375 Refs: nodejs/build#3014 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Nitzan Uziely <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
See nodejs/reliability#354 for reference, it seems the earliest CI run that exhibited this was https://ci.nodejs.org/job/node-test-binary-windows-js-suites/16283/, which is for a PR that hasn't landed rebased on top of a doc commit nodejs/node@dcc9589 so it might be infra issues.
The text was updated successfully, but these errors were encountered: