Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test-dgram-pingpong flaky on parallel runs #4526

Closed
jbergstroem opened this issue Jan 4, 2016 · 15 comments
Closed

test-dgram-pingpong flaky on parallel runs #4526

jbergstroem opened this issue Jan 4, 2016 · 15 comments
Labels
arm Issues and PRs related to the ARM platform. dgram Issues and PRs related to the dgram subsystem / UDP. test Issues and PRs related to the tests.

Comments

@jbergstroem
Copy link
Member

It's been very frequent on the ppc machines during -J runs as of late: https://ci.nodejs.org/job/node-test-commit-plinux/nodes=ppcbe-fedora20/ws/test.tap/*view*/

@jbergstroem jbergstroem added the test Issues and PRs related to the tests. label Jan 4, 2016
@mscdex mscdex added the dgram Issues and PRs related to the dgram subsystem / UDP. label Jan 4, 2016
@Trott
Copy link
Member

Trott commented Jan 4, 2016

I doubt this is the problem, but just in case: Is there any special firewalling or throttling on those machines? It looks like this test fires 1500 UDP packets in (let's say) half a second or so.

@jbergstroem
Copy link
Member Author

@Trott: iptables looks squeaky clean and the vm admin dashboard doesn't point to any firewalling.

@Trott
Copy link
Member

Trott commented Jan 5, 2016

At least one other "flaky on PPC when run in parallel" test is a test that is also network intensive. In #4527, you report that test-http-regr-gh-2928 is flaky on PPC. That test checks that the HTTP flood prevention mechanism kicks in.

Maybe there's something the provider is doing that we can't see from the host itself? (I think all this stuff happens on localhost, so it wouldn't be on the subnet or anything like that, I imagine...)

@jbergstroem
Copy link
Member Author

Regarding provider: that's why i checked the dashboard (there's usually ways to access firewalling from there). I'll create a support ticket and ask.

@Trott
Copy link
Member

Trott commented Jan 8, 2016

Maybe this test should just be moved to sequential. Maybe it's just too network (or other resrouce) intensive for some platforms.

@Trott
Copy link
Member

Trott commented Jan 11, 2016

Probable fix: #4615

@Trott
Copy link
Member

Trott commented Jan 14, 2016

I think #4615 fixed this. Please re-open if I'm mistaken.

@Trott Trott closed this as completed Jan 14, 2016
@Trott
Copy link
Member

Trott commented Feb 3, 2016

@Trott Trott reopened this Feb 3, 2016
@Trott Trott changed the title test-dgram-pingpong flakey on parallel runs test-dgram-pingpong flaky on parallel runs Feb 3, 2016
@jbergstroem
Copy link
Member Author

:/

@Trott
Copy link
Member

Trott commented Feb 4, 2016

Trott added a commit to Trott/io.js that referenced this issue Feb 4, 2016
A few tests have started failing on Raspberry Pi devices in CI.
https://ci.nodejs.org/job/node-test-binary-arm/943/

Ref: nodejs#4830
Ref: nodejs#3635
Ref: nodejs#4526
@Trott
Copy link
Member

Trott commented Feb 4, 2016

The sudden-seeming spike in this failing on Raspberry Pi suggests either a regression in the code or else a change in CI devices or other CI infrastructure.

@Trott Trott added the arm Issues and PRs related to the ARM platform. label Feb 4, 2016
@Trott
Copy link
Member

Trott commented Feb 4, 2016

All of them failing. :-(

@Trott
Copy link
Member

Trott commented Feb 5, 2016

I added some logging for each dgram message received. It looks like what happens is that a message is dropped from time to time. I interpret this as Not A Bug because there is no guarantee that UDP packets will be received. This test opens three UDP ports and sends 500 messages to them relatively quickly. No problem for a typical host, but the Raspberry Pi devices seem to have trouble from time to time.

So my inclination would be to alter the test to tolerate an occasional missed message.

Thoughts?

/cc @rvagg (who, as best as I can tell, is The Raspberry Pi Person around here to the extent that we have one)

Here's the stress test using the additional logging: https://ci.nodejs.org/job/node-stress-single-test/409/nodes=pi2-raspbian-wheezy/console

Trott added a commit to Trott/io.js that referenced this issue Feb 5, 2016
There is no guarantee UDP messages will be received. Accommodate the
occasional dropped message.

Fixes: nodejs#4526
Trott added a commit that referenced this issue Feb 6, 2016
A few tests have started failing on Raspberry Pi devices in CI.
https://ci.nodejs.org/job/node-test-binary-arm/943/

PR-URL: #5082
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: Roman Klauke <[email protected]>
Ref: #4830
Ref: #3635
Ref: #4526
Trott added a commit to Trott/io.js that referenced this issue Feb 6, 2016
There is no guarantee UDP messages will be received. Accommodate the
occasional dropped message.

This is a functionality test, not a performance benchmark. Speed up the
test by sending 5 messages per server rather than 500.

Fixes: nodejs#4526
Trott added a commit to Trott/io.js that referenced this issue Feb 7, 2016
There is no guarantee UDP messages will be received. Accommodate the
occasional dropped message.

This is a functionality test, not a performance benchmark. Speed up the
test by not sending 1500 messages across three ports.

Fixes: nodejs#4526
@jasnell jasnell closed this as completed in 987e9e3 Feb 7, 2016
@rvagg
Copy link
Member

rvagg commented Feb 8, 2016

sorry for the delay, I have nothing useful to add here sorry, let me know if I'm needed to dive deeper here cause I'm only passing a cursory glance over the issue right now

@rvagg rvagg reopened this Feb 8, 2016
@rvagg rvagg closed this as completed Feb 8, 2016
rvagg pushed a commit that referenced this issue Feb 8, 2016
A few tests have started failing on Raspberry Pi devices in CI.
https://ci.nodejs.org/job/node-test-binary-arm/943/

PR-URL: #5082
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: Roman Klauke <[email protected]>
Ref: #4830
Ref: #3635
Ref: #4526
rvagg pushed a commit that referenced this issue Feb 8, 2016
There is no guarantee UDP messages will be received. Accommodate the
occasional dropped message.

This is a functionality test, not a performance benchmark. Speed up the
test by not sending 1500 messages across three ports.

Fixes: #4526
PR-URL: #5125
Reviewed-By: Brian White <[email protected]>
Reviewed-By: Ben Noordhuis <[email protected]>
Reviewed-By: James M Snell <[email protected]>
rvagg pushed a commit that referenced this issue Feb 9, 2016
A few tests have started failing on Raspberry Pi devices in CI.
https://ci.nodejs.org/job/node-test-binary-arm/943/

PR-URL: #5082
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: Roman Klauke <[email protected]>
Ref: #4830
Ref: #3635
Ref: #4526
rvagg pushed a commit that referenced this issue Feb 9, 2016
There is no guarantee UDP messages will be received. Accommodate the
occasional dropped message.

This is a functionality test, not a performance benchmark. Speed up the
test by not sending 1500 messages across three ports.

Fixes: #4526
PR-URL: #5125
Reviewed-By: Brian White <[email protected]>
Reviewed-By: Ben Noordhuis <[email protected]>
Reviewed-By: James M Snell <[email protected]>
MylesBorins pushed a commit that referenced this issue Feb 18, 2016
A few tests have started failing on Raspberry Pi devices in CI.
https://ci.nodejs.org/job/node-test-binary-arm/943/

PR-URL: #5082
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: Roman Klauke <[email protected]>
Ref: #4830
Ref: #3635
Ref: #4526
MylesBorins pushed a commit that referenced this issue Feb 18, 2016
There is no guarantee UDP messages will be received. Accommodate the
occasional dropped message.

This is a functionality test, not a performance benchmark. Speed up the
test by not sending 1500 messages across three ports.

Fixes: #4526
PR-URL: #5125
Reviewed-By: Brian White <[email protected]>
Reviewed-By: Ben Noordhuis <[email protected]>
Reviewed-By: James M Snell <[email protected]>
MylesBorins pushed a commit that referenced this issue Feb 18, 2016
A few tests have started failing on Raspberry Pi devices in CI.
https://ci.nodejs.org/job/node-test-binary-arm/943/

PR-URL: #5082
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: Roman Klauke <[email protected]>
Ref: #4830
Ref: #3635
Ref: #4526
MylesBorins pushed a commit that referenced this issue Feb 18, 2016
There is no guarantee UDP messages will be received. Accommodate the
occasional dropped message.

This is a functionality test, not a performance benchmark. Speed up the
test by not sending 1500 messages across three ports.

Fixes: #4526
PR-URL: #5125
Reviewed-By: Brian White <[email protected]>
Reviewed-By: Ben Noordhuis <[email protected]>
Reviewed-By: James M Snell <[email protected]>
MylesBorins pushed a commit that referenced this issue Mar 2, 2016
A few tests have started failing on Raspberry Pi devices in CI.
https://ci.nodejs.org/job/node-test-binary-arm/943/

PR-URL: #5082
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: Roman Klauke <[email protected]>
Ref: #4830
Ref: #3635
Ref: #4526
MylesBorins pushed a commit that referenced this issue Mar 2, 2016
There is no guarantee UDP messages will be received. Accommodate the
occasional dropped message.

This is a functionality test, not a performance benchmark. Speed up the
test by not sending 1500 messages across three ports.

Fixes: #4526
PR-URL: #5125
Reviewed-By: Brian White <[email protected]>
Reviewed-By: Ben Noordhuis <[email protected]>
Reviewed-By: James M Snell <[email protected]>
scovetta pushed a commit to scovetta/node that referenced this issue Apr 2, 2016
A few tests have started failing on Raspberry Pi devices in CI.
https://ci.nodejs.org/job/node-test-binary-arm/943/

PR-URL: nodejs#5082
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: Roman Klauke <[email protected]>
Ref: nodejs#4830
Ref: nodejs#3635
Ref: nodejs#4526
scovetta pushed a commit to scovetta/node that referenced this issue Apr 2, 2016
There is no guarantee UDP messages will be received. Accommodate the
occasional dropped message.

This is a functionality test, not a performance benchmark. Speed up the
test by not sending 1500 messages across three ports.

Fixes: nodejs#4526
PR-URL: nodejs#5125
Reviewed-By: Brian White <[email protected]>
Reviewed-By: Ben Noordhuis <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arm Issues and PRs related to the ARM platform. dgram Issues and PRs related to the dgram subsystem / UDP. test Issues and PRs related to the tests.
Projects
None yet
Development

No branches or pull requests

4 participants