Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate UDP drops #209

Closed
1 task
kamirr opened this issue Sep 14, 2022 · 3 comments
Closed
1 task

Investigate UDP drops #209

kamirr opened this issue Sep 14, 2022 · 3 comments

Comments

@kamirr
Copy link
Contributor

kamirr commented Sep 14, 2022 •
edited
Loading

Todo:

  • calculate/estimate do we overload the UDP buffer, i.e. why do we loose packets on localhost?

Running cargo run --release --example perf -- -a udp://127.0.0.1:7464 load --connections=200,400 --neighbours-weight=1 --neighbours-size=10 --ping-weight=1 --find-node-weight=2 --requests-per-connection=1000

<snip>
[DEBUG ya_relay_client::session] Finding Node info [0xb8d297dca1ecf7727a5690ce33a3864a3284934e], using session d7acf0585e8ec64c74470296393f35be.
[DEBUG ya_relay_client::session] Finding Node info [0x13acb447bc7e22720867974b5e2976013d974d08], using session d12df133a0abf686ca77167466d7d46a.
[DEBUG ya_relay_client::session] Finding Node info [0x18e49954df83929a5c53a0de3fbd7eff189fb33e], using session 4551e2fa8444c7c23a353eed8fb82a81.
[DEBUG ya_relay_client::session] Finding Node info [0x8d520fe0f3184b2bdd2cf7e7b938f10e34dd4517], using session 4534d918218b80c7f3c350ec314159d8.
[DEBUG ya_relay_client::session] Finding Node info [0x92b60603c34c5f9ba3903ecaedc866853ec7a3bf], using session 8d7d1c86bf03fe4fc95a30c416fe0896.
[DEBUG ya_relay_client::session] Finding Node info [0xc81b243e8d1e9d47a38be8bb97b732cbdb95a98f], using session 2eaeeac6c881a3752da3f76dd03bc9aa.
[DEBUG ya_relay_client::session] Finding Node info [0x2eeb0e08477345809817fa2b2cf1f85b540b71ac], using session 86454e2f7d85f6b5d27935be44126479.
[DEBUG ya_relay_client::session] Finding Node info [0xdcb065a8647d2640693651cc43a5fa005d2e8a7f], using session 9727db92509a628d214c4b0e3804d4cb.
[DEBUG ya_relay_client::session] Finding Node info [0x324625162416ee383e7ebb46f14a2cada8a57ec5], using session 03e7204e0d512aa5581c2f17d51e8e4e.
[DEBUG ya_relay_client::session] Finding Node info [0x777497814c717368e5e345001d4858ddb34ab2e9], using session 6eddcbd6c31098e338e67e4efeab146e.
[DEBUG ya_relay_client::session] Finding Node info [0x7cdf32a972b919d54a0cd5cd03f208ccc87d451b], using session a64eac7ac8f34baa97bb55c4fa26f32d.
[DEBUG ya_relay_client::session] Finding Node info [0x295d66192f14ff4b0da0728380924aed5385b62a], using session 8d60398acf6b2239708f7323403dacdd.
[DEBUG ya_relay_client::client] Asking NET relay Server for neighborhood (10).
[DEBUG ya_relay_client::session] Finding Node info [0x13acb447bc7e22720867974b5e2976013d974d08], using session d12df133a0abf686ca77167466d7d46a.
[DEBUG ya_relay_client::session] Finding Node info [0x18e49954df83929a5c53a0de3fbd7eff189fb33e], using session 4551e2fa8444c7c23a353eed8fb82a81.
[DEBUG ya_relay_client::session] Finding Node info [0x8d520fe0f3184b2bdd2cf7e7b938f10e34dd4517], using session 4534d918218b80c7f3c350ec314159d8.
[DEBUG ya_relay_client::session] Finding Node info [0x92b60603c34c5f9ba3903ecaedc866853ec7a3bf], using session 8d7d1c86bf03fe4fc95a30c416fe0896.
[DEBUG ya_relay_client::session] Finding Node info [0xc81b243e8d1e9d47a38be8bb97b732cbdb95a98f], using session 2eaeeac6c881a3752da3f76dd03bc9aa.
[DEBUG ya_relay_client::session] Finding Node info [0x2eeb0e08477345809817fa2b2cf1f85b540b71ac], using session 86454e2f7d85f6b5d27935be44126479.
[DEBUG ya_relay_client::session] Finding Node info [0xdcb065a8647d2640693651cc43a5fa005d2e8a7f], using session 9727db92509a628d214c4b0e3804d4cb.
[DEBUG ya_relay_client::session] Finding Node info [0x324625162416ee383e7ebb46f14a2cada8a57ec5], using session 03e7204e0d512aa5581c2f17d51e8e4e.
[DEBUG ya_relay_client::session] Finding Node info [0x777497814c717368e5e345001d4858ddb34ab2e9], using session 6eddcbd6c31098e338e67e4efeab146e.
[DEBUG ya_relay_client::session] Finding Node info [0x7cdf32a972b919d54a0cd5cd03f208ccc87d451b], using session a64eac7ac8f34baa97bb55c4fa26f32d.
[DEBUG ya_relay_client::session] Finding Node info [0x295d66192f14ff4b0da0728380924aed5385b62a], using session 8d60398acf6b2239708f7323403dacdd.
[DEBUG ya_relay_client::session] Finding Node info [0x13acb447bc7e22720867974b5e2976013d974d08], using session d12df133a0abf686ca77167466d7d46a.
[DEBUG ya_relay_client::client] Asking NET relay Server for neighborhood (10).
[DEBUG ya_relay_client::session] Finding Node info [0x92b60603c34c5f9ba3903ecaedc866853ec7a3bf], using session 8d7d1c86bf03fe4fc95a30c416fe0896.
[DEBUG ya_relay_client::session] Finding Node info [0xdcb065a8647d2640693651cc43a5fa005d2e8a7f], using session 9727db92509a628d214c4b0e3804d4cb.
[DEBUG ya_relay_client::session] Finding Node info [0x777497814c717368e5e345001d4858ddb34ab2e9], using session 6eddcbd6c31098e338e67e4efeab146e.
[DEBUG ya_relay_client::session] Finding Node info [0x295d66192f14ff4b0da0728380924aed5385b62a], using session 8d60398acf6b2239708f7323403dacdd.
[DEBUG ya_relay_client::session] Finding Node info [0x13acb447bc7e22720867974b5e2976013d974d08], using session d12df133a0abf686ca77167466d7d46a.
[DEBUG ya_relay_client::session] Finding Node info [0x92b60603c34c5f9ba3903ecaedc866853ec7a3bf], using session 8d7d1c86bf03fe4fc95a30c416fe0896.
[INFO  perf::load_test] Couldn't complete with rate limit None
    error: Request timed out after 3000 ms
<snip>
@kamirr kamirr mentioned this issue Sep 14, 2022
Closed
9 tasks
@kamirr kamirr self-assigned this Sep 15, 2022
@nieznanysprawiciel
Copy link
Contributor

May be related to: golemfactory/yagna#2053

@nieznanysprawiciel
Copy link
Contributor

and: golemfactory/yagna#2148

@kamirr
Copy link
Contributor Author

kamirr commented Oct 3, 2022

From #191 (comment):

Average packet size for this setup during loading in 27.8 bytes, which shouldn't overflow a 32MiB buffers I've set via sysctl -w net.core.{r,w}mem_{max,default}=33554432.

@kamirr kamirr changed the title perf reports timeouts locally for ~400+ connections during load tests Investigate UDP drops Oct 6, 2022
@kamirr kamirr assigned kamirr and unassigned kamirr Oct 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@prekucki @kamirr @nieznanysprawiciel