network/litep2p: Slower sync time compared to libp2p #4986
Comments
|
This may free up some litep2p resources: Based on comment litep2p handles considerably more connections. Might also be the cause of a lower networking bytes sent:
|
|
After implementing connection limits in litep2p:
Based on origin/master at After implementing connection limits, we have effectively improved the sync time with around 50%, from 16m 24s to 10m 48s. We are also seeing a regression for libp2p, related to: In the current state of things, litep2p is twice is fast as libp2p. Even compared to the previous better libp2p state, we are still 30% faster in syncing. It is clear we need more performance testing, both for:
|
|
Why are the numbers for |
|
It could be some noise for sure. I've synced to kusama both nodes, believe I've used the same params here We could double check using more data points, and probably improve https://github.com/lexnv/sub-triage-logs/ to do this for us. |
1 task
github-merge-queue bot
pushed a commit
that referenced
this issue
Sep 10, 2024
This release introduces several new features, improvements, and fixes to the litep2p library. Key updates include enhanced error handling, configurable connection limits, and a new API for managing public addresses. For a detailed set of changes, see [litep2p changelog](https://github.com/paritytech/litep2p/blob/master/CHANGELOG.md#070---2024-09-05). This PR makes use of: - connection limits to optimize network throughput - better errors that are propagated to substrate metrics - public addresses API to report healthy addresses to the Identify protocol ### Warp sync time improvement Measuring warp sync time is a bit inaccurate since the network is not deterministic and we might end up using faster peers (peers with more resources to handle our requests). However, I did not see warp sync times of 16 minutes, instead, they are roughly stabilized between 8 and 10 minutes. For measuring warp-sync time, I've used [sub-trige-logs](https://github.com/lexnv/sub-triage-logs/?tab=readme-ov-file#warp-time) ### Litep2p Phase | Time -|- Warp | 426.999999919s State | 99.000000555s Total | 526.000000474s ### Libp2p Phase | Time -|- Warp | 731.999999837s State | 71.000000882s Total | 803.000000719s Closes: #4986 ### Low peer count After exposing the `litep2p::public_addresses` interface, we can report to litep2p confirmed external addresses. This should mitigate or at least improve: #4925. Will keep the issue around to confirm this. ### Improved metrics We are one step closer to exposing similar metrics as libp2p: #4681. cc @paritytech/networking ### Next Steps - [x] Use public address interface to confirm addresses to identify protocol --------- Signed-off-by: Alexandru Vasile <[email protected]>
mordamax
pushed a commit
to paritytech-stg/polkadot-sdk
that referenced
this issue
Sep 11, 2024
This release introduces several new features, improvements, and fixes to the litep2p library. Key updates include enhanced error handling, configurable connection limits, and a new API for managing public addresses. For a detailed set of changes, see [litep2p changelog](https://github.com/paritytech/litep2p/blob/master/CHANGELOG.md#070---2024-09-05). This PR makes use of: - connection limits to optimize network throughput - better errors that are propagated to substrate metrics - public addresses API to report healthy addresses to the Identify protocol ### Warp sync time improvement Measuring warp sync time is a bit inaccurate since the network is not deterministic and we might end up using faster peers (peers with more resources to handle our requests). However, I did not see warp sync times of 16 minutes, instead, they are roughly stabilized between 8 and 10 minutes. For measuring warp-sync time, I've used [sub-trige-logs](https://github.com/lexnv/sub-triage-logs/?tab=readme-ov-file#warp-time) ### Litep2p Phase | Time -|- Warp | 426.999999919s State | 99.000000555s Total | 526.000000474s ### Libp2p Phase | Time -|- Warp | 731.999999837s State | 71.000000882s Total | 803.000000719s Closes: paritytech#4986 ### Low peer count After exposing the `litep2p::public_addresses` interface, we can report to litep2p confirmed external addresses. This should mitigate or at least improve: paritytech#4925. Will keep the issue around to confirm this. ### Improved metrics We are one step closer to exposing similar metrics as libp2p: paritytech#4681. cc @paritytech/networking ### Next Steps - [x] Use public address interface to confirm addresses to identify protocol --------- Signed-off-by: Alexandru Vasile <[email protected]>
lexnv
added a commit
that referenced
this issue
Nov 15, 2024
This release introduces several new features, improvements, and fixes to the litep2p library. Key updates include enhanced error handling, configurable connection limits, and a new API for managing public addresses. For a detailed set of changes, see [litep2p changelog](https://github.com/paritytech/litep2p/blob/master/CHANGELOG.md#070---2024-09-05). This PR makes use of: - connection limits to optimize network throughput - better errors that are propagated to substrate metrics - public addresses API to report healthy addresses to the Identify protocol Measuring warp sync time is a bit inaccurate since the network is not deterministic and we might end up using faster peers (peers with more resources to handle our requests). However, I did not see warp sync times of 16 minutes, instead, they are roughly stabilized between 8 and 10 minutes. For measuring warp-sync time, I've used [sub-trige-logs](https://github.com/lexnv/sub-triage-logs/?tab=readme-ov-file#warp-time) Phase | Time -|- Warp | 426.999999919s State | 99.000000555s Total | 526.000000474s Phase | Time -|- Warp | 731.999999837s State | 71.000000882s Total | 803.000000719s Closes: #4986 After exposing the `litep2p::public_addresses` interface, we can report to litep2p confirmed external addresses. This should mitigate or at least improve: #4925. Will keep the issue around to confirm this. We are one step closer to exposing similar metrics as libp2p: #4681. cc @paritytech/networking - [x] Use public address interface to confirm addresses to identify protocol --------- Signed-off-by: Alexandru Vasile <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
litep2p
Libp2p
Data points
litep2p warp sync: 14:06:42.414 - 14:16:34.688 = 9m 52s
libp2p warp sync: 14:07:27.569 - 14:15:54.968 = 8m 27s
litep2p state sync: 14:16:34.688 - 14:23:06.679 = 6m 32s
libp2p state sync: 14:15:54.968 - 14:21:24.293 = 5m 30s
The text was updated successfully, but these errors were encountered: