Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[State sync] Add client-side of new state syncing mode. #5866

Merged
merged 1 commit into from
Dec 17, 2022
Merged

Conversation

JoshLind
Copy link
Contributor

@JoshLind JoshLind commented Dec 12, 2022

Note: Over 1k lines of this PR are just new tests. The rest is boring boilerplate for supporting client-side data requests and a new syncing mode.

Description

This PR adds the client-side implementation of the new semi-intelligent syncing mode that switches between output application and transaction execution depending on whichever is faster. To achieve this, we:

  1. Add a new set of state sync modes to fetch transactions or outputs.
  2. Add all the boilerplate for the new state sync modes to fetch the correct data from the server (e.g., via data streams).
  3. Set the maximum transaction output chunk size to be 1k (from 2k). Based on data in mainnet, this is more optimal as it results in fewer pre-fetcher misses.
  4. Add a new fallback mechanism to the semi-intelligent state syncing modes. Specifically, if state sync tries to execute transactions that are no longer compatible, it switches to output syncing for 2 minutes to get past the problematic transactions. This is to the handle the worst case and should hopefully only ever happen in rare circumstances.
  5. Adds a bunch of new tests around the driver (i.e., bootstrapper and continuous syncer), data streaming service and aptos data client. Smoke tests are also added.

Test Plan

New unit tests and smoke tests!

@JoshLind JoshLind added the CICD:run-e2e-tests when this label is present github actions will run all land-blocking e2e tests from the PR label Dec 12, 2022
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions
Copy link
Contributor

✅ Forge suite land_blocking success on 8c331dac1dd183da96e90145a76945a200b9db0b

performance benchmark with full nodes : 6131 TPS, 6479 ms latency, 9900 ms p99 latency,(!) expired 280 out of 2618400 txns
Test Ok

@github-actions
Copy link
Contributor

✅ Forge suite compat success on testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 8c331dac1dd183da96e90145a76945a200b9db0b

Compatibility test results for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 8c331dac1dd183da96e90145a76945a200b9db0b (PR)
1. Check liveness of validators at old version: testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 7178 TPS, 5355 ms latency, 7600 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 8c331dac1dd183da96e90145a76945a200b9db0b
compatibility::simple-validator-upgrade::single-validator-upgrade : 4423 TPS, 9392 ms latency, 11700 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 8c331dac1dd183da96e90145a76945a200b9db0b
compatibility::simple-validator-upgrade::half-validator-upgrade : 4433 TPS, 8997 ms latency, 12200 ms p99 latency,no expired txns
4. upgrading second batch to new version: 8c331dac1dd183da96e90145a76945a200b9db0b
compatibility::simple-validator-upgrade::rest-validator-upgrade : 6495 TPS, 6050 ms latency, 10100 ms p99 latency,no expired txns
5. check swarm health
Compatibility test for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 8c331dac1dd183da96e90145a76945a200b9db0b passed
Test Ok

@JoshLind JoshLind merged commit 1e9c313 into main Dec 17, 2022
@JoshLind JoshLind deleted the ss_new_mode_5 branch December 17, 2022 07:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CICD:run-e2e-tests when this label is present github actions will run all land-blocking e2e tests from the PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants