Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[cli] Increase REST endpoint timeout to 30s #4628

Merged
merged 1 commit into from
Sep 29, 2022

Conversation

igor-aptos
Copy link
Contributor

@igor-aptos igor-aptos commented Sep 29, 2022

We saw bunch of timeouts with 10s timeout during loadtests, and then the experience is confusing - transaction actually goes through, but that is not clear.

We used 30s timeout for txn emitter, and that worked well.

together with #4456, CLI experience should be better now

Description

Test Plan


This change is Reviewable

We saw bunch of timeouts with 10s timeout during loadtests,
and then the experience is confusing - transcation actually goes through,
but that is not clear
@banool
Copy link
Contributor

banool commented Sep 29, 2022

I think this is long enough that many users will think something went wrong / the request hung. Could we instead decrease the expiration time of the requests the CLI sends and then decrease this timeout accordingly?

Alternatively it would be okay to do this if the CLI made it clear that it was still waiting for the request, e.g. with some logging to stderr.

@banool
Copy link
Contributor

banool commented Sep 29, 2022

Okay we spoke offline, I was misunderstanding how time is tracked on chain (blockchain time vs absolute time, which can differ under high load). This makes sense then.


/// Connection timeout in seconds, used for the REST endpoint of the fullnode
#[clap(long, default_value = "30")]
pub connection_timeout_s: u64,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you clarify where is this set to 10s default today?

@@ -823,7 +831,10 @@ impl RestOptions {
}

pub fn client(&self, profile: &str) -> CliTypedResult<Client> {
Ok(Client::new(self.url(profile)?))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here, implementation is:

pub fn new(base_url: Url) -> Self {
    Self::new_with_timeout(base_url, Duration::from_secs(10))
}

@igor-aptos igor-aptos enabled auto-merge (squash) September 29, 2022 20:51
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions
Copy link
Contributor

✅ Forge suite land_blocking success on 4aa063f3a68448c4e5c5a4007d70d49036ce5842

performance benchmark with full nodes : 7422 TPS, 5341 ms latency, 9900 ms p99 latency,(!) expired 40 out of 3169560 txns
Test Ok

@github-actions
Copy link
Contributor

✅ Forge suite compat success on 843b204dce971d98449b82624f4f684c7a18b991 ==> 4aa063f3a68448c4e5c5a4007d70d49036ce5842

Compatibility test results for 843b204dce971d98449b82624f4f684c7a18b991 ==> 4aa063f3a68448c4e5c5a4007d70d49036ce5842 (PR)
1. Check liveness of validators at old version: 843b204dce971d98449b82624f4f684c7a18b991
compatibility::simple-validator-upgrade::liveness-check : 7863 TPS, 4681 ms latency, 7300 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 4aa063f3a68448c4e5c5a4007d70d49036ce5842
compatibility::simple-validator-upgrade::single-validator-upgrade : 5745 TPS, 6524 ms latency, 8600 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 4aa063f3a68448c4e5c5a4007d70d49036ce5842
compatibility::simple-validator-upgrade::half-validator-upgrade : 4998 TPS, 8457 ms latency, 13400 ms p99 latency,no expired txns
4. upgrading second batch to new version: 4aa063f3a68448c4e5c5a4007d70d49036ce5842
compatibility::simple-validator-upgrade::rest-validator-upgrade : 7850 TPS, 4651 ms latency, 8300 ms p99 latency,no expired txns
5. check swarm health
Compatibility test for 843b204dce971d98449b82624f4f684c7a18b991 ==> 4aa063f3a68448c4e5c5a4007d70d49036ce5842 passed
Test Ok

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants