Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add connectivity timeout #6703

Merged
merged 2 commits into from
Nov 27, 2024
Merged

feat: add connectivity timeout #6703

merged 2 commits into from
Nov 27, 2024

Conversation

hansieodendaal
Copy link
Contributor

@hansieodendaal hansieodendaal commented Nov 26, 2024 •
edited
Loading

Description

Added a wallet connectivity timeout so that long-winded dial-peer failures can be managed timeously.

Motivation and Context

On some mobile wallets, the dial_peer command can take up to 6 minutes to return an error or timeout. See the nextnet mobile wallet log below and notice the timestamp difference between Attempting base node peer and Connectivity error: ConnectionFailed:

2024-11-26 09:48:39.078496504 [wallet::connectivity] DEBUG Attempting base node peer '1c8b393640e1cd7d1a86957eb3'... (last attempt Some(895ns))
2024-11-26 09:54:32.664270603 [wallet::connectivity] WARN  Connectivity error: ConnectionFailed: Failed to connect to peer within the maximum number of attempts
2024-11-26 09:54:32.667991713 [wallet::connectivity] DEBUG Attempting base node peer 'a04e78a02df887a26ffa9eadaa'... (last attempt Some(122ns))
2024-11-26 09:58:35.639455170 [wallet::connectivity] WARN  Connectivity error: ConnectionFailed: Failed to connect to peer within the maximum number of attempts
2024-11-26 09:58:35.642903697 [wallet::connectivity] DEBUG Attempting base node peer 'e108eb7fbf44d58199908aa928'... (last attempt Some(244ns))
2024-11-26 10:00:03.975743624 [wallet::connectivity] WARN  Connectivity error: ConnectionFailed: Failed to connect to peer within the maximum number of attempts
2024-11-26 10:00:03.978222425 [wallet::connectivity] DEBUG Attempting base node peer '6ea597117476676d5ddcb18153'... (last attempt Some(163ns))
2024-11-26 10:04:24.017030169 [wallet::connectivity] WARN  Connectivity error: ConnectionFailed: Failed to connect to peer within the maximum number of attempts
2024-11-26 10:04:24.022077451 [wallet::connectivity] DEBUG Attempting base node peer '63718f177e749a0484a7050f8d'... (last attempt Some(366ns))
2024-11-26 10:09:52.935971247 [wallet::connectivity] WARN  Connectivity error: ConnectionFailed: Failed to connect to peer within the maximum number of attempts
2024-11-26 10:09:52.939635269 [wallet::connectivity] DEBUG Attempting base node peer 'db5ab85a84ce79bcd2ea700f13'... (last attempt Some(285ns))
2024-11-26 10:16:01.668541211 [wallet::connectivity] WARN  Connectivity error: ConnectionFailed: Failed to connect to peer within the maximum number of attempts
2024-11-26 10:16:01.672009107 [wallet::connectivity] DEBUG Attempting base node peer '81e258cb5e8419dfdd863282c3'... (last attempt Some(610ns))
2024-11-26 10:20:25.197734699 [wallet::connectivity] WARN  Connectivity error: ConnectionFailed: Failed to connect to peer within the maximum number of attempts
2024-11-26 10:20:25.202187051 [wallet::connectivity] DEBUG Attempting base node peer '6d9043bba0c196ded52a9f79da'... (last attempt Some(244ns))
2024-11-26 10:20:26.038427123 [wallet::connectivity] DEBUG Established peer connection to base node '6d9043bba0c196ded52a9f79da'

This PR, as tested on Windows console wallet, with many dysfunctional peer seeds on purpose

2024-11-26 16:29:06.603475100 [wallet::connectivity] DEBUG Attempting base node peer '1c8b393640e1cd7d1a86957eb3'... (last attempt Some(100ns))
2024-11-26 16:29:06.603488500 [wallet::connectivity] TRACE Attempt dial with client timeout 1s
2024-11-26 16:29:07.615599600 [wallet::connectivity] WARN  Connectivity error: Client cancelled: 'Could not connect to '1c8b393640e1cd7d1a86957eb3' in 1s'
2024-11-26 16:29:07.615740400 [wallet::connectivity] DEBUG Attempting base node peer 'e108eb7fbf44d58199908aa928'... (last attempt Some(100ns))
2024-11-26 16:29:07.615751700 [wallet::connectivity] TRACE Attempt dial with client timeout 1s
2024-11-26 16:29:08.617694200 [wallet::connectivity] WARN  Connectivity error: Client cancelled: 'Could not connect to 'e108eb7fbf44d58199908aa928' in 1s'
2024-11-26 16:29:08.617884800 [wallet::connectivity] DEBUG Attempting base node peer '63718f177e749a0484a7050f8d'... (last attempt Some(100ns))
2024-11-26 16:29:08.617903900 [wallet::connectivity] TRACE Attempt dial with client timeout 1s
2024-11-26 16:29:09.631693800 [wallet::connectivity] WARN  Connectivity error: Client cancelled: 'Could not connect to '63718f177e749a0484a7050f8d' in 1s'
2024-11-26 16:29:09.631830300 [wallet::connectivity] DEBUG Attempting base node peer 'b02156f7ec30db5d661dd9b6e2'... (last attempt Some(0ns))
2024-11-26 16:29:09.631840600 [wallet::connectivity] TRACE Attempt dial with client timeout 1s
2024-11-26 16:29:09.650843600 [wallet::connectivity] WARN  Connectivity error: ConnectionFailed: Noise handshake error: An established connection was aborted by the software in your host machine. (os error 10053)
2024-11-26 16:29:09.650927000 [wallet::connectivity] DEBUG Attempting base node peer 'a04e78a02df887a26ffa9eadaa'... (last attempt Some(0ns))
2024-11-26 16:29:09.650937900 [wallet::connectivity] TRACE Attempt dial with client timeout 1s
2024-11-26 16:29:10.663273200 [wallet::connectivity] WARN  Connectivity error: Client cancelled: 'Could not connect to 'a04e78a02df887a26ffa9eadaa' in 1s'
2024-11-26 16:29:10.663415700 [wallet::connectivity] DEBUG Attempting base node peer 'a12b205abe9023e621910f0ee9'... (last attempt Some(100ns))
2024-11-26 16:29:10.663430500 [wallet::connectivity] TRACE Attempt dial with client timeout 1s
2024-11-26 16:29:11.678154700 [wallet::connectivity] WARN  Connectivity error: Client cancelled: 'Could not connect to 'a12b205abe9023e621910f0ee9' in 1s'
2024-11-26 16:29:11.678311700 [wallet::connectivity] DEBUG Attempting base node peer '81e258cb5e8419dfdd863282c3'... (last attempt Some(0ns))
2024-11-26 16:29:11.678338700 [wallet::connectivity] TRACE Attempt dial with client timeout 1s
2024-11-26 16:29:12.695260800 [wallet::connectivity] WARN  Connectivity error: Client cancelled: 'Could not connect to '81e258cb5e8419dfdd863282c3' in 1s'
2024-11-26 16:29:12.695391800 [wallet::connectivity] DEBUG Attempting base node peer '6d9043bba0c196ded52a9f79da'... (last attempt Some(100ns))
2024-11-26 16:29:12.695401300 [wallet::connectivity] TRACE Attempt dial with client timeout 1s
2024-11-26 16:29:13.708776300 [wallet::connectivity] WARN  Connectivity error: Client cancelled: 'Could not connect to '6d9043bba0c196ded52a9f79da' in 1s'
2024-11-26 16:29:13.708929300 [wallet::connectivity] DEBUG Attempting base node peer 'fb593780d69882b938429beb69'... (last attempt Some(0ns))
2024-11-26 16:29:13.708951900 [wallet::connectivity] TRACE Attempt dial with client timeout 1s
2024-11-26 16:29:14.721569600 [wallet::connectivity] WARN  Connectivity error: Client cancelled: 'Could not connect to 'fb593780d69882b938429beb69' in 1s'
2024-11-26 16:29:14.721697500 [wallet::connectivity] DEBUG Attempting base node peer 'db5ab85a84ce79bcd2ea700f13'... (last attempt Some(100ns))
2024-11-26 16:29:14.721708200 [wallet::connectivity] TRACE Attempt dial with client timeout 1s
2024-11-26 16:29:15.734914000 [wallet::connectivity] WARN  Connectivity error: Client cancelled: 'Could not connect to 'db5ab85a84ce79bcd2ea700f13' in 1s'
2024-11-26 16:29:15.735066300 [wallet::connectivity] DEBUG Attempting base node peer '6ea597117476676d5ddcb18153'... (last attempt Some(0ns))
2024-11-26 16:29:15.735099300 [wallet::connectivity] TRACE Attempt dial with client timeout 1s
2024-11-26 16:29:16.740185800 [wallet::connectivity] WARN  Connectivity error: Client cancelled: 'Could not connect to '6ea597117476676d5ddcb18153' in 1s'
2024-11-26 16:29:16.740313200 [wallet::connectivity] DEBUG Attempting base node peer '647db13c611aa70dcd9f06a8b1'... (last attempt Some(100ns))
2024-11-26 16:29:16.740324900 [wallet::connectivity] TRACE Attempt dial with client timeout 1s
2024-11-26 16:29:17.740956000 [wallet::connectivity] WARN  Connectivity error: Client cancelled: 'Could not connect to '647db13c611aa70dcd9f06a8b1' in 1s'
2024-11-26 16:29:17.741152500 [wallet::connectivity] DEBUG Attempting base node peer 'a4dbaf87693acaa0fe92f8f4f0'... (last attempt Some(0ns))
2024-11-26 16:29:17.741166200 [wallet::connectivity] TRACE Attempt dial with client timeout 1s
2024-11-26 16:29:18.744368800 [wallet::connectivity] WARN  Connectivity error: Client cancelled: 'Could not connect to 'a4dbaf87693acaa0fe92f8f4f0' in 1s'
2024-11-26 16:29:18.744526000 [wallet::connectivity] DEBUG Attempting base node peer 'e7b0d91465d5386e9d07e53cff'... (last attempt Some(0ns))
2024-11-26 16:29:18.744538900 [wallet::connectivity] TRACE Attempt dial with client timeout 1s
2024-11-26 16:29:19.745518100 [wallet::connectivity] WARN  Connectivity error: Client cancelled: 'Could not connect to 'e7b0d91465d5386e9d07e53cff' in 1s'
2024-11-26 16:29:19.745680000 [wallet::connectivity] DEBUG Attempting base node peer '17bfe6ee47f45491ea9407ad89'... (last attempt Some(0ns))
2024-11-26 16:29:19.745797100 [wallet::connectivity] TRACE Attempt dial with client timeout 1s
2024-11-26 16:29:20.747956700 [wallet::connectivity] WARN  Connectivity error: Client cancelled: 'Could not connect to '17bfe6ee47f45491ea9407ad89' in 1s'
2024-11-26 16:29:20.748180600 [wallet::connectivity] DEBUG Attempting base node peer 'bd11a65bb14f8188e0bde642d0'... (last attempt Some(0ns))
2024-11-26 16:29:20.748227700 [wallet::connectivity] TRACE Attempt dial with client timeout 1s
2024-11-26 16:29:21.755291100 [wallet::connectivity] WARN  Connectivity error: Client cancelled: 'Could not connect to 'bd11a65bb14f8188e0bde642d0' in 1s'
2024-11-26 16:29:21.755439100 [wallet::connectivity] DEBUG Attempting base node peer 'b7c07e293f2186689e88978bff'... (last attempt Some(0ns))
2024-11-26 16:29:21.755453200 [wallet::connectivity] TRACE Attempt dial with client timeout 10s
2024-11-26 16:29:23.810412000 [wallet::connectivity] WARN  Connectivity error: ConnectionFailed: Failed to connect to peer within the maximum number of attempts
2024-11-26 16:29:23.810491600 [wallet::connectivity] DEBUG Attempting base node peer '1c8b393640e1cd7d1a86957eb3'... (last attempt Some(0ns))
2024-11-26 16:29:23.810501100 [wallet::connectivity] TRACE Attempt dial with client timeout 10s
2024-11-26 16:29:33.811236800 [wallet::connectivity] WARN  Connectivity error: Client cancelled: 'Could not connect to '1c8b393640e1cd7d1a86957eb3' in 10s'
2024-11-26 16:29:33.811393800 [wallet::connectivity] DEBUG Attempting base node peer 'e108eb7fbf44d58199908aa928'... (last attempt Some(100ns))
2024-11-26 16:29:33.811404200 [wallet::connectivity] TRACE Attempt dial with client timeout 10s
2024-11-26 16:29:43.822083900 [wallet::connectivity] WARN  Connectivity error: Client cancelled: 'Could not connect to 'e108eb7fbf44d58199908aa928' in 10s'
2024-11-26 16:29:43.822250500 [wallet::connectivity] DEBUG Attempting base node peer '63718f177e749a0484a7050f8d'... (last attempt Some(100ns))
2024-11-26 16:29:43.822263700 [wallet::connectivity] TRACE Attempt dial with client timeout 10s
2024-11-26 16:29:53.850035700 [wallet::connectivity] WARN  Connectivity error: Client cancelled: 'Could not connect to '63718f177e749a0484a7050f8d' in 10s'
2024-11-26 16:29:53.850223800 [wallet::connectivity] DEBUG Attempting base node peer 'b02156f7ec30db5d661dd9b6e2'... (last attempt Some(0ns))
2024-11-26 16:29:53.850241100 [wallet::connectivity] TRACE Attempt dial with client timeout 10s
2024-11-26 16:29:53.870034300 [wallet::connectivity] WARN  Connectivity error: ConnectionFailed: Noise handshake error: An existing connection was forcibly closed by the remote host. (os error 10054)
2024-11-26 16:29:53.870101100 [wallet::connectivity] DEBUG Attempting base node peer 'a04e78a02df887a26ffa9eadaa'... (last attempt Some(100ns))
2024-11-26 16:29:53.870110100 [wallet::connectivity] TRACE Attempt dial with client timeout 10s
2024-11-26 16:30:00.043621300 [wallet::connectivity] WARN  Connectivity error: ConnectionFailed: Failed to connect to peer within the maximum number of attempts
2024-11-26 16:30:00.047536900 [wallet::connectivity] DEBUG Attempting base node peer 'a12b205abe9023e621910f0ee9'... (last attempt Some(100ns))
2024-11-26 16:30:00.047550400 [wallet::connectivity] TRACE Attempt dial with client timeout 10s
2024-11-26 16:30:10.053350900 [wallet::connectivity] WARN  Connectivity error: Client cancelled: 'Could not connect to 'a12b205abe9023e621910f0ee9' in 10s'
2024-11-26 16:30:10.053487900 [wallet::connectivity] DEBUG Attempting base node peer '81e258cb5e8419dfdd863282c3'... (last attempt Some(100ns))
2024-11-26 16:30:10.053501000 [wallet::connectivity] TRACE Attempt dial with client timeout 10s
2024-11-26 16:30:20.068640100 [wallet::connectivity] WARN  Connectivity error: Client cancelled: 'Could not connect to '81e258cb5e8419dfdd863282c3' in 10s'
2024-11-26 16:30:20.068766200 [wallet::connectivity] DEBUG Attempting base node peer '6d9043bba0c196ded52a9f79da'... (last attempt Some(100ns))
2024-11-26 16:30:20.068777100 [wallet::connectivity] TRACE Attempt dial with client timeout 10s
2024-11-26 16:30:20.068813900 [wallet::connectivity] DEBUG Established peer connection to base node '6d9043bba0c196ded52a9f79da'

How Has This Been Tested?

System-level testing

What process can a PR reviewer use to test or verify this change?

Code review
System-level testing

Breaking Changes

  • None
  • Requires data directory on base node to be deleted
  • Requires hard fork
  • Other - Please specify

@hansieodendaal
Add connectivity timeout
dada390 
Added a wallet connectivity timeout so long-winded failures can be managed timeously.
@hansieodendaal hansieodendaal requested a review from a team as a code owner November 26, 2024 12:28
@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 26, 2024 •
edited
Loading

Test Results (CI)

    3 files    129 suites   39m 15s ⏱️
1 344 tests 1 344 ✅ 0 💤 0 ❌
4 030 runs  4 030 ✅ 0 💤 0 ❌

Results for commit 669d2f0.

♻️ This comment has been updated with latest results.

@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 26, 2024 •
edited
Loading

Test Results (Integration tests)

36 tests   36 ✅  15m 22s ⏱️
11 suites   0 💤
 2 files     0 ❌

Results for commit 669d2f0.

♻️ This comment has been updated with latest results.

@hansieodendaal
Copy link
Contributor Author

hansieodendaal commented Nov 26, 2024 •
edited
Loading

Maybe we need the timeout to be adaptive... 1st time round maybe 20s, then 40s, and max out at 60s ?
See the measured data below, both connected in just over 40s:

Windows PC, console wallet

2024-11-26 12:31:25.139034200 [wallet::connectivity] DEBUG Attempting base node peer '6d9043bba0c196ded52a9f79da'... (last attempt Some(100ns))
2024-11-26 12:32:06.760160100 [wallet::connectivity] DEBUG Established peer connection to base node '6d9043bba0c196ded52a9f79da'

Android, mobile wallet

2024-11-26 10:20:25.202187051 [wallet::connectivity] DEBUG Attempting base node peer '6d9043bba0c196ded52a9f79da'... (last attempt Some(244ns))
2024-11-26 10:20:26.038427123 [wallet::connectivity] DEBUG Established peer connection to base node '6d9043bba0c196ded52a9f79da'

@SWvheerden SWvheerden merged commit e22bc0e into tari-project:development Nov 27, 2024
18 checks passed
@hansieodendaal hansieodendaal deleted the ho_connectivity_timeout branch November 27, 2024 07:14
hansieodendaal added a commit to hansieodendaal/tari that referenced this pull request Nov 28, 2024
@hansieodendaal
Merge development
3beaa5f 
PRs: tari-project#6703, tari-project#6701, tari-project#6705, tari-project#6704
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@SWvheerden SWvheerden SWvheerden approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@hansieodendaal @SWvheerden