Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace roko.TryForever with exponential backoff for ~24 hours #2588

Merged
merged 1 commit into from
Jan 16, 2024

Conversation

tessereth
Copy link
Contributor

@tessereth tessereth commented Jan 16, 2024

Occasionally something goes wrong and the agent gets stuck retrying job log uploads or job finish calls forever (eg because the agent's access token is revoked). Forever is a really long time. Instead just retry for ~24 hours. I also changed these to exponential backoff rather than constant time because if a request fails, it's likely to either be fixed very quickly (network blip) or take a long time (incident). So no point retrying every second if it's already failed a hundred times. I also made them both the same and 2 seconds initial interval to match other parts of the code.

Using the ExponentialSubsecond algorithm with a 2 second initial interval, this is what the retry intervals looks like:

Retry attempt Delay Cumulative delay
0 0:00:02 0:00:02
1 0:00:03 0:00:05
2 0:00:05 0:00:10
3 0:00:08 0:00:19
4 0:00:13 0:00:32
5 0:00:22 0:00:54
6 0:00:35 0:01:28
7 0:00:56 0:02:24
8 0:01:29 0:03:53
9 0:02:24 0:06:17
10 0:03:51 0:10:08
11 0:06:12 0:16:20
12 0:09:58 0:26:18
13 0:16:02 0:42:20
14 0:25:47 1:08:07
15 0:41:27 1:49:35
16 1:06:40 2:56:15
17 1:47:12 4:43:27
18 2:52:24 7:35:51
19 4:37:14 12:13:05
20 7:25:50 19:38:55
21 11:56:56 31:35:51

Copy link
Contributor

@moskyb moskyb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

forever is a long time

[citation needed]

@tessereth tessereth merged commit 00c74ac into main Jan 16, 2024
1 check passed
@tessereth tessereth deleted the remove-try-forever branch January 16, 2024 03:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants