discovery: add upper limit for waiting on a retry

[fanminshi]

Adding upper limit ensures that expoential backoff doesn't reach more than 5 min on a re-try.

FIX #6648

[fanminshi]

PLTAL cc/ @xiang90 @crawford

[xiang90]

use floor(log2(maxWaitInSeconds)) instead?

[fanminshi]

yep, that's better

[fanminshi]

due to type conversion. the code looks a bit ugly

    if d.retries < uint(math.Floor(math.Log2(float64(maxWaitInSeconds)))) {
        retryTime = 0x1 << d.retries
    } else {
        retryTime = maxWaitInSeconds
    }

I can change it to

    d.retries++
    var retryTimeInSecond time.Duration
    // logAndBackoffForRetry stops exponential backoff when the retries are more than the maxExpoentialRetries and is set to a constant backoff afterward.
    if d.retries < maxExpoentialRetries {
        retryTimeInSecond = time.Duration(0x1<<d.retries) * time.Second
    } else {
        retryTimeInSecond = time.Duration(0x1<<maxExpoentialRetries) * time.Second
    }
    plog.Infof("%s: error connecting to %s, retrying in %s", step, d.url, retryTimeInSecond)
    d.clock.Sleep(retryTimeInSecond)

[xiang90]

ok. sure.

[fanminshi]

PTAL cc @xiang90

[xiang90]

lgtm.

[crawford]

I prefer something closer to https://github.com/coreos/ignition/blob/5dd13c9c174b40dfefb09d777d2bd890e8ba6fa7/internal/resource/http.go#L31. That way you can actually keep track of the number of attempts. It's probably okay though.

[fanminshi]

@crawford My code does not modify the retries counter. So I could still track number of the failed attempts.
https://github.com/fanminshi/etcd/blob/discovery_max_wait/discovery/discovery.go#L271-L273

[crawford]

Okay. This is no longer an exponential backoff.

[fanminshi]

@crawford yeah, the exponential back off stops at 8th retry attempt. every retry afterward will be a constant backoff about 2^8 = 256 second. Is this a reasonable backoff strategy?

[crawford]

It's linear not exponential though. Unless I'm missing something.

[xiang90]

this should still be (0x1 << retries)

[fanminshi]

good catch

[xiang90]

@crawford @fanminshi

For the first N retries, wait time grows exponentially. After that the wait time should fixed. The upper bound is 1 << N seconds.

[fanminshi]

All Fixed. PTAL cc/@xiang90 @crawford

[crawford]

LGTM

[xiang90]

CI fails due to #6535. So not relevant to this PR.