fix vault retry logic on failed calls

[eikenb]

The original problem was that for non-renewable vault secrets that it
was having trouble fetching, it would wait the standard exponential
backoff time plus the configured sleep time (like it does between
successful fetches). When what it should do is use the sleep time
between successful fetches and exponential backoff on failures.

While fixing this I cleaned up the code to make the logic clear.
The issue existed in both vault_read and vault_write, and they shared a
common chunk of renew logic between them and with vault_token. So I
refactored that out into a common function.

Fixes #1224

[mkeeler]

Overall this looks great. Just the few very minor points.

[mkeeler]

LGTM

[vapopov]

@mkeeler @eikenb
Hello, this part of the code fail blocking operation of lease duration, when i use it with vault pki endpoint it just recursively creates tons of certificates

consul-template/dependency/vault_write.go

Lines 61 to 64 in a96de95

	select {
	case <-d.sleepCh:
	default:
	}

How this part of code should work at all? We just ignore time channel and always pass through default:

[vapopov]

In previous implementation version 0.20.0, it was working as expected.
it terms of vault pki, certificates can't be renewable, but we have to block Fetch operation until lease duration is not reached, otherwise we get loop of certificate generation

consul-template/dependency/vault_write.go

Lines 96 to 101 in 9a0f301

	select {
	case <-time.After(dur):
	// The lease is almost expired, it's time to request a new one.
	case <-d.stopCh:
	return nil, nil, ErrStopped
	}

[eikenb]

Hey @vapopvo, thanks for the comments. I missed these at the time but others reported the issue. I just pushed the fix for this in 0.21.3. In the future if you see these things please open a new issue and then reference things like this. I don't check my github notifications every day and can easily miss comments on already merged PRs (or closed issues).

Thanks.

[vapopov]

@eikenb good, thanks a lot will do, the template of issues just prevented me to spend time a lot of writing it :)