Dynamically resolve reverse tunnel address

[rosstimothy]

The reverse tunnel address is currently a static string that is
retrieved from config and passed around for the duration of a
services lifetime. When the tunnel_public_address is changed
on the proxy and the proxy is then restarted, all established
reverse tunnels over the old address will fail indefinintely.
As a means to get around this, #8102 introduced a mechanism
that would cause nodes to restart if their connection to the
auth server was down for a period of time. While this did
allow the nodes to pickup the new address after the nodes
restarted it was meant to be a stop gap until a more robust
solution could be applid.

Instead of using a static address, the reverse tunnel address
is now resolved via a reversetunnel.Resolver. Anywhere that
previoulsy relied on the static proxy address now will fetch
the actual reverse tunnel address via the webclient by using
the Resolver. In addition this builds on the refactoring done
in #4290 to further simplify the reversetunnel package. Since
we no longer track multiple proxies, all the left over bits
that did so have been removed to accomodate using a dynamic
reverse tunnel address.

[Joerger]

LGTM, just some minor comments

[Joerger]

Why do we set count to 1 here? Is it obvious and I'm missing it, or could we add a comment?

[rosstimothy]

My educated guess is that setting the workpool target to anything < 1 would cause it to close the underlying workgroup and thus require a new one to be created in the future when the target becomes >= 1. By instead setting the workpool target to 1 it resets the workgroup without deleting it.

@fspmarshall it looks like it has been like this since you added the tracker can you please correct me if I am wrong here?

[fspmarshall]

The count is the number of proxies we expect to discover, based on the heartbeats we've seen. We don't start off knowing about any proxies since it is the first proxy that we connect to that tells us about its peers. Therefore if we don't know about any proxies yet, we just try to find at least one, and then wait for it to tell us the real number.

[fspmarshall]

I'm a bit concerned about the increased network load if proxies are restarted in very large clusters. Might create another thundering herd problem if we reload the address on every call to the resolver. I'm thinking it might be good if the resolver had some basic ttl-based caching capabilities, so that if it were called many times over a small period (say 3-5 seconds), only one network call is actually made.

We already use lib/cache/fncache.go to do the same thing to reduce thundering herd issues when the cache is unhealthy. That helper doesn't rely on anything in lib/cache, so we could easily move it somewhere under utils, and use it to provide similar functionality here. Ex:

func CachingResolver(resolver Resolver) Resolver {
    cache := utils.NewFnCache(3 * time.Second)
    return func() (*utils.NetAddr, error) {
        a, err := cache.Get(context.TODO(), "resolver", func() (interface{}, error) {
            addr, err := resolver()
            return addr, err
        })
        if err != nil {
            return nil, err
        }
        return a.(*utils.NetAddr), nil
    }
}

[bernardjkim]

Hi @rosstimothy do we plan on back porting this into v7?

In certain situations v7 agents will fail to reconnect after a proxy config change. More details here https://github.com/gravitational/cloud/issues/1441#issuecomment-1067391237. If we don't plan on back porting this change into v7 I can create a new issue to patch up the current implementation.