metrics: Add RPC rate metrics to endpoints that validate TLS names

[tgross]

Continues the work done in #15876 by extending rate metrics measurement over the various RPCs that are authenticated via TLS name validation. This changeset also canonicalizes the order-of-operations between Authenticate and that validation. We call Authenticate first so that we get identity info like the remote IP address. This changeset doesn't change the existing behavior that drops the RPC before forwarding; I think that's worth having a discussion about but I didn't want to bloat this PR with behavior changes.

[shoenig]

LGTM; just one idea: what if the MeasureRPCRate signature accepted a Name() interface as the first parameter instead of a string, then instead of hardcoding endpoint names you'd just pass in the endpoint object, e.g.

k.srv.MeasureRPFRate(k, structs.RateMetricRead, args)

Or maybe that's less clear, I dunno

[tgross]

LGTM; just one idea: what if the MeasureRPCRate signature accepted a Name() interface as the first parameter instead of a string, then instead of hardcoding endpoint names you'd just pass in the endpoint object,

I like that. We could probably use it in other places like the logger too. That'll touch a lot of handlers though (including the others I have in review for this feature), so I'll break that out into its own PR.

[tgross]

LGTM; just one idea: what if the MeasureRPCRate signature accepted a Name() interface as the first parameter instead of a string, then instead of hardcoding endpoint names you'd just pass in the endpoint object, e.g.

So I started implementing this and I think it ends up being kind of boilerplate-y without a lot of readability improvement. I checked the disassembly and the strings all get interned anyways. What do you think about:

func (j *Job) measureRPCRate(op string, args structs.RequestWithIdentity) {
	j.srv.MeasureRPCRate("job", op, args)
}

which results in callers like:

// Register is used to upsert a job for scheduling
func (j *Job) Register(args *structs.JobRegisterRequest, reply *structs.JobRegisterResponse) error {
	authErr := j.srv.Authenticate(j.ctx, args)
	if done, err := j.srv.forward("Job.Register", args, args, reply); done {
		return err
	}
	j.measureRPCRate(structs.RateMetricWrite, args) // <-- HERE
	if authErr != nil {
		return structs.ErrPermissionDenied
	}
	defer metrics.MeasureSince([]string{"nomad", "job", "register"}, time.Now())

That's still pretty boilerplate-y and doesn't win us that much though. This is all the sort of thing I'd like to have rolled up in the generic RPC middleware we've discussed too, so maybe not worth the effort to do right now until we have that figured out.