schedulerlatency: improve percentile computation

[irfansharif]

Is your feature request related to a problem? Please describe.

The way we're computing scheduling latencies at specific percentiles (typically p99) is naive thanks to yours truly. It just picks the mid point of the bucket we've found for the given percentile, instead of linearly interpolating. Experimentally we've noticed that this makes us slightly insensitive to changes in admission.elastic_cpu.scheduler_latency_target -- a target of 1ms behaves no differently than, say, 950us.

cockroach/pkg/util/schedulerlatency/sampler.go

Lines 337 to 346 in e397faf

	// TODO(irfansharif): Implement proper linear interpolation instead and then
	// write test comparing against it against metric.ValueAtQuantileWindowed.
	// If pmax, we'll return the bucket max. If pmin, we'll return the bucket
	// min. If the 90th and 100th percentile values lie within some bucket, and
	// we're looking for 99.9th percentile, we'll interpolate to the 99% point
	// between bucket start and end. Because we're doing this naive mid-point
	// thing, it makes for a confusing difference when comparing the p99
	// computed off of go.scheduler_latency vs.
	// admission.scheduler_latency_listener.p99_nanos.
	}

This is partly due to how the bucket boundaries are configured in the Go runtime, something we have no say over (yet):

cockroach/pkg/util/schedulerlatency/sampler.go

Lines 285 to 289 in e397faf

	// bucket[270] width=16.384µs boundary=[737.28µs, 753.664µs)
	// bucket[271] width=16.384µs boundary=[753.664µs, 770.048µs)
	// bucket[272] width=278.528µs boundary=[770.048µs, 1.048576ms)
	// bucket[273] width=32.768µs boundary=[1.048576ms, 1.081344ms)
	// bucket[274] width=32.768µs boundary=[1.081344ms, 1.114112ms)

+cc @cockroachdb/admission-control.

Jira issue: CRDB-20459

[irfansharif]

@aadityasondhi expressed interest in picking this up, assigning.