kvserver: gossip frequency is implicitly 10 seconds

[kvoli]

Description

Store gossip occurs more than what is expected. It occurs every ~10 seconds under no changes whilst it should occur every minute currently.

These updates are triggered by capacity changes, specifically lease add events .

This function declares itself as idempotent however that is not the case, as each call may cause a new gossip to kick off of the latest store descriptor.

Reproduce

To reproduce, run https://github.com/cockroachdb/cockroach/compare/master...kvoli:220519.gossip-metrics?expand=1 and open up DB console. Using kv.allocator.staleness you can examine the histogram of gossip store descriptor staleness used in allocation decisions.

What triggers gossip updates

We only update the storepool state with newer information here. We also update the storepool state following lease transfers and replica changes with the estimated impact.

Gossip updates occur for the local store every 1 minute and will also be triggered if between now and the last gossip, any of these are true:

Condition	Condition Check Trigger	Description
(replica count delta) > 2	Replica change	code
(replica count delta)/(last gossipped replica count) > 1% of last gossipped value	Replica change	code
(lease count delta) > 0 and (lease count delta)/(last gossipped lease count) >1%	Lease transfer	code
(qps delta)/(last gossipped qps) > 50% and qps delta > 100	Every 10 seconds (when updating metrics gauges)	code

Expected behavior

Gossip should occur only when the above conditions are met. Additionally we may wish to investigate lowering the timer from 1 minute, given this bug has existed for some time without issue. Making the interval explicit rather than implicit.

Jira issue: CRDB-16019

[kvoli]

Update: In the original experiments, the histogram that was being used could not record values greater than 10s - due to

cockroach/pkg/util/metric/metric.go

Line 30 in 933b684

MaxLatency = 10 * time.Second

.

In an updated experiment where this is set to 120s, we see expected theoretical results:

commit: kvoli@904f5c7

The updated values are:

%	staleness
p99	60s
p90	53s
p75	45s
p50	25s

This invalidates the previous theory that the gossip interval is implicity 10s. Instead, under no triggers it is the stated value in the code, 60s.

Closing this issue in favor of another issue for lowering the gossip interval #83841.