kvserver: set StoresInterval to 10s

[tbg]

In #81669 @kvoli discovered that we often gossip much more frequently
than at a 60s interval. Since we are also adding I/O liveness signals
to the store capacity to use in a solution for #79215, we want to make
this more reactive interval explicit.

This change has the side effect of reducing the minimum allowed value
for server.time_until_store_dead to 25s¹; this seems reasonable.

Release note: None

Footnotes

1. https://github.com/cockroachdb/cockroach/blob/263fbb7c8fcf001fcf47d7d35894b5824c78dc14/pkg/kv/kvserver/allocator/storepool/store_pool.go#L94-L99 ↩

[cockroach-teamcity]

This change is 

[kvoli]

I think we should instrument a fix for the implicit level or add this as a gossip if you haven't in 10 seconds.

Otherwise this will compound with the existing implicit rate.

I'm looking at a fix atm for #81669

I think before we change the interval we should note with the amount of data transferred, at different intervals.

[kvoli]

Running some experiments at 60/30/10s

10s seems perfectly reasonable. Only increasing by 1% egress/ingress % of total network bandwidth.

repro

export gce_project=cockroach-ephemeral
export email=austen-gossip-interval
roachprod create $email -n 32 --gce-machine-type=n1-standard-16

roachprod put $email cockroach-remote cockroach
roachprod start $email

roachprod grafana-start $email

roachprod sql $email:1 -- -e "set cluster setting gossip.stores.interval = '60s';"
sleep 600
roachprod sql $email:1 -- -e "set cluster setting gossip.stores.interval = '30s';"
sleep 600
roachprod sql $email:1 -- -e "set cluster setting gossip.stores.interval = '10s';"
sleep 600
roachprod sql $email:1 -- -e "set cluster setting gossip.stores.interval = '60s';"
roachprod run $email -- './cockroach workload run kv --tolerate-errors --min-block-bytes=127 --max-rate=1000 --max-block-bytes=256 --read-percent=95 --concurrency=256 --drop --duration=10m'

roachprod sql $email:1 -- -e "set cluster setting gossip.stores.interval = '30s';"
roachprod run $email -- './cockroach workload run kv --tolerate-errors --min-block-bytes=127 --max-rate=1000 --max-block-bytes=256 --read-percent=95 --concurrency=256 --drop --duration=10m'

roachprod sql $email:1 -- -e "set cluster setting gossip.stores.interval = '10s';"
roachprod run $email -- './cockroach workload run kv --tolerate-errors --min-block-bytes=127 --max-rate=1000 --max-block-bytes=256 --read-percent=95 --concurrency=256 --drop --duration=10m'

[tbg]

Thanks for the experiment!

bors r=kvoli

[craig]

Build succeeded:

• GitHub CI (Cockroach)