kv: large-disk cluster has poor performance post-restart

[jbowens]

Reproduction steps

Create a cluster with 10 TB disks, eg:

roachprod create jackson-bank-200b --username jackson --clouds aws -n 8 \
  --local-ssd=false --aws-machine-type c5.4xlarge --aws-ebs-volume-size=10000
roachprod put    jackson-bank-200b ./cockroach-linux-2.6.32-gnu-amd64 ./cockroach
roachprod start  jackson-bank-200b
roachprod sql    jackson-bank-200b:1 -- -e \
   "set cluster setting kv.range_merge.queue_enabled = false"
roachprod sql    jackson-bank-200b:1 -- -e \
   "set cluster setting kv.bulk_io_write.concurrent_addsstable_requests = 8"

Import a large bank dataset. (This takes ~28 hours.)

roachprod run    jackson-bank-200b:1 -- \
   "./cockroach workload fixtures import bank --rows 200000000000 {pgurl:1} >/dev/null 2>&1 &"

Run the bank workload:

roachprod stop jackson-bank-200b
roachprod start jackson-bank-200b
roachprod run jackson-bank-200b:1 -- \
    ./cockroach workload run bank --ramp=5m --rows=200000000000 --duration=15m

Throughput is around ~40 ops/sec. See https://docs.google.com/document/d/1rfWNGFZ6gulKqb6BMXeMzq9GkMgdVfXnC4SJ3FM3ilE/edit?usp=sharing

Jira issue: CRDB-2897

[blathers-crl]

Hi @jbowens, please add a C-ategory label to your issue. Check out the label system docs.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[jbowens]

I left this cluster for a bit and eventually all but one node was oom killed. The logs showed persistent liveness errors and lots of duplicate connection gossip errors:

I201119 00:09:36.443177 8449909 gossip/client.go:124 ⋮ [n6] started gossip client to ‹10.12.21.44:26257›
I201119 00:09:36.444866 8449909 gossip/client.go:129 ⋮ [n6] closing client to n3 (‹10.12.21.44:26257›): ‹rpc error: code = Unknown desc = duplicate connection from node at 10.12.28.144:26257›
I201119 00:09:37.763208 8451167 gossip/server.go:277 ⋮ [n6] refusing gossip from n3 (max 3 conns); forwarding to n1 (‹10.12.16.145:26257›)
I201119 00:09:38.474971 8451309 gossip/client.go:124 ⋮ [n6] started gossip client to ‹10.12.27.108:26257›
I201119 00:09:38.485140 8451309 gossip/client.go:129 ⋮ [n6] closing client to n4 (‹10.12.27.108:26257›): stopping outgoing client to n4 (‹10.12.27.108:26257›); already have incoming
I201119 00:09:38.972033 8452297 gossip/server.go:277 ⋮ [n6] refusing gossip from n2 (max 3 conns); forwarding to n4 (‹10.12.27.108:26257›)
I201119 00:09:39.478998 8451560 gossip/client.go:124 ⋮ [n6] started gossip client to ‹10.12.23.41:26257›

[lunevalex]

[lunevalex]

This maybe the same issue with the raft scheduler that we have seen here #56851 given the number of ranges.

[nvanbenschoten]

#56943 is introducing a new raft.scheduler.latency metric, which will allow us to verify whether Raft scheduler latency is the issue here if we decide to run this experiment again.

[github-actions]

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!

[jbowens]

I'll close this out, and we should revisit node density as a part of the planned scalability limits work (cc @williamkulju). I don't think this one data point from 3 years ago is providing much context, and we'll hopefully be able to identify specific concrete obstacles to high node density in that work.