scatter: downreplication takes a long time

[benesch]

Discovered while implementing a proper scatter in #16249. AdminScatter is able to very quickly upreplicate (e.g., in a few minutes), but the replicate queue can take ten minutes to downreplicate. Figure out why; @a-robinson claims the downreplication should take approximately as long as the upreplication.

This behavior is easy to reproduce with a four-node local cluster with preeexisting data: https://github.com/benesch/crdb-playground-scatter

[a-robinson]

Was this fixed by your DeleteRange change?

[benesch]

Don't think so, but you'd know better. Here's the downreplication after a 15GB restore with ~1k ranges.

Takes about 10m.

[benesch]

And here's a screenshot that's, like, actually useful:

[benesch]

Try not to look too hard at the leaseholders per store graph.

[a-robinson]

Assuming I'm squinting properly at that graph, it looks like it took around 7 minutes, so I guess it's still not fixed.

[benesch]

Yeah, sorry, it's ~10m. How long would you expect it to take?

[a-robinson]

About the same amount of time as up-replicating. The problem, though, is that's the wrong graph to be looking at, since it's including not just the down-replication time, but also the replica GC time, since the metric behind the replicas-per-store graph only gets decremented when a replica gets GC'ed, not when it gets removed from the consensus group. Usually those events are close together, but they aren't guaranteed to be. The "Range Operations" graph would be more useful.

I'll check it out sometime, but it doesn't seem urgent.

[benesch]

That is incredibly useful to know. Yeah, no rush at all; I may look into it too.

[tbg]

I wouldn't be surprised if this were fixed now. Don't the decommissioning tests verify something similar?

[a-robinson]

Scatter no longer scatters replicas, and when we bring that aspect of it back (#26438) it won't add all replicas before removing any. I think it's safe to close this.