Support more than one store per node

[rmloveland]

As noted in #7666 and elsewhere, CockroachDB does not balance replicas among multiple stores per node. This limits the usefulness of running with multiple stores.

There is work ongoing to remove this limitation in the 20.2 cycle. Therefore, we will need to update docs accordingly when that work gets done.

Estimated scope of work:

• Update the cockroach start docs accordingly. In particular, there is a note there about rebalancing that can probably (?) be removed as a result of this work (and that note might actually close Document that we do not balance replicas among multiple stores per node #7666, linked below).

• Possibly update the recommended production settings on storage to note that you can "put more stores on it"

• Research other places this info might be needed in the docs (?), and do something about it

Related issues:

• Document that we do not balance replicas among multiple stores per node #7666

[johnrk-zz]

Possibly update the recommended production settings on storage to note that you can "put more stores on it"

This hasn't been tested. cc @darinpp

[jseldess]

@rmloveland, @mwang1026, what's the state of rebalancing across multiple stores? We have a number of docs issues around this, but I'm not sure if/how we should update the docs.

[mwang1026]

We're in the same place re: rebalancing. It's actually been a problem of late so I'm hesitant to provide that recommendation.

[jseldess]

Thanks, @mwang1026. Is work ongoing on this? Should we close this issue and reopen to reflect current projects? This ticket is from the 20.2 roadmap...

[johnrk-zz]

@jseldess , yes, @lunevalex worked on this in 20.2:

• JIRA Epic
• kvserver: Allow rebalances between stores on the same nodes. #51567

Alex's change resulted in our safely rebalancing between stores that already existed on a node. IIRC, it does not support adding stores and rebalancing to an existing node and was not going to address the root problem of storing more per node. Ultimately, we decided to not prioritize documenting this update because it was not sufficiently customer impacting.