acceptance: stats-based rebalancing makes Allocator Test 3 to 5 10G flaky

[a-robinson]

https://teamcity.cockroachdb.com/viewType.html?buildTypeId=Cockroach_AllocatorTest3to510g&tab=buildTypeHistoryList&branch_Cockroach_Nightlies=__all_branches__

Opening this to make it clear it's on my plate.

[cuongdo]

Could this be a case of the passed in max standard deviation being too strict for the latest allocator changes? It seems like we were more aggressively trying to balance range counts before

[a-robinson]

The two most likely explanations are:

1. The goal of the new allocator isn't to balance range counts - it's to balance the combination of range counts, writes per second, and disk usage per store. If writes and disk usage are balanced but the number of ranges isn't, that's considered fine. This test only verifies range counts, so the allocator could have succeeded in its new goals while failing the test. Changing the test would be needed to fix this.
2. If writes and disk usage are evenly distributed, you might expect that balancing them is approximately the same as balancing range count. However, the test could still be failing because we only require writes per second and disk usage to be within 20% of the mean on each store, but are testing that the standard deviation of range count is less than 10%. Reducing the setting for the duration of the test may fix this.

There is a third explanation (that the allocator is actually broken), but both of the above explanations are likely to lead to flakiness of this test even if everything is working perfectly in the allocator.

[a-robinson]

And for future reference, here are the artifacts from last night's failure:
Nightlies_Allocator_Test_3_to_5_10G_461_artifacts.zip

[TestRebalance_3To5Small] allocator_test.go:162: 3m15s822ms670us elapsed without changes, but replica count standard deviation is 21.05 (>14.00)

[a-robinson]

It's looking like #17733 may have fixed this. The only failures since it went in have been due to acceptance test infrastructure issues, which were fixed in #17776. I'll continue to monitor, but things are looking good at this point.

[tbg]

Closing as not actionable.