feat: add 'argocd admin cluster shard balance' command

[alexmt]

Signed-off-by: Alexander Matyushentsev [email protected]

WIP PR that is supposed to resolve #6125

PR adds command argocd cluster shards balance that distribute clusters between available controller shards based on number of resources in each shard. During the testing, I've discovered that resources count and amount of used memory do not correlate at all. The table below contains snapshot of argocd that manages ~300 clusters:

shard  resources      memory
0      181653 (108%)  105.50 %
1      109621 (65%)   103.27 %
2      301413 (180%)  61.30 %
3      232993 (139%)  137.5 %
4      189908 (113%)  79.25 %
5      85075  (51%)   175.96 %
6      188501 (112%)  79.80 %
7      149737 (89%)   88.25 %
8      106461 (64%)   120.45 %
9      130819 (78%)   146.7 %

Shard #5 manages clusters with ~50% less than average resources but consumes 75% more memory. After rebalancing memory usage got even more uneven. So keep it as a draft for now and try to find a better metric than the number of resources.

[codecov]

Codecov Report

Merging #8002 (09db887) into master (487db97) will decrease coverage by 0.15%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master    #8002      +/-   ##
==========================================
- Coverage   41.49%   41.34%   -0.16%     
==========================================
  Files         173      173              
  Lines       22591    22673      +82     
==========================================
  Hits         9375     9375              
- Misses      11885    11967      +82     
  Partials     1331     1331              

Impacted Files	Coverage Δ
cmd/argocd/commands/admin/cluster.go	0.00% <0.00%> (ø)
util/settings/settings.go	46.85% <0.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 487db97...09db887. Read the comment docs.

[pasha-codefresh]

@alexmt i would close this one, please feel free to reopen if you plan to work on it