Kubernetes / containers: Advise adjusting GOMAXPROCS based on cpu requests/limits

[exalate-issue-sync]

Bilal Akhtar (itsbilal) commented:

Currently, our docs on using Cockroach with Kubernetes mention CPU requests and limits:

https://www.cockroachlabs.com/docs/v20.2/kubernetes-performance#resource-requests

The same knobs also exist in all cases of running Cockroach in cgroup-driven clusters (eg. Docker), but for simplicity I'm going to use k8s as an umbrella term to refer to all container-based setups with CPU resource limiting in this issue.

Our docs make little reference made to the need to adjust the GOMAXPROCS environment variable based on either setting. The Cockroach process has visibility into CPU limits when it reports CPU usage percentages (since cockroachdb/cockroach#56461), however it doesn't adjust GOMAXPROCS on its own based on CPU limits. Furthermore, the cockroach process cannot calculate the percentage of all CPU shares in the parent cgroup that it has access to, as it doesn't have visibility into the shares given to other sibling cgroup controllers / k8s pods.

If GOMAXPROCS is higher than the CPU limit set, then some goroutines will get scheduled for execution by the Go runtime but will then have to block for one of the (fewer) available cpus to be ready to execute it, resulting in degraded concurrent performance.

Similarly, for CPU requests, let's say if there are 8 CPUs on a machine, and there are 4 pods that all request the same CPU request amount, each pod would get 25% of CPUs (so ~2 cores or core-equivalent runtime) when all pods are maxing out CPU utilization. If such a workload persists for a while, and GOMAXPROCS is greater than 2, then it's possible for performance to be less ideal than if GOMAXPROCS was right at 2.

The ideal GOMAXPROCS setting is going to vary based on the workload, but it's probably a good idea to advise setting it to something like this:

GOMAXPROCS = floor(num_cores * min(cpu_limit_ratio_for_this_container, 2 * cpu_share_ratio_for_this_container))

The 2 * is there to allow for some burst utilization, in cases where cpu requests are much lower than cpu limits.

Jira Issue: DOC-834

[exalate-issue-sync]

Bilal Akhtar (itsbilal) commented:
Assigning @taroface for now, feel free to re-assign as appropriate

[exalate-issue-sync]

Raphael 'kena' Poss (knz) commented:
Discussed offline - @itsbilal to fish out how this is computed in CC, as an example best practice.

[exalate-issue-sync]

Bilal Akhtar (itsbilal) commented:
Had a conversation with CC engineers, seems like I was mistaken on CC adjusting for this. cockroachdb/cockroach#57390 is what they expect Cockroach to do in this situation, so I've gone ahead and implemented it. The advice outlined in this issue would still be necessary for all users of containers, especially those that use cpu weights/shares.

[mari-crl]

This issue is an accidental duplicate created by automation. The correct issue that is synced to the corresponding Jira issue is #9001.