runtime: make `GOMAXPROCS` cfs-aware on `GOOS=linux` #33803

jcorbin · 2019-08-23T16:11:53Z

Problem

The default setting of runtime.GOMAXPROCS() (to be the number of os-apparent processors) can be greatly misaligned with container cpu quota (e.g. as implemented through cfs bandwidth control by docker).

This can lead to large latency artifacts in programs, especially under peak load, or when saturating all processors during background GC phases.

The smaller the container / larger the machine = the worse this effect becomes: let's say you deploy a fleet of micro service workers, each container having a cpu quota of 4, on a fleet of 32 processor[1] machines.

To understand why, you really have to understand the CFS quota mechanism; this blog post does well (with pictures); this kubernetes issue further explores the topic (especially as it relates to a recently resolved kernel cpu accounting bug). But to summarize it briefly for this issue:

there is a quota period, say 100ms
and there is then a quota, say 400ms to affect a 4-processor quota
within any period, once the process group exceeds its quota it is throttled

Running an application workload at a reasonable level of cpu efficiency makes it quite likely that you'll be spiking up to your full quota and getting throttled.

Background waste workload, like concurrent GC[2], is especially likely to cause quota exhaustion.

I hesitate to even call this a "tail latency" problem; the artifacts are visible in the main body of and can shift the entire latency distribution.

Solution

If you care about latency, reliability, predictability (... insert more *ilities to taste), then the correct thing to do is to never exceed your cpu quota, by setting GOMAXPROCS=max(1, floor(cpu_quota)).

Using this as a default for GOMAXPROCS makes the world safe again, which is why we use uber-go/automaxprocs in all of our microservices.

NOTEs

intentionally avoiding use of the word "core"; the matter of hyper-threading and virtual-vs-physical cores is another topic
/digression: can't not mention userspace scheduler pressure induced by background GC; where are we at with goroutine preemption again?

The text was updated successfully, but these errors were encountered:

jcorbin · 2019-08-23T16:23:24Z

I really have to disagree with some of the latter suggestions in kubernetes/kubernetes#67577
like
kubernetes/kubernetes#67577 (comment): using ceil(quota)+2, or in any way over-provisioning GOMAXPROCS vs cpu quota, is at best a statistical gamble to ameliorate current shortcomings in Go's userspace scheduler.

Some background on uber-go/automaxprocs#13 (changing from ceil(quota) to floor(quota)):

over-provisioning seemed reasonable at first when the guess was "if we provision fractional cores, use them"
but later on we ended up needing fractionals to add margin for other supporting processes injected into your container (e.g. by Mesos Aurora's Thermos executor)

I'll reprise (copied with some edits) my description from that issue here for easy reading:

as far as the Go scheduler is concerned, there's no such thing as a fractional CPU

so let's say you have your quota set to N + p for some integer N and some 0.0 < p < 1.0

the only safe assumption then is that you're using that p value as a hedge for something like "systems effects" or "c libraries"

in that latter case, what you really might want is to be able give maxprocs some value K of CPUs stolen for some parasite like a C library or sidecar; but this will always need to be application-config specific

jcorbin · 2019-08-23T16:37:47Z

Noting: #19378 (comment) explores some GC-CFS relationship

jcorbin · 2019-08-23T18:28:20Z

For comparative purposes, Oracle blog post about Java adding similar support ( especially for GC threads )

ianlancetaylor · 2019-08-23T20:09:58Z

CC @aclements @mknyszek

(I'm not sure this has to be a proposal at all. This is more like a bug report. See https://golang.org/s/proposal.)

ianlancetaylor · 2019-09-03T20:52:09Z

Changing this from a proposal into a feature request for the runtime package.

Perhaps due to etcd failing due to mismatch between the CPU limits and the apparent available CPUs according to GOMAXPROCS See golang/go#33803

GOMAXPROCS subsumes NumCPU for the purpose of sizing semaphores. If users set CPU affinity, then GOMAXPROCS will reflect that. If users only set GOMAXPROCS, then NumCPU would be inaccurate. Additionally, there are plans to make GOMAXPROCS aware of CPU quotas (golang/go#33803). Users are still advised to set CPU affinity instead of relying on GOMAXPROCS to limit CPU usage, because Staticcheck shells out to the underlying build system, which together with Staticcheck would be able to use more CPU than intended if limited by just GOMAXPROCS.

This makes "new" and "old" setup_env functions. In subsequent commits, all callers of the "old" form will be fixed, and the "new" will be renamed back. The old and new functions diff: ```diff --- /tmp/old 2023-05-06 12:03:23.913951720 -0700 +++ /tmp/new 2023-05-06 12:06:37.688300514 -0700 --- a 2023-05-27 10:20:55.477856825 -0700 +++ b 2023-05-27 10:23:13.253192669 -0700 @@ -1,4 +1,4 @@ -kube::golang::old::setup_env() { +kube::golang::new::setup_env() { kube::golang::verify_go_version # Set up GOPATH. We have tools which depend on being in a GOPATH (see @@ -7,9 +7,9 @@ # Even in module mode, we need to set GOPATH for `go build` and `go install` # to work. We build various tools (usually via `go install`) from a lot of # scripts. - # * We can't set GOBIN because that does not work on cross-compiles. - # * We could use `go build -o <something>`, but it's subtle when it comes - # to cross-compiles and whether the <something> is a file or a directory, + # * We can't just set GOBIN because that does not work on cross-compiles. + # * We could always use `go build -o <something>`, but it's subtle wrt + # cross-compiles and whether the <something> is a file or a directory, # and EVERY caller has to get it *just* right. # * We could leave GOPATH alone and let `go install` write binaries # wherever the user's GOPATH says (or doesn't say). @@ -20,7 +20,6 @@ # # Eventually, when we no longer rely on run-in-gopath.sh we may be able to # simplify this some. - kube::golang::old::create_gopath_tree export GOPATH="${KUBE_GOPATH}" # If these are not set, set them now. This ensures that any subsequent @@ -31,23 +30,12 @@ # Make sure our own Go binaries are in PATH. export PATH="${KUBE_GOPATH}/bin:${PATH}" - # Change directories so that we are within the GOPATH. Some tools get really - # upset if this is not true. We use a whole fake GOPATH here to collect the - # resultant binaries. - local subdir - subdir=$(kube::realpath . | sed "s|${KUBE_ROOT}||") - cd "${KUBE_GOPATH}/src/${KUBE_GO_PACKAGE}/${subdir}" || return 1 - - # Set GOROOT so binaries that parse code can work properly. - GOROOT=$(go env GOROOT) - export GOROOT - # Unset GOBIN in case it already exists in the current session. # Cross-compiles will not work with it set. unset GOBIN - # This seems to matter to some tools - export GO15VENDOREXPERIMENT=1 + # Explicitly turn on modules. + export GO111MODULE=on # GOMAXPROCS by default does not reflect the number of cpu(s) available # when running in a container, please see golang/go#33803 @@ -58,8 +46,6 @@ # shellcheck disable=SC2164 popd >/dev/null fi - - GOMAXPROCS=${GOMAXPROCS:-$(ncpu)} - export GOMAXPROCS - kube::log::status "Setting GOMAXPROCS: ${GOMAXPROCS}" + export GOMAXPROCS=${GOMAXPROCS:-$(ncpu)} + V=3 kube::log::status "Setting GOMAXPROCS: ${GOMAXPROCS}" } ``` Result: `make` is the same as previous commit ``` $ export PATH=/home/thockin/sdk/gotip/bin:$PATH $ export FORCE_HOST_GO=true $ make kubectl +++ [0527 10:25:24] Building go targets for linux/amd64 k8s.io/kubernetes/cmd/kubectl (static) $ make WHAT=./cmd/kubectl/ +++ [0527 10:25:57] Building go targets for linux/amd64 k8s.io/kubernetes/./cmd/kubectl/ (non-static) $ make WHAT=k8s.io/kubernetes/cmd/kubectl +++ [0527 10:26:29] Building go targets for linux/amd64 k8s.io/kubernetes/cmd/kubectl (static) $ make test WHAT=./cmd/kubectl +++ [0527 10:26:59] Setting GOMAXPROCS: 12 +++ [0527 10:26:59] Running tests without code coverage and with -race cmd/kubectl/kubectl.go:25:2: cannot find package "k8s.io/client-go/plugin/pkg/client/auth" in any of: /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/plugin/pkg/client/auth (vendor tree) /home/thockin/sdk/gotip/src/k8s.io/client-go/plugin/pkg/client/auth (from $GOROOT) /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/client-go/plugin/pkg/client/auth (from $GOPATH) cmd/kubectl/kubectl.go:20:2: cannot find package "k8s.io/component-base/cli" in any of: /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/component-base/cli (vendor tree) /home/thockin/sdk/gotip/src/k8s.io/component-base/cli (from $GOROOT) /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/component-base/cli (from $GOPATH) cmd/kubectl/kubectl.go:21:2: cannot find package "k8s.io/kubectl/pkg/cmd" in any of: /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd (vendor tree) /home/thockin/sdk/gotip/src/k8s.io/kubectl/pkg/cmd (from $GOROOT) /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubectl/pkg/cmd (from $GOPATH) cmd/kubectl/kubectl.go:22:2: cannot find package "k8s.io/kubectl/pkg/cmd/util" in any of: /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util (vendor tree) /home/thockin/sdk/gotip/src/k8s.io/kubectl/pkg/cmd/util (from $GOROOT) /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubectl/pkg/cmd/util (from $GOPATH) make: *** [Makefile:184: test] Error 1 $ make test WHAT=k8s.io/kubernetes/cmd/kubectl +++ [0527 10:27:13] Setting GOMAXPROCS: 12 +++ [0527 10:27:13] Running tests without code coverage and with -race cmd/kubectl/kubectl.go:25:2: cannot find package "k8s.io/client-go/plugin/pkg/client/auth" in any of: /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/plugin/pkg/client/auth (vendor tree) /home/thockin/sdk/gotip/src/k8s.io/client-go/plugin/pkg/client/auth (from $GOROOT) /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/client-go/plugin/pkg/client/auth (from $GOPATH) cmd/kubectl/kubectl.go:20:2: cannot find package "k8s.io/component-base/cli" in any of: /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/component-base/cli (vendor tree) /home/thockin/sdk/gotip/src/k8s.io/component-base/cli (from $GOROOT) /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/component-base/cli (from $GOPATH) cmd/kubectl/kubectl.go:21:2: cannot find package "k8s.io/kubectl/pkg/cmd" in any of: /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd (vendor tree) /home/thockin/sdk/gotip/src/k8s.io/kubectl/pkg/cmd (from $GOROOT) /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubectl/pkg/cmd (from $GOPATH) cmd/kubectl/kubectl.go:22:2: cannot find package "k8s.io/kubectl/pkg/cmd/util" in any of: /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util (vendor tree) /home/thockin/sdk/gotip/src/k8s.io/kubectl/pkg/cmd/util (from $GOROOT) /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubectl/pkg/cmd/util (from $GOPATH) make: *** [Makefile:184: test] Error 1 $ make WHAT=./staging/src/k8s.io/api +++ [0527 10:27:28] Building go targets for linux/amd64 k8s.io/kubernetes/./staging/src/k8s.io/api (non-static) cannot find module providing package k8s.io/kubernetes/staging/src/k8s.io/api: import lookup disabled by -mod=vendor (Go version in go.work is at least 1.14 and vendor directory exists.) !!! [0527 10:27:28] Call tree: !!! [0527 10:27:28] 1: /home/thockin/src/kubernetes/hack/lib/golang.sh:846 kube::golang::build_some_binaries(...) !!! [0527 10:27:28] 2: /home/thockin/src/kubernetes/hack/lib/golang.sh:1008 kube::golang::build_binaries_for_platform(...) !!! [0527 10:27:28] 3: hack/make-rules/build.sh:27 kube::golang::build_binaries(...) !!! [0527 10:27:28] Call tree: !!! [0527 10:27:28] 1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...) !!! [0527 10:27:28] Call tree: !!! [0527 10:27:28] 1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...) make: *** [Makefile:92: all] Error 1 $ make WHAT=k8s.io/api +++ [0527 10:27:40] Building go targets for linux/amd64 k8s.io/api (non-static) $ make test WHAT=./staging/src/k8s.io/api +++ [0527 10:27:56] Setting GOMAXPROCS: 12 +++ [0527 10:27:56] Running tests without code coverage and with -race code in directory /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/staging/src/k8s.io/api expects import "k8s.io/api" make: *** [Makefile:184: test] Error 1 $ make test WHAT=k8s.io/api +++ [0527 10:28:14] Setting GOMAXPROCS: 12 !!! [0527 10:28:14] specified test path '${GOPATH}/src/k8s.io/api' does not exist make: *** [Makefile:184: test] Error 1 ```

This makes "new" and "old" setup_env functions. In subsequent commits, all callers of the "old" form will be fixed, and the "new" will be renamed back. The old and new functions diff: ```diff --- /tmp/old 2023-05-06 12:03:23.913951720 -0700 +++ /tmp/new 2023-05-06 12:06:37.688300514 -0700 --- a 2023-05-27 10:20:55.477856825 -0700 +++ b 2023-05-27 10:23:13.253192669 -0700 @@ -1,4 +1,4 @@ -kube::golang::old::setup_env() { +kube::golang::new::setup_env() { kube::golang::verify_go_version # Set up GOPATH. We have tools which depend on being in a GOPATH (see @@ -7,9 +7,9 @@ # Even in module mode, we need to set GOPATH for `go build` and `go install` # to work. We build various tools (usually via `go install`) from a lot of # scripts. - # * We can't set GOBIN because that does not work on cross-compiles. - # * We could use `go build -o <something>`, but it's subtle when it comes - # to cross-compiles and whether the <something> is a file or a directory, + # * We can't just set GOBIN because that does not work on cross-compiles. + # * We could always use `go build -o <something>`, but it's subtle wrt + # cross-compiles and whether the <something> is a file or a directory, # and EVERY caller has to get it *just* right. # * We could leave GOPATH alone and let `go install` write binaries # wherever the user's GOPATH says (or doesn't say). @@ -20,7 +20,6 @@ # # Eventually, when we no longer rely on run-in-gopath.sh we may be able to # simplify this some. - kube::golang::old::create_gopath_tree export GOPATH="${KUBE_GOPATH}" # If these are not set, set them now. This ensures that any subsequent @@ -31,23 +30,12 @@ # Make sure our own Go binaries are in PATH. export PATH="${KUBE_GOPATH}/bin:${PATH}" - # Change directories so that we are within the GOPATH. Some tools get really - # upset if this is not true. We use a whole fake GOPATH here to collect the - # resultant binaries. - local subdir - subdir=$(kube::realpath . | sed "s|${KUBE_ROOT}||") - cd "${KUBE_GOPATH}/src/${KUBE_GO_PACKAGE}/${subdir}" || return 1 - - # Set GOROOT so binaries that parse code can work properly. - GOROOT=$(go env GOROOT) - export GOROOT - # Unset GOBIN in case it already exists in the current session. # Cross-compiles will not work with it set. unset GOBIN - # This seems to matter to some tools - export GO15VENDOREXPERIMENT=1 + # Explicitly turn on modules. + export GO111MODULE=on # GOMAXPROCS by default does not reflect the number of cpu(s) available # when running in a container, please see golang/go#33803 @@ -58,8 +46,6 @@ # shellcheck disable=SC2164 popd >/dev/null fi - - GOMAXPROCS=${GOMAXPROCS:-$(ncpu)} - export GOMAXPROCS - kube::log::status "Setting GOMAXPROCS: ${GOMAXPROCS}" + export GOMAXPROCS=${GOMAXPROCS:-$(ncpu)} + V=3 kube::log::status "Setting GOMAXPROCS: ${GOMAXPROCS}" } ``` Result: `make` is the same as previous commit ``` $ make kubectl +++ [0527 10:25:24] Building go targets for linux/amd64 k8s.io/kubernetes/cmd/kubectl (static) $ make WHAT=./cmd/kubectl/ +++ [0527 10:25:57] Building go targets for linux/amd64 k8s.io/kubernetes/./cmd/kubectl/ (non-static) $ make WHAT=k8s.io/kubernetes/cmd/kubectl +++ [0527 10:26:29] Building go targets for linux/amd64 k8s.io/kubernetes/cmd/kubectl (static) $ make test WHAT=./cmd/kubectl +++ [0527 10:26:59] Setting GOMAXPROCS: 12 +++ [0527 10:26:59] Running tests without code coverage and with -race cmd/kubectl/kubectl.go:25:2: cannot find package "k8s.io/client-go/plugin/pkg/client/auth" in any of: /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/plugin/pkg/client/auth (vendor tree) /home/thockin/sdk/gotip/src/k8s.io/client-go/plugin/pkg/client/auth (from $GOROOT) /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/client-go/plugin/pkg/client/auth (from $GOPATH) cmd/kubectl/kubectl.go:20:2: cannot find package "k8s.io/component-base/cli" in any of: /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/component-base/cli (vendor tree) /home/thockin/sdk/gotip/src/k8s.io/component-base/cli (from $GOROOT) /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/component-base/cli (from $GOPATH) cmd/kubectl/kubectl.go:21:2: cannot find package "k8s.io/kubectl/pkg/cmd" in any of: /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd (vendor tree) /home/thockin/sdk/gotip/src/k8s.io/kubectl/pkg/cmd (from $GOROOT) /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubectl/pkg/cmd (from $GOPATH) cmd/kubectl/kubectl.go:22:2: cannot find package "k8s.io/kubectl/pkg/cmd/util" in any of: /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util (vendor tree) /home/thockin/sdk/gotip/src/k8s.io/kubectl/pkg/cmd/util (from $GOROOT) /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubectl/pkg/cmd/util (from $GOPATH) make: *** [Makefile:184: test] Error 1 $ make test WHAT=k8s.io/kubernetes/cmd/kubectl +++ [0527 10:27:13] Setting GOMAXPROCS: 12 +++ [0527 10:27:13] Running tests without code coverage and with -race cmd/kubectl/kubectl.go:25:2: cannot find package "k8s.io/client-go/plugin/pkg/client/auth" in any of: /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/plugin/pkg/client/auth (vendor tree) /home/thockin/sdk/gotip/src/k8s.io/client-go/plugin/pkg/client/auth (from $GOROOT) /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/client-go/plugin/pkg/client/auth (from $GOPATH) cmd/kubectl/kubectl.go:20:2: cannot find package "k8s.io/component-base/cli" in any of: /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/component-base/cli (vendor tree) /home/thockin/sdk/gotip/src/k8s.io/component-base/cli (from $GOROOT) /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/component-base/cli (from $GOPATH) cmd/kubectl/kubectl.go:21:2: cannot find package "k8s.io/kubectl/pkg/cmd" in any of: /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd (vendor tree) /home/thockin/sdk/gotip/src/k8s.io/kubectl/pkg/cmd (from $GOROOT) /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubectl/pkg/cmd (from $GOPATH) cmd/kubectl/kubectl.go:22:2: cannot find package "k8s.io/kubectl/pkg/cmd/util" in any of: /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util (vendor tree) /home/thockin/sdk/gotip/src/k8s.io/kubectl/pkg/cmd/util (from $GOROOT) /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubectl/pkg/cmd/util (from $GOPATH) make: *** [Makefile:184: test] Error 1 $ make WHAT=./staging/src/k8s.io/api +++ [0527 10:27:28] Building go targets for linux/amd64 k8s.io/kubernetes/./staging/src/k8s.io/api (non-static) cannot find module providing package k8s.io/kubernetes/staging/src/k8s.io/api: import lookup disabled by -mod=vendor (Go version in go.work is at least 1.14 and vendor directory exists.) !!! [0527 10:27:28] Call tree: !!! [0527 10:27:28] 1: /home/thockin/src/kubernetes/hack/lib/golang.sh:846 kube::golang::build_some_binaries(...) !!! [0527 10:27:28] 2: /home/thockin/src/kubernetes/hack/lib/golang.sh:1008 kube::golang::build_binaries_for_platform(...) !!! [0527 10:27:28] 3: hack/make-rules/build.sh:27 kube::golang::build_binaries(...) !!! [0527 10:27:28] Call tree: !!! [0527 10:27:28] 1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...) !!! [0527 10:27:28] Call tree: !!! [0527 10:27:28] 1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...) make: *** [Makefile:92: all] Error 1 $ make WHAT=k8s.io/api +++ [0527 10:27:40] Building go targets for linux/amd64 k8s.io/api (non-static) $ make test WHAT=./staging/src/k8s.io/api +++ [0527 10:27:56] Setting GOMAXPROCS: 12 +++ [0527 10:27:56] Running tests without code coverage and with -race code in directory /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/staging/src/k8s.io/api expects import "k8s.io/api" make: *** [Makefile:184: test] Error 1 $ make test WHAT=k8s.io/api +++ [0527 10:28:14] Setting GOMAXPROCS: 12 !!! [0527 10:28:14] specified test path '${GOPATH}/src/k8s.io/api' does not exist make: *** [Makefile:184: test] Error 1 ```

This makes "new" and "old" setup_env functions. In subsequent commits, all callers of the "old" form will be fixed, and the "new" will be renamed back. The old and new functions diff: ```diff --- /tmp/old 2023-05-06 12:03:23.913951720 -0700 +++ /tmp/new 2023-05-06 12:06:37.688300514 -0700 --- a 2023-05-27 10:20:55.477856825 -0700 +++ b 2023-05-27 10:23:13.253192669 -0700 @@ -1,4 +1,4 @@ -kube::golang::old::setup_env() { +kube::golang::new::setup_env() { kube::golang::verify_go_version # Set up GOPATH. We have tools which depend on being in a GOPATH (see @@ -7,9 +7,9 @@ # Even in module mode, we need to set GOPATH for `go build` and `go install` # to work. We build various tools (usually via `go install`) from a lot of # scripts. - # * We can't set GOBIN because that does not work on cross-compiles. - # * We could use `go build -o <something>`, but it's subtle when it comes - # to cross-compiles and whether the <something> is a file or a directory, + # * We can't just set GOBIN because that does not work on cross-compiles. + # * We could always use `go build -o <something>`, but it's subtle wrt + # cross-compiles and whether the <something> is a file or a directory, # and EVERY caller has to get it *just* right. # * We could leave GOPATH alone and let `go install` write binaries # wherever the user's GOPATH says (or doesn't say). @@ -20,7 +20,6 @@ # # Eventually, when we no longer rely on run-in-gopath.sh we may be able to # simplify this some. - kube::golang::old::create_gopath_tree export GOPATH="${KUBE_GOPATH}" # If these are not set, set them now. This ensures that any subsequent @@ -31,23 +30,12 @@ # Make sure our own Go binaries are in PATH. export PATH="${KUBE_GOPATH}/bin:${PATH}" - # Change directories so that we are within the GOPATH. Some tools get really - # upset if this is not true. We use a whole fake GOPATH here to collect the - # resultant binaries. - local subdir - subdir=$(kube::realpath . | sed "s|${KUBE_ROOT}||") - cd "${KUBE_GOPATH}/src/${KUBE_GO_PACKAGE}/${subdir}" || return 1 - - # Set GOROOT so binaries that parse code can work properly. - GOROOT=$(go env GOROOT) - export GOROOT - # Unset GOBIN in case it already exists in the current session. # Cross-compiles will not work with it set. unset GOBIN - # This seems to matter to some tools - export GO15VENDOREXPERIMENT=1 + # Explicitly turn on modules. + export GO111MODULE=on # GOMAXPROCS by default does not reflect the number of cpu(s) available # when running in a container, please see golang/go#33803 @@ -58,8 +46,6 @@ # shellcheck disable=SC2164 popd >/dev/null fi - - GOMAXPROCS=${GOMAXPROCS:-$(ncpu)} - export GOMAXPROCS - kube::log::status "Setting GOMAXPROCS: ${GOMAXPROCS}" + export GOMAXPROCS=${GOMAXPROCS:-$(ncpu)} + V=3 kube::log::status "Setting GOMAXPROCS: ${GOMAXPROCS}" } ``` Result: `make` works for k/k: ``` $ make kubectl +++ [1211 11:07:31] Building go targets for linux/amd64 k8s.io/kubernetes/cmd/kubectl (static) $ make WHAT=./cmd/kubectl/ +++ [1211 11:08:19] Building go targets for linux/amd64 k8s.io/kubernetes/./cmd/kubectl/ (non-static) $ make WHAT=k8s.io/kubernetes/cmd/kubectl +++ [1211 11:08:52] Building go targets for linux/amd64 k8s.io/kubernetes/cmd/kubectl (static) ``` Result: `make` works for stagin by package: ``` $ make WHAT=k8s.io/api +++ [1211 11:11:37] Building go targets for linux/amd64 k8s.io/api (non-static) ``` Result: `make` fails for staging by path: ``` $ make WHAT=./staging/src/k8s.io/api +++ [1211 11:12:44] Building go targets for linux/amd64 k8s.io/kubernetes/./staging/src/k8s.io/api (non-static) cannot find module providing package k8s.io/kubernetes/staging/src/k8s.io/api: import lookup disabled by -mod=vendor (Go version in go.work is at least 1.14 and vendor directory exists.) !!! [1211 11:12:44] Call tree: !!! [1211 11:12:44] 1: /home/thockin/src/kubernetes/hack/lib/golang.sh:850 kube::golang::build_some_binaries(...) !!! [1211 11:12:44] 2: /home/thockin/src/kubernetes/hack/lib/golang.sh:1012 kube::golang::build_binaries_for_platform(...) !!! [1211 11:12:44] 3: hack/make-rules/build.sh:27 kube::golang::build_binaries(...) !!! [1211 11:12:44] Call tree: !!! [1211 11:12:44] 1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...) !!! [1211 11:12:44] Call tree: !!! [1211 11:12:44] 1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...) make: *** [Makefile:96: all] Error 1 ``` Result: `make test` fails: ``` $ make test WHAT=./cmd/kubectl +++ [1211 11:13:38] Set GOMAXPROCS automatically to 6 +++ [1211 11:13:38] Running tests without code coverage and with -race cmd/kubectl/kubectl.go:25:2: cannot find package "k8s.io/client-go/plugin/pkg/client/auth" in any of: /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/plugin/pkg/client/auth (vendor tree) /home/thockin/sdk/gotip/src/k8s.io/client-go/plugin/pkg/client/auth (from $GOROOT) /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/client-go/plugin/pkg/client/auth (from $GOPATH) cmd/kubectl/kubectl.go:20:2: cannot find package "k8s.io/component-base/cli" in any of: /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/component-base/cli (vendor tree) /home/thockin/sdk/gotip/src/k8s.io/component-base/cli (from $GOROOT) /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/component-base/cli (from $GOPATH) cmd/kubectl/kubectl.go:21:2: cannot find package "k8s.io/kubectl/pkg/cmd" in any of: /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd (vendor tree) /home/thockin/sdk/gotip/src/k8s.io/kubectl/pkg/cmd (from $GOROOT) /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubectl/pkg/cmd (from $GOPATH) cmd/kubectl/kubectl.go:22:2: cannot find package "k8s.io/kubectl/pkg/cmd/util" in any of: /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util (vendor tree) /home/thockin/sdk/gotip/src/k8s.io/kubectl/pkg/cmd/util (from $GOROOT) /home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubectl/pkg/cmd/util (from $GOPATH) make: *** [Makefile:191: test] Error 1 ```

gopherbot added this to the Proposal milestone Aug 23, 2019

gopherbot added the Proposal label Aug 23, 2019

This comment has been minimized.

Sign in to view

ianlancetaylor changed the title ~~proposal: make GOMAXPROCS cfs-aware on GOOS=linux~~ runtime: make GOMAXPROCS cfs-aware on GOOS=linux Sep 3, 2019

ianlancetaylor added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. and removed Proposal labels Sep 3, 2019

ianlancetaylor modified the milestones: Proposal, Go1.14 Sep 3, 2019

This comment was marked as outdated.

Sign in to view

griesemer mentioned this issue Sep 3, 2019

proposal: review meeting minutes #33502

Open

rsc modified the milestones: Go1.14, Backlog Oct 9, 2019

DavadDi mentioned this issue Dec 6, 2019

The variable GOMAXPROCS does not work properly in Container environment prometheus/prometheus#6413

Closed

jcorbin mentioned this issue Jan 7, 2020

Support cgroups v2 and its unified hierarchy uber-go/automaxprocs#21

Closed

alphastorm mentioned this issue May 25, 2020

Automatically set GOMAXPROCS to match Linux container CPU quota keep-network/keep-core#1827

Open

julz mentioned this issue Aug 5, 2020

Automatically set GOMAXPROCS in sharedmain knative/pkg#1583

Closed

drwatsno mentioned this issue Aug 31, 2020

Automatically set GOMAXPROCS according to linux container cpu quota ory/hydra#2033

Closed

islishude mentioned this issue Nov 15, 2020

容器资源可见性问题与 GOMAXPROCS 配置 islishude/blog#216

Open

uhthomas mentioned this issue Mar 5, 2021

automaxprocs? buildbarn/bb-storage#111

Closed

ivan-valkov mentioned this issue Aug 6, 2021

Auto set GOMAXPROCS based on container limits in executor and operator SeldonIO/seldon-core#3471

Merged

bt90 mentioned this issue Jan 19, 2024

Set GOMAXPROCS quota aware in Linux containers syncthing/syncthing#9357

Closed

seankhliao mentioned this issue Feb 11, 2024

runtime: make the proportion of CPU the GC uses based on actual available CPU time and not GOMAXPROCS #59715

Open

serathius mentioned this issue Feb 15, 2024

feat(helm chart): Make env vars configurable and auto configure go runtime kubernetes-sigs/metrics-server#1412

Closed

jtackaberry mentioned this issue Jun 13, 2024

Investigate autosetting GOMAXPROCS and GOMEMLIMIT env vars rqlite/helm-charts#27

Open

f9n mentioned this issue Jul 22, 2024

Alloy should be cgroup aware grafana/alloy#1340

Open

jharveyb mentioned this issue Sep 16, 2024

fn+proof: use GOMAXPROCS for worker pool size lightninglabs/taproot-assets#1123

Merged

DarVoid mentioned this issue Sep 26, 2024

ref: GOMAXPROCS=1 antonputra/tutorials#288

Merged

patrickdappollonio mentioned this issue Oct 24, 2024

Add support for appropriate GOMAXPROCS that's aware of Kubernetes limits. patrickdappollonio/http-server#136

Merged

uhthomas mentioned this issue Nov 5, 2024

Set number of workers to allocated cgroup CPUs? envoyproxy/envoy#36988

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime: make `GOMAXPROCS` cfs-aware on `GOOS=linux` #33803

runtime: make `GOMAXPROCS` cfs-aware on `GOOS=linux` #33803

jcorbin commented Aug 23, 2019 •

edited

Loading

jcorbin commented Aug 23, 2019

jcorbin commented Aug 23, 2019

jcorbin commented Aug 23, 2019

This comment has been minimized.

This comment has been minimized.

ianlancetaylor commented Aug 23, 2019

ianlancetaylor commented Sep 3, 2019

This comment was marked as outdated.

runtime: make GOMAXPROCS cfs-aware on GOOS=linux #33803

runtime: make GOMAXPROCS cfs-aware on GOOS=linux #33803

Comments

jcorbin commented Aug 23, 2019 • edited Loading

Problem

Solution

NOTEs

jcorbin commented Aug 23, 2019

jcorbin commented Aug 23, 2019

jcorbin commented Aug 23, 2019

This comment has been minimized.

This comment has been minimized.

ianlancetaylor commented Aug 23, 2019

ianlancetaylor commented Sep 3, 2019

This comment was marked as outdated.

runtime: make `GOMAXPROCS` cfs-aware on `GOOS=linux` #33803

runtime: make `GOMAXPROCS` cfs-aware on `GOOS=linux` #33803

jcorbin commented Aug 23, 2019 •

edited

Loading