cmd/link: fatal error: runtime: out of memory

[hska06]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.10.3 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/hs/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/hs/go"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build755635740=/tmp/go-build -gno-record-gcc-switches"

What did you do?

$ go get -d k8s.io/kubernetes
$ cd $GOPATH/src/k8s.io/kubernetes
$ make

What did you expect to see?

make to pass

What did you see instead?

:~/go/src/k8s.io/kubernetes$ make
+++ [0626 06:46:31] Building go targets for linux/amd64:
./vendor/k8s.io/code-generator/cmd/deepcopy-gen
+++ [0626 06:46:47] Building go targets for linux/amd64:
./vendor/k8s.io/code-generator/cmd/defaulter-gen
+++ [0626 06:46:55] Building go targets for linux/amd64:
./vendor/k8s.io/code-generator/cmd/conversion-gen
+++ [0626 06:47:04] Building go targets for linux/amd64:
./vendor/k8s.io/code-generator/cmd/openapi-gen
+++ [0626 06:47:16] Building go targets for linux/amd64:
./vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0626 06:47:18] Building go targets for linux/amd64:
cmd/kube-proxy
cmd/kube-apiserver
cmd/kube-controller-manager
cmd/cloud-controller-manager
cmd/kubelet
cmd/kubeadm
cmd/hyperkube
cmd/kube-scheduler
vendor/k8s.io/kube-aggregator
vendor/k8s.io/apiextensions-apiserver
cluster/gce/gci/mounter
cmd/kubectl
cmd/gendocs
cmd/genkubedocs
cmd/genman
cmd/genyaml
cmd/genswaggertypedocs
cmd/linkcheck
vendor/github.com/onsi/ginkgo/ginkgo
test/e2e/e2e.test
cmd/kubemark
vendor/github.com/onsi/ginkgo/ginkgo
test/e2e_node/e2e_node.test
/usr/local/go/pkg/tool/linux_amd64/link: signal: killed

k8s.io/kubernetes/cmd/genman
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x5fda68, 0x16)
/usr/local/go/src/runtime/panic.go:616 +0x81
runtime.sysMap(0xc4674d0000, 0x100000, 0x548f00, 0x7178d8)
/usr/local/go/src/runtime/mem_linux.go:216 +0x20a
runtime.(*mheap).sysAlloc(0x6fef40, 0x100000, 0x7f745d115c98)
/usr/local/go/src/runtime/malloc.go:470 +0xd4
runtime.(*mheap).grow(0x6fef40, 0x1f, 0x0)
/usr/local/go/src/runtime/mheap.go:907 +0x60
runtime.(*mheap).allocSpanLocked(0x6fef40, 0x1f, 0x7178e8, 0x7ffd7fcc8a40)
/usr/local/go/src/runtime/mheap.go:820 +0x301
runtime.(*mheap).alloc_m(0x6fef40, 0x1f, 0x410100, 0xc41dc5a3ff)
/usr/local/go/src/runtime/mheap.go:686 +0x118
runtime.(*mheap).alloc.func1()
/usr/local/go/src/runtime/mheap.go:753 +0x4d
runtime.(*mheap).alloc(0x6fef40, 0x1f, 0x7ffd7f010100, 0x412e8c)
/usr/local/go/src/runtime/mheap.go:752 +0x8a
runtime.largeAlloc(0x3c8c0, 0x440001, 0x7f745da5a6c8)
/usr/local/go/src/runtime/malloc.go:826 +0x94
runtime.mallocgc.func1()
/usr/local/go/src/runtime/malloc.go:721 +0x46
runtime.systemstack(0x0)
/usr/local/go/src/runtime/asm_amd64.s:409 +0x79
runtime.mstart()
/usr/local/go/src/runtime/proc.go:1175

goroutine 1 [running]:
runtime.systemstack_switch()
/usr/local/go/src/runtime/asm_amd64.s:363 fp=0xc42062e390 sp=0xc42062e388 pc=0x451b50
runtime.mallocgc(0x3c8c0, 0x5f2780, 0x301, 0xc4552f01a0)
/usr/local/go/src/runtime/malloc.go:720 +0x8a2 fp=0xc42062e430 sp=0xc42062e390 pc=0x40f5c2
runtime.makeslice(0x5f2780, 0x3e8, 0x3e8, 0xc5, 0x7174a0, 0xc466a34c00)
/usr/local/go/src/runtime/slice.go:61 +0x77 fp=0xc42062e460 sp=0xc42062e430 pc=0x43e247
cmd/link/internal/sym.(*Symbols).Newsym(...)
/usr/local/go/src/cmd/link/internal/sym/symbols.go:56
cmd/link/internal/sym.(*Symbols).Lookup(0xc42050e000, 0xc4674b0ea0, 0xc5, 0x0, 0xc4674967c8)
/usr/local/go/src/cmd/link/internal/sym/symbols.go:79 +0x299 fp=0xc42062e4c8 sp=0xc42062e460 pc=0x4d71a9
cmd/link/internal/objfile.(*objReader).readRef(0xc42062e8a0)
/usr/local/go/src/cmd/link/internal/objfile/objfile.go:390 +0x10d fp=0xc42062e5f0 sp=0xc42062e4c8 pc=0x4f7b0d
cmd/link/internal/objfile.(*objReader).loadObjFile(0xc42062e8a0)
/usr/local/go/src/cmd/link/internal/objfile/objfile.go:109 +0x2ce fp=0xc42062e7e8 sp=0xc42062e5f0 pc=0x4f4f3e
cmd/link/internal/objfile.Load(0x6ed3a0, 0xc42050e000, 0xc436847c50, 0xc44341c240, 0x88bb8, 0xc447614150, 0x2b)
/usr/local/go/src/cmd/link/internal/objfile/objfile.go:69 +0x1a1 fp=0xc42062e9c8 sp=0xc42062e7e8 pc=0x4f4a61
cmd/link/internal/ld.ldobj(0xc420512000, 0xc436847c50, 0xc44341c240, 0x88c24, 0xc447614150, 0x2b, 0xc4205a8470, 0x23, 0x1, 0x0)
/usr/local/go/src/cmd/link/internal/ld/lib.go:1515 +0xcb1 fp=0xc42062ec80 sp=0xc42062e9c8 pc=0x54f611
cmd/link/internal/ld.loadobjfile(0xc420512000, 0xc44341c240)
/usr/local/go/src/cmd/link/internal/ld/lib.go:839 +0x50d fp=0xc42062ee68 sp=0xc42062ec80 pc=0x548fbd
cmd/link/internal/ld.(*Link).loadlib(0xc420512000)
/usr/local/go/src/cmd/link/internal/ld/lib.go:365 +0x183 fp=0xc42062f110 sp=0xc42062ee68 pc=0x545a33
cmd/link/internal/ld.Main(0x6ed3a0, 0x10, 0x20, 0x1, 0x7, 0x10, 0x5ff52e, 0x1b, 0x5fc864, 0x14, ...)
/usr/local/go/src/cmd/link/internal/ld/main.go:201 +0xa73 fp=0xc42062f2a0 sp=0xc42062f110 pc=0x55d4d3
main.main()
/usr/local/go/src/cmd/link/main.go:62 +0x277 fp=0xc42062ff88 sp=0xc42062f2a0 pc=0x594ad7
runtime.main()
/usr/local/go/src/runtime/proc.go:198 +0x212 fp=0xc42062ffe0 sp=0xc42062ff88 pc=0x429f62
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42062ffe8 sp=0xc42062ffe0 pc=0x454541
!!! [0626 06:57:13] Call tree:
!!! [0626 06:57:13] 1: /home/hs/go/src/k8s.io/kubernetes/hack/lib/golang.sh:610 kube::golang::build_binaries_for_platform(...)
!!! [0626 06:57:13] 2: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0626 06:57:13] Call tree:
!!! [0626 06:57:13] 1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0626 06:57:13] Call tree:
!!! [0626 06:57:13] 1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
Makefile:92: recipe for target 'all' failed
make: *** [all] Error 1

[mvdan]

I'm not sure what the bug you are reporting here is. If the Go program you are running runs out of memory, it will panic.

Unless this issue is about the compiler unexpectedly running out of memory. If so, please give details of exactly what go build or go install command fails for you, and how much free memory your system had when you ran the command.

[ianlancetaylor]

As far as I can tell, the linker ran out of memory while executing.

[cherrymui]

cc @thanm @aclements

[hska06]

@mvdan was expecting linker to handle available memory rather than expecting more memory.

[mvdan]

@harsha-s how much memory was available? this is very likely a bug if you had a spare 8GB, but may not be if there was little memory left.

[hska06]

@mvdan had 4GB when I tried

[agnivade]

@harsha-s - Just to clarify, you mean you had 4GB of memory still left when the crash happened ? Or you had a total of 4GB RAM in your system ?

If it is the former, just to rule out any unrelated issues, could you post the output of ulimit -a on your system ?

Also, if possible, try to trigger a coredump of the build process (set GOTRACEBACK=crash) so that we get more info.

[hska06]

@agnivade

had a total of 4GB RAM in your system.
setting GOTRACEBACK=crash produced the same output as mentioned in issue

hs@ubuntu:~/go/src/k8s.io/kubernetes$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 15231
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 15231
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimit

[agnivade]

had a total of 4GB RAM in your system.

Thanks. Then I would guess your system itself ran out of available memory ? Or do you see available memory when the crash happens ?

setting GOTRACEBACK=crash produced the same output

You need to enable coredumps in your system. Your ulimit -c is 0. Set it to unlimited.

[ianlancetaylor]

Closing in favor of #26186 which is about reducing linker memory requirements.