cmd/go: cgo crashes with GO15VENDOREXPERIMENT=1

[michaeledgar]

When performing a clean build of our project with GO15VENDOREXPERIMENT=1 we need to run "go install" twice. The first time always fails in cgo-related compile or link steps. The second time always works fine.

I'm running "go version go1.5beta2 linux/amd64".

The error from cgo is usually different from run to run. I've also seen cgo crash with a nil pointer error. These are the environment variables we set when building:

BUILD=//home/adgar/go/src//vendor/libgit2/build
PCFILE=//home/adgar/go/src//vendor/libgit2/build/libgit2.pc
PKG_CONFIG_PATH=//home/adgar/go/src//vendor/libgit2/build:
FLAGS='-L//home/adgar/go/src//vendor/libgit2/install/lib -lgit2 -lcurl -lrt -lssl -lcrypto -ldl -lz '
CGO_LDFLAGS='//home/adgar/go/src//vendor/libgit2/build/libgit2.a -L//home/adgar/go/src//vendor/libgit2/build -L//home/adgar/go/src//vendor/libgit2/install/lib -lgit2 -lcurl -lrt -lssl -lcrypto -ldl -lz '
CGO_CFLAGS=-I//home/adgar/go/src//vendor/libgit2/include

Here are 3 example errors I've seen in the past ~5 minutes of reproducing the issue.

[run 1]
cannot parse gcc output $WORK//vendor/github.com/libgit2/git2go/_obj//cgo.o as ELF, Mach-O, PE object

[run 2]
vendor/github.com/libgit2/git2go/diff.go:20:31: unable to find value of constant C.GIT_DIFF_FLAG_BINARY
vendor/github.com/libgit2/git2go/diff.go:21:31: unable to find value of constant C.GIT_DIFF_FLAG_NOT_BINARY
vendor/github.com/libgit2/git2go/diff.go:22:31: unable to find value of constant C.GIT_DIFF_FLAG_VALID_ID
vendor/github.com/libgit2/git2go/diff.go:42:38: unable to find value of constant C.GIT_DIFF_LINE_CONTEXT
vendor/github.com/libgit2/git2go/diff.go:43:38: unable to find value of constant C.GIT_DIFF_LINE_ADDITION
vendor/github.com/libgit2/git2go/diff.go:44:38: unable to find value of constant C.GIT_DIFF_LINE_DELETION
vendor/github.com/libgit2/git2go/diff.go:45:38: unable to find value of constant C.GIT_DIFF_LINE_CONTEXT_EOFNL
vendor/github.com/libgit2/git2go/diff.go:46:38: unable to find value of constant C.GIT_DIFF_LINE_ADD_EOFNL
vendor/github.com/libgit2/git2go/diff.go:47:38: unable to find value of constant C.GIT_DIFF_LINE_DEL_EOFNL
vendor/github.com/libgit2/git2go/diff.go:49:33: unable to find value of constant C.GIT_DIFF_LINE_FILE_HDR
vendor/github.com/libgit2/git2go/diff.go:50:33: unable to find value of constant C.GIT_DIFF_LINE_HUNK_HDR
vendor/github.com/libgit2/git2go/diff.go:51:33: unable to find value of constant C.GIT_DIFF_LINE_BINARY
vendor/github.com/libgit2/git2go/diff.go:62:26: type C.git_diff_file: undefined C type 'git_diff_file'
vendor/github.com/libgit2/git2go/diff.go:64:10: call of non-function C.GoString
vendor/github.com/libgit2/git2go/diff.go:80:28: type C.git_diff_delta: undefined C type 'git_diff_delta'
vendor/github.com/libgit2/git2go/diff.go:98:26: type C.git_diff_hunk: undefined C type 'git_diff_hunk'
vendor/github.com/libgit2/git2go/diff.go:104:13: call of non-function C.GoStringN
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x40ec96]

goroutine 1 [running]:
main.(_Package).rewriteRef(0xc8200942a0, 0xc8202fc300)
//home/adgar/go15/go/src/cmd/cgo/gcc.go:610 +0x1466
main.(_Package).Translate(0xc8200942a0, 0xc8202fc300)
//home/adgar/go15/go/src/cmd/cgo/gcc.go:170 +0x197
main.main()
//home/adgar/go15/go/src/cmd/cgo/main.go:269 +0x1059

[run 3, mojibaked]

/cmd/ws-daemon

//home/adgar/go/pkg/linux_amd64//vendor/github.com/libgit2/git2go.�): not an object file
//home/adgar/go/pkg/linux_amd64//vendor/github.com/libgit2/git2go.a���): not an object file
//home/adgar/go/pkg/linux_amd64//vendor/github.com/libgit2/git2go.aþ �): not an object file
//home/adgar/go/pkg/linux_amd64//vendor/github.com/libgit2/git2go.a(ü ): not an object file
//home/adgar/go/pkg/linux_amd64//vendor/github.com/libgit2/git2go.a(�8
): not an object file
//home/adgar/go/pkg/linux_amd64//vendor/github.com/libgit2/git2go.a(X
): not an object file
//home/adgar/go/pkg/linux_amd64//vendor/github.com/libgit2/git2go.a(ex
): not an object file
//home/adgar/go/pkg/linux_amd64//vendor/github.com/libgit2/git2go.a(�
%p�
): not an object file
//home/adgar/go/pkg/linux_amd64//vendor/github.com/libgit2/git2go.a(�°
2): not an object file
//home/adgar/go/pkg/linux_amd64//vendor/github.com/libgit2/git2go.a( 
¡): not an object file
//home/adgar/go/pkg/linux_amd64//vendor/github.com/libgit2/git2go.a(
): not an object file
//home/adgar/go/pkg/linux_amd64//vendor/github.com/libgit2/git2go.a(): not an object file
//home/adgar/go/pkg/linux_amd64//vendor/github.com/libgit2/git2go.a(E¬): not an object file
//home/adgar/go/pkg/linux_amd64//vendor/github.com/libgit2/git2go.a(W.
): not an object file
//home/adgar/go/pkg/linux_amd64//vendor/github.com/libgit2/git2go.a(
=�): not an object file
//home/adgar/go/pkg/linux_amd64//vendor/github.com/libgit2/git2go.a(£.
): not an object file
//home/adgar/go/pkg/linux_amd64//vendor/github.com/libgit2/git2go.a(
«�): not an object file
//home/adgar/go/pkg/linux_amd64//vendor/github.com/libgit2/git2go.a(0/
): not an object file
//home/adgar/go/pkg/linux_amd64//vendor/github.com/libgit2/git2go.a(
Ə): not an object file
//home/adgar/go/pkg/linux_amd64//vendor/github.com/libgit2/git2go.a(¤/
): not an object file
//home/adgar/go/pkg/linux_amd64//vendor/github.com/libgit2/git2go.a(
��): not an object file
//home/adgar/go15/go/pkg/tool/linux_amd64/link: too many errors

[ianlancetaylor]

Is your environment list complete? In particular, do you have GOROOT set in the environment?

Can you describe a sequence of steps we can use to recreate the problem?

[michaeledgar]

Thanks for the quick response, Ian! rsc@ reached out privately for access to the source repositories necessary to reproduce the issue, which I've granted - I can add you to the projects as well if you'd like.

Here's the output of "go env" - note that GOROOT is indeed set, to my Go 1.5 beta 2 installation. I have Go 1.4 installed but it should be unused - it's on a different mount that isn't referenced below.

GOARCH="amd64"
GOBIN="//home/adgar/go/bin"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="//home/adgar/go"
GORACE=""
GOROOT="//home/adgar/go15/go"
GOTOOLDIR="//home/adgar/go15/go/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT="1"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

[michaeledgar]

As an aside, I'm building the newly released Go 1.5 beta 3 to see if maybe this was already fixed, and will update the issue with the result.

[michaeledgar]

The misbehavior is still present when building with Go 1.5 beta 3.

[gopherbot]

CL https://golang.org/cl/13022 mentions this issue.

[marete]

The fix for this change broke "go get -v all" for me:

$ go get -v all
internal error: duplicate loads of unsafe
internal error: duplicate loads of runtime
internal error: duplicate loads of errors
internal error: duplicate loads of sync/atomic
internal error: duplicate loads of sync
internal error: duplicate loads of io
internal error: duplicate loads of unicode
internal error: duplicate loads of unicode/utf8
internal error: duplicate loads of bytes
internal error: duplicate loads of bufio
internal error: duplicate loads of math
internal error: duplicate loads of strconv
internal error: duplicate loads of reflect
internal error: duplicate loads of encoding/binary
internal error: duplicate loads of syscall
internal error: duplicate loads of time
internal error: duplicate loads of os
internal error: duplicate loads of fmt
internal error: duplicate loads of sort
internal error: duplicate loads of strings
internal error: duplicate loads of path/filepath
internal error: duplicate loads of path
internal error: duplicate loads of io/ioutil
internal error: duplicate loads of flag
internal error: duplicate loads of log
internal error: duplicate loads of cmd/internal/obj
internal error: duplicate loads of cmd/internal/obj/arm
internal error: duplicate loads of cmd/internal/obj/arm64
internal error: duplicate loads of cmd/internal/obj/ppc64
internal error: duplicate loads of cmd/internal/obj/x86
internal error: duplicate loads of cmd/asm/internal/arch
internal error: duplicate loads of cmd/asm/internal/flags
internal error: duplicate loads of text/scanner
internal error: duplicate loads of cmd/asm/internal/lex
internal error: duplicate loads of cmd/asm/internal/asm
internal error: duplicate loads of hash
internal error: duplicate loads of math/rand
internal error: duplicate loads of cmd/compile/internal/big
internal error: duplicate loads of cmd/internal/gcprog
internal error: duplicate loads of crypto
internal error: duplicate loads of crypto/md5
internal error: duplicate loads of text/tabwriter
internal error: duplicate loads of runtime/pprof
internal error: duplicate loads of cmd/compile/internal/gc
internal error: duplicate loads of cmd/compile/internal/amd64
internal error: duplicate loads of cmd/compile/internal/arm
internal error: duplicate loads of cmd/compile/internal/arm64
internal error: duplicate loads of cmd/compile/internal/ppc64
internal error: duplicate loads of cmd/compile/internal/x86
internal error: duplicate loads of compress/flate
internal error: duplicate loads of hash/crc32
internal error: duplicate loads of cmd/internal/goobj
internal error: duplicate loads of cmd/internal/rsc.io/arm/armasm
internal error: duplicate loads of cmd/internal/rsc.io/x86/x86asm
internal error: duplicate loads of debug/dwarf
internal error: duplicate loads of debug/elf
internal error: duplicate loads of debug/gosym
internal error: duplicate loads of debug/macho
internal error: duplicate loads of debug/pe
internal error: duplicate loads of debug/plan9obj
internal error: duplicate loads of regexp/syntax
internal error: duplicate loads of regexp
internal error: duplicate loads of cmd/internal/objfile
internal error: duplicate loads of crypto/sha1
internal error: duplicate loads of os/exec
internal error: duplicate loads of cmd/link/internal/ld
internal error: duplicate loads of cmd/link/internal/amd64
internal error: duplicate loads of cmd/link/internal/arm
internal error: duplicate loads of cmd/link/internal/arm64
internal error: duplicate loads of cmd/link/internal/ppc64
internal error: duplicate loads of cmd/link/internal/x86
internal error: duplicate loads of compress/gzip
internal error: duplicate loads of cmd/pprof/internal/profile
internal error: duplicate loads of cmd/pprof/internal/plugin
internal error: duplicate loads of encoding
internal error: duplicate loads of encoding/base64
internal error: duplicate loads of unicode/utf16
internal error: duplicate loads of encoding/json
internal error: duplicate loads of html
internal error: duplicate loads of net/url
internal error: duplicate loads of text/template/parse
internal error: duplicate loads of text/template
internal error: duplicate loads of html/template
internal error: duplicate loads of cmd/pprof/internal/report
internal error: duplicate loads of cmd/pprof/internal/svg
internal error: duplicate loads of cmd/pprof/internal/tempfile
internal error: duplicate loads of cmd/pprof/internal/commands
internal error: duplicate loads of cmd/pprof/internal/driver
internal error: duplicate loads of container/list
internal error: duplicate loads of crypto/subtle
internal error: duplicate loads of crypto/cipher
internal error: duplicate loads of crypto/aes
internal error: duplicate loads of crypto/des
internal error: duplicate loads of math/big
internal error: duplicate loads of crypto/elliptic
internal error: duplicate loads of crypto/sha512
internal error: duplicate loads of encoding/asn1
internal error: duplicate loads of crypto/ecdsa
internal error: duplicate loads of crypto/hmac
internal error: duplicate loads of internal/syscall/unix
internal error: duplicate loads of crypto/rand
internal error: duplicate loads of crypto/rc4
internal error: duplicate loads of crypto/rsa
internal error: duplicate loads of crypto/sha256
internal error: duplicate loads of crypto/dsa
internal error: duplicate loads of crypto/x509/pkix
internal error: duplicate loads of encoding/hex
internal error: duplicate loads of encoding/pem
internal error: duplicate loads of internal/singleflight
internal error: duplicate loads of runtime/cgo
internal error: duplicate loads of net
internal error: duplicate loads of crypto/x509
internal error: duplicate loads of crypto/tls
internal error: duplicate loads of mime
internal error: duplicate loads of mime/quotedprintable
internal error: duplicate loads of net/textproto
internal error: duplicate loads of mime/multipart
internal error: duplicate loads of net/http/internal
internal error: duplicate loads of net/http
internal error: duplicate loads of cmd/pprof/internal/fetch
internal error: duplicate loads of cmd/pprof/internal/symbolizer
internal error: duplicate loads of cmd/pprof/internal/symbolz
internal error: duplicate loads of cmd/vet/whitelist
internal error: duplicate loads of container/heap
internal error: duplicate loads of encoding/xml
internal error: duplicate loads of go/token
internal error: duplicate loads of go/scanner
internal error: duplicate loads of go/ast
internal error: duplicate loads of go/doc
internal error: duplicate loads of go/parser
internal error: duplicate loads of go/build
internal error: duplicate loads of go/constant
internal error: duplicate loads of go/printer
internal error: duplicate loads of internal/format
internal error: duplicate loads of go/format
internal error: duplicate loads of go/types
internal error: duplicate loads of go/internal/gccgoimporter
internal error: duplicate loads of go/internal/gcimporter
internal error: duplicate loads of go/importer
internal error: duplicate loads of internal/trace
internal error: duplicate loads of os/signal

My environment:

$ go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/marebri/devel/go"
GORACE=""
GOROOT="/opt/go"
GOTOOLDIR="/opt/go/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT="1"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

After re-building Go, I am currently working around this problem by invoking "go install -v all" instead", to re-build my packages.