cmd/go: permission issue building CGO with Go 1.10

[jaricftw]

What version of Go are you using (go version)?

1.10

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/<foo>/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/<foo>/go"
GORACE=""
GOROOT="/usr/lib/go-1.10"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go-1.10/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build884921568=/tmp/go-build -gno-record-gcc-switches"

Intel(R) Xeon(R) CPU E5-2680

What did you do?

We have been running a go program that access rocksdb (written in c) via cgo, and it has been running fine under go 1.9. However, after upgrading to 1.10, we get the following error:

go build runtime/cgo: open /usr/lib/go-1.10/pkg/linux_amd64/runtime/cgo.a: permission denied

And the permissions of the cgo files:

ls -l /usr/lib/go-1.10/pkg/linux_amd64/runtime/
total 720
-rw-r--r-- 1 root root  93876 Mar 29 23:49 cgo.a
-rw-r--r-- 1 root root  15674 Mar 29 23:49 debug.a
drwxr-xr-x 2 root root   4096 Mar 29 23:49 internal
drwxr-xr-x 3 root root   4096 Mar 29 23:49 pprof
-rw-r--r-- 1 root root 231096 Mar 29 23:49 pprof.a
-rw-r--r-- 1 root root 378548 Mar 29 23:49 race.a
-rw-r--r-- 1 root root   2614 Mar 29 23:49 trace.a

[AlexRouSg]

Have you checked the read permissions of the /usr/lib/go-1.10 folder and that specific file?

[davecheney]

How did you install go, it looks like it was provided by your operating system distribution based on the pathnames. Also, please check that you do not have goroot set.

…

On 4 Apr 2018, at 10:45, Chotepud Teo ***@***.***> wrote: Have you checked the read permissions of the /usr/lib/go-1.10 folder and that specific file? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[jaricftw]

@AlexRouSg updated the description. It has read permission so probably something is going on here to write it.

[davecheney]

The reason the go tool would be trying to write to this directory is if the files are considered stale (ie, the .a files do not match the source). In most cases this is because of a corrupt Go installation.

How did you install Go? It looks like it was provided by your operating system distribution.

[jaricftw]

@davecheney So I've found out the issue and a possible fix, but don't know why it only happens after our go upgrade (I don't have enough details about our go upgrade process yet). Basically when we build the program, we've included the -i flags for go build as well as go test, and that somehow tries to recompile cgo and causes the permission issue. After deleting all -i from our build script, everything works fine now. What's your thoughts on this?

[davecheney]

I recommend avoiding the package provided by your os district and using the tarball from the golang.org website, they are well tested and are known to work.

[jaricftw]

@davecheney thanks. I am not convinced that it's the issue with the go installation, as removing the -i flag in go build and go test solves the issues. And we are only have issues with cgo users, not any other Go programs so far.

Based on what I read so far, The -i flag installs the packages that are dependencies of the target, discussed here.

I also noticed with go 1.10, it suggest not to use -i anymore with the introduction of go build cache

The go build command now maintains a cache of recently built packages, separate from the installed packages in $GOROOT/pkg or $GOPATH/pkg. The effect of the cache should be to speed builds that do not explicitly install packages or when switching between different copies of source code (for example, when changing back and forth between different branches in a version control system). The old advice to add the -i flag for speed, as in go build -i or go test -i, is no longer necessary: builds run just as fast without -i.

Given these suggestions, I consider the immediately fix on my side is to remove the -i flags, but I wonder what has changed to go 1.10 that causes writing to cgo in GOROOT.

[AlexRouSg]

I am unable to reproduce this on linux amd64 with 1.10 and 1.10.1 with freshly downloaded binaries and the entire folder set to read only.

[davecheney]

I believe the installation provided by your os vendor is fault. If the timestamps and checksums of the std library .a files were correct then -i would have no effect. But it does, therefore I think the problem lies in your os vendor's packaging.

…

On 6 April 2018 at 14:16, jaricftw ***@***.***> wrote: @davecheney <https://github.com/davecheney> thanks. I am not convinced that it's the issue with the go installation, as removing the -i flag in go build and go test solves the issues. And we are only have issues with cgo users, not any other Go programs so far. Based on what I read so far, The -i flag installs the packages that are dependencies of the target, discussed here <https://golang.org/cmd/go/#hdr-Compile_packages_and_dependencies>. I also noticed with go 1.10, it suggest not to use -i anymore with the introduction of go build cache The go build command now maintains a cache of recently built packages, separate from the installed packages in $GOROOT/pkg or $GOPATH/pkg. The effect of the cache should be to speed builds that do not explicitly install packages or when switching between different copies of source code (for example, when changing back and forth between different branches in a version control system). The old advice to add the -i flag for speed, as in go build -i or go test -i, is no longer necessary: builds run just as fast without -i. Given these suggestions, I consider the immediately fix on my side is to remove the -i flags, but I wonder what has changed to go 1.10 that causes writing to cgo in GOROOT. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#24674 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAcAw0_KsJyZ5v11zA2n8airzpJ759Iks5tluwxgaJpZM4TF-HF> .

[funny-falcon]

Same issue on Ubuntu 17.10 MATE with snap installation of go:

$ go install -i somepackage
go install runtime/cgo: open /snap/go/current/pkg/linux_amd64/runtime/cgo.a: read-only file system

Note: it is not variant to remove "-i" because build cache somewhat doesn't work with testing.
We have huge project with numerous packages. With "-i" whole testing consumes 20 minutes. Without "-i" it consumes 50 minutes. Why build cache doesn't work? I don't know :-( Installation of dependencies works well, because there is a lot of common dependencies between packages.
But flag "-i" is wired into our build command utility, and it runs with the same flag on my developer machine leading to silly error above.

[funny-falcon-at-joomcode]

Note: there were no problem with go 1.9 although it also installed dependencies.
So, why it fails with 1.10?

[davecheney]

@funny-falcon the issue lies with process that produced the snap, it has messed up the timestamps causing the go tool to try to rebuild the runtime. There isn't anything we can do. Sorry.

[davecheney]

@funny-falcon

Why build cache doesn't work? I don't know

Please raise a new issue with steps to reproduce the test caching failure. Thank you.

[funny-falcon-at-joomcode]

@davecheney

it has messed up the timestamps causing the go tool to try to rebuild the runtime.

How this can be checked? Is there way to trace how decision to recompile cgo were accepted?
Why cgo.a triggers recompilation in both cases (original issue case, and mine)? Is it because it is first file in folder "linux_amd64/runtime/" ?
Why it didn't happen with go 1.9 ?

[davecheney]

How this can be checked? Is there way to trace how decision to recompile cgo were accepted?

Try

% go list -f '{{.Stale}}' runtime
false
% go list -f '{{.StaleReason }}' runtime

If .Stale reports true the .StaleReason may indicate why

Why cgo.a triggers recompilation in both cases (original issue case, and mine)?

Possibly the Go installation was compiled with cgo explicitly disabled, or may have happened as a side effect of cross compilation.

Is it because it is first file in folder "linux_amd64/runtime/" ?

That's just the base of almost all dependency trees, so it's the first one to be compiled and thus hits the permission error.

Why it didn't happen with go 1.9 ?

You'll have to ask the people who packaged 1.9.

[funny-falcon-at-joomcode]

Nope:

$ go list -f '{{.Stale}}' runtime
false
$ go list -f '{{.StaleReason }}' runtime

[funny-falcon-at-joomcode]

Ops, that is for runtime/cgo:

$ go list -f '{{.Stale}}' runtime/cgo
true
$ go list -f '{{.StaleReason}}' runtime/cgo
build ID mismatch

[funny-falcon]

I've build go 1.10.1 in a temporary directory, and its cgo.a differs from cgo.a from snap (~100kb in temp dir, and ~ 32kb in snap dir), while other files in runtime directory matches.
So, looks like builder of snap package really does strange thing with it.

[AlexRouSg]

Found a way to contact someone who I hope is still the snap maintainer or at least in contact with them. My post is still under review since it's a google groups post on golang-dev.

https://groups.google.com/forum/#!topic/golang-dev/iRauvZsPrNE%5B1-25%5D

[ianlancetaylor]

It's not obvious to me that there is anything the Go project can do to fix this problem.

[mwhudson]

Feel free to ping me in any bugs to do with debian/ubuntu/snap packaging!

For the snap packaging, I do know why runtime/cgo appears stale. It's fairly obscure though: the snap packaging builds runtime/cgo with particular C compiler flags to ensure that the generated .o files can be handled by older compilers. The go tool now keeps track of these, so unless you happen to use exactly the same flags, the go tool considers runtime/cgo to be stale:

mwhudson@aeglos:~$ /snap/bin/go list  -f '{{ .ImportPath }}: {{ if .Stale }}stale because "{{ .StaleReason }}"{{ else }}not stale {{ end }}' runtime/cgo
runtime/cgo: stale because "build ID mismatch"
mwhudson@aeglos:~$ CGO_CFLAGS='-Wa,-mrelax-relocations=no' /snap/bin/go list  -f '{{ .ImportPath }}: {{ if .Stale }}stale because "{{ .StaleReason }}"{{ else }}not stale {{ end }}' runtime/cgo
runtime/cgo: not stale 

In a sense, runtime/cgo being stale is not actually a bad thing because it means that the C parts are always going to be built with your local C compiler (once, and then saved into the cache, so this doesn't even have much of a time cost).

It's not obvious to me that there is anything the Go project can do to fix this problem.

Well, I think the go tool could swallow failures to write to GOROOT, at least when they are being made because of a -i flag. I presume the initial variant of the report (i.e. not the one for the snap) is something like this, which I don't think is terribly sensible behaviour:

ubuntu@bionic:~$ go build -i -gcflags all=-N trivial.go 
go build runtime/internal/sys: open /usr/lib/go-1.10/pkg/linux_amd64/runtime/internal/sys.a: permission denied
go build runtime/internal/atomic: open /usr/lib/go-1.10/pkg/linux_amd64/runtime/internal/atomic.a: permission denied

(the alternative is training people to stop using -i, I guess)

[ianlancetaylor]

Swallowing failures sounds like a bad idea, but we could consider changing -i to not install to GOROOT at all.

But likely better is to convince people to stop using -i unnecessarily, as with 1.10 it's no longer particularly useful.

[mwhudson]

Swallowing failures sounds like a bad idea, but we could consider changing -i to not install to GOROOT at all.

That works for me (maybe only for release versions?)

But likely better is to convince people to stop using -i unnecessarily, as with 1.10 it's no longer particularly useful.

Well yes, but it takes a while for people's scripts and tooling to catch up I guess.

[akshayjshah]

I'm so sorry that this has bubbled up to all of you.

This is purely an artifact of our internal build system, which has to support Go 1.9, installing multiple Go compilers on the same host, and a handful of other charming quirks. There's nothing broken in Go, and there's nothing that the Go project or Debian package maintainers can or should do about this. Slightly altering the internal build configuration for this particular project fully resolves the issue.

Again, I'm sorry that our internal triaging process failed to catch this before it surfaced here.

[ianlancetaylor]

Thanks, closing.

[NatoBoram]

I still have this issue with Go 1.11.

neofetch --off

OS: Ubuntu 18.04.1 LTS x86_64 
Kernel: 4.15.0-34-generic 
Uptime: 1 day, 47 mins 
Packages: 2213 
Shell: bash 4.4.19 
Resolution: 1920x1080 
DE: GNOME 3.28.3 
WM: GNOME Shell 
WM Theme: Adwaita 
Theme: Adwaita-dark [GTK2/3] 
Icons: Ubuntu-mono-dark [GTK2/3] 
Terminal: gnome-terminal 
CPU: Intel i5-3570K (4) @ 4.100GHz 
GPU: NVIDIA GeForce GTX 660 Ti 
Memory: 4952MiB / 7922MiB 

snap info go

name:      go
summary:   Go programming language compiler, linker, stdlib
publisher: Michael Hudson-Doyle (mwhudson)
contact:   [email protected]
license:   unset
description: |
  This snap provides an assembler, compiler, linker, and compiled libraries
  for the Go programming language.
commands:
  - go
  - go.gofmt
snap-id:      Md1HBASHzP4i0bniScAjXGnOII9cEK6e
tracking:     candidate
refresh-date: 25 days ago, at 05:30 EDT
installed:        1.11          (2635) 82MB classic

go env

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/nato/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/nato/go"
GOPROXY=""
GORACE=""
GOROOT="/snap/go/2635"
GOTMPDIR=""
GOTOOLDIR="/snap/go/2635/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build648960057=/tmp/go-build -gno-record-gcc-switches"

go build runtime/cgo: open /snap/go/2635/pkg/linux_amd64/runtime/cgo.a: read-only file system

[ghost]

Same issue

go build runtime/cgo: open /snap/go/2635/pkg/linux_amd64/runtime/cgo.a: read-only file system

[davecheney]

@NatoBoram @x-t0xid this issue relates to Go 1.10 and is closed.

Please open a new issue for the problems you are having with Go 1.11. You can open seperate issues if you like, we'll take care of handling any duplication.