cmd/go: mod vendor fails with unlinkat error

[miroslav-vulovic]

What version of Go are you using (go version)?

$ go version
go version go1.13.10 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build266767394=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I'm trying to build blackbox_exporter within Docker container. I have successfully built it on my CentOS 7 system, now I'm trying to build it in Docker container.

My Dockerfile is:

FROM golang:1.13-alpine AS go

RUN mkdir -p /app/blackbox-exporter

WORKDIR /app/blackbox-exporter

RUN apk add --update --no-cache git curl make gcc libc-dev

ENV PATH $PATH:/go/bin

RUN git clone https://github.com/prometheus/blackbox_exporter.git /app/blackbox-exporter

RUN make

What did you expect to see?

I expect to see no errors when building blackbox_exporter in Docker container.

What did you see instead?

Instead I get the following error:

>> running check for unused packages in vendor/
GO111MODULE=on go mod vendor
go mod vendor: unlinkat /app/blackbox-exporter/vendor/github.com/alecthomas/template/parse: directory not empty
make: *** [/app/blackbox-exporter/Makefile.common:204: common-unused] Error 1
The command '/bin/sh -c make -f /app/blackbox-exporter/blackbox-exporter-makefile' returned a non-zero code: 2

[toothrot]

Thanks for filing. What is the docker command you're using to build? My guess is that you're mounting a local module cache into the container, which is readonly. (see #27161)

[jayconrod]

Couldn't reproduce this with Docker on macOS.

Could you give a more minimal example that reproduces the problem? There's a moderately sized Makefile here, and I don't know whether it's doing something that might cause issues.

Is there any chance that more than one instance of go mod vendor is run concurrently?

[miroslav-vulovic]

@toothrot This is the command I'm using to build:
docker build --tag test_img_2 /home/mirov/workspace/sa20sre/blackbox-exporter

[miroslav-vulovic]

@jayconrod I've looked into Makefile.common and from what I see I don't think there is more than one instance of go mod vendor running when I start the build. I have to admit I don't quite understand you when you say 'Could you give a more minimal example that reproduces the problem?'. The make file I use is the one that's already there as it came with go sources. I don't want to mess with it.

[miroslav-vulovic]

@toothroot @jayconrod Temporarily I'm avoiding this problem by removing 'unused' flag in Makefile.common. I have replaced the original line in Makefile.common:

common-all: precheck style check_license lint unused build test

with

common-all: precheck style check_license lint build test

so that 'go mod vendor' is not getting executed.

I did it by putting this line in my Docker file (before calling make):

RUN sed -i 's/precheck style check_license lint unused build test/precheck style check_license lint build test/g' Makefile.common

Now the make goes on without 'go mod vendor' error about directory not being empty.

[jayconrod]

@jayconrod I've looked into Makefile.common and from what I see I don't think there is more than one instance of go mod vendor running when I start the build. I have to admit I don't quite understand you when you say 'Could you give a more minimal example that reproduces the problem?'. The make file I use is the one that's already there as it came with go sources. I don't want to mess with it.

I'm wondering if there's a way to reproduce this using a few go commands without the Makefile. There's a lot going on in the Makefile, and it's not really an easy language to debug.

From the error message, it sounds like something is writing files in the vendor directory at the same time that go mod vendor runs. go mod vendor starts by calling os.RemoveAll on the vendor directory, and that would fail if it were racing with another process that's adding files. The other process could be another instance of go mod vendor. If this indeed the case, I'm not sure there's much we can do.

I tried this on a couple systems, but I still couldn't reproduce the failure you're seeing. It does however fail with:

>> running all tests
GO111MODULE=on go test -race  -mod=vendor ./...
# runtime/race
/usr/lib/gcc/x86_64-alpine-linux-musl/9.2.0/../../../../x86_64-alpine-linux-musl/bin/ld: race_linux_amd64.syso: in function `__sanitizer::GetArgv()':
gotsan.cc:(.text+0x4183): undefined reference to `__libc_stack_end'
/usr/lib/gcc/x86_64-alpine-linux-musl/9.2.0/../../../../x86_64-alpine-linux-musl/bin/ld: race_linux_amd64.syso: in function `__sanitizer::ReExec()':
gotsan.cc:(.text+0x9797): undefined reference to `__libc_stack_end'
/usr/lib/gcc/x86_64-alpine-linux-musl/9.2.0/../../../../x86_64-alpine-linux-musl/bin/ld: race_linux_amd64.syso: in function `__sanitizer::InternalAlloc(unsigned long, __sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator32<__sanitizer::AP32> >*, unsigned long)':
gotsan.cc:(.text+0xaac1): undefined reference to `__libc_malloc'
/usr/lib/gcc/x86_64-alpine-linux-musl/9.2.0/../../../../x86_64-alpine-linux-musl/bin/ld: race_linux_amd64.syso: in function `__sanitizer::InternalRealloc(void*, unsigned long, __sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator32<__sanitizer::AP32> >*)':
gotsan.cc:(.text+0xca20): undefined reference to `__libc_realloc'
/usr/lib/gcc/x86_64-alpine-linux-musl/9.2.0/../../../../x86_64-alpine-linux-musl/bin/ld: race_linux_amd64.syso: in function `__sanitizer::InternalFree(void*, __sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator32<__sanitizer::AP32> >*)':
gotsan.cc:(.text+0x66e8): undefined reference to `__libc_free'
collect2: error: ld returned 1 exit status

[miroslav-vulovic]

@jayconrod Thanks for answering. Regarding the race error you mentioned, I've also bumped into it. I have worked around it by removing - race flag frm the Makefile.common file. I know it's not the solution but it enabled me to go on with the build. The 'go mod vendor' problem I avoided by removing the flag unused from the Makefile.common file. Again, not the solution but after that I was able to build blackbox_exporter.

[bcmills]

The link error looks like #14481, fixed in release-branch.go1.14.

[jayconrod]

Closing since we haven't been able to reproduce this, and there doesn't seem to be a way to move forward.

[marcpar]

delete vendor folder then docker-compose exec [container] go mod vendor