plugin: spurious version-mismatch error for packages that differ only in debug information

[mem]

What version of Go are you using (go version)?

$ go version
go version go1.12.1 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/mem/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/mem/go"
GOPROXY=""
GORACE=""
GOROOT="/home/mem/devel/go/go-1.12"
GOTMPDIR=""
GOTOOLDIR="/home/mem/devel/go/go-1.12/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/mem/devel/go/plugin-modules/go-plugin-modules/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build318788666=/tmp/go-build -gno-record-gcc-switches"

What did you do?

The code to reproduce the issue is available at https://github.com/mem/go-plugin-modules

You can use the makefile in there to build the code and run the tests using make test.

When building a program and a plugin using the -mod=vendor flag (and vendoring the corresponding packages in the respective modules using go mod vendor), the plugin code fails to load:

$ cd go-plugin-loader/ && go build -mod=vendor -o /home/mem/devel/go/plugin-modules/go-plugin-modules/go-plugin-loader/go-plugin-loader-vendor main.go

$ cd go-plugin-hook/ && go build -mod=vendor -buildmode=plugin -o /home/mem/devel/go/plugin-modules/go-plugin-modules/go-plugin-hook/go-plugin-hook-vendor.so main.go

And then running this program as:

$ go-plugin-loader/go-plugin-loader-vendor go-plugin-hook/go-plugin-hook-vendor.so

produces the error:

plugin.Open("go-plugin-hook/go-plugin-hook-vendor"): plugin was built with a different version of package golang.org/x/xerrors/internal

What did you expect to see?

The plugin is loaded and the code runs. This is demonstrated in the above code by running make test-no-vendor, which simply does not pass the -mod=vendor flag when building.

What did you see instead?

plugin.Open("go-plugin-hook/go-plugin-hook-vendor"): plugin was built with a different version of package golang.org/x/xerrors/internal

It seems the recorded import path when passing -mod=vendor is different.

This is related (duplicate?) of #28983, but I found that issue only until I was submitting this one, so I think there's missing documentation that explains what the issue is and how it should be dealt with.

[bcmills]

Have you verified that go-plugin-loader and go-plugin-hook are in fact using the same version of golang.org/x/xerrors/internal? (What does go list -m golang.org/x/xerrors report in your two repos?)

CC @jayconrod

[mem]

Thanks for looking at this Bryan.

Each module has a go.mod file that cotains:

require golang.org/x/xerrors v0.0.0-20190315151331-d61658bd2e18

the line is the same in both cases:

https://github.com/mem/go-plugin-modules/blob/master/go-plugin-hook/go.mod#L5
https://github.com/mem/go-plugin-modules/blob/master/go-plugin-loader/go.mod#L5

The output of go list -m golang.org/x/xerrors is:

golang.org/x/xerrors v0.0.0-20190315151331-d61658bd2e18

in both cases.

If I look at the stored hashes by looking at the data stored in the ELF data, for the program, I see:

82d0bd0dbd4a8b94a351681599480d1f3b408d77 golang.org/x/xerrors/internal
55d8bb599e8f087d57740811285a4d346d44bb3f golang.org/x/xerrors

For the plugin that works (the one without the -mod=vendor flag), I see:

82d0bd0dbd4a8b94a351681599480d1f3b408d77 golang.org/x/xerrors/internal
55d8bb599e8f087d57740811285a4d346d44bb3f golang.org/x/xerrors

(they match)

For the plugin that does not work (with the -mod=vendor flag), I see:

a09889ca55ea57a92def3699da80ea747cebcf8f golang.org/x/xerrors/internal
6813cacc76ce0a32b951b92256570909a5533418 golang.org/x/xerrors

(does not match).

I've been looking the linker code but I can't figure out how these hashes are computed.

Investigating a little further (and guessing a lot), I can get this to work if I pass -mod=vendor and -gcflags=all=-trimpath="${dir_with_go_mod}" -asmflags=all=-trimpath="${dir_with_go_mod}", where ${dir_with_go_mod} is the directory containing the go.mod file.

It does not interoperate, and I'm not sure if it should. What I mean is that if I build the program with -mod=vendor -gcflags=... -asmflags=... and the plugin without (or the other way around), the plugin cannot be loaded (and the hashes are again different).

I gues the questions are this point are:

• -mod=vendor seems to require -trimpath, is this correct? expected? a bug?
• programs and plugins must either both use -mod=vendor or not use it all. Is this expected? A bug?

[mem]

Additional info. For the non-working case, the hashes are:

# plugin, without -mod=vendor, without -trimpath
82d0bd0dbd4a8b94a351681599480d1f3b408d77 golang.org/x/xerrors/internal
55d8bb599e8f087d57740811285a4d346d44bb3f golang.org/x/xerrors
# program, without -mod=vendor, without -trimpath
82d0bd0dbd4a8b94a351681599480d1f3b408d77 golang.org/x/xerrors/internal
55d8bb599e8f087d57740811285a4d346d44bb3f golang.org/x/xerrors
# program, with -mod=vendor, without -trimpath
62b9c62049b9fb4057ab676a77ad6696b811da27 golang.org/x/xerrors/internal
0298110ff71bc3fd8845a72084d36e114c6376ee golang.org/x/xerrors
# plugin, with -mod=vendor, without -trimpath
a09889ca55ea57a92def3699da80ea747cebcf8f golang.org/x/xerrors/internal
6813cacc76ce0a32b951b92256570909a5533418 golang.org/x/xerrors

Adding -trimpath to get it working:

# plugin, without -mod=vendor, with -trimpath
82d0bd0dbd4a8b94a351681599480d1f3b408d77 golang.org/x/xerrors/internal
55d8bb599e8f087d57740811285a4d346d44bb3f golang.org/x/xerrors
# program, without -mod=vendor, with -trimpath
82d0bd0dbd4a8b94a351681599480d1f3b408d77 golang.org/x/xerrors/internal
55d8bb599e8f087d57740811285a4d346d44bb3f golang.org/x/xerrors
# program, with -mod=vendor, with -trimpath
31a9229bcad8c38e314d35314f119a1c9a34069b golang.org/x/xerrors/internal
44d0e2b49162ae0a15e4e14f9332611651e7d836 golang.org/x/xerrors
# plugin, with -mod=vendor, with -trimpath
31a9229bcad8c38e314d35314f119a1c9a34069b golang.org/x/xerrors/internal
44d0e2b49162ae0a15e4e14f9332611651e7d836 golang.org/x/xerrors

Since -trimpath doesn't seem to have an effect with / without -mod=vendor, I'm guessing $GOPATH/pkg/mod/ is somehow involved in the computation of the hash?

Just to be clear, everything above has been done without GOPATH set (so it's using the $HOME/go default).

If I set GOPATH to something (just to hold the pkg/mod files in a different directory), I can see the hashes in the "without -mod=vendor" case changing, but the "with -mod=vendor" case stays the same.

So the end result is that I have to use -mod=vendor (otherwise I have to build using the same path for the code) and I have to set -trimpath (otherwise the code cannot be loaded).

Should this be captured in a document?

[bcmills]

Since -trimpath doesn't seem to have an effect with / without -mod=vendor, I'm guessing $GOPATH/pkg/mod/ is somehow involved in the computation of the hash?

Probably at least one difference here is in the debug information, which by default includes the full file paths. That's why you need -trimpath in -mod=vendor mode, and why the two versions (with and without -mod=vendor) differ.

If that's correct, then you may find that if you set -trimpath for both builds, then you can combine a plugin and binary with and without -mod=vendor.

[bcmills]

@ianlancetaylor and @cherrymui: is there something we can do in the plugin package to make the version-mismatch detection code less sensitive to differences that don't affect the executable code or ABI of the package?

[ianlancetaylor]

As far as I can tell the hash value is based only on the externally visible ABI. Changes in the debug information shouldn't affect it. But I don't think anybody really understands this other than @crawshaw.

[mem]

While revisiting this for a different reason, I also noticed that -trimpath is a string (flag.StringVar(&pathPrefix, "trimpath", "", "remove prefix from recorded source file paths") from src/cmd/compile/internal/gc/main.go).

The build command uses the -trimpath flag internally, calling compile for example as /.../pkg/tool/linux_amd64/compile -o $WORK/b003/_pkg_.a -trimpath $WORK/b003 ...

That means if I pass -gcflags=all=-trimpath=... on the command line, I end up with a compile command with two -trimpath flags (verified this using go build -x ...) and because of the way the flag is defined, only the second value is used, dropping the one the compiler uses internally.

Does this matter?

[bcmills]

@mem, now that #16860 is resolved, you might be able to use go build -trimpath as a workaround.

[mem]

Checking, thanks @bcmills

[bcmills]

@mem, note that the -trimpath approach might still not work due to #34326. (You could try a build from source at release-branch.go1.13 instead, or use golang.org/dl/gotip.)

[seankhliao]

sounds like the solution is -trimpath.