Build fails on Alpine due to bundled .a files

[bschofield]

Firstly, thanks a lot for making this. I was experiencing DataDog/zstd#22 and switching over to this library seems to have solved that problem, with a minimum of code changes.

My issue is that I deploy my golang application in docker containers. I use Alpine Linux as my deployment base in order to reduce the container sizes. Because Alpine uses musl instead of glibc, the pre-compiled binaries you distribute cannot work with Alpine. Instead, at build time errors like this are generated:

39 # github.com/valyala/gozstd
40 /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/valyala/[email protected]/libzstd_linux_amd64.a(zdict.o): in function `ZDICT_analyzeEntropy':
41 zdict.c:(.text+0x7ce): undefined reference to `__fprintf_chk'
42 /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/valyala/[email protected]/libzstd_linux_amd64.a(zdict.o): in function `ZDICT_analyzePos':
43 zdict.c:(.text+0x1832): undefined reference to `__fprintf_chk'
44 /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: zdict.c:(.text+0x18c2): undefined reference to `__fprintf_chk'
45 /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/valyala/[email protected]/libzstd_linux_amd64.a(zdict.o): in function `ZDICT_finalizeDictionary':
46 zdict.c:(.text+0x1b24): undefined reference to `__fprintf_chk'
47 /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/valyala/[email protected]/libzstd_linux_amd64.a(zdict.o): in function `ZDICT_trainFromBuffer_unsafe_legacy':
48 zdict.c:(.text+0x1d54): undefined reference to `__fprintf_chk'
49 /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/valyala/[email protected]/libzstd_linux_amd64.a(zdict.o):zdict.c:(.text+0x1dd4): more undefined references to `__fprintf_chk' follow
50 /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/valyala/[email protected]/libzstd_linux_amd64.a(entropy_common.o): in function `FSE_readNCount':
51 entropy_common.c:(.text+0x345): undefined reference to `__memcpy_chk'
52 /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/valyala/[email protected]/libzstd_linux_amd64.a(fastcover.o): in function `FASTCOVER_buildDictionary.isra.6':
53 fastcover.c:(.text+0x3a9): undefined reference to `__fprintf_chk'
54 /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: fastcover.c:(.text+0x44f): undefined reference to `__fprintf_chk'
55 /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: fastcover.c:(.text+0x488): undefined reference to `__fprintf_chk'
56 /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/valyala/[email protected]/libzstd_linux_amd64.a(fastcover.o): in function `FASTCOVER_ctx_init':
57 fastcover.c:(.text+0x8a0): undefined reference to `__fprintf_chk'
58 /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: fastcover.c:(.text+0x8e6): undefined reference to `__fprintf_chk'
59 /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/valyala/[email protected]/libzstd_linux_amd64.a(fastcover.o):fastcover.c:(.text+0xbc7): more undefined references to `__fprintf_chk' follow
60 collect2: error: ld returned 1 exit status

I can work around this by putting in special case code into my build process just for this module, something like

go get -d github.com/valyala/gozstd; cd /go/pkg/mod/github.com/valyala/gozstd@*; if [[ ! -f _rebuilt ]]; then chmod -R +w .; make -j8 clean; make -j8 libzstd.a; touch _rebuilt; fi; cd /go/src/myproject

...but it feels quite unusual to have to do that.

What changes would be required to avoid having to distribute a binary library?

[valyala]

What changes would be required to avoid having to distribute a binary library?

In this case you should run make libzstd.a after go get github.com/valyala/gozstd. Unfortunately Go cannot build libzstd.a during go get from the unmodified upstream code inside zstd directory. Go can automatically build only *.c and *.h files inside the same directory as *.go files. It doesn't know how to build *.c files inside other directories such as zstd directory :(

[bschofield]

OK, thanks!

For anyone else having this issue, here is an updated one-liner which can be placed in your .gitlab-ci.yaml to automatically pull the version of this library specified in go.mod before cleaning and rebuilding the .a files:

    - BUILD_DIR=$(pwd); GOZSTD_VER=$(cat go.mod | fgrep github.com/valyala/gozstd | awk '{print $NF}'); go get -d github.com/valyala/gozstd@${GOZSTD_VER}; cd ${GOPATH}/pkg/mod/github.com/valyala/gozstd@${GOZSTD_VER}; if [[ ! -f _rebuilt ]]; then chmod -R +w .; make -j8 clean; make -j8 libzstd.a; touch _rebuilt; fi; cd ${BUILD_DIR}

The one-liner makes a marker file _rebuilt inside the module directory, so if you are caching build dependencies then the rebuild only has to happen once. The chmod is required because of #6.

[f41gh7]

FYI, bundle for alpine linux with musl is supported now, you have to provide build tag for it. It can be done with command:
go build -tag 'musl'

https://github.com/valyala/gozstd/blob/master/libzstd_linux_musl_amd64.go

[manoj398]

Should be
go build -tags 'musl'