CFLAGS #cgo pragma in libbpfgo.go cannot be ignored for static building

[grantseltzer]

When building a project that uses libbpfgo, the go compiler must be given a linker flag pointing to libbpf. If libbpf is installed in a standard location -lbpf can be passed. They can also point to a specific location. Since go get attempts to build the package without any ldflags, this will cause the errors pasted below.

Users could do something like: CGO_LDFLAGS="-lbpf" go get github.com/aquasecurity/tracee. Alternatively we can add a line for CGO_LDFLAGS in the CGO code.

This shouldn't cause any related issues if not using go get.

For example:

[*] go get github.com/aquasecurity/libbpfgo

go: downloading github.com/aquasecurity/libbpfgo v0.0.0-20210517223445-485e1cdc10ae
go: downloading golang.org/x/sys v0.0.0-20210514084401-e8d321eab015
# github.com/aquasecurity/libbpfgo
/usr/bin/ld: $WORK/b001/_x003.o: in function `init_ring_buf':
../../../../../pkg/mod/github.com/aquasecurity/[email protected]/libbpfgo.go:40: undefined reference to `ring_buffer__new'
/usr/bin/ld: $WORK/b001/_x003.o: in function `init_perf_buf':
../../../../../pkg/mod/github.com/aquasecurity/[email protected]/libbpfgo.go:54: undefined reference to `perf_buffer__new'
/usr/bin/ld: $WORK/b001/_x003.o: in function `attach_kprobe_legacy':
../../../../../pkg/mod/github.com/aquasecurity/[email protected]/libbpfgo.go:158: undefined reference to `bpf_program__attach_perf_event'
/usr/bin/ld: ../../../../../pkg/mod/github.com/aquasecurity/[email protected]/libbpfgo.go:159: undefined reference to `libbpf_get_error'
/usr/bin/ld: $WORK/b001/_x003.o: in function `_cgo_a66f50300837_Cfunc_bpf_link__destroy':
/tmp/go-build/cgo-gcc-prolog:74: undefined reference to `bpf_link__destroy'
/usr/bin/ld: $WORK/b001/_x003.o: in function `_cgo_a66f50300837_Cfunc_bpf_map__fd':
/tmp/go-build/cgo-gcc-prolog:93: undefined reference to `bpf_map__fd'
/usr/bin/ld: $WORK/b001/_x003.o: in function `_cgo_a66f50300837_Cfunc_bpf_map__max_entries':
/tmp/go-build/cgo-gcc-prolog:112: undefined reference to `bpf_map__max_entries'
/usr/bin/ld: $WORK/b001/_x003.o: in function `_cgo_a66f50300837_Cfunc_bpf_map__pin':
/tmp/go-build/cgo-gcc-prolog:132: undefined reference to `bpf_map__pin'
/usr/bin/ld: $WORK/b001/_x003.o: in function `_cgo_a66f50300837_Cfunc_bpf_map__resize':
/tmp/go-build/cgo-gcc-prolog:153: undefined reference to `bpf_map__resize'
/usr/bin/ld: $WORK/b001/_x003.o: in function `_cgo_a66f50300837_Cfunc_bpf_map__set_pin_path':
/tmp/go-build/cgo-gcc-prolog:173: undefined reference to `bpf_map__set_pin_path'
/usr/bin/ld: $WORK/b001/_x003.o: in function `_cgo_a66f50300837_Cfunc_bpf_map__unpin':
/tmp/go-build/cgo-gcc-prolog:193: undefined reference to `bpf_map__unpin'
/usr/bin/ld: $WORK/b001/_x003.o: in function `_cgo_a66f50300837_Cfunc_bpf_map_delete_elem':
/tmp/go-build/cgo-gcc-prolog:214: undefined reference to `bpf_map_delete_elem'
/usr/bin/ld: $WORK/b001/_x003.o: in function `_cgo_a66f50300837_Cfunc_bpf_map_lookup_elem':
/tmp/go-build/cgo-gcc-prolog:236: undefined reference to `bpf_map_lookup_elem'
/usr/bin/ld: $WORK/b001/_x003.o: in function `_cgo_a66f50300837_Cfunc_bpf_map_update_elem':
/tmp/go-build/cgo-gcc-prolog:259: undefined reference to `bpf_map_update_elem'
/usr/bin/ld: $WORK/b001/_x003.o: in function `_cgo_a66f50300837_Cfunc_bpf_object__find_map_by_name':
/tmp/go-build/cgo-gcc-prolog:290: undefined reference to `bpf_object__find_map_by_name'
/usr/bin/ld: $WORK/b001/_x003.o: in function `_cgo_a66f50300837_Cfunc_bpf_object__find_program_by_name':
/tmp/go-build/cgo-gcc-prolog:309: undefined reference to `bpf_object__find_program_by_name'
/usr/bin/ld: $WORK/b001/_x003.o: in function `_cgo_a66f50300837_Cfunc_bpf_object__load':
/tmp/go-build/cgo-gcc-prolog:328: undefined reference to `bpf_object__load'
/usr/bin/ld: $WORK/b001/_x003.o: in function `_cgo_a66f50300837_Cfunc_bpf_object__open':
/tmp/go-build/cgo-gcc-prolog:346: undefined reference to `bpf_object__open'
/usr/bin/ld: $WORK/b001/_x003.o: in function `_cgo_a66f50300837_Cfunc_bpf_object__open_buffer':
/tmp/go-build/cgo-gcc-prolog:366: undefined reference to `bpf_object__open_buffer'
/usr/bin/ld: $WORK/b001/_x003.o: in function `_cgo_a66f50300837_Cfunc_bpf_program__attach_kprobe':
/tmp/go-build/cgo-gcc-prolog:387: undefined reference to `bpf_program__attach_kprobe'
/usr/bin/ld: $WORK/b001/_x003.o: in function `_cgo_a66f50300837_Cfunc_bpf_program__attach_lsm':
/tmp/go-build/cgo-gcc-prolog:405: undefined reference to `bpf_program__attach_lsm'
/usr/bin/ld: $WORK/b001/_x003.o: in function `_cgo_a66f50300837_Cfunc_bpf_program__attach_raw_tracepoint':
/tmp/go-build/cgo-gcc-prolog:424: undefined reference to `bpf_program__attach_raw_tracepoint'
/usr/bin/ld: $WORK/b001/_x003.o: in function `_cgo_a66f50300837_Cfunc_bpf_program__attach_tracepoint':
/tmp/go-build/cgo-gcc-prolog:444: undefined reference to `bpf_program__attach_tracepoint'
/usr/bin/ld: $WORK/b001/_x003.o: in function `_cgo_a66f50300837_Cfunc_bpf_program__fd':
/tmp/go-build/cgo-gcc-prolog:463: undefined reference to `bpf_program__fd'
/usr/bin/ld: $WORK/b001/_x003.o: in function `_cgo_a66f50300837_Cfunc_bpf_program__get_type':
/tmp/go-build/cgo-gcc-prolog:482: undefined reference to `bpf_program__get_type'
/usr/bin/ld: $WORK/b001/_x003.o: in function `_cgo_a66f50300837_Cfunc_bpf_program__set_autoload':
/tmp/go-build/cgo-gcc-prolog:503: undefined reference to `bpf_program__set_autoload'
/usr/bin/ld: $WORK/b001/_x003.o: in function `_cgo_a66f50300837_Cfunc_bpf_program__set_tracepoint':
/tmp/go-build/cgo-gcc-prolog:522: undefined reference to `bpf_program__set_tracepoint'
/usr/bin/ld: $WORK/b001/_x003.o: in function `_cgo_a66f50300837_Cfunc_perf_buffer__poll':
/tmp/go-build/cgo-gcc-prolog:607: undefined reference to `perf_buffer__poll'
/usr/bin/ld: $WORK/b001/_x003.o: in function `_cgo_a66f50300837_Cfunc_ring_buffer__poll':
/tmp/go-build/cgo-gcc-prolog:661: undefined reference to `ring_buffer__poll'
/usr/bin/ld: $WORK/b001/_x003.o: in function `set_print_fn':
../../../../../pkg/mod/github.com/aquasecurity/[email protected]/libbpfgo.go:35: undefined reference to `libbpf_set_print'
/usr/bin/ld: $WORK/b001/_x003.o: in function `_cgo_a66f50300837_Cfunc_bpf_object__close':
/tmp/go-build/cgo-gcc-prolog:274: undefined reference to `bpf_object__close'
/usr/bin/ld: $WORK/b001/_x003.o: in function `_cgo_a66f50300837_Cfunc_perf_buffer__free':
/tmp/go-build/cgo-gcc-prolog:589: undefined reference to `perf_buffer__free'
/usr/bin/ld: $WORK/b001/_x003.o: in function `_cgo_a66f50300837_Cfunc_ring_buffer__free':
/tmp/go-build/cgo-gcc-prolog:643: undefined reference to `ring_buffer__free'
collect2: error: ld returned 1 exit status

[rafaeldtinoco]

This was somehow handled by:

#28

But now we have to address the following:

#28 (comment)

So I would say this issue has become the issue described above.

[rafaeldtinoco]

The Makefile file in tracee uses pkgconfig for the libbpf static compilation:

libbpfgo could do the same and use:

// #cgo pkg-config: libelf zlib

to have this issue addressed.

[ShubhamPalriwala]

Just faced the exact same bug when trying to run a test here in Tracee with:

tracee/cmd/tracee-ebpf/internal/printer
go test -v

[MaciekLeks]

I've struggled the same or almost the same "thing".
So, this code does not work:

CGO_CFLAGS_STATIC = "-I$(abspath $(LIBBPF_DIR))"
CGO_LDFLAGS_STATIC = "-lelf -lz $(LIBBPF_STATIC_LIB)"
CGO_EXTLDFLAGS_STATIC = '-w -extldflags "-static"'
.PHONY: k8s-build-cmd
k8s-build-cmd:  $(CMD_K8S_GO_SOURCE) $(TARGET_BPF) 
	CGO_CFLAGS=$(CGO_CFLAGS_STATIC) \
	CGO_LDFLAGS=$(CGO_LDFLAGS_STATIC) \
	$(GO) build -x \
	-tags netgo -ldflags $(CGO_EXTLDFLAGS_STATIC) \
	-o $(TARGET_K8S) ./cmd/kubernetes/$(MAIN).go

but this works well:

CGO_CFLAGS_STATIC = "-I$(abspath $(LIBBPF_DIR))"
CGO_LDFLAGS_STATIC = "-lelf -lz $(LIBBPF_STATIC_LIB)"
GO_EXTLDFLAGS_STATIC = '-w -extldflags "-static $(LIBBPF_STATIC_LIB) -lelf -lz"'
#^ librabbry order is important for GO_EXTLDFLAGS_STATIC
k8s-build-cmd: $(CMD_K8S_GO_SOURCE) $(TARGET_BPF)
	CGO_CFLAGS=$(CGO_CFLAGS_STATIC) \
	$(GO) build -x \
	-tags netgo -ldflags $(GO_EXTLDFLAGS_STATIC) \
	-o $(TARGET_K8S) ./cmd/kubernetes/$(MAIN).go

So, the only difference is moving LDFLAGS on the go build level and changing order of the libs.