Makefile: improvements #28

rafaeldtinoco · 2021-07-01T03:55:11Z

Makefile: improvements for the entire tree

Currently you will find the following GNU Makefile rules:

Makefile Rule	Description
all	builds libbpfgo (dynamic)
clean	cleans entire tree
selftest	builds all selftests (static)
selftest-run	runs all selftests (static)

libbpf dynamically linked (libbpf from OS)

Makefile Rule	Description
libbpfgo-dynamic	builds dynamic libbpfgo (libbpf)
libbpfgo-dynamic-test	'go test' with dynamic libbpfgo
selftest-dynamic	build tests with dynamic libbpfgo
selftest-dynamic-run	run tests using dynamic libbpfgo

statically compiled (libbpf submodule)

Makefile Rule	Description
libbpfgo-static	builds static libbpfgo (libbpf)
libbpfgo-static-test	'go test' with static libbpfgo
selftest-static	build tests with static libbpfgo
selftest-static-run	run tests using static libbpfgo

examples

$ make libbpfgo-static => libbpfgo statically linked with libbpf
$ make -C selftest/perfbuffers => single selftest build (static libbpf)\
$ make -C selftest/perfbuffers run-dynamic => single selftest run (dynamic libbpf)
$ make selftest-static-run => will build & run all static selftests

Note 01: dynamic builds need your OS to have a recent enough libbpf
package (and its headers) installed. Sometimes, recent features might
require the use of backported OS packages in order for your OS to
contain latest libbpf features (sometimes required by libbpfgo).

Note 02: static builds need git submodule init first. Make sure to
sync the libbpf git submodule before trying to statically compile or
test the libbpfgo repository.

rafaeldtinoco · 2021-07-03T17:21:17Z

Showing the benefits of this arrangement...

Checking if libbpfgo builds with libbpf v0.2:

~/.../ebpf/libbpfgo/libbpf$ git checkout v0.2
Previous HEAD position was 051a400 pkgconfig: use literal ${prefix} to allow override
HEAD is now at d1fd50d helpers: add `struct bpf_redir_neigh` forward declaration
~/.../ebpf/libbpfgo/libbpf$ cd ..
~/.../sources/ebpf/libbpfgo$ make libbpfgo-static-test
# ./selftest/build already depends on libbpfgo-static
make -C ./selftest/build
make[1]: Entering directory '/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/selftest/build'
make -C /home/rafaeldtinoco/work/sources/ebpf/libbpfgo libbpfgo-static
make[2]: Entering directory '/home/rafaeldtinoco/work/sources/ebpf/libbpfgo'
mkdir -p output/libbpf
/sbin/bpftool btf dump file /sys/kernel/btf/vmlinux format c > /home/rafaeldtinoco/work/sources/ebpf/libbpfgo/output/vmlinux.h
CC="gcc" CFLAGS="-g -O2 -Wall -fpie" LD_FLAGS="" \
   make -C /home/rafaeldtinoco/work/sources/ebpf/libbpfgo/libbpf/src \
	BUILD_STATIC_ONLY=1 \
	OBJDIR=/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/output/libbpf \
	DESTDIR=/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/output \
	INCLUDEDIR= LIBDIR= UAPIDIR= install
make[3]: Entering directory '/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/libbpf/src'
  MKDIR    staticobjs
  INSTALL  bpf.h libbpf.h btf.h xsk.h libbpf_util.h bpf_helpers.h bpf_helper_defs.h bpf_tracing.h bpf_endian.h bpf_core_read.h libbpf_common.h
  CC       bpf.o
  CC       btf.o
  CC       libbpf.o
  CC       libbpf_errno.o
  CC       netlink.o
  CC       nlattr.o
  CC       str_error.o
  CC       libbpf_probes.o
  CC       bpf_prog_linfo.o
  CC       xsk.o
  CC       btf_dump.o
  CC       hashmap.o
  CC       ringbuf.o
  INSTALL  libbpf.pc
  AR       libbpf.a
  INSTALL  libbpf.a
make[3]: Leaving directory '/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/libbpf/src'
CC=clang CGO_CFLAGS="-I/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/output" CGO_LDFLAGS="/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/output/libbpf.a -lelf -lz" go build .
make[2]: Leaving directory '/home/rafaeldtinoco/work/sources/ebpf/libbpfgo'
clang -g -O2 -Wall -fpie -target bpf -I../../output -c libbpfgo_test.bpf.c -o libbpfgo_test.bpf.o
In file included from libbpfgo_test.bpf.c:3:
../../output/bpf/bpf_helpers.h:77:1: warning: unused function 'bpf_tail_call_static' [-Wunused-function]
bpf_tail_call_static(void *ctx, const void *map, const __u32 slot)
^
1 warning generated.
make[1]: Leaving directory '/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/selftest/build'
CC=clang CGO_CFLAGS="-I/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/output" CGO_LDFLAGS="/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/output/libbpf.a -lelf -lz" sudo -E go test .
ok  	github.com/aquasecurity/libbpfgo	(cached)

Builds and libbpfgo_test.go works with one compile warning.

Now checking perfbuffers selftest with v0.2:

~/.../sources/ebpf/libbpfgo$ make -C selftest/perfbuffers/ run ; echo $?
make: Entering directory '/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/selftest/perfbuffers'
make -C /home/rafaeldtinoco/work/sources/ebpf/libbpfgo libbpfgo-static
make[1]: Entering directory '/home/rafaeldtinoco/work/sources/ebpf/libbpfgo'
CC=clang CGO_CFLAGS="-I/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/output" CGO_LDFLAGS="/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/output/libbpf.a -lelf -lz" go build .
make[1]: Leaving directory '/home/rafaeldtinoco/work/sources/ebpf/libbpfgo'
clang -g -O2 -Wall -fpie -target bpf -I../../output -c self.bpf.c -o self.bpf.o
In file included from self.bpf.c:3:
../../output/bpf/bpf_helpers.h:77:1: warning: unused function 'bpf_tail_call_static' [-Wunused-function]
bpf_tail_call_static(void *ctx, const void *map, const __u32 slot)
^
1 warning generated.
CC=clang CGO_CFLAGS="-I../../output" CGO_LDFLAGS="../../output/libbpf.a -lelf -lz" \
   go build ./main.go
sudo ./main
make: Leaving directory '/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/selftest/perfbuffers'
0

Now checking perfbuffers selftest with v0.4.0

~/.../sources/ebpf/libbpfgo$ cd libbpf
~/.../ebpf/libbpfgo/libbpf$ git checkout v0.4.0
Previous HEAD position was d1fd50d helpers: add `struct bpf_redir_neigh` forward declaration
HEAD is now at db9614b libbpf: Add support for new llvm bpf relocations
~/.../sources/ebpf/libbpfgo$ make -C selftest/perfbuffers/ run ; echo $?
make: Entering directory '/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/selftest/perfbuffers'
make -C /home/rafaeldtinoco/work/sources/ebpf/libbpfgo libbpfgo-static
make[1]: Entering directory '/home/rafaeldtinoco/work/sources/ebpf/libbpfgo'
mkdir -p output/libbpf
/sbin/bpftool btf dump file /sys/kernel/btf/vmlinux format c > /home/rafaeldtinoco/work/sources/ebpf/libbpfgo/output/vmlinux.h
CC="gcc" CFLAGS="-g -O2 -Wall -fpie" LD_FLAGS="" \
   make -C /home/rafaeldtinoco/work/sources/ebpf/libbpfgo/libbpf/src \
	BUILD_STATIC_ONLY=1 \
	OBJDIR=/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/output/libbpf \
	DESTDIR=/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/output \
	INCLUDEDIR= LIBDIR= UAPIDIR= install
make[2]: Entering directory '/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/libbpf/src'
  MKDIR    staticobjs
  INSTALL  bpf.h libbpf.h btf.h xsk.h bpf_helpers.h bpf_helper_defs.h bpf_tracing.h bpf_endian.h bpf_core_read.h libbpf_common.h
  CC       bpf.o
  CC       btf.o
  CC       libbpf.o
  CC       libbpf_errno.o
  CC       netlink.o
  CC       nlattr.o
  CC       str_error.o
  CC       libbpf_probes.o
  CC       bpf_prog_linfo.o
  CC       xsk.o
  CC       btf_dump.o
  CC       hashmap.o
  CC       ringbuf.o
  CC       strset.o
  CC       linker.o
  INSTALL  libbpf.pc
  AR       libbpf.a
  INSTALL  libbpf.a
make[2]: Leaving directory '/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/libbpf/src'
CC=clang CGO_CFLAGS="-I/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/output" CGO_LDFLAGS="/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/output/libbpf.a -lelf -lz" go build .
make[1]: Leaving directory '/home/rafaeldtinoco/work/sources/ebpf/libbpfgo'
clang -g -O2 -Wall -fpie -target bpf -I../../output -c self.bpf.c -o self.bpf.o
CC=clang CGO_CFLAGS="-I../../output" CGO_LDFLAGS="../../output/libbpf.a -lelf -lz" \
   go build ./main.go
sudo ./main
make: Leaving directory '/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/selftest/perfbuffers'
0

Both v0.2 and v0.4.0 works with perfbuffers selftest.

grantseltzer · 2021-07-05T19:08:40Z

I get the following error:

fedo vagrant in ~> libbpfgo on ~> 0f0f981
[*] make selftest                                                                                                                                                                                                                                                       130 ↵
building dir: ./selftest/libbpf-module...
make[1]: *** No targets specified and no makefile found.  Stop.
building dir: ./selftest/dist...
make[1]: *** No targets specified and no makefile found.  Stop.
building dir: ./selftest/build...
make -C /home/vagrant/go/src/github.com/aquasecurity/libbpfgo libbpfgo-static
CC=clang CGO_CFLAGS="-I/home/vagrant/go/src/github.com/aquasecurity/libbpfgo/output" CGO_LDFLAGS="/home/vagrant/go/src/github.com/aquasecurity/libbpfgo/output/libbpf.a -lelf -lz" go build .
clang -g -O2 -Wall -fpie -target bpf -I../../output -c libbpfgo_test.bpf.c -o libbpfgo_test.bpf.o
building dir: ./selftest/iterators...
make -C /home/vagrant/go/src/github.com/aquasecurity/libbpfgo libbpfgo-static
CC=clang CGO_CFLAGS="-I/home/vagrant/go/src/github.com/aquasecurity/libbpfgo/output" CGO_LDFLAGS="/home/vagrant/go/src/github.com/aquasecurity/libbpfgo/output/libbpf.a -lelf -lz" go build .
clang -g -O2 -Wall -fpie -target bpf -I../../output -c self.bpf.c -o self.bpf.o
CC=clang CGO_CFLAGS="-I../../output" CGO_LDFLAGS="../../output/libbpf.a -lelf -lz" \
   go build ./main.go
# runtime/cgo
clang-11: error: no such file or directory: '../../output/libbpf.a'
make[1]: *** [Makefile:32: main] Error 2
building dir: ./selftest/ringbuffers...
make -C /home/vagrant/go/src/github.com/aquasecurity/libbpfgo libbpfgo-static
CC=clang CGO_CFLAGS="-I/home/vagrant/go/src/github.com/aquasecurity/libbpfgo/output" CGO_LDFLAGS="/home/vagrant/go/src/github.com/aquasecurity/libbpfgo/output/libbpf.a -lelf -lz" go build .
clang -g -O2 -Wall -fpie -target bpf -I../../output -c self.bpf.c -o self.bpf.o
CC=clang CGO_CFLAGS="-I../../output" CGO_LDFLAGS="../../output/libbpf.a -lelf -lz" \
   go build ./main.go
# runtime/cgo
clang-11: error: no such file or directory: '../../output/libbpf.a'
make[1]: *** [Makefile:32: main] Error 2
building dir: ./selftest/uprobe...
make -C /home/vagrant/go/src/github.com/aquasecurity/libbpfgo libbpfgo-static
CC=clang CGO_CFLAGS="-I/home/vagrant/go/src/github.com/aquasecurity/libbpfgo/output" CGO_LDFLAGS="/home/vagrant/go/src/github.com/aquasecurity/libbpfgo/output/libbpf.a -lelf -lz" go build .
clang -g -O2 -Wall -fpie -target bpf -I../../output -c self.bpf.c -o self.bpf.o
CC=clang CGO_CFLAGS="-I../../output" CGO_LDFLAGS="../../output/libbpf.a -lelf -lz" \
   go build ./main.go
# runtime/cgo
clang-11: error: no such file or directory: '../../output/libbpf.a'
make[1]: *** [Makefile:32: main] Error 2
building dir: ./selftest/perfbuffers...
make -C /home/vagrant/go/src/github.com/aquasecurity/libbpfgo libbpfgo-static
CC=clang CGO_CFLAGS="-I/home/vagrant/go/src/github.com/aquasecurity/libbpfgo/output" CGO_LDFLAGS="/home/vagrant/go/src/github.com/aquasecurity/libbpfgo/output/libbpf.a -lelf -lz" go build .
clang -g -O2 -Wall -fpie -target bpf -I../../output -c self.bpf.c -o self.bpf.o
CC=clang CGO_CFLAGS="-I../../output" CGO_LDFLAGS="../../output/libbpf.a -lelf -lz" \
   go build ./main.go
# runtime/cgo
clang-11: error: no such file or directory: '../../output/libbpf.a'
make[1]: *** [Makefile:32: main] Error 2
make: *** [Makefile:68: selftest] Error 2

fedo vagrant in ~> libbpfgo on ~> 0f0f981
[*] ls output                                                                                                                                                                                                                                                             2 ↵
bpf  libbpf  libbpf.a  pkgconfig  vmlinux.h

fedo vagrant in ~> libbpfgo on ~> 0f0f981
[*] pwd
/home/vagrant/go/src/github.com/aquasecurity/libbpfgo

I'm not sure why it can't find libbpf.a, any idea?

rafaeldtinoco · 2021-07-05T19:50:38Z

Would u mind trying it again ?

$ make -C selftest/perfbuffers/ run ; echo $?
make: Entering directory '/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/selftest/perfbuffers'
make -C /home/rafaeldtinoco/work/sources/ebpf/libbpfgo libbpfgo-static
make[1]: Entering directory '/home/rafaeldtinoco/work/sources/ebpf/libbpfgo'
mkdir -p output/libbpf
/sbin/bpftool btf dump file /sys/kernel/btf/vmlinux format c > /home/rafaeldtinoco/work/sources/ebpf/libbpfgo/output/vmlinux.h
CC="gcc" CFLAGS="-g -O2 -Wall -fpie" LD_FLAGS="" \
   make -C /home/rafaeldtinoco/work/sources/ebpf/libbpfgo/libbpf/src \
	BUILD_STATIC_ONLY=1 \
	OBJDIR=/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/output/libbpf \
	DESTDIR=/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/output \
	INCLUDEDIR= LIBDIR= UAPIDIR= install
make[2]: Entering directory '/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/libbpf/src'
  MKDIR    staticobjs
  INSTALL  bpf.h libbpf.h btf.h libbpf_common.h libbpf_legacy.h xsk.h bpf_helpers.h bpf_helper_defs.h bpf_tracing.h bpf_endian.h bpf_core_read.h skel_internal.h
  CC       bpf.o
  CC       btf.o
  CC       libbpf.o
  CC       libbpf_errno.o
  CC       netlink.o
  CC       nlattr.o
  CC       str_error.o
  CC       libbpf_probes.o
  CC       bpf_prog_linfo.o
  CC       xsk.o
  CC       btf_dump.o
  CC       hashmap.o
  CC       ringbuf.o
  CC       strset.o
  CC       linker.o
  CC       gen_loader.o
  INSTALL  libbpf.pc
  AR       libbpf.a
  INSTALL  libbpf.a
make[2]: Leaving directory '/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/libbpf/src'
CC=clang CGO_CFLAGS="-I/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/output" CGO_LDFLAGS="/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/output/libbpf.a -lelf -lz" go build .
make[1]: Leaving directory '/home/rafaeldtinoco/work/sources/ebpf/libbpfgo'
clang -g -O2 -Wall -fpie -target bpf -I../../output -c self.bpf.c -o self.bpf.o
CC=clang CGO_CFLAGS="-I../../output" CGO_LDFLAGS="/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/output/libbpf.a -lelf -lz" \
   go build ./main.go
sudo ./main
make: Leaving directory '/home/rafaeldtinoco/work/sources/ebpf/libbpfgo/selftest/perfbuffers'
0

I'm using $(abspath ..) for the ../ directory, let's see if that works out in your setup.

rafaeldtinoco · 2021-07-05T19:52:12Z

Oh, I just realized you have:

building dir: ./selftest/libbpf-module...
make[1]: *** No targets specified and no makefile found.  Stop.
building dir: ./selftest/dist...

You might have to do a git submodule deinit --all --force and then checkout this repo. Then git submodule init libbpf again. I changed upstream libbpf location to ./libbpf and kept all tests inside ./libbpf/. I'm in doubt of ./selftest/build as it is not a selftest but a 'bpf' dependency for the build test (we could have it as ./libbpfgo_test.bpf.c, for example).

Anyway, it is just a suggestion on better organizing the tree and having a way to test regressions and bisections in an easier way. The ./selftest/*/Makefile can call a "run.sh" script inside the test directory (like you had).. for now I have only done 'perftest' as example.

grantseltzer · 2021-07-05T23:51:44Z

Alright the change you pushed appears to have fixed it! I'm now getting linker errors which I suspect have to do with my environment, but haven't tracked it down yet:

CC=clang CGO_CFLAGS="-I../../output" CGO_LDFLAGS="/home/vagrant/go/src/github.com/aquasecurity/libbpfgo/output/libbpf.a -lelf -lz" \
   go build ./main.go
# github.com/aquasecurity/libbpfgo
../../libbpfgo.go:1049:37: could not determine kind of name for C.BPF_TC_CUSTOM
../../libbpfgo.go:1047:37: could not determine kind of name for C.BPF_TC_EGRESS
../../libbpfgo.go:1055:26: could not determine kind of name for C.BPF_TC_F_REPLACE
../../libbpfgo.go:1046:37: could not determine kind of name for C.BPF_TC_INGRESS
../../libbpfgo.go:1152:9: could not determine kind of name for C.bpf_tc_attach
../../libbpfgo.go:1163:9: could not determine kind of name for C.bpf_tc_detach
../../libbpfgo.go:1133:9: could not determine kind of name for C.bpf_tc_hook_create
../../libbpfgo.go:1142:9: could not determine kind of name for C.bpf_tc_hook_destroy
../../libbpfgo.go:1174:9: could not determine kind of name for C.bpf_tc_query
../../libbpfgo.go:1098:12: could not determine kind of name for C.sizeof_struct_bpf_tc_hook
../../libbpfgo.go:1075:12: could not determine kind of name for C.sizeof_struct_bpf_tc_opts

rafaeldtinoco · 2021-07-06T10:57:54Z

Okay, so.. if you are compiling the libbpfgo-static, then you have to make sure your git submodule ./libbpf/ has a master branch checked to at least v0.4.0. I think this part shall be okay for you.

If you are compiling libbpfgo-dynamic (which is the default, and the one I think you are referring to) then you have to make sure the OS you are using (fedora ?) has at least the package libbpf v0.4.0. In my case, for example, I'm using:

Distributor ID:	Ubuntu
Description:	Ubuntu 21.04
Release:	21.04
Codename:	hirsute

BUT the default version for libbpf0 package (and libbpf-dev) is:

$ rmadison libbpf
 libbpf | 0.1.0-1        | groovy/universe | source
 libbpf | 0.3-2ubuntu1   | hirsute         | source
 libbpf | 0.4.0-1ubuntu1 | impish          | source

So I had to bring the 0.4.0-1ubuntu1 version from Ubuntu Impish (the -devel version) to make sure it contained the following libbpf upstream commit:

$ git log -1 --grep BPF_TC_CUSTOM | cat
commit d71ff87
Author: Kumar Kartikeya Dwivedi <[email protected]>
Date:   Wed May 12 20:41:22 2021

    libbpf: Add low level TC-BPF management API

    This adds functions that wrap the netlink API used for adding, manipulating,
    and removing traffic control filters.

    [snip]...

and

$ git tag --contains d71ff87
v0.4.0

grantseltzer · 2021-07-06T13:27:28Z

My libbpf submodule is certainly on a new enough commit, I just don't know why the self test compile isn't linking against it properly, it appears to be pointing to it.

rafaeldtinoco · 2021-07-06T13:31:46Z

These happened to be:

../../libbpfgo.go:1049:37: could not determine kind of name for C.BPF_TC_CUSTOM
../../libbpfgo.go:1047:37: could not determine kind of name for C.BPF_TC_EGRESS
../../libbpfgo.go:1055:26: could not determine kind of name for C.BPF_TC_F_REPLACE
../../libbpfgo.go:1046:37: could not determine kind of name for C.BPF_TC_INGRESS

when I had 0.3.0 from the OS, thus the previous comment. But, now I see that you had:

CC=clang CGO_CFLAGS="-I../../output" CGO_LDFLAGS="/home/vagrant/go/src/github.com/aquasecurity/libbpfgo/output/libbpf.a -lelf -lz" \

... so... maybe the env variables are not being taken into consideration ? Then what will be used will be the #cgo LDFLAGS: -lelf -lz -lbpf only. Weird :(.

grantseltzer · 2021-07-06T13:43:22Z

Ah ok I finally got it working. I had divergent packages for libbpf and libbpf-static on my system. I uninstalled and installed manually from source.

But you make a good point, this means that the self tests are building against my environment instead of the git submodule. Investigating

grantseltzer · 2021-07-06T16:19:22Z

Alright now that I have my environment issues figured out, this looks very good to me!

Can you please:

Add the run target to the other selftests
Add a short summary of how to run the selftests/build in the README?

rafaeldtinoco · 2021-07-06T16:34:54Z

Yes! I'm glad you liked it. I'll fix the RUN and improve instructions how to have this working. Thanks for reviewing it!

grantseltzer · 2021-07-06T16:46:29Z

Also no reason to keep the 'old' directories. I like this system better :-)

rafaeldtinoco · 2021-07-06T17:13:52Z

Definitely! It was kept as a reference until the +1 only. Will do.

rafaeldtinoco · 2021-07-07T14:59:16Z

Ha... I got it.. (why you are having issues). I did a small change to libbpfgo.go to have "-lbpf", that made it to ALWAYS link to shared libbpf. If i remove that, I can have static or dynamic linking.

$ ldd main
	linux-vdso.so.1 (0x00007fffa9ff8000)
	libelf.so.1 => /lib/x86_64-linux-gnu/libelf.so.1 (0x00007ff224beb000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007ff224bcf000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff224bad000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff2249c1000)
	/lib64/ld-linux-x86-64.so.2 (0x00007ff224c19000)

But then I need to find a way to automatically dynamic link to libbpf when just doing 'go build & go install' without having to manually specify CGO_LDFLAGS containing "-lbpf".

grantseltzer · 2021-07-07T15:07:19Z

Yup that makes sense. We should avoid having -lbpf so that consumers of libbpfgo can bring their own libbpf.

rafaeldtinoco · 2021-07-08T16:49:39Z

Alright, this is ready if you don't find anything else worth doing together in this PR.

Thanks for reviewing @grantseltzer

grantseltzer

Just a couple comments, but otherwise this is great!

Readme.md

itaysk

made it halfway through and had to drop it, will continue next time

Makefile

Readme.md

Makefile

Closes: #33 $ make libbpfgo-static-test ./libbpfgo.go:790:15: errptrError format %s has arg pid of wrong type int ./libbpfgo.go:802:14: Sprintf format %s has arg offset of wrong type uint32

grantseltzer

This looks very good to me! I will follow up to this merge with a PR handling #35

@itaysk Do you have any additional comments before merging?

libbpfgo.go

selftest/build/Makefile

rafaeldtinoco · 2021-07-09T15:21:56Z

There is a small orthogonal, to this change, problem:

I'm removing CFLAGS #cgo pragma from libbpfgo.go so we can have full dynamic and static builds for the library test, consumers (local) and selftests. Problem is... without:

// #cgo LDFLAGS: -lelf -lz -lbpf

An end user of this module (through go.mod -> require) won't be able to compile it locally (even if having libbpf package installed). Problem raises exactly there:

If I add the pragma there is no way I can override it (thus making static builds, important for development/regression/etc).
If I don't have it then simply having 'go build' in a shared lib environment won't work.

Best idea I had so far: I could add a wrapper for CC (clang.sh) to remove CFLAGS being automatically added by the pragma WHEN building statically and/or locally.

Do either @itaysk or @grantseltzer have a better idea how to solve this ?

grantseltzer · 2021-07-10T03:33:38Z

An end user of this module (through go.mod -> require) won't be able to compile it locally (even if having libbpf package installed). Problem raises exactly there:

Unless they explicitly specify the path to the libraries, right?

If I add the pragma there is no way I can override it (thus making static builds, important for development/regression/etc).

If I don't have it then simply having 'go build' in a shared lib environment won't work.

Best idea I had so far: I could add a wrapper for CC (clang.sh) to remove CFLAGS being automatically added by the pragma WHEN building statically and/or locally.

Do either @itaysk or @grantseltzer have a better idea how to solve this ?

What do you think about keeping -lelf and -lz but removing -lbpf, and document how consumers should bring their own libbpf? At least until libbpf is more stable?

itaysk · 2021-07-10T04:59:42Z

What do you think about keeping -lelf and -lz but removing -lbpf, and document how consumers should bring their own libbpf?

Yes this is what I meant when making the comment. Sorry, I thought it was obvious. (Please very I'm correct, but I think this is how it works today)

BTW - maybe we need a sanity test for that use case but not a blocker for this PR

rafaeldtinoco · 2021-07-11T01:41:22Z

An end user of this module (through go.mod -> require) won't be able to compile it locally (even if having libbpf package installed). Problem raises exactly there:

Unless they explicitly specify the path to the libraries, right?

To be honest I think the libbpfgo package should use CGO PKG-CONFIG stanza. This would make very easy for all distros to use it as they all rely in PKG-CONFIG for CFLAGS/LDFLAGS:

#cgo pkg-config: bpf

Problem is, by adding any stanza 'by default' we lose the static/dynamic capability (I could not figure a way to override them).

If I add the pragma there is no way I can override it (thus making static builds, important for development/regression/etc).

If I don't have it then simply having 'go build' in a shared lib environment won't work.

Best idea I had so far: I could add a wrapper for CC (clang.sh) to remove CFLAGS being automatically added by the pragma WHEN building statically and/or locally.

Do either @itaysk or @grantseltzer have a better idea how to solve this ?

What do you think about keeping -lelf and -lz but removing -lbpf, and document how consumers should bring their own libbpf? At least until libbpf is more stable?

I think 'go build' will have to work out of the box so users will simply include it and use it without worrying much. I have seen things such as go generate that perhaps could help us out with this, but haven't dig into it yet =(.

rafaeldtinoco · 2021-07-11T01:48:27Z

Alright, I'm returning the:

// #cgo LDFLAGS: -lelf -lz

it works for the dynamic and static builds. For now the 'go build' won't work out-of-the-box because it will miss -lbpf LDFLAG. Maybe we can open an issue later to check if we can use 'go generate', something like having wrappers such as this example here:

cilium/ebpf#305

rafaeldtinoco · 2021-07-11T01:52:44Z

What do you think about keeping -lelf and -lz but removing -lbpf, and document how consumers should bring their own libbpf?

I'm working on this (low priority, as we've spoken). I'm porting my portablebpf example to libbpfgo and I was thinking in keeping both in an 'examples/' folder, with a good Readme.md (for example). This go towards with what @grantseltzer was saying about keeping documentation in a single place (don't want to keep in my personal repo as it will be interesting for libbpfgo). It shall explain how to use libbpfgo both dynamically and statically for a simple portable libbpfgo project.

Currently you will find the following GNU Makefile rules: | Makefile Rule | Description | |--------------------------|-----------------------------------| | all | builds libbpfgo (dynamic) | | clean | cleans entire tree | | selftest | builds all selftests (static) | | selftest-run | runs all selftests (static) | * libbpf dynamically linked (libbpf from OS) | Makefile Rule | Description | |--------------------------|-----------------------------------| | libbpfgo-dynamic | builds dynamic libbpfgo (libbpf) | | libbpfgo-dynamic-test | 'go test' with dynamic libbpfgo | | selftest-dynamic | build tests with dynamic libbpfgo | | selftest-dynamic-run | run tests using dynamic libbpfgo | * statically compiled (libbpf submodule) | Makefile Rule | Description | |--------------------------|-----------------------------------| | libbpfgo-static | builds static libbpfgo (libbpf) | | libbpfgo-static-test | 'go test' with static libbpfgo | | selftest-static | build tests with static libbpfgo | | selftest-static-run | run tests using static libbpfgo | * examples $ make libbpfgo-static => libbpfgo statically linked with libbpf $ make -C selftest/perfbuffers => single selftest build (static libbpf) $ make -C selftest/perfbuffers run-dynamic => single selftest run (dynamic libbpf) $ make selftest-static-run => will build & run all static selftests > Note 01: dynamic builds need your OS to have a *recent enough* libbpf package (and its headers) installed. Sometimes, recent features might require the use of backported OS packages in order for your OS to contain latest *libbpf* features (sometimes required by libbpfgo). > Note 02: static builds need `git submodule init` first. Make sure to sync the *libbpf* git submodule before trying to statically compile or test the *libbpfgo* repository.

All static binaries (and selftests) were being built statically against libbpf only. This change makes them full static binaries.

rafaeldtinoco · 2021-07-11T02:56:41Z

I'm using the following to test this change:

$ make libbpfgo-dynamic
$ make libbpfgo-dynamic-test
$ make clean
$ make libbpfgo-dynamic-test
$ make clean
$ make output/libbpf.h
$ make clean
$ make libbpfgo-static
$ make libbpfgo-static-test
$ make clean
$ make libbpfgo-static-test
$ make clean
$ make selftest-dynamic
$ make selftest-static
$ make clean
$ make -C selftest/perfbuffers libbpfgo-static
$ make -C selftest/perfbuffers libbpfgo-dynamic
$ make clean
$ make -C selftest/perfbuffers main-static
$ make -C selftest/perfbuffers main-dynamic
$ make clean

grantseltzer

I think this is all in the right spot, lgtm! I will handle #35 shortly

@itaysk anything you still want to address?

itaysk · 2021-07-11T05:15:48Z

just want to make sure, if someone just imports libbpfgo into a go program and go builds it. will it work now? did it work before? if there's no regression then it lgtm

rafaeldtinoco · 2021-07-11T23:52:59Z

Since this commit:

commit c0c9e4d
Author: Yaniv Agman <[email protected]>
Date:   Wed Apr 7 08:36:20 2021

    Fix build with libbpfgo

diff --git a/libbpfgo.go b/libbpfgo.go
index 316e46d..6ab03cb 100644
--- a/libbpfgo.go
+++ b/libbpfgo.go
@@ -1,7 +1,7 @@
 package libbpfgo

 /*
-#cgo LDFLAGS: -lelf -lz -lbpf
+#cgo LDFLAGS: -lelf -lz

$ git tag --contains c0c9e4d
v0.1.0
v0.1.1

the cgo LDFLAGS stanza did not have -lbpf. I guess tracee is dealing with that on its own. Any other consumers might be doing the same. I'd say we're good (wih no regressions) but this is something we have to address (the 'go install' out of the box in a dynamic build AND allowing the static builds). What about merging this one, to unblock @grantseltzer's other commits, and we can address this in a merge on its own ?

itaysk · 2021-07-12T08:37:21Z

What about merging this one, to unblock @grantseltzer's other commits, and we can address this in a merge on its own ?

👍

This comment has been minimized.

Sign in to view

grantseltzer mentioned this pull request Jul 7, 2021

Change BPFMap method Update to unsafe.Pointer #32

Merged

This was referenced Jul 8, 2021

consumers using CO.RE might need vmlinux.h generation from BTF: it might not be available OR be different compile/run time #30

Closed

errptrError format %s has arg pid of wrong type int #33

Closed

grantseltzer suggested changes Jul 8, 2021

View reviewed changes

Readme.md Outdated Show resolved Hide resolved

Readme.md Outdated Show resolved Hide resolved

itaysk reviewed Jul 8, 2021

View reviewed changes

Makefile Show resolved Hide resolved

Readme.md Show resolved Hide resolved

Makefile Outdated Show resolved Hide resolved

Makefile Outdated Show resolved Hide resolved

rafaeldtinoco requested a review from grantseltzer July 8, 2021 22:18

rafaeldtinoco added 3 commits July 9, 2021 09:35

Move selftest/libbpf-module to libbpf

8388ce5

Fix: small fix after uprobe support

56ad7c8

Closes: #33 $ make libbpfgo-static-test ./libbpfgo.go:790:15: errptrError format %s has arg pid of wrong type int ./libbpfgo.go:802:14: Sprintf format %s has arg offset of wrong type uint32

Readme.md: small rephrasings

5dadae2

grantseltzer mentioned this pull request Jul 9, 2021

Fix broken selftests #35

Closed

grantseltzer previously approved these changes Jul 9, 2021

View reviewed changes

itaysk reviewed Jul 9, 2021

View reviewed changes

libbpfgo.go Outdated Show resolved Hide resolved

selftest/build/Makefile Show resolved Hide resolved

rafaeldtinoco dismissed grantseltzer’s stale review via db7701b July 9, 2021 15:13

rafaeldtinoco added 6 commits July 10, 2021 22:56

Update Readme.md with build instructions and minor fixes

ec5e8a1

selftest/map-update: fixes for the new tree structure

e91772f

Makefile: better info on errors

cbf3905

Makefile: turn static builds into full static

a7d9828

All static binaries (and selftests) were being built statically against libbpf only. This change makes them full static binaries.

Makefile: un-phony vmlinux.h

90ca338

grantseltzer approved these changes Jul 11, 2021

View reviewed changes

rafaeldtinoco merged commit a683635 into aquasecurity:main Jul 12, 2021

rafaeldtinoco mentioned this pull request Jul 30, 2021

CFLAGS #cgo pragma in libbpfgo.go cannot be ignored for static building #1

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Makefile: improvements #28

Makefile: improvements #28

rafaeldtinoco commented Jul 1, 2021 •

edited

Loading

rafaeldtinoco commented Jul 3, 2021

grantseltzer commented Jul 5, 2021

rafaeldtinoco commented Jul 5, 2021 •

edited

Loading

rafaeldtinoco commented Jul 5, 2021 •

edited

Loading

grantseltzer commented Jul 5, 2021

rafaeldtinoco commented Jul 6, 2021 •

edited

Loading

grantseltzer commented Jul 6, 2021

rafaeldtinoco commented Jul 6, 2021

grantseltzer commented Jul 6, 2021

grantseltzer commented Jul 6, 2021

rafaeldtinoco commented Jul 6, 2021

grantseltzer commented Jul 6, 2021

rafaeldtinoco commented Jul 6, 2021

This comment has been minimized.

rafaeldtinoco commented Jul 7, 2021 •

edited

Loading

grantseltzer commented Jul 7, 2021

rafaeldtinoco commented Jul 8, 2021

grantseltzer left a comment

itaysk left a comment

grantseltzer left a comment

rafaeldtinoco commented Jul 9, 2021 •

edited

Loading

grantseltzer commented Jul 10, 2021

itaysk commented Jul 10, 2021 •

edited

Loading

rafaeldtinoco commented Jul 11, 2021

rafaeldtinoco commented Jul 11, 2021 •

edited

Loading

rafaeldtinoco commented Jul 11, 2021 •

edited

Loading

rafaeldtinoco commented Jul 11, 2021

grantseltzer left a comment

itaysk commented Jul 11, 2021 •

edited

Loading

rafaeldtinoco commented Jul 11, 2021 •

edited

Loading

itaysk commented Jul 12, 2021

Makefile: improvements #28

Makefile: improvements #28

Conversation

rafaeldtinoco commented Jul 1, 2021 • edited Loading

rafaeldtinoco commented Jul 3, 2021

grantseltzer commented Jul 5, 2021

rafaeldtinoco commented Jul 5, 2021 • edited Loading

rafaeldtinoco commented Jul 5, 2021 • edited Loading

grantseltzer commented Jul 5, 2021

rafaeldtinoco commented Jul 6, 2021 • edited Loading

grantseltzer commented Jul 6, 2021

rafaeldtinoco commented Jul 6, 2021

grantseltzer commented Jul 6, 2021

grantseltzer commented Jul 6, 2021

rafaeldtinoco commented Jul 6, 2021

grantseltzer commented Jul 6, 2021

rafaeldtinoco commented Jul 6, 2021

This comment has been minimized.

rafaeldtinoco commented Jul 7, 2021 • edited Loading

grantseltzer commented Jul 7, 2021

rafaeldtinoco commented Jul 8, 2021

grantseltzer left a comment

Choose a reason for hiding this comment

itaysk left a comment

Choose a reason for hiding this comment

grantseltzer left a comment

Choose a reason for hiding this comment

rafaeldtinoco commented Jul 9, 2021 • edited Loading

grantseltzer commented Jul 10, 2021

itaysk commented Jul 10, 2021 • edited Loading

rafaeldtinoco commented Jul 11, 2021

rafaeldtinoco commented Jul 11, 2021 • edited Loading

rafaeldtinoco commented Jul 11, 2021 • edited Loading

rafaeldtinoco commented Jul 11, 2021

grantseltzer left a comment

Choose a reason for hiding this comment

itaysk commented Jul 11, 2021 • edited Loading

rafaeldtinoco commented Jul 11, 2021 • edited Loading

itaysk commented Jul 12, 2021

rafaeldtinoco commented Jul 1, 2021 •

edited

Loading

rafaeldtinoco commented Jul 5, 2021 •

edited

Loading

rafaeldtinoco commented Jul 5, 2021 •

edited

Loading

rafaeldtinoco commented Jul 6, 2021 •

edited

Loading

rafaeldtinoco commented Jul 7, 2021 •

edited

Loading

rafaeldtinoco commented Jul 9, 2021 •

edited

Loading

itaysk commented Jul 10, 2021 •

edited

Loading

rafaeldtinoco commented Jul 11, 2021 •

edited

Loading

rafaeldtinoco commented Jul 11, 2021 •

edited

Loading

itaysk commented Jul 11, 2021 •

edited

Loading

rafaeldtinoco commented Jul 11, 2021 •

edited

Loading