Make Centipede a default engine #9427

DonggeLiu · 2023-01-16T00:02:29Z

Make Centipede a default fuzzing engine and apply it to more projects:

Add its build steps to expected_build_steps.json.
- Should we delete the previous test_get_centipede_build_steps, given it has been covered in test_get_build_steps now?

DonggeLiu · 2023-01-16T00:03:00Z

/gcbrun trial_build.py all --sanitizer none --fuzzing-engine centipede

DonggeLiu · 2023-01-16T04:13:24Z

/gcbrun trial_build.py all --sanitizer none --fuzzing-engine centipede

Looks like until google#9427 is merged it should be specified explicitly.

evverx · 2023-01-16T09:11:53Z

To judge from https://github.com/google/oss-fuzz/actions/runs/3928694793/jobs/6716605398 systemd fails to build with centipede due to I assume another variation on mesonbuild/meson#4542 or something like that. I can exclude centipede myself by explicitly listing the three fuzzing engines the build system currently supports later but to prevent the next systemd build from failing on CF it would be great if it could be excluded here.

evverx · 2023-01-16T09:24:15Z

Just to clarify I enabled it explicitly in #9428. It doesn't fail anywhere else yet.

evverx · 2023-01-16T15:11:13Z

Before I forget other projects using meson like dbus-broker fail to build with centipede too. I think if by analogy with COVERAGE_FLAGS_coverage OSS-Fuzz prepended -Wl to linker options it would make centipede work with meson as well because mesonbuild/meson#4542 was fixed by filtering out external linker flags passed via CFLAGS starting with -Wl. I'm not sure meson is going to start filtering out anything else.

DonggeLiu · 2023-01-17T06:58:48Z

Before I forget other projects using meson like dbus-broker fail to build with centipede too. I think if by analogy with COVERAGE_FLAGS_coverage OSS-Fuzz prepended -Wl to linker options it would make centipede work with meson as well because mesonbuild/meson#4542 was fixed by filtering out external linker flags passed via CFLAGS starting with -Wl. I'm not sure meson is going to start filtering out anything else.

Thanks, @evverx!
Yeah, Centipede fails on all projects that have -Werror -Wunused-command-line-argument. We made an attempt to address it, but that does not fix all projects, as you pointed out.

-Wl is a good point, thanks for suggesting that!
While I was experimenting -Wl on systemd, I got a different error:

../../src/systemd/src/basic/alloc-util.h:123:35: error: static declaration of 'reallocarray' follows non-static declaration
_alloc_(2, 3) static inline void *reallocarray(void *p, size_t need, size_t size) {
                                  ^
/usr/include/stdlib.h:559:14: note: previous declaration is here
extern void *reallocarray (void *__ptr, size_t __nmemb, size_t __size)

Would you happen to know how to fix it? Thanks!

By the way, we plan to make Centipede support all projects, as it will replace libFuzzer.

DonggeLiu · 2023-01-17T07:04:18Z

/gcbrun trial_build.py all --sanitizer none --fuzzing-engine centipede

DonggeLiu · 2023-01-17T09:05:49Z

/gcbrun trial_build.py all --sanitizer none --fuzzing-engine centipede

evverx · 2023-01-17T09:57:07Z

While I was experimenting -Wl on systemd, I got a different error:

Looks like cc.has_function went sideways somewhere because it's supposed to set HAVE_REALLOCARRAY when reallocarray is shipped with glibc to prevent build failures exactly like that. I'll try to reproduce it locally to get the meson build logs. Just to make sure did it fail when LIBRARIES_FLAGS="-Wno-unused-command-line-argument -Wl,-ldl,-lrt,-lpthread,$SRC/centipede/weak.o" was added to CFLAGS?

DonggeLiu · 2023-01-17T10:44:02Z

While I was experimenting -Wl on systemd, I got a different error:

Looks like cc.has_function went sideways somewhere because it's supposed to set HAVE_REALLOCARRAY when reallocarray is shipped with glibc to prevent build failures exactly like that. I'll try to reproduce it locally to get the meson build logs. Just to make sure did it fail when LIBRARIES_FLAGS="-Wno-unused-command-line-argument -Wl,-ldl,-lrt,-lpthread,$SRC/centipede/weak.o" was added to CFLAGS?

Yep, the same as in this commit Use -Wl for linker flags.

evverx · 2023-01-17T11:45:55Z

I rebuilt the base-builder image with those flags and reproduced the build failure. The good news is that systemd got to that point :-) (which means that -Wl works and get_supported_arguments(possible_cc_flags) no longer fails). cc.has_function fails to build its test programs with something like

Command line:  clang -L/usr/lib/clang/15.0.0/lib/linux /work/build/meson-private/tmp77ei02ar/testfile.c -o /work/build/meson-private/tmp77ei02ar/output.exe -O1 -fno-omit-frame-pointer -gline-tables-only -fsanitize-coverage=trace-loads -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fno-builtin -fsanitize-coverage=trace-pc-guard,pc-table,trace-cmp -O2 -gline-tables-only -Wno-unused-command-line-argument -fsanitize=fuzzer-no-link -D_FILE_OFFSET_BITS=64 -O0 -Werror=implicit-function-declaration -std=gnu11 -D_GNU_SOURCE
...
Compiler stderr:^M
 /usr/bin/ld: /usr/bin/ld: DWARF error: invalid or unhandled FORM value: 0x25^M
/tmp/testfile-bd5938.o: in function `main':^M
testfile.c:(.text.main[main]+0x41): undefined reference to `__sanitizer_cov_load8'^M
/usr/bin/ld: testfile.c:(.text.main[main]+0x52): undefined reference to `__sanitizer_cov_load8'^M
clang-15: error: linker command failed with exit code 1 (use -v to see invocation)^M

I'm not sure what to do about it yet. I don't think I can fix it in the systemd OSS-Fuzz build script because it's somewhere between meson and OSS-Fuzz.

evverx · 2023-01-17T12:11:34Z

dbus-broker no longer fails to build though (because cc.has_function isn't used there). lxc fails due to the same cc.has_function issue.

DonggeLiu · 2023-01-17T12:29:14Z

I rebuilt the base-builder image with those flags and reproduced the build failure. The good news is that systemd got to that point :-) (which means that -Wl works and get_supported_arguments(possible_cc_flags) no longer fails). cc.has_function fails to build its test programs with something like
Command line:  clang -L/usr/lib/clang/15.0.0/lib/linux /work/build/meson-private/tmp77ei02ar/testfile.c -o /work/build/meson-private/tmp77ei02ar/output.exe -O1 -fno-omit-frame-pointer -gline-tables-only -fsanitize-coverage=trace-loads -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fno-builtin -fsanitize-coverage=trace-pc-guard,pc-table,trace-cmp -O2 -gline-tables-only -Wno-unused-command-line-argument -fsanitize=fuzzer-no-link -D_FILE_OFFSET_BITS=64 -O0 -Werror=implicit-function-declaration -std=gnu11 -D_GNU_SOURCE
...
Compiler stderr:^M
 /usr/bin/ld: /usr/bin/ld: DWARF error: invalid or unhandled FORM value: 0x25^M
/tmp/testfile-bd5938.o: in function `main':^M
testfile.c:(.text.main[main]+0x41): undefined reference to `__sanitizer_cov_load8'^M
/usr/bin/ld: testfile.c:(.text.main[main]+0x52): undefined reference to `__sanitizer_cov_load8'^M
clang-15: error: linker command failed with exit code 1 (use -v to see invocation)^M
I'm not sure what to do about it yet. I don't think I can fix it in the systemd OSS-Fuzz build script because it's somewhere between meson and OSS-Fuzz.

My experience is that errors like /usr/bin/ld: testfile.c:(.text.main[main]+0x52): undefined reference to '__sanitizer_cov_load8'^M can be fixed by making use of weak.o from Centipede.
This weak.o defines the weak symbols of those functions.

However, I am not sure what is the best way to make use of it in systemd.
Would you happen to have some thoughts?

evverx · 2023-01-17T12:48:21Z

Ideally OSS-Fuzz should pass linker flags via LDFLAGS (at least that's where meson expects to find them: mesonbuild/meson#4542 (comment)). Having said that as far as I can remember LDFLAGS broke something else and were rolled back at some point.

With $SRC/centipede/weak.o added to LDFLAGS explicitly systemd can be built with centipede as far as I can see but it would have to be added to every project where has_function is used and given that I already have to carry kludges like 7ac2b25 I don't think it's very convenient.

evverx · 2023-01-17T14:22:13Z

FWIW I think in theory it could work if OSS-Fuzz passed those flags via LDFLAGS as well. It didn't work out with FI because it needed to switch linkers and some projects didn't expect it so CFLAGS was the way to go there but with centipede the linker is intact as far as I can tell.

DonggeLiu · 2023-01-17T20:27:22Z

FWIW I think in theory it could work if OSS-Fuzz passed those flags via LDFLAGS as well. It didn't work out with FI because it needed to switch linkers and some projects didn't expect it so CFLAGS was the way to go there but with centipede the linker is intact as far as I can tell.

I see... Yeah adding it to LDFLAGS is not convenient.
But it's good to know that works! Thanks a lot!

I will try to think of ways to do it in one place : )

Mirrors [the fixes from OSS-Fuzz](google/oss-fuzz#9427): 1. Use [`-Wl` on the linker flags](google/oss-fuzz#9427 (comment)). 2. Use [`LDFLAGS`](google/oss-fuzz#9427 (comment)).

DonggeLiu · 2023-01-19T12:05:26Z

/gcbrun trial_build.py all --sanitizer none --fuzzing-engine centipede

jonathanmetzman · 2023-01-23T14:28:21Z

Related: #9299

DonggeLiu · 2023-02-02T03:24:54Z

/gcbrun trial_build.py all --sanitizer none --fuzzing-engine centipede

DonggeLiu · 2023-02-02T11:24:30Z

/gcbrun trial_build.py all --sanitizer none --fuzzing-engine centipede

DonggeLiu · 2023-02-05T12:07:29Z

/gcbrun trial_build.py all --sanitizer none --fuzzing-engine centipede

DonggeLiu · 2023-02-05T23:18:18Z

@jonathanmetzman, 334aff6 compiles weak.o with $CC instead of $CXX as we discussed.
The undefined reference to operator new()/delete() error reemerged.
I suppose we will still need to compile/link target with -lc++ : )

This reverts commit 334aff6.

DonggeLiu · 2023-02-05T23:21:05Z

/gcbrun trial_build.py all --sanitizer none --fuzzing-engine centipede

jonathanmetzman · 2023-02-06T14:40:11Z

@jonathanmetzman, 334aff6 compiles weak.o with $CC instead of $CXX as we discussed. The undefined reference to operator new()/delete() error reemerged. I suppose we will still need to compile/link target with -lc++ : )

What is LIB_FUZZING_ENGINE compiled with? Also clang++?

jonathanmetzman · 2023-02-06T14:40:51Z

@jonathanmetzman, 334aff6 compiles weak.o with $CC instead of $CXX as we discussed. The undefined reference to operator new()/delete() error reemerged. I suppose we will still need to compile/link target with -lc++ : )

What is LIB_FUZZING_ENGINE compiled with? Also clang++?

I think this doesn't happen with AFL(++) because I'm pretty sure they only use C for the part that gets linked to user code.

In #9427, some projects [failed on `bad_build_check`](https://pantheon.corp.google.com/cloud-build/builds;region=us-central1/31a7e835-32aa-49c8-81f1-f8d2eab9e3c1;step=20?project=oss-fuzz) (e.g., [`matio`](https://github.com/google/oss-fuzz/tree/master/projects/matio) has seed corpus) without explicit reason. This is because `Centipede` [does not skip seed corpus in `run_fuzzer`](https://github.com/google/oss-fuzz/blob/a5ee351ed1cfb97b73eab96787c0a86434cfd99e/infra/base-images/base-runner/run_fuzzer#L89) like [other fuzzers](https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-runner/bad_build_check#L147). This PR sets the environment variable so that `Centipede` also skips seeds and pass the `bad_build_check`.

jonathanmetzman · 2023-01-23T14:30:38Z

infra/base-images/base-builder/compile_centipede

+# For Meson.
+export LDFLAGS='/src/centipede/weak.o'


I'm still not clear why centipede is so unique that we need to support LDFLAGS after six years of not needing it.

Which projects does this help? Most meson projects seem broken still.

Yeah, I will test removing it here.
I've added weak.o to ld wrapper, not sure if that works for meson.

jonathanmetzman · 2023-01-23T14:30:43Z

infra/base-images/base-builder/compile_centipede

+# For Meson.
+export LDFLAGS='/src/centipede/weak.o'
+
+# For Centipede's ld


end with period.

jonathanmetzman · 2023-01-23T14:31:07Z

infra/base-images/base-builder/compile_centipede

+
+# For Centipede's ld
+export CENTIPEDE_BIN="$SRC/centipede/bin"
+export PATH="$CENTIPEDE_BIN":$PATH


Put ending quote on the end of this line.

jonathanmetzman · 2023-02-06T18:07:05Z

It's extremely hard for me to test anything today because archive.ubuntu.com is down. But you can see here that a project that is having issues with libc++ not being linked in, is doing something incorrect. They are using CC to link fuzz targets despite our rule requiring use of CXX for this purpose

jonathanmetzman · 2023-02-06T18:09:37Z

/gcbrun trial_build.py all --sanitizer none --fuzzing-engine centipede

jonathanmetzman · 2023-02-06T19:48:29Z

Related: #9299

jonathanmetzman · 2023-02-06T20:40:19Z

It's extremely hard for me to test anything today because archive.ubuntu.com is down. But you can see here that a project that is having issues with libc++ not being linked in, is doing something incorrect. They are using CC to link fuzz targets despite our rule requiring use of CXX for this purpose

Looks like this hunch was correct #9599

jonathanmetzman · 2023-02-06T21:30:42Z

/gcbrun trial_build.py all --sanitizer none --fuzzing-engine centipede

jonathanmetzman · 2023-02-06T23:49:01Z

/gcbrun trial_build.py all --sanitizer none --fuzzing-engine centipede

jonathanmetzman · 2023-02-06T23:49:35Z

/gcbrun trial_build.py skcms systemd --sanitizer none address --fuzzing-engine libfuzzer afl honggfuzz

DonggeLiu · 2023-02-07T00:21:29Z

/gcbrun trial_build.py all --sanitizer none --fuzzing-engine centipede

How do we confirm the exact build ID of this command? (It's the second one above this.)
The build history page has two builds (4285852f, 4edf647e) on the latest commit 38d53f2, but I am not sure why the first one failed and the second passed.

There is a third one on the top, 827d915c. I presume that is the last one ( /gcbrun trial_build.py skcms systemd --sanitizer none address --fuzzing-engine libfuzzer afl honggfuzz)

jonathanmetzman · 2023-02-07T00:50:00Z

How do we confirm the exact build ID of this command? (It's the second one above this.)
The build history page has two builds (4285852f, 4edf647e) on the latest commit 38d53f2, but I am not sure why the first one failed and the second passed.

I think youre right. But you have to look at what gets built to figure this out.

DonggeLiu · 2023-02-07T02:45:13Z

How do we confirm the exact build ID of this command? (It's the second one above this.)
The build history page has two builds (4285852f, 4edf647e) on the latest commit 38d53f2, but I am not sure why the first one failed and the second passed.

I think youre right. But you have to look at what gets built to figure this out.

Both 4edf647e and 827d915c only tested skcms and systemd, which is why they passed.

Only 4285852f tested Centipede on all projects, and it failed as expected.

jonathanmetzman · 2023-02-07T03:36:50Z

infra/base-images/base-builder/precompile_centipede

+#!/bin/bash
+/usr/bin/ld \$@ -ldl -lrt -lpthread -lc++ /src/centipede/weak.o
+EOF
+chmod 777 ${CENTIPEDE_BIN}/ld


Are we setting this to be the linker somewhere?

Line 37-39 in compile_centipede.

Ah got it. Do you mind giving me an hour to test #9300? I have a hunch most of the major fixes have been pushed and I think this PR has some unecessary complexity. I suspect this LD wrapper isn't necessary

Sure, I am working on google/fuzzbench#1632 now.

In google#9427, some projects [failed on `bad_build_check`](https://pantheon.corp.google.com/cloud-build/builds;region=us-central1/31a7e835-32aa-49c8-81f1-f8d2eab9e3c1;step=20?project=oss-fuzz) (e.g., [`matio`](https://github.com/google/oss-fuzz/tree/master/projects/matio) has seed corpus) without explicit reason. This is because `Centipede` [does not skip seed corpus in `run_fuzzer`](https://github.com/google/oss-fuzz/blob/a5ee351ed1cfb97b73eab96787c0a86434cfd99e/infra/base-images/base-runner/run_fuzzer#L89) like [other fuzzers](https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-runner/bad_build_check#L147). This PR sets the environment variable so that `Centipede` also skips seeds and pass the `bad_build_check`.

DonggeLiu marked this pull request as draft January 16, 2023 00:02

Make Centipede a default engine

be42b22

DonggeLiu force-pushed the make_centipede_default_engine branch from f623023 to be42b22 Compare January 16, 2023 02:16

Add Centipede's build steps to expected build steps

e2a15d8

evverx added a commit to evverx/oss-fuzz that referenced this pull request Jan 16, 2023

[systemd] bring centipede explicitly

9e02032

Looks like until google#9427 is merged it should be specified explicitly.

evverx mentioned this pull request Jan 16, 2023

Projects using meson fail to compile with centipede #9431

Closed

Use -Wl for linker flags

87b26c4

Fix flag

3b3c1da

Add week.o to LDFLAGS

3e7af92

DonggeLiu mentioned this pull request Jan 18, 2023

Fix centipede linker flags google/fuzzbench#1610

Merged

A wrapper of ld that always append centipede flags

1b4b623

Add -lc++ to ld

9c536a4

evverx mentioned this pull request Feb 2, 2023

oss-fuzz: build libcap and libmount along with systemd systemd/systemd#25842

Closed

DonggeLiu mentioned this pull request Feb 5, 2023

Make Centipede compatible with as many existing projects as possible. #9583

Open

Build weak.o with $CC and omit -lc++

334aff6

Revert "Build weak.o with $CC and omit -lc++"

d43e235

This reverts commit 334aff6.

DonggeLiu mentioned this pull request Feb 6, 2023

Centipede skips seed corpus in build test #9587

Merged

jonathanmetzman reviewed Feb 6, 2023

View reviewed changes

Merge branch 'master' into make_centipede_default_engine

e633dfc

Merge branch 'master' into make_centipede_default_engine

38d53f2

jonathanmetzman reviewed Feb 7, 2023

View reviewed changes

DonggeLiu closed this May 4, 2023

Make Centipede a default engine #9427

Make Centipede a default engine #9427

Conversation

DonggeLiu commented Jan 16, 2023 • edited Loading

DonggeLiu commented Jan 16, 2023

DonggeLiu commented Jan 16, 2023

evverx commented Jan 16, 2023

evverx commented Jan 16, 2023

evverx commented Jan 16, 2023

DonggeLiu commented Jan 17, 2023 • edited Loading

DonggeLiu commented Jan 17, 2023

DonggeLiu commented Jan 17, 2023

evverx commented Jan 17, 2023

DonggeLiu commented Jan 17, 2023

evverx commented Jan 17, 2023

evverx commented Jan 17, 2023 • edited Loading

DonggeLiu commented Jan 17, 2023 • edited Loading

evverx commented Jan 17, 2023

evverx commented Jan 17, 2023

DonggeLiu commented Jan 17, 2023

DonggeLiu commented Jan 19, 2023

jonathanmetzman commented Jan 23, 2023

DonggeLiu commented Feb 2, 2023

DonggeLiu commented Feb 2, 2023

DonggeLiu commented Feb 5, 2023

DonggeLiu commented Feb 5, 2023

DonggeLiu commented Feb 5, 2023

jonathanmetzman commented Feb 6, 2023

jonathanmetzman commented Feb 6, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonathanmetzman commented Feb 6, 2023

jonathanmetzman commented Feb 6, 2023

jonathanmetzman commented Feb 6, 2023

jonathanmetzman commented Feb 6, 2023

jonathanmetzman commented Feb 6, 2023

jonathanmetzman commented Feb 6, 2023

jonathanmetzman commented Feb 6, 2023

DonggeLiu commented Feb 7, 2023 • edited Loading

jonathanmetzman commented Feb 7, 2023

DonggeLiu commented Feb 7, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonathanmetzman Feb 7, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DonggeLiu commented Jan 16, 2023 •

edited

Loading

DonggeLiu commented Jan 17, 2023 •

edited

Loading

evverx commented Jan 17, 2023 •

edited

Loading

DonggeLiu commented Jan 17, 2023 •

edited

Loading

DonggeLiu commented Feb 7, 2023 •

edited

Loading

jonathanmetzman Feb 7, 2023 •

edited

Loading