Integrate afl++ with OSS-Fuzz, deprecate vanilla afl. #4280

vanhauser-thc · 2020-08-07T23:29:23Z

@jonathanmetzman @lszekeres @inferno-chromium

The following are variations on how you can run afl++ effectively.

building afl++

git clone https://github.com/AFLplusplus/AFLplusplus afl++
cd afl++
checkout stable
make all
make -C llvm_mode
make -C examples/aflpp_driver

The last entry builds the libafl driver (see https://github.com/google/fuzzbench/blob/master/fuzzers/aflplusplus/builder.Dockerfile)

building targets
Just use afl-clang-fast
afl-clang-lto is a full step better (faster, auto dictionary) plus has a high impact on build time.
Plain, or better with two useful options:

AFL_LLVM_CMPLOG=1     <= cmplog/redqueen, highly effective., not compatible with afl spinoffs
AFL_LLVM_LAF_ALL=1     <= highly improved laf-intel (floats, non-literal string support, etc.), compatible with afl spinoffs

Note that the binary comparability with afl and spin-offs are limited, as afl++ support variable map sizes - means: afl-fuzz and
the target choose the correct size and have a non-colliding coverage. So far some targets the map will be 15k entries, and
these are no problem. For a few large targets it can be > 64k and then these binaries cannot be used with others.

Note that for cmplog you are faster if you compile one version with cmplog and one without. the cmplog version is passed with the -c parameter to afl-fuzz. However if you do not care to loose a bit of speed, then you can use just a cmplog compiled one for both.

running targets
The normal havoc mode is improved on afl and better than MOpt, and if it has a cycle without finds then it addionally activates MOpt. so your choices are:

  -L 0   <- MOpt only
  -L -1 <- start with mixed mode
 nothing  <- mixed mode is turned on after a cycle without finds
  AFL_EXPAND_HAVOC_NOW=1  <- dont wait for a cycle without finds to enable advanced havoc mode

And then you can select a schedule. we have 9 schedules and the best are: seek, rare, explore (the default). (-p schedule).

as you are fuzzing with an existing and growing corpus I highly recommend to use cmplog and laf-intel, use AFL_EXPAND_HAVOC_NOW and -p rare or -p seek.
These options are the more effective ones to find new paths.

If you have further questions - just put them here.

The text was updated successfully, but these errors were encountered:

inferno-chromium · 2020-08-08T02:29:01Z

Thank you so much @vanhauser-thc , this is a great start. We will keep posting the updates as we proceed here.

inferno-chromium · 2021-01-25T17:48:28Z

First patch landed, this replaces afl - 665e489, need to let it stabilize for a day or two. tests on a couple of projects seem fine, with both fuzzing and reproduction, lets see at scale if it all works.

Next steps,

need to add afl++ in a dozen of places in docs (waiting for 1-2 days to let afl++ stick).
we run 2 hr fuzzing sessions at a time on saturated corpora, do you recommend changing runtime options like -L?
Does cmplog, lafintel require lto instrumentation? i guess we should enable these one by one without lto first ? Do env options like AFL_LLVM_LAF_ALL fail when build instrumentation is not there, reason being we dont want to break regression testing with old build. Is it safe to set AFL_LLVM_CMPLOG=1 and AFL_LLVM_LAF_ALL=1 even when build instrumentation is not there.

Thanks for your support @vanhauser-thc !

vanhauser-thc · 2021-01-25T18:12:38Z

* we run 2 hr fuzzing sessions at a time on saturated corpora, do you recommend changing runtime options like -L?

I would recommend randomly setting either -L0 or not at all (25% for setting it if you do have a fine-grained choice mechanism)

* Does cmplog, lafintel require lto instrumentation? i guess we should enable these one by one without lto first ? Do env options like AFL_LLVM_LAF_ALL fail when build instrumentation is not there, reason being we dont want to break regression testing with old build. Is it safe to set AFL_LLVM_CMPLOG=1 and AFL_LLVM_LAF_ALL=1 even when build instrumentation is not there.

neither do require LTO.
As targets get compile regularly and fuzzed in 2h intervals I would recommend to stick to afl-clang-fast, because the compile time needed for afl-clang-lto would not be healthy then.
And these env vars are build options. afl-fuzz does not know or care about them.
For building I would recommend 60% CMPLOG, 20% LAF, 20% none - and not CMPLOG and LAF together.

Another thing you can do:
AFL_LLVM_DICT2FILE=/full/path/to/dictfile.dic
this generates a dictionary from the compiled data that (integers, strings, ...) that you can also load into libfuzzer and honggfuzz (and obviously afl-fuzz -x, afl++ can load multiple dictionaries with -x foo -x bar). fuzzbench showed that this is very effective (about 2.5% gain in coverage)

vanhauser-thc · 2021-01-25T18:23:35Z

and for -p SCHEDULE I recommend randomly selecting one of: fast, coe, explore, exploit, rare (equally).

Always set AFL_EXPAND_HAVOC_NOW=1

50% change for AFL_DISABLE_TRIM=1

In the initial post I put "-L -1" and a few other items as alternatives, but with the afl++ development this has changed.
Once the improved cmplog is done (needs 1-2 more fuzzbench runs) there will a new command line option that might be beneficial to set for it.

vanhauser-thc · 2021-01-26T07:23:19Z

One more thing that came to my mind. if you fuzz in 2 hours intervals then this is IMHO too short. The fuzzers have to calibrate the starting corpus, as with oss-fuzz they are already saturated and therefore large, this can take quite a while.

I did a test on my current fuzzing experiment containing now a queue of 7000 entries. for that target it took 20 minutes until the calibration phase was done. Using AFL_FAST_CAL=1 takes the time down to 10 minutes - so that it something I strongly recommend to set. But it also means that even then ~5-10% of the runtime is lost to the calibration the fuzzers have to do at startup or over time. hence I recommend to run fuzzers for at least 4 hours so the calibration overhead is reduced.

inferno-chromium · 2021-01-26T16:49:26Z

Good news, nothing broke w.r.t builds (currently building with clang, afl_driver, not aflpp_driver)_

google/clusterfuzz#2208

Ok, time changed to 4hr (AFL_FAST_CAL is already a strategy, see decide_fast_cal_* - https://github.com/google/clusterfuzz/blob/master/src/python/bot/fuzzers/afl/launcher.py#L328)
Enabled AFL_EXPAND_HAVOC_NOW=1 always

google/clusterfuzz#2206
Account for default -S behavior, keep it backward compatible with AFL builds on chromium (need migration to afl++)

Before we do furthur optimization, need to debug this afl-showmap issue, did anything change in call style
Just one exception left, any idea why showmap output file is not written. This binary is build with llvm's afl_driver, so it should be fed data from stdin.

message: "afl-showmap didn't output any coverage. Command: ['/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-afl_wireshark_9de6374568df96eba97b9288a3fce517c93d2636/revisions/afl-showmap', '-o/mnt/scratch0/clusterfuzz/bot/inputs/fuzzer-testcases-disk/temp-87120/afl_showmap_output', '-mnone', '/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-afl_wireshark_9de6374568df96eba97b9288a3fce517c93d2636/revisions/fuzzshark_udp_port-dns', '1']
Return code: -14
Time executed: 2.159447193145752
Output: ======================= INFO =========================
This binary is built for AFL-fuzz.
To run the target function on individual input(s) execute this:
  /mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-afl_wireshark_9de6374568df96eba97b9288a3fce517c93d2636/revisions/fuzzshark_udp_port-dns < INPUT_FILE
or
  /mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-afl_wireshark_9de6374568df96eba97b9288a3fce517c93d2636/revisions/fuzzshark_udp_port-dns INPUT_FILE1 [INPUT_FILE2 ... ]
To fuzz with afl-fuzz execute this:
  afl-fuzz [afl-flags] /mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-afl_wireshark_9de6374568df96eba97b9288a3fce517c93d2636/revisions/fuzzshark_udp_port-dns [-N]
afl-fuzz will run N iterations before re-spawning the process (default: 1000)
======================================================
oss-fuzzshark: disabling: ip
oss-fuzzshark: disabling: udp
oss-fuzzshark: disabling: udplite
oss-fuzzshark: disabling: ospf
oss-fuzzshark: disabling: bgp
oss-fuzzshark: disabling: dhcp
oss-fuzzshark: disabling: json
oss-fuzzshark: disabling: snort
oss-fuzzshark: configured for dissector: dns in table: udp.port
WARNING: using the deprecated call style `/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-afl_wireshark_9de6374568df96eba97b9288a3fce517c93d2636/revisions/fuzzshark_udp_port-dns 1`
/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-afl_wireshark_9de6374568df96eba97b9288a3fce517c93d2636/revisions/fuzzshark_udp_port-dns: successfully executed 1 input(s)

Code - https://github.com/google/clusterfuzz/blob/master/src/python/bot/fuzzers/afl/launcher.py#L1039

vanhauser-thc · 2021-01-26T17:00:50Z

Try: export AFL_MAP_SIZE=4194304 ... there is a disadvantage with collision free coverage ... (btw this is also necessary for afl-fuzz, however afl-fuzz will tell you that)

inferno-chromium · 2021-01-26T18:10:40Z

afl-showmap seems to run fine locally, so still investigating what is going on.

./afl-showmap -t1000+ -o/tmp/p -mnone /out/fuzz_db 1 < /tmp/ii
[!] WARNING: Mistyped AFL environment variable: AFL_FUZZER_ARGS=-m none
afl-showmap++3.01a by Michal Zalewski
[*] Executing '/out/fuzz_db'...
-- Program output begins --
======================= INFO =========================
This binary is built for AFL-fuzz.
To run the target function on individual input(s) execute this:
  /out/fuzz_db < INPUT_FILE
or
  /out/fuzz_db INPUT_FILE1 [INPUT_FILE2 ... ]
To fuzz with afl-fuzz execute this:
  afl-fuzz [afl-flags] /out/fuzz_db [-N]
afl-fuzz will run N iterations before re-spawning the process (default: 1000)
======================================================
WARNING: using the deprecated call style `/out/fuzz_db 1`
/out/fuzz_db: successfully executed 1 input(s)
-- Program output ends --
[+] Captured 776 tuples (highest value 8, total values 1790) in '/tmp/p'.

inferno-chromium · 2021-01-26T21:34:20Z

afl-showmap issue is not a blocker, although these new -14 return code are weird, it is succeeding for most other cases

I 2021-01-26T21:32:59.665801568Z afl-showmap succeedded for file /mnt/scratch0/clusterfuzz/bot/inputs/data-bundles/pcapplusplus_FuzzTarget/448160ea3409c5487451c2fea14e913617746937: 1700 features 
I 2021-01-26T21:32:59.665715911Z afl-showmap succeedded for file /mnt/scratch0/clusterfuzz/bot/inputs/data-bundles/ibmswtpm2_fuzz_tpm_server/radamsa-01190-f3b75f730fa1870a377afd3651f6a72b0978f020: 257 features 
I 2021-01-26T21:32:59.661712300Z afl-showmap succeedded for file /mnt/scratch0/clusterfuzz/bot/inputs/data-bundles/ibmswtpm2_fuzz_tpm_server/radamsa-00220-cfcf98c6346119b3f37cfd80bf3f8dbdaa10c8b5: 209 features 
I 2021-01-26T21:32:59.657761543Z afl-showmap succeedded for file /mnt/scratch0/clusterfuzz/bot/inputs/data-bundles/pcapplusplus_FuzzTarget/radamsa-00508-9348627e74fadd62b70de18a3f5a7c545f99e525: 1286 features 
I 2021-01-26T21:32:59.653803200Z afl-showmap succeedded for file /mnt/scratch0/clusterfuzz/bot/inputs/data-bundles/pcapplusplus_FuzzTarget/radamsa-01987-102956ae52ecacc1b9a9dfc1cde9a235.00000039.honggfuzz.cov: 1261 features 
I 2021-01-26T21:32:59.650982473Z afl-showmap succeedded for file /mnt/scratch0/clusterfuzz/bot/inputs/data-bundles/nginx_http_request_fuzzer/43cc6b6f54335c3c5faa7847285f718f536f319d: 1788 features

vanhauser-thc · 2021-01-26T22:09:35Z

the -14 is curious yes. are you using the afl-showmap from afl++ or is it the vanilla afl one?
and I think it would help if you run one afl-showmap from hand to see what the exact output is that generates that return code.

btw:

[!] WARNING: Mistyped AFL environment variable: AFL_FUZZER_ARGS=-m none

we check for unknown env vars because we have so many :)
you do not need this env. -m none is default since 3.0 for afl-fuzz, and in afl-tmin and afl-showmap soon on the next push to stable.

inferno-chromium · 2021-01-26T22:42:42Z

the -14 is curious yes. are you using the afl-showmap from afl++ or is it the vanilla afl one?
and I think it would help if you run one afl-showmap from hand to see what the exact output is that generates that return code.

btw:
[!] WARNING: Mistyped AFL environment variable: AFL_FUZZER_ARGS=-m none
we check for unknown env vars because we have so many :)
you do not need this env. -m none is default since 3.0 for afl-fuzz, and in afl-tmin and afl-showmap soon on the next push to stable.

Cannot remove '-m none` since keeping it compatible with afl used in chromium (should get replaced sometime with afl++)
I can ignore warning for now, but you should add AFL_DRIVER_STDERR_DUPLICATE_FILENAME sometime (i think it is missing from your list since it is in aflpp_driver and afl_driver :) From my side, i will try to reduce use of AFL_ vars, but warnings is not our worry atm :)

So, previously we never saw any errors in afl-showmap, and now all projects should be on afl++. i did run the afl-showmap for afl++ for several projects locally and it was executing fine, so that is why i added more debugging code now - google/clusterfuzz@6a2c07e. I do see it succeeding for most runs, so logging to see what input was it when this error occurs. We are using clang+trace-pc-guard with afl_driver [https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-builder/compile_afl#L27], not afl_clang_fast or afl_clang_lto, could that instrumentation be an issue.

inferno-chromium · 2021-01-26T23:30:31Z

AFL++ now in all docs - 65d4f8e and 89603f3

inferno-chromium · 2021-01-27T00:44:25Z

@vanhauser-thc - this afl-showmap issue is not frequent, but annoying (no specific targets, have seen this on libreoffice, leveldb, etc), basically, it returns with -14 error code, does not write coverage output and does not write "-- Program output ends --" section and tuples. happening on regular corpus files (non-zero, example -

"afl-showmap didn't output any coverage for file /mnt/scratch0/clusterfuzz/bot/inputs/data-bundles/mupdf_pdf_fuzzer/0b9bc25420abea6c9f53a5afae0c5e6df660f571 (49 bytes).
Command: ['/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-afl_mupdf_72598fa5eaf39798954f7e369768d97a68ec5677/revisions/afl-showmap', '-o/mnt/scratch0/clusterfuzz/bot/inputs/fuzzer-testcases-disk/temp-5152/afl_showmap_output', '-mnone', '/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-afl_mupdf_72598fa5eaf39798954f7e369768d97a68ec5677/revisions/pdf_fuzzer', '1']
Return code: -14
Time executed: 3.4761881828308105
Output: ======================= INFO =========================
This binary is built for AFL-fuzz.
To run the target function on individual input(s) execute this:
  /mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-afl_mupdf_72598fa5eaf39798954f7e369768d97a68ec5677/revisions/pdf_fuzzer < INPUT_FILE
or
  /mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-afl_mupdf_72598fa5eaf39798954f7e369768d97a68ec5677/revisions/pdf_fuzzer INPUT_FILE1 [INPUT_FILE2 ... ]
To fuzz with afl-fuzz execute this:
  afl-fuzz [afl-flags] /mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-afl_mupdf_72598fa5eaf39798954f7e369768d97a68ec5677/revisions/pdf_fuzzer [-N]
afl-fuzz will run N iterations before re-spawning the process (default: 1000)
======================================================
"

Let me try AFL_MAP_SIZE=4194304

vanhauser-thc · 2021-01-27T07:25:42Z

I dont see any afl-showmap output in the paste, only from the target.
A paste of what exactly afl-showmap is printing would help. (plus setting AFL_DEBUG=1 and not using -q)

inferno-chromium · 2021-01-31T19:01:40Z

New instrumentation change to afl-clang-fast should enable AFL_DEBUG.
Looks like AFL_MAP_SIZE is required for larger targets, so had to do - 93eb602 [ClusterFuzz side runtime change is still needed, this is just needed when we run basic checks when we archived builds, those were crashing on startup].

@vanhauser-thc - should we not do a high AFL_MAP_SIZE=4194304 as default in afl++ itself, are there really bad memory implications / fuzzing efficiency issue with that high map size ? Still thinking if we should fix this in ClusterFuzz, or would it better to do a high default in afl++, rather than crashing on startup [this was breaking projects like libsass, matio, so seems common and annoying]

vanhauser-thc · 2021-01-31T19:10:20Z

afl-fuzz and target only use as much from the map as needed. Shared memory is a limited resource hence we keep it at 64kb - which is more than enough for many targets. But I am thinking about getting the map size first and the resetting the required shared mem size - which would eliminate the need for the env.

inferno-chromium · 2021-01-31T19:19:57Z

afl-fuzz and target only use as much from the map as needed. Shared memory is a limited resource hence we keep it at 64kb - which is more than enough for many targets. But I am thinking about getting the map size first and the resetting the required shared mem size - which would eliminate the need for the env.

Yes that would be great if we can eliminate this completely . If you make a new release, please let me know here or update this hash here- https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-builder/Dockerfile#L152. at this point, we want to ensure hundreds of these projects build with afl-clang-fast [and will test afl++ at scale :)], so AFL_MAP_SIZE workaround is fine, but on CF, will wait for this fix.

inferno-chromium · 2021-02-01T15:41:45Z

aflpp_driver+afl-clang-fast change broke many projects, trying to list issues here. reverting for now.

Project using meson, there is some unused command line arg in afl-clang-fast, e.g. systemd, irssi, dav1d
[systemd] turn off AFL #5084

Compiler stderr:
 clang-12: error: /src/aflplusplus/afl-compiler-rt.o: 'linker' input unused [-Werror,-Wunused-command-line-argument]
clang-12: error: -Wl,--dynamic-list=/src/aflplusplus/dynamic_list.txt: 'linker' input unused [-Werror,-Wunused-command-line-argument]

meson.build:647:16: ERROR: Problem encountered: unable to determine gperf len type
+ exit 1

Crpytofuzz, njs, mruby, bignum-fuzzer
https://oss-fuzz-build-logs.storage.googleapis.com/log-60a3aa59-7b18-4692-ae45-2315ff241083.txt
https://oss-fuzz-build-logs.storage.googleapis.com/log-bf011061-b6d2-415c-8680-ff1c0ad59e36.txt
https://oss-fuzz-build-logs.storage.googleapis.com/log-ef630a07-882e-41fc-a66d-a730101b0e71.txt
https://oss-fuzz-build-logs.storage.googleapis.com/log-871a183a-01cc-496b-a977-9bda4ce792de.txt

Step #32: loadelf: /src/go/pkg/linux_amd64/runtime/cgo.a(_x006.o): 224863: sym#7: ignoring symbol in section 9 (type 0)
Step #32: net(.text._cgo_26061493d47f_C2func_getaddrinfo): relocation target __afl_area_ptr not defined
Step #32: net(.text._cgo_26061493d47f_Cfunc_freeaddrinfo): relocation target __afl_area_ptr not defined
Step #32: net(.text._cgo_26061493d47f_Cfunc_gai_strerror): relocation target __afl_area_ptr not defined
Step #32: runtime/cgo(.text.x_cgo_notify_runtime_init_done): relocation target __afl_area_ptr not defined
Step #32: runtime/cgo(.text.x_cgo_mmap): relocation target __afl_area_ptr not defined
Step #32: runtime/cgo(.text.x_cgo_munmap): relocation target __afl_area_ptr not defined
Step #32: runtime/cgo(.text.x_cgo_setenv): relocation target __afl_area_ptr not defined
Step #32: runtime/cgo(.text.x_cgo_unsetenv): relocation target __afl_area_ptr not defined
Step #32: runtime/cgo(.text.x_cgo_sigaction): relocation target __afl_area_ptr not defined
Step #32: runtime/cgo(.text.x_cgo_callers): relocation target __afl_area_ptr not defined
Step #32: runtime/cgo(.text.x_cgo_thread_start): relocation target __afl_area_ptr not defined
Step #32: runtime/cgo(.text.x_cgo_thread_start): relocation target _cgo_sys_thread_start not defined
Step #32: _cgo_init: relocation target x_cgo_init not defined

usrsctp project, protobuf-c, libraw, libplist, flac
E.g. https://oss-fuzz-build-logs.storage.googleapis.com/log-aa06cd51-bfc1-4d97-b67b-da7789adc9c4.txt
https://oss-fuzz-build-logs.storage.googleapis.com/log-fb4a50b1-bab2-4197-96b6-d5a3e7446469.txt
https://oss-fuzz-build-logs.storage.googleapis.com/log-7c317136-01b7-498a-a387-40b6fef311f4.txt

Step #32: [100%] Linking C executable fuzzer_connect_multi
Step #32: /src/aflplusplus/afl-compiler-rt.o/src/aflplusplus/afl-compiler-rt.o/src/aflplusplus/afl-compiler-rt.o: In function `: In function `: In function `__cmplog_ins_hook1__cmplog_ins_hook1__cmplog_ins_hook1':
Step #32: ':
Step #32: ':
Step #32: llvm_mode/instrumentation/afl-compiler-rt.o.c:llvm_mode/instrumentation/afl-compiler-rt.o.c:llvm_mode/instrumentation/afl-compiler-rt.o.c:121212121212: multiple definition of `: multiple definition of `: multiple definition of `__sanitizer_cov_trace_cmp1__sanitizer_cov_trace_cmp1'
Step #32: __sanitizer_cov_trace_cmp1'
Step #32: '
Step #32: /src/aflplusplus/afl-compiler-rt.o: In function `__cmplog_ins_hook1':

tpm2-tss, ibmswtpm2
https://oss-fuzz-build-logs.storage.googleapis.com/log-75ef812e-92d2-4d01-abd1-ec71854abc67.txt
https://oss-fuzz-build-logs.storage.googleapis.com/log-58ba4bb5-ba15-476b-bffd-e6a9f0163e49.txt

tep #32: /usr/lib/gcc/x86_64-linux-gnu/5.4.0/../../../libFuzzingEngine.a(aflpp_driver.o): In function `main':
Step #32: /src/aflplusplus/utils/aflpp_driver/aflpp_driver.c:235: undefined reference to `__afl_manual_init'
Step #32: /src/aflplusplus/utils/aflpp_driver/aflpp_driver.c:243: undefined reference to `__afl_manual_init'
Step #32: /src/aflplusplus/utils/aflpp_driver/aflpp_driver.c:250: undefined reference to `__afl_persistent_loop'
Step #32: /src/aflplusplus/utils/aflpp_driver/aflpp_driver.c:250: undefined reference to `__afl_persistent_loop'
Step #32: /src/aflplusplus/utils/aflpp_driver/aflpp_driver.c:261: undefined reference to `__afl_fuzz_len'
Step #32: /src/aflplusplus/utils/aflpp_driver/aflpp_driver.c:264: undefined reference to `__afl_fuzz_ptr'
Step #32: make: *** [test/fuzz/Tss2_Sys_PolicyRestart_Complete.fuzz] Error 1

qt, simple fix, need to bring back afl repo or redirect to aflpp

Step #4: cp: cannot stat '/src/afl/dictionaries/xml.dict': No such file or directory
Step #4: ********************************************************************************
Step #4: Failed to build.
Step #4: To reproduce, run:
Step #4: python infra/helper.py build_image qt
Step #4: python infra/helper.py build_fuzzers --sanitizer address --engine libfuzzer --architecture x86_64 qt
Step #4: ********************************************************************************

vanhauser-thc · 2021-02-01T16:10:41Z

I have never used oss-fuzz - how can I easily try and debug a build?

inferno-chromium · 2021-02-01T16:18:13Z

First pull latest images

python infra/helper.py pull_images

Make a change in base-builder, like apply the cl above. This creates a local build of the docker image.

python infra/helper.py build_image base-builder

Build the project using this image built above

python infra/helper.py build_fuzzers --sanitizer address --engine afl --architecture x86_64 dav1d

This reproduces the failures, but for debugging inside docker
then do

python infra/helper.py shell dav1d
./compile

afl++ code, built bins in /src/aflplusplus
this compile_afl file - https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-builder/compile_afl is copied to /usr/local/bin in https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-builder/Dockerfile#L163, so if you want to modify this, you can do that and then run ./compile again

vanhauser-thc · 2021-02-01T16:57:46Z

I tried this but it seems something is missing or I misunderstood:

python infra/helper.py pull_images
python infra/helper.py build_image base-builder
python infra/helper.py build_fuzzers --sanitizer address --engine afl --architecture x86_64 tpm2-tss
python infra/helper.py shell tpm2-tss
=>
compile
compile_afl
compile    <= still compiles for libfuzzer not for afl++

and there is no ./compile

Edit: there is a "./compile" after running "compile", but that doesnt do anything

inferno-chromium · 2021-02-01T17:14:50Z

I tried this but it seems something is missing or I misunderstood:

python infra/helper.py pull_images
python infra/helper.py build_image base-builder
python infra/helper.py build_fuzzers --sanitizer address --engine afl --architecture x86_64 tpm2-tss
python infra/helper.py shell tpm2-tss
=>
compile
compile_afl
compile    <= still compiles for libfuzzer not for afl++

and there is no ./compile

sorry my bad

python infra/helper.py shell --sanitizer address --engine afl --architecture x86_64 tpm2-tss
compile

vanhauser-thc · 2021-02-01T17:33:44Z

The first issue is compile_afl.
the third to last line:

ls afl-* *.a *.o *.so | sort -u | xargs cp -t $OUT

this additionally needs *.txt - this is the reason several projects do not compile.

not checking for other issues.

vanhauser-thc · 2021-02-01T17:44:48Z

#5087

vanhauser-thc · 2021-02-01T17:46:49Z

with this PR dav1d and tpm2-tss compile.
this should fix everything except the wrong dict for qt - and usrsctp project, protobuf-c, libraw, libplist, flac ... will check them next.

vanhauser-thc · 2021-02-01T18:37:09Z

another issue I see in #5074 is that afl-clang-fast is used, however CFLAGS/CXXFLAGS still contain -fsanitize-fuzzer... which is obviously a bad idea :)
I have no clue where this is set.

inferno-chromium · 2021-02-02T02:36:24Z

Regarding this comment - #4280 (comment), which project was it. Sometimes project's makefile can put that incorrectly, which is a bug they should fix. Your stripping logic is fine too.

systemd, irssi, dav1d, njs, protobuf-c, libraw, libplist, flac work great

remaining breakages (i dont think we need to revert now, we can see if there is an easy fix here since will affect other afl++ users too, take your time, thanks a lot!). -

usrsctp

[100%] Linking C executable fuzzer_listen
/src/aflplusplus/afl-compiler-rt.o: In function `__cmplog_ins_hook1':
llvm_mode/instrumentation/afl-compiler-rt.o.c:1216: multiple definition of `__sanitizer_cov_trace_cmp1'
/src/aflplusplus/afl-compiler-rt.o/src/aflplusplus/afl-compiler-rt.o/src/aflplusplus/afl-compiler-rt.o: In function `: In function `: In function `__cmplog_ins_hook1__cmplog_ins_hook1__cmplog_ins_hook1':

bignum-fuzzer

Step #22: loadelf: /src/go/pkg/linux_amd64/runtime/cgo.a(_x006.o): 181418: sym#7: ignoring symbol in section 9 (type 0)
Step #22: net(.text._cgo_26061493d47f_C2func_getaddrinfo): relocation target __afl_area_ptr not defined
Step #22: net(.text._cgo_26061493d47f_Cfunc_freeaddrinfo): relocation target __afl_area_ptr not defined
Step #22: net(.text._cgo_26061493d47f_Cfunc_gai_strerror): relocation target __afl_area_ptr not defined
Step #22: runtime/cgo(.text.x_cgo_notify_runtime_init_done): relocation target __afl_area_ptr not defined

tpm2-tss

  CXXLD    test/fuzz/Tss2_Sys_Rewrap_Prepare.fuzz
/usr/lib/gcc/x86_64-linux-gnu/5.4.0/../../../libFuzzingEngine.a(aflpp_driver.o): In function `main':
/src/aflplusplus/utils/aflpp_driver/aflpp_driver.c:236: undefined reference to `__afl_manual_init'
/src/aflplusplus/utils/aflpp_driver/aflpp_driver.c:244: undefined reference to `__afl_manual_init'
/src/aflplusplus/utils/aflpp_driver/aflpp_driver.c:251: undefined reference to `__afl_persistent_loop'
/src/aflplusplus/utils/aflpp_driver/aflpp_driver.c:251: undefined reference to `__afl_persistent_loop'
/src/aflplusplus/utils/aflpp_driver/aflpp_driver.c:262: undefined reference to `__afl_fuzz_len'
/src/aflplusplus/utils/aflpp_driver/aflpp_driver.c:265: undefined reference to `__afl_fuzz_ptr'

ibmswtpm2

clang++  -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link -stdlib=libc++ fuzzer.o AlgorithmCap.o AlgorithmTests.o AsymmetricCommands.o Attest_spt.o AttestationCommands.o AuditCommands.o Bits.o BnConvert.o BnEccData.o BnMath.o BnMemory.o Cancel.o CapabilityCommands.o Clock.o ClockCommands.o CommandAudit.o CommandCodeAttributes.o CommandDispatcher.o ContextCommands.o Context_spt.o CryptDes.o CryptCmac.o CryptEccKeyExchange.o CryptEccMain.o CryptEccSignature.o CryptHash.o CryptHashData.o CryptPrime.o CryptPrimeSieve.o CryptRand.o CryptRsa.o CryptSelfTest.o CryptSmac.o CryptSym.o CryptUtil.o DA.o DictionaryCommands.o DuplicationCommands.o EACommands.o EncryptDecrypt_spt.o Entity.o Entropy.o EphemeralCommands.o ExecCommand.o Global.o Handle.o HashCommands.o Hierarchy.o HierarchyCommands.o IoBuffers.o IntegrityCommands.o Locality.o LocalityPlat.o ManagementCommands.o Manufacture.o Marshal.o MathOnByteBuffers.o Memory.o NVCommands.o NVDynamic.o NVMem.o NVReserved.o NV_spt.o Object.o ObjectCommands.o Object_spt.o PCR.o PP.o PPPlat.o PlatformData.o Policy_spt.o Power.o PowerPlat.o PrimeData.o PropertyCap.o RandomCommands.o Response.o ResponseCodeProcessing.o RsaKeyCache.o RunCommand.o Session.o SessionCommands.o SessionProcess.o SigningCommands.o StartupCommands.o SymmetricCommands.o TPMCmdp.o TPMCmds.o TestingCommands.o Ticket.o Time.o TpmFail.o TpmSizeChecks.o TpmToOsslDesSupport.o TpmToOsslMath.o TpmToOsslSupport.o Unique.o Unmarshal.o Vendor_TCG_Test.o ntc2lib.o ntc2.o TcpServerPosix.o -ggdb -DTPM_POSIX -DTPM_NUVOTON -lcrypto -lpthread -lrt -I. -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION /usr/lib/libFuzzingEngine.a -o fuzz_tpm_server
/usr/lib/libFuzzingEngine.a(aflpp_driver.o): In function `main':
/src/aflplusplus/utils/aflpp_driver/aflpp_driver.c:236: undefined reference to `__afl_manual_init'
/src/aflplusplus/utils/aflpp_driver/aflpp_driver.c:244: undefined reference to `__afl_manual_init'
/src/aflplusplus/utils/aflpp_driver/aflpp_driver.c:251: undefined reference to `__afl_persistent_loop'
/src/aflplusplus/utils/aflpp_driver/aflpp_driver.c:251: undefined reference to `__afl_persistent_loop'
/src/aflplusplus/utils/aflpp_driver/aflpp_driver.c:262: undefined reference to `__afl_fuzz_len'
/src/aflplusplus/utils/aflpp_driver/aflpp_driver.c:265: undefined reference to `__afl_fuzz_ptr'

cryptofuzz

Step #32: # cmd/trace
Step #32: loadelf: /src/go/pkg/linux_amd64/runtime/cgo.a(_x006.o): 181418: sym#7: ignoring symbol in section 9 (type 0)
Step #32: net(.text._cgo_26061493d47f_C2func_getaddrinfo): relocation target __afl_area_ptr not defined
Step #32: net(.text._cgo_26061493d47f_Cfunc_freeaddrinfo): relocation target __afl_area_ptr not defined
Step #32: net(.text._cgo_26061493d47f_Cfunc_gai_strerror): relocation target __afl_area_ptr not defined
Step #32: runtime/cgo(.text.x_cgo_notify_runtime_init_done): relocation target __afl_area_ptr not defined
Step #32: runtime/cgo(.text.x_cgo_mmap): relocation target __afl_area_ptr not defined
Step #32: runtime/cgo(.text.x_cgo_munmap): relocation target __afl_area_ptr not defined
Step #32: runtime/cgo(.text.x_cgo_setenv): relocation target __afl_area_ptr not defined
Step #32: runtime/cgo(.text.x_cgo_unsetenv): relocation target __afl_area_ptr not defined
Step #32: runtime/cgo(.text.x_cgo_sigaction): relocation target __afl_area_ptr not defined
Step #32: runtime/cgo(.text.x_cgo_callers): relocation target __afl_area_ptr not defined
Step #32: runtime/cgo(.text.x_cgo_thread_start): relocation target __afl_area_ptr not defined
Step #32: runtime/cgo(.text.x_cgo_thread_start): relocation target _cgo_sys_thread_start not defined
Step #32: _cgo_init: relocation target x_cgo_init not defined

if these are just the -fsanitize=fuzzer issues, i can check tmrw on why your stripping is not working.

vanhauser-thc · 2021-02-06T18:47:38Z

@vanhauser-thc - renaming 2 issues , next can focus on instrumentation, dict enabling (my team can start on this)
1. that afl map issue you mentioned above, i have just set AFL_MAP_SIZE= 4194304 for now.

I prepared a PR for the optimal default setup, this also makes libreoffice work for 40 out of 45 fuzzers.
#5129

caolanm · 2021-02-06T19:20:12Z

@inferno-chromium
I found a workaround that works for most libreoffice fuzzers ... but ...
maybe you have seen this issue before - when building libreoffice this happens:
[LNK] Executable/xlsxfuzzer
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_extra_spill_area'
clang-12: error: unable to execute command: Killed
clang-12: error: linker command failed due to signal (use -v to see invocation)
/src/libreoffice/vcl/Executable_xlsxfuzzer.mk:13: recipe for target '/work/instdir/program/xlsxfuzzer' failed

i havent seen this, but @caolanm is our expert here on libreoffice. once you fix cmd line issue, i think lets ask him to take a look on this one.

Is it maybe just getting killed by OOM? They are quite large with static linking. (LibreOffice: Based on technology proudly breaking your toolchain since 1985)

vanhauser-thc · 2021-02-06T22:41:02Z

@caolanm might be OOM, this "killed" message sounds like it.
I just looked I thought it is 16GB RAM machine I compile on, but it only has 8 :D
can you maybe test if #5129 builds libreoffice for you or if there is still a fuzzer failing to build? thanks!

vanhauser-thc · 2021-02-07T06:05:19Z

cmplog test is in #5130

rlohning · 2021-02-11T22:21:31Z

@rlohning - i think you are looking at ibmswtpm2 and tpm2-tss already
lua looks super simple fix, i am waiting on dev to fix $CC in sed

Sorry for the late reply, I had some days off.
I don't expect to fix anything else than Qt. Please give me some time. I'll have to work around some issues in our code to make this build correctly.

inferno-chromium · 2021-02-22T06:58:13Z

@vanhauser-thc - some segfaults seen in some projects+configs (probably from the new cmplog, lafintel picks).

Due to randomness of options, we dont see 2 consecutive build failures and hence project devs are not auto-notified.
Probably the last set of issues from OSS-Fuzz side, rest would be just minor CF side changes (e.g. afl++.dict use)

vanhauser-thc · 2021-02-22T08:31:15Z

@inferno-chromium is there a way to get that exact failing container?

vanhauser-thc · 2021-02-22T13:44:56Z

fixes in #5235

vanhauser-thc · 2021-02-28T09:29:13Z

are there any more build failures I need to be aware of?

inferno-chromium · 2021-02-28T17:42:05Z

Right now, only 4 failures and could be a project issue, so best to just check
https://oss-fuzz-build-logs.storage.googleapis.com/index.html
check using
https://oss-fuzz-build-logs.storage.googleapis.com/index.html#<project_name>

dropbear
zeek
postgresql
monero

we can close this bug once clusterfuzz side lands, stabilizes, and then we test on runtime if any startup crashes stick.

vanhauser-thc · 2021-03-01T10:42:05Z

zeek

zeek build successfully for 1st March although I do not see any change in oss-fuzz or their repo between 28th and 1st ... weird

postgresql

the input crashes the harness I kinda doubt this is afl++

dropbear
monero

these crash on startup, I will try to find out what is happening there.

vanhauser-thc · 2021-03-01T18:34:04Z

#5271 fixes dropbear und monero. such a large target base is very good to find even very rare bug occurances :)

vanhauser-thc · 2021-03-15T16:15:18Z

ibmswtpm2 is re-bugged and does not honor CXX anymore:

Step #32: clang++ -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope  -stdlib=libc++ fuzzer.o AlgorithmCap.o AlgorithmTests.o AsymmetricCommands.o Attest_spt.o AttestationCommands.o AuditCommands.o Bits.o BnConvert.o BnEccData.o BnMath.o BnMemory.o Cancel.o CapabilityCommands.o Clock.o ClockCommands.o CommandAudit.o CommandCodeAttributes.o CommandDispatcher.o ContextCommands.o Context_spt.o CryptDes.o CryptCmac.o CryptEccKeyExchange.o CryptEccMain.o CryptEccSignature.o CryptHash.o CryptHashData.o CryptPrime.o CryptPrimeSieve.o CryptRand.o CryptRsa.o CryptSelfTest.o CryptSmac.o CryptSym.o CryptUtil.o DA.o DictionaryCommands.o DuplicationCommands.o EACommands.o EncryptDecrypt_spt.o Entity.o Entropy.o EphemeralCommands.o ExecCommand.o Global.o Handle.o HashCommands.o Hierarchy.o HierarchyCommands.o IoBuffers.o IntegrityCommands.o Locality.o LocalityPlat.o ManagementCommands.o Manufacture.o Marshal.o MathOnByteBuffers.o Memory.o NVCommands.o NVDynamic.o NVMem.o NVReserved.o NV_spt.o Object.o ObjectCommands.o Object_spt.o PCR.o PP.o PPPlat.o PlatformData.o Policy_spt.o Power.o PowerPlat.o PrimeData.o PropertyCap.o RandomCommands.o Response.o ResponseCodeProcessing.o RsaKeyCache.o RunCommand.o Session.o SessionCommands.o SessionProcess.o SigningCommands.o StartupCommands.o SymmetricCommands.o TPMCmdp.o TPMCmds.o TestingCommands.o Ticket.o Time.o TpmFail.o TpmSizeChecks.o TpmToOsslDesSupport.o TpmToOsslMath.o TpmToOsslSupport.o Unique.o Unmarshal.o Vendor_TCG_Test.o ntc2lib.o ntc2.o TcpServerPosix.o -ggdb -DTPM_POSIX -DTPM_NUVOTON -lcrypto -lpthread -lrt -I. -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION /usr/lib/libFuzzingEngine.a -o fuzz_tpm_server
Step #32: /usr/lib/libFuzzingEngine.a(aflpp_driver.o): In function `main':
Step #32: /src/aflplusplus/utils/aflpp_driver/aflpp_driver.c:236: undefined reference to `__afl_manual_init'
Step #32: /src/aflplusplus/utils/aflpp_driver/aflpp_driver.c:244: undefined reference to `__afl_manual_init'
Step #32: /src/aflplusplus/utils/aflpp_driver/aflpp_driver.c:251: undefined reference to `__afl_persistent_loop'
Step #32: /src/aflplusplus/utils/aflpp_driver/aflpp_driver.c:251: undefined reference to `__afl_persistent_loop'
Step #32: /src/aflplusplus/utils/aflpp_driver/aflpp_driver.c:262: undefined reference to `__afl_fuzz_len'
Step #32: /src/aflplusplus/utils/aflpp_driver/aflpp_driver.c:265: undefined reference to `__afl_fuzz_ptr'

libreoffice builds but test cases time out in some wait_.... call. I do not know what the issue is.

zeek and skia work only without cmplog. with cmplog they time out.
so far I was unsuccessful in finding out why - where things go wrong seems not to be anything that afl-clang-fast touches.

vanhauser-thc · 2021-03-21T15:42:02Z

@inferno-chromium

I took a quick look if any targets still have problems with afl++.

ibmswtpm2: (still) does not honor $CC/$CXX

libreoffice: the known timeout error - the instrumentation is fine, this needs someone from the libreoffice project to take a look why the afl++ compiler runtime results in hang.

zeek: there was a software change so now on startup:

Running LLVMFuzzerInitialize ...
internal error in /out/./oss-fuzz-zeek-scripts/base/init-bare.zeek, line 5443: built-in function Option::any_set_to_any_vec missing

so IMHO needs to be fixed by zeek maintainers

capstone: it fails because an instrumented libcapstone.so is loaded into a python script, so the runtime is not there, hence __afl_map_ptr is not there and the load fails. looking at honggfuzz + libfuzzer - they both seem not to build that .so instrumented and hence they do not fail. maybe needs a look by the maintainer as well.
EDIT: Ah found out why - when the .so is built, CFLAGS="" are dropped and hence no instrumentation. Needs a fix in libcapstone.

anything else I overlooked?

inferno-chromium · 2021-03-22T01:53:01Z

@inferno-chromium

I took a quick look if any targets still have problems with afl++.

ibmswtpm2: (still) does not honor $CC/$CXX

should get fixed in #5451 (thanks @jonathanmetzman )

libreoffice: the known timeout error - the instrumentation is fine, this needs someone from the libreoffice project to take a look why the afl++ compiler runtime results in hang.

this is filed in #5441, thoughts?

zeek: there was a software change so now on startup:
Running LLVMFuzzerInitialize ...
internal error in /out/./oss-fuzz-zeek-scripts/base/init-bare.zeek, line 5443: built-in function Option::any_set_to_any_vec missing
so IMHO needs to be fixed by zeek maintainers

yes ignore.

capstone: it fails because an instrumented libcapstone.so is loaded into a python script, so the runtime is not there, hence __afl_map_ptr is not there and the load fails. looking at honggfuzz + libfuzzer - they both seem not to build that .so instrumented and hence they do not fail. maybe needs a look by the maintainer as well.
EDIT: Ah found out why - when the .so is built, CFLAGS="" are dropped and hence no instrumentation. Needs a fix in libcapstone.

anything else I overlooked?

No, i only see zeek and libreoffice failing, everything else seems fine. Thank you so much @vanhauser-thc for being on top of this and helping the OSS community with AFL++ use here.

evverx · 2021-04-29T18:47:53Z

I moved a couple of fuzz targets into the lxc repo (where they are built with -fPIE and -pie by default). Since then lxc has been (sporadically) failing to compile with aflpp

Step #32: libtool: link: /src/aflplusplus/afl-clang-fast++ -fPIE -Wvla -std=gnu11 -fms-extensions -fdiagnostics-color -Wcast-align -Wstrict-prototypes -fno-strict-aliasing -fstack-clash-protection -fstack-protector-strong --param=ssp-buffer-size=4 -g -fcf-protection -Werror=implicit-function-declaration -Wmissing-include-dirs -Wold-style-definition -Winit-self -Wfloat-equal -Werror=return-type -Werror=incompatible-pointer-types -Wformat=2 -Wshadow -Wendif-labels -Werror=overflow -fdiagnostics-show-option -Werror=shift-count-overflow -Wdate-time -Wnested-externs -fasynchronous-unwind-tables -pipe -fexceptions -Warray-bounds -DLXCROOTFSMOUNT=\"/usr/local/lib/lxc/rootfs\" -DLXCPATH=\"/usr/local/var/lib/lxc\" -DLXC_GLOBAL_CONF=\"/usr/local/etc/lxc/lxc.conf\" -DLXCINITDIR=\"/usr/local/libexec\" -DLIBEXECDIR=\"/usr/local/libexec\" -DLOGPATH=\"/usr/local/var/log/lxc\" -DLXCTEMPLATEDIR=\"/usr/local/share/lxc/templates\" -DLXC_DEFAULT_CONFIG=\"/usr/local/etc/lxc/default.conf\" -DDEFAULT_CGROUP_PATTERN=\"\" -DRUNTIME_PATH=\"/run\" -DSBINDIR=\"/usr/local/sbin\" -I ../../src -I ../../src/lxc -I ../../src/lxc/cgroups -I ../../src/lxc/tools -I ../../src/lxc/storage -pthread -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -stdlib=libc++ -Wl,--as-needed -Wl,--gc-sections -Wl,-z -Wl,relro -Wl,-z -Wl,now -pie -Wl,-fuse-ld=gold -o fuzz-lxc-define-load fuzz_lxc_define_load-fuzz-lxc-define-load.o  ../lxc/.libs/liblxc.a /usr/lib/libFuzzingEngine.a -lpthread -pthread
Step #32: /usr/bin/ld: /usr/lib/libFuzzingEngine.a(aflpp_driver.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
Step #32: /usr/lib/libFuzzingEngine.a: error adding symbols: Bad value
Step #32: /usr/bin/ld: /usr/lib/libFuzzingEngine.a(aflpp_driver.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
Step #32: /usr/lib/libFuzzingEngine.a: error adding symbols: Bad value

@vanhauser-thc I wonder what can be done to make it work properly? My knee-jerk reaction was to turn off AFL in #5685. Another option seems to be to stop building the fuzz targets with -pie (lxc/lxc#3815).

evverx · 2021-04-29T21:43:11Z

FWIW Trying to figure out how libFuzzingEngine.a is built I've run /usr/local/bin/precompile_honggfuzz and /usr/local/bin/precompile_afl. Looks like aflpp_driver is compiled without -fPIC

+ make -C utils/aflpp_driver
make: Entering directory '/src/aflplusplus/utils/aflpp_driver'
/usr/local/bin/clang -I. -I../../include -O3 -funroll-loops -g -c aflpp_driver.c
ar ru libAFLDriver.a aflpp_driver.o
ar: `u' modifier ignored since `D' is the default (see `U')
ar: creating libAFLDriver.a
cp -vf libAFLDriver.a ../../
'libAFLDriver.a' -> '../../libAFLDriver.a'

while honggfuzz is built with -fPIC (because of LIBS_CFLAGS ?= -fPIC -fno-stack-protector -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 set in Makefile:

clang -c -O3 -funroll-loops -D_HF_LINUX_NO_BFD -std=c11 -I/usr/local/include -D_GNU_SOURCE -Wall -Wextra -Werror -Wno-format-truncation -Wno-override-init -I. -D_FILE_OFFSET_BITS=64 -Wno-initializer-overrides -Wno-unknown-warning-option -Wno-gnu-empty-initializer -Wno-format-pedantic -Wno-gnu-statement-expression -mllvm -inline-threshold=2000 -D_HF_ARCH_LINUX -fPIC -fno-stack-protector -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0   -o libhfcommon/files.o libhfcommon/files.c
clang -c -O3 -funroll-loops -D_HF_LINUX_NO_BFD -std=c11 -I/usr/local/include -D_GNU_SOURCE -Wall -Wextra -Werror -Wno-format-truncation -Wno-override-init -I. -D_FILE_OFFSET_BITS=64 -Wno-initializer-overrides -Wno-unknown-warning-option -Wno-gnu-empty-initializer -Wno-format-pedantic -Wno-gnu-statement-expression -mllvm -inline-threshold=2000 -D_HF_ARCH_LINUX -fPIC -fno-stack-protector -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0   -o libhfcommon/log.o libhfcommon/log.c
clang -c -O3 -funroll-loops -D_HF_LINUX_NO_BFD -std=c11 -I/usr/local/include -D_GNU_SOURCE -Wall -Wextra -Werror -Wno-format-truncation -Wno-override-init -I. -D_FILE_OFFSET_BITS=64 -Wno-initializer-overrides -Wno-unknown-warning-option -Wno-gnu-empty-initializer -Wno-format-pedantic -Wno-gnu-statement-expression -mllvm -inline-threshold=2000 -D_HF_ARCH_LINUX -fPIC -fno-stack-protector -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0   -o libhfcommon/ns.o libhfcommon/ns.c
clang -c -O3 -funroll-loops -D_HF_LINUX_NO_BFD -std=c11 -I/usr/local/include -D_GNU_SOURCE -Wall -Wextra -Werror -Wno-format-truncation -Wno-override-init -I. -D_FILE_OFFSET_BITS=64 -Wno-initializer-overrides -Wno-unknown-warning-option -Wno-gnu-empty-initializer -Wno-format-pedantic -Wno-gnu-statement-expression -mllvm -inline-threshold=2000 -D_HF_ARCH_LINUX -fPIC -fno-stack-protector -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0   -o libhfcommon/util.o libhfcommon/util.c
clang -c -O3 -funroll-loops -D_HF_LINUX_NO_BFD -std=c11 -I/usr/local/include -D_GNU_SOURCE -Wall -Wextra -Werror -Wno-format-truncation -Wno-override-init -I. -D_FILE_OFFSET_BITS=64 -Wno-initializer-overrides -Wno-unknown-warning-option -Wno-gnu-empty-initializer -Wno-format-pedantic -Wno-gnu-statement-expression -mllvm -inline-threshold=2000 -D_HF_ARCH_LINUX -fPIC -fno-stack-protector -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0   -o libhfuzz/fetch.o libhfuzz/fetch.c
clang -c -O3 -funroll-loops -D_HF_LINUX_NO_BFD -std=c11 -I/usr/local/include -D_GNU_SOURCE -Wall -Wextra -Werror -Wno-format-truncation -Wno-override-init -I. -D_FILE_OFFSET_BITS=64 -Wno-initializer-overrides -Wno-unknown-warning-option -Wno-gnu-empty-initializer -Wno-format-pedantic -Wno-gnu-statement-expression -mllvm -inline-threshold=2000 -D_HF_ARCH_LINUX -fPIC -fno-stack-protector -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0   -o libhfuzz/instrument.o libhfuzz/instrument.c
clang -c -O3 -funroll-loops -D_HF_LINUX_NO_BFD -std=c11 -I/usr/local/include -D_GNU_SOURCE -Wall -Wextra -Werror -Wno-format-truncation -Wno-override-init -I. -D_FILE_OFFSET_BITS=64 -Wno-initializer-overrides -Wno-unknown-warning-option -Wno-gnu-empty-initializer -Wno-format-pedantic -Wno-gnu-statement-expression -mllvm -inline-threshold=2000 -D_HF_ARCH_LINUX -fPIC -fno-stack-protector -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0   -o libhfuzz/linux.o libhfuzz/linux.c
clang -c -O3 -funroll-loops -D_HF_LINUX_NO_BFD -std=c11 -I/usr/local/include -D_GNU_SOURCE -Wall -Wextra -Werror -Wno-format-truncation -Wno-override-init -I. -D_FILE_OFFSET_BITS=64 -Wno-initializer-overrides -Wno-unknown-warning-option -Wno-gnu-empty-initializer -Wno-format-pedantic -Wno-gnu-statement-expression -mllvm -inline-threshold=2000 -D_HF_ARCH_LINUX -fPIC -fno-stack-protector -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0   -o libhfuzz/memorycmp.o libhfuzz/memorycmp.c
clang -c -O3 -funroll-loops -D_HF_LINUX_NO_BFD -std=c11 -I/usr/local/include -D_GNU_SOURCE -Wall -Wextra -Werror -Wno-format-truncation -Wno-override-init -I. -D_FILE_OFFSET_BITS=64 -Wno-initializer-overrides -Wno-unknown-warning-option -Wno-gnu-empty-initializer -Wno-format-pedantic -Wno-gnu-statement-expression -mllvm -inline-threshold=2000 -D_HF_ARCH_LINUX -fPIC -fno-stack-protector -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0   -o libhfuzz/performance.o libhfuzz/performance.c
clang -c -O3 -funroll-loops -D_HF_LINUX_NO_BFD -std=c11 -I/usr/local/include -D_GNU_SOURCE -Wall -Wextra -Werror -Wno-format-truncation -Wno-override-init -I. -D_FILE_OFFSET_BITS=64 -Wno-initializer-overrides -Wno-unknown-warning-option -Wno-gnu-empty-initializer -Wno-format-pedantic -Wno-gnu-statement-expression -mllvm -inline-threshold=2000 -D_HF_ARCH_LINUX -fPIC -fno-stack-protector -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0   -o libhfuzz/persistent.o libhfuzz/persistent.c
clang -c -O3 -funroll-loops -D_HF_LINUX_NO_BFD -std=c11 -I/usr/local/include -D_GNU_SOURCE -Wall -Wextra -Werror -Wno-format-truncation -Wno-override-init -I. -D_FILE_OFFSET_BITS=64 -Wno-initializer-overrides -Wno-unknown-warning-option -Wno-gnu-empty-initializer -Wno-format-pedantic -Wno-gnu-statement-expression -mllvm -inline-threshold=2000 -D_HF_ARCH_LINUX -fPIC -fno-stack-protector -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0   -o libhfnetdriver/netdriver.o libhfnetdriver/netdriver.c

It would be great if both aflpp and honggfuzz were built consistently to avoid running into weird linking issues.

evverx · 2021-04-30T01:23:56Z

Looking at the build logs I noticed that afl-clang-fast apparently misreports its pie support:

https://oss-fuzz-build-logs.storage.googleapis.com/log-13f7449a-d1a4-4091-aba8-023f424d8e5c.txt (where lxc compiled sucessfully because -pie wasn't "supported" and wasn't passed to the linker):

Step #32: checking if /src/aflplusplus/afl-clang-fast supports flag -pie in envvar LDFLAGS... no

https://oss-fuzz-build-logs.storage.googleapis.com/log-d218339e-4ebc-4413-b19c-084e9b4b1cb3.txt (where lxc failed to compile)

Step #32: checking if /src/aflplusplus/afl-clang-fast supports flag -pie in envvar LDFLAGS... yes

so it seems lxc/lxc#3815 should fix it once and for all but I think there is a bug in afl-clang-fast.

evverx · 2021-04-30T03:16:31Z

Looks like it boils down to AFL_LLVM_INSTRUMENT=CLASSIC,CTX-2 (which is set when $(($RANDOM % 10)) -lt 3) (which seems to explain why it fails from time to time).

# printf 'int main(void) { return 0; }' >conftest.c
# /src/aflplusplus/afl-clang-fast -o conftest -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope    -Werror -pie conftest.c -lpthread
afl-cc ++3.13a by Michal Zalewski, Laszlo Szekeres, Marc Heuse - mode: LLVM-PCGUARD
SanitizerCoveragePCGUARD++3.13a
[+] Instrumented 1 locations with no collisions (non-hardened mode).
/usr/bin/ld: /tmp/conftest-8d496c.o: relocation R_X86_64_32 against `__start___sancov_guards' can not be used when making a shared object; recompile with -fPIC
/tmp/conftest-8d496c.o: error adding symbols: Bad value
clang-12: error: linker command failed with exit code 1 (use -v to see invocation)

# AFL_LLVM_INSTRUMENT=CLASSIC,CTX-2 /src/aflplusplus/afl-clang-fast -o conftest -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope    -Werror -pie conftest.c -lpthread
afl-cc ++3.13a by Michal Zalewski, Laszlo Szekeres, Marc Heuse - mode: LLVM-CLASSIC + K-CTX-2
afl-llvm-pass++3.13a by <[email protected]> and <[email protected]>
[+] Instrumented 1 locations (non-hardened mode, ratio 100%).

It seems LXC is failing to compile with AFL with ``` ../../src/lxc/storage -pthread -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -stdlib=libc++ -Wl,--as-needed -Wl,--gc-sections -Wl,-z -Wl,relro -Wl,-z -Wl,now -pie -Wl,-fuse-ld=gold -o fuzz-lxc-define-load fuzz_lxc_define_load-fuzz-lxc-define-load.o ../lxc/.libs/liblxc.a /usr/lib/libFuzzingEngine.a -lpthread -pthread Step google#32: /usr/bin/ld: /usr/lib/libFuzzingEngine.a(aflpp_driver.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC Step google#32: /usr/lib/libFuzzingEngine.a: error adding symbols: Bad value Step google#32: clang-12: [0;1;31merror: [0m[1mlinker command failed with exit code 1 (use -v to see invocation)[0m Step google#32: make[3]: *** [fuzz-lxc-config-read] Error 1 ``` Apparently aflpp tends to misdetect compiler/linker features 30% (70%?) of the time: google#4280 (comment)

It seems LXC is failing to compile with AFL with ``` ../../src/lxc/storage -pthread -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -stdlib=libc++ -Wl,--as-needed -Wl,--gc-sections -Wl,-z -Wl,relro -Wl,-z -Wl,now -pie -Wl,-fuse-ld=gold -o fuzz-lxc-define-load fuzz_lxc_define_load-fuzz-lxc-define-load.o ../lxc/.libs/liblxc.a /usr/lib/libFuzzingEngine.a -lpthread -pthread Step #32: /usr/bin/ld: /usr/lib/libFuzzingEngine.a(aflpp_driver.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC Step #32: /usr/lib/libFuzzingEngine.a: error adding symbols: Bad value Step #32: clang-12: [0;1;31merror: [0m[1mlinker command failed with exit code 1 (use -v to see invocation)[0m Step #32: make[3]: *** [fuzz-lxc-config-read] Error 1 ``` Apparently aflpp tends to misdetect compiler/linker features 30% (70%?) of the time: #4280 (comment)

vanhauser-thc · 2021-04-30T07:50:47Z

@evverx thank you for debugging this issue!
I can confirm your PoC.
The culprit is SantitizeCoveragePCGUARD.cc however I do not understand yet why. It is the same base code as in llvm (thats where it is copied from). will investigate ...

vanhauser-thc · 2021-04-30T08:41:19Z

@evverx can you please test if #5691 works for you?

evverx · 2021-04-30T10:36:58Z

@vanhauser-thc thanks! I've just started rebuilding the base-builder image so it might take a while. I'll report back there once it's finished.

Another issue I found is that it appears the autoconf code run by ./configure to figure out whether -pie is supported is kind of broken in the sense that it just passes -pie (without -fPIE). It has nothing to do with aflpp though. If the check was correct it would be just easier to reproduce the build failure because lxc would always fail to compile trying to link with libFuzzingEngine.a compiled without -fPIC.

inferno-chromium changed the title ~~afl++ setup~~ Integrate afl++ with OSS-Fuzz, deprecate vanilla afl. Aug 14, 2020

jonathanmetzman mentioned this issue Aug 24, 2020

Build AFL++ in AFL builds and include in /out #4360

Closed

inferno-chromium mentioned this issue Feb 1, 2021

Revert "Switch to using afl-clang-fast for afl++." #5086

Merged

inferno-chromium mentioned this issue Mar 15, 2021

zeek: enable afl and simplify project.yaml #5148

Merged

xavier2k6 mentioned this issue Mar 18, 2021

Apply to/make use of Google's OSS-Fuzz (Fuzz testing/Sanitizers) qbittorrent/qBittorrent#10360

Open

This was referenced Apr 30, 2021

oss-fuzz: build the fuzzers without -PIE -pie lxc/lxc#3815

Closed

[lxc] turn off AFL for now #5685

Merged

vanhauser-thc mentioned this issue Apr 30, 2021

fix afl++ for pie targets #5691

Merged

inferno-chromium closed this as completed in #5691 Apr 30, 2021

Integrate afl++ with OSS-Fuzz, deprecate vanilla afl. #4280

Integrate afl++ with OSS-Fuzz, deprecate vanilla afl. #4280

Comments

vanhauser-thc commented Aug 7, 2020 • edited Loading

inferno-chromium commented Aug 8, 2020

inferno-chromium commented Jan 25, 2021

vanhauser-thc commented Jan 25, 2021

vanhauser-thc commented Jan 25, 2021

vanhauser-thc commented Jan 26, 2021

inferno-chromium commented Jan 26, 2021 • edited Loading

vanhauser-thc commented Jan 26, 2021 • edited Loading

inferno-chromium commented Jan 26, 2021

inferno-chromium commented Jan 26, 2021

vanhauser-thc commented Jan 26, 2021

inferno-chromium commented Jan 26, 2021

inferno-chromium commented Jan 26, 2021

inferno-chromium commented Jan 27, 2021 • edited Loading

vanhauser-thc commented Jan 27, 2021

inferno-chromium commented Jan 31, 2021

vanhauser-thc commented Jan 31, 2021 via email

inferno-chromium commented Jan 31, 2021

inferno-chromium commented Feb 1, 2021

vanhauser-thc commented Feb 1, 2021

inferno-chromium commented Feb 1, 2021 • edited Loading

First pull latest images

Make a change in base-builder, like apply the cl above. This creates a local build of the docker image.

Build the project using this image built above

vanhauser-thc commented Feb 1, 2021 • edited Loading

inferno-chromium commented Feb 1, 2021

vanhauser-thc commented Feb 1, 2021

vanhauser-thc commented Feb 1, 2021

vanhauser-thc commented Feb 1, 2021

vanhauser-thc commented Feb 1, 2021 • edited Loading

inferno-chromium commented Feb 2, 2021 • edited Loading

vanhauser-thc commented Feb 6, 2021

caolanm commented Feb 6, 2021

vanhauser-thc commented Feb 6, 2021

vanhauser-thc commented Feb 7, 2021

rlohning commented Feb 11, 2021

inferno-chromium commented Feb 22, 2021 • edited Loading

vanhauser-thc commented Feb 22, 2021

vanhauser-thc commented Feb 22, 2021

vanhauser-thc commented Feb 28, 2021

inferno-chromium commented Feb 28, 2021

vanhauser-thc commented Mar 1, 2021

vanhauser-thc commented Mar 1, 2021

vanhauser-thc commented Mar 15, 2021

vanhauser-thc commented Mar 21, 2021 • edited Loading

inferno-chromium commented Mar 22, 2021

evverx commented Apr 29, 2021 • edited Loading

evverx commented Apr 29, 2021

evverx commented Apr 30, 2021

evverx commented Apr 30, 2021

vanhauser-thc commented Apr 30, 2021

vanhauser-thc commented Apr 30, 2021

evverx commented Apr 30, 2021

vanhauser-thc commented Aug 7, 2020 •

edited

Loading

inferno-chromium commented Jan 26, 2021 •

edited

Loading

vanhauser-thc commented Jan 26, 2021 •

edited

Loading

inferno-chromium commented Jan 27, 2021 •

edited

Loading

inferno-chromium commented Feb 1, 2021 •

edited

Loading

vanhauser-thc commented Feb 1, 2021 •

edited

Loading

vanhauser-thc commented Feb 1, 2021 •

edited

Loading

inferno-chromium commented Feb 2, 2021 •

edited

Loading

inferno-chromium commented Feb 22, 2021 •

edited

Loading

vanhauser-thc commented Mar 21, 2021 •

edited

Loading

evverx commented Apr 29, 2021 •

edited

Loading