Another LibAFL compaison on new programs of SBFT'23 #1894

tokatoka · 2023-09-02T12:41:17Z

#1890
Created another PR for this

@alan32liu
I added build files for the additional 17 fuzzers in this branch.

DonggeLiu · 2023-09-02T12:43:14Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks arduinojson_json_fuzzer assimp_assimp_fuzzer astc-encoder_fuzz_astc_physical_to_symbolic botan_tls_server brotli_decode_fuzzer double-conversion_string_to_double_fuzzer draco_draco_pc_decoder_fuzzer dropbear_fuzzer-postauth_nomaths firestore_firestore_serializer_fuzzer fmt_chrono-duration-fuzzer guetzli_guetzli_fuzzer icu_unicode_string_codepage_create_fuzzer jansson_json_load_dump_fuzzer libaom_av1_dec_fuzzer libcoap_pdu_parse_fuzzer libhevc_hevc_dec_fuzzer librdkafka_fuzz_regex --fuzzers libafl_fuzzbench_cmplog libafl_fuzzbench_cov_accounting libafl_fuzzbench_explore libafl_fuzzbench_fast libafl_fuzzbench_mopt --experiment-name 2023-09-02-libafl-1

DonggeLiu · 2023-09-02T12:44:44Z

OK, I will launch the experiments as we discussed in #1890.
I hope you are feeling better now : )

DonggeLiu · 2023-09-02T12:51:41Z

Experiment 2023-09-02-libafl-1 data and results will be available later at:
The experiment data.
The experiment report.

DonggeLiu · 2023-09-02T12:52:08Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks arduinojson_json_fuzzer assimp_assimp_fuzzer astc-encoder_fuzz_astc_physical_to_symbolic botan_tls_server brotli_decode_fuzzer double-conversion_string_to_double_fuzzer draco_draco_pc_decoder_fuzzer dropbear_fuzzer-postauth_nomaths firestore_firestore_serializer_fuzzer fmt_chrono-duration-fuzzer guetzli_guetzli_fuzzer icu_unicode_string_codepage_create_fuzzer jansson_json_load_dump_fuzzer libaom_av1_dec_fuzzer libcoap_pdu_parse_fuzzer libhevc_hevc_dec_fuzzer librdkafka_fuzz_regex --fuzzers libafl_fuzzbench_naive libafl_fuzzbench_naive_ctx libafl_fuzzbench_ngram4 libafl_fuzzbench_ngram8 --experiment-name 2023-09-02-libafl-2

DonggeLiu · 2023-09-02T12:53:07Z

Experiment 2023-09-02-libafl-2 data and results will be available later at:
The experiment data.
The experiment report.

tokatoka · 2023-09-02T13:12:11Z

seems like one of the rust dependency is complaining an error in CI...

let me fix this first

tokatoka · 2023-09-03T11:10:29Z

now it looks good

could you start the experiment again?

DonggeLiu · 2023-09-03T13:58:56Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks arduinojson_json_fuzzer assimp_assimp_fuzzer astc-encoder_fuzz_astc_physical_to_symbolic botan_tls_server brotli_decode_fuzzer double-conversion_string_to_double_fuzzer draco_draco_pc_decoder_fuzzer dropbear_fuzzer-postauth_nomaths firestore_firestore_serializer_fuzzer fmt_chrono-duration-fuzzer guetzli_guetzli_fuzzer icu_unicode_string_codepage_create_fuzzer jansson_json_load_dump_fuzzer libaom_av1_dec_fuzzer libcoap_pdu_parse_fuzzer libhevc_hevc_dec_fuzzer librdkafka_fuzz_regex --fuzzers libafl_fuzzbench_cmplog libafl_fuzzbench_cov_accounting libafl_fuzzbench_explore libafl_fuzzbench_fast libafl_fuzzbench_mopt --experiment-name 2023-09-02-libafl-1

DonggeLiu · 2023-09-03T13:59:14Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks arduinojson_json_fuzzer assimp_assimp_fuzzer astc-encoder_fuzz_astc_physical_to_symbolic botan_tls_server brotli_decode_fuzzer double-conversion_string_to_double_fuzzer draco_draco_pc_decoder_fuzzer dropbear_fuzzer-postauth_nomaths firestore_firestore_serializer_fuzzer fmt_chrono-duration-fuzzer guetzli_guetzli_fuzzer icu_unicode_string_codepage_create_fuzzer jansson_json_load_dump_fuzzer libaom_av1_dec_fuzzer libcoap_pdu_parse_fuzzer libhevc_hevc_dec_fuzzer librdkafka_fuzz_regex --fuzzers libafl_fuzzbench_cmplog libafl_fuzzbench_cov_accounting libafl_fuzzbench_explore libafl_fuzzbench_fast libafl_fuzzbench_mopt --experiment-name 2023-09-03-libafl-1

DonggeLiu · 2023-09-04T01:32:41Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks arduinojson_json_fuzzer assimp_assimp_fuzzer astc-encoder_fuzz_astc_physical_to_symbolic botan_tls_server brotli_decode_fuzzer double-conversion_string_to_double_fuzzer draco_draco_pc_decoder_fuzzer dropbear_fuzzer-postauth_nomaths firestore_firestore_serializer_fuzzer fmt_chrono-duration-fuzzer guetzli_guetzli_fuzzer icu_unicode_string_codepage_create_fuzzer jansson_json_load_dump_fuzzer libaom_av1_dec_fuzzer libcoap_pdu_parse_fuzzer libhevc_hevc_dec_fuzzer librdkafka_fuzz_regex --fuzzers libafl_fuzzbench_cmplog libafl_fuzzbench_cov_accounting libafl_fuzzbench_explore libafl_fuzzbench_fast libafl_fuzzbench_mopt --experiment-name 2023-09-04-libafl-1

DonggeLiu · 2023-09-04T01:38:26Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks arduinojson_json_fuzzer assimp_assimp_fuzzer astc-encoder_fuzz_astc_physical_to_symbolic botan_tls_server brotli_decode_fuzzer double-conversion_string_to_double_fuzzer draco_draco_pc_decoder_fuzzer dropbear_fuzzer-postauth_nomaths firestore_firestore_serializer_fuzzer fmt_chrono-duration-fuzzer guetzli_guetzli_fuzzer icu_unicode_string_codepage_create_fuzzer jansson_json_load_dump_fuzzer libaom_av1_dec_fuzzer libcoap_pdu_parse_fuzzer libhevc_hevc_dec_fuzzer librdkafka_fuzz_regex --fuzzers libafl_fuzzbench_naive libafl_fuzzbench_naive_ctx libafl_fuzzbench_ngram4 libafl_fuzzbench_ngram8 --experiment-name 2023-09-04-libafl-2

DonggeLiu · 2023-09-04T01:42:17Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks arduinojson_json_fuzzer assimp_assimp_fuzzer astc-encoder_fuzz_astc_physical_to_symbolic botan_tls_server brotli_decode_fuzzer double-conversion_string_to_double_fuzzer draco_draco_pc_decoder_fuzzer dropbear_fuzzer-postauth_nomaths firestore_firestore_serializer_fuzzer fmt_chrono-duration-fuzzer guetzli_guetzli_fuzzer icu_unicode_string_codepage_create_fuzzer jansson_json_load_dump_fuzzer libaom_av1_dec_fuzzer libcoap_pdu_parse_fuzzer libhevc_hevc_dec_fuzzer librdkafka_fuzz_regex --fuzzers libafl_fuzzbench_rand_scheduler libafl_fuzzbench_value_profile libafl_fuzzbench_weighted--experiment-name 2023-09-04-libafl-3

DonggeLiu · 2023-09-04T01:43:21Z

Experiment 2023-09-04-libafl-1 data and results will be available later at:
The experiment data.
The experiment report.

DonggeLiu · 2023-09-04T01:43:28Z

Experiment 2023-09-04-libafl-2 data and results will be available later at:
The experiment data.
The experiment report.

DonggeLiu · 2023-09-04T01:43:36Z

Experiment 2023-09-04-libafl-3 data and results will be available later at:
The experiment data.
The experiment report.

DonggeLiu · 2023-09-04T23:38:18Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks arduinojson_json_fuzzer assimp_assimp_fuzzer astc-encoder_fuzz_astc_physical_to_symbolic botan_tls_server brotli_decode_fuzzer double-conversion_string_to_double_fuzzer draco_draco_pc_decoder_fuzzer dropbear_fuzzer-postauth_nomaths firestore_firestore_serializer_fuzzer fmt_chrono-duration-fuzzer guetzli_guetzli_fuzzer icu_unicode_string_codepage_create_fuzzer jansson_json_load_dump_fuzzer libaom_av1_dec_fuzzer libcoap_pdu_parse_fuzzer libhevc_hevc_dec_fuzzer librdkafka_fuzz_regex --fuzzers libafl_fuzzbench_rand_scheduler libafl_fuzzbench_value_profile libafl_fuzzbench_weighted --experiment-name 2023-09-05-libafl-3

DonggeLiu · 2023-09-04T23:39:58Z

Sorry, there is a type in my command to run 2023-09-04-libafl-3.
It's fixed above in experiment 2023-09-05-libafl-3, its data and results will be available later at:
The experiment data.
The experiment report.

tokatoka · 2023-09-06T12:05:09Z

Hi @alan32liu

I see there's some missing data, NaNs in the experiment result.
https://storage.googleapis.com/www.fuzzbench.com/reports/experimental/2023-09-04-libafl-1/index.html
https://storage.googleapis.com/www.fuzzbench.com/reports/experimental/2023-09-04-libafl-2/index.html
These two have 9 and 3 NaNs respectively.
I don't think this is a bug in the fuzzer because it ran perfectly in the usual 23 benchmarks before

I guess maybe 5 fuzzers are still too many and they couldn't generate coverage data in the end.
I want to get the data missing in NaNs.
For this purpose, Could you do the following experiment?

Run1 (For missing values in `libafl_fuzzbench_cmplog` and `libafl_fuzzbench_explore`)

- assimp_assimp_fuzzer
- brotli_decode_fuzzer
- guetzli_guetzli_fuzzer
- libcoap_pdu_parse_fuzzer

for

- libafl_fuzzbench_cmplog
- libafl_fuzzbench_explore

Run2 (For missing values in `libafl_fuzzbench_fast`, `libafl_fuzzbench_cov_accounting`, and `libafl_fuzzbench_mopt`)

- astc-encoder_fuzz_astc_physical_to_symbolic
- double-conversion_string_to_double_fuzzer
- draco_draco_pc_decoder_fuzzer
- fmt_chrono-duration-fuzzer
- libcoap_pdu_parse_fuzzer
- libhevc_hevc_dec_fuzzer

for

- libafl_fuzzbench_fast
- libafl_fuzzbench_cov_accounting
- libafl_fuzzbench_mopt

Run3 (For missing values in `libafl_fuzzbench_ngram8` and `libafl_fuzzbench_naive_ctx`)

- brotli_decode_fuzzer
- libaom_av1_dec_fuzzer

for

- libafl_fuzzbench_ngram8
- libafl_fuzzbench_naive_ctx

tokatoka · 2023-09-06T12:15:28Z

The commands would be

Run1:
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks assimp_assimp_fuzzer brotli_decode_fuzzer guetzli_guetzli_fuzzer libcoap_pdu_parse_fuzzer --fuzzers libafl_fuzzbench_cmplog libafl_fuzzbench_explore --experiment-name 2023-09-06-libafl-1

Run2:
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks astc-encoder_fuzz_astc_physical_to_symbolic double-conversion_string_to_double_fuzzer draco_draco_pc_decoder_fuzzer fmt_chrono-duration-fuzzer libcoap_pdu_parse_fuzzer libhevc_hevc_dec_fuzzer --fuzzers libafl_fuzzbench_cov_accounting libafl_fuzzbench_fast libafl_fuzzbench_mopt --experiment-name 2023-09-06-libafl-2

Run3:
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks brotli_decode_fuzzer libaom_av1_dec_fuzzer --fuzzers libafl_fuzzbench_naive_ctx libafl_fuzzbench_ngram8 --experiment-name 2023-09-06-libafl-3

tokatoka · 2023-09-06T13:28:22Z

oh wait.
we do actually have a perfect experiment here: https://www.fuzzbench.com/reports/experimental/2023-09-03-libafl-1/index.html
for

- libafl_fuzzbench_cmplog
- libafl_fuzzbench_explore
- libafl_fuzzbench_fast
- libafl_fuzzbench_cov_accounting
- libafl_fuzzbench_mopt

Sorry I didn't notice this.

In this case could you only run
Run3:

- brotli_decode_fuzzer
- libaom_av1_dec_fuzzer

for

- libafl_fuzzbench_ngram8
- libafl_fuzzbench_naive_ctx

?

DonggeLiu · 2023-09-07T01:55:22Z

Hi @alan32liu

I see there's some missing data, NaNs in the experiment result.

Hi @tokatoka, thanks for the summary.
Unfortunately, some of them are due to build failures. For example benchmark-botan_tls_server-fuzzer-libafl_fuzzbench_cmplog.
They are recorded in the build-logs (e.g., https://storage.googleapis.com/fuzzbench-data/index.html?prefix=<experiment_name>/build-logs/).

In this case could you only run
Run3:

brotli_decode_fuzzer

libaom_av1_dec_fuzzer
for

libafl_fuzzbench_ngram8

libafl_fuzzbench_naive_ctx

Sure, they build succifully, but got the following error during runitme:

Executed command: "nice -n 5 python3 -u -c from fuzzers.libafl_fuzzbench_ngram8 import fuzzer; fuzzer.fuzz("/out/seeds", "/out/corpus", "/out/av1_dec_fuzzer")" returned: 1.

I will rerun them below to double-check this is not flaky.

DonggeLiu · 2023-09-07T01:56:04Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks brotli_decode_fuzzer libaom_av1_dec_fuzzer --fuzzers libafl_fuzzbench_naive_ctx libafl_fuzzbench_ngram8 --experiment-name 2023-09-06-libafl-3

tokatoka · 2023-09-07T14:51:28Z

Unfortunately, some of them are due to build failures.

Yes for botan-tls, the issue was that their build system doesn't recognize our compiler wrapper.

Sure, they build succifully, but got the following error during runitme:

Yes this seems to be a problem in our ngram instrumentation.

I will rerun them below to double-check this is not flaky.

Thanks,
https://www.fuzzbench.com/reports/experimental/2023-09-06-libafl-3/index.html
this result is good for me.

DonggeLiu · 2023-09-07T22:06:20Z

Thanks, https://www.fuzzbench.com/reports/experimental/2023-09-06-libafl-3/index.html this result is good for me.

OK, I will post the data link below for future reference.

Experiment 2023-09-06-libafl-3 data and results will be available later at:
The experiment data.
The experiment report.

tokatoka · 2023-09-18T15:39:45Z

Hi @alan32liu

Actually last time I forgot to add one fuzzer; libafl_fuzzbench_grimoire.. 😞
Could you run the experiment for this fuzzer on all of the 17 target please?

FIY, The command would be

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks arduinojson_json_fuzzer assimp_assimp_fuzzer astc-encoder_fuzz_astc_physical_to_symbolic botan_tls_server brotli_decode_fuzzer double-conversion_string_to_double_fuzzer draco_draco_pc_decoder_fuzzer dropbear_fuzzer-postauth_nomaths firestore_firestore_serializer_fuzzer fmt_chrono-duration-fuzzer guetzli_guetzli_fuzzer icu_unicode_string_codepage_create_fuzzer jansson_json_load_dump_fuzzer libaom_av1_dec_fuzzer libcoap_pdu_parse_fuzzer libhevc_hevc_dec_fuzzer librdkafka_fuzz_regex --fuzzers libafl_fuzzbench_grimoire --experiment-name 2023-09-18-libafl

Thanks.

tokatoka · 2023-09-30T13:24:14Z

Sorry we found another problem and we want to request another experiment.

tokatoka · 2023-09-30T13:30:33Z

@alan32liu
We found that with assimp_assimp_fuzzer sometimes our fuzzer can't catch the abort in the allocator, leading to the premature death of the fuzzer. so in the latest commit, we decided to use jemalloc instead to patch it.

Could you run the experiment on
assimp_assimp_fuzzer
for these fuzzers?

libafl_fuzzbench_cmplog
libafl_fuzzbench_cov_accounting
libafl_fuzzbench_explore
libafl_fuzzbench_fast
libafl_fuzzbench_mopt
libafl_fuzzbench_naive
libafl_fuzzbench_naive_ctx
libafl_fuzzbench_ngram4
libafl_fuzzbench_ngram8
libafl_fuzzbench_rand_scheduler
libafl_fuzzbench_value_profile
libafl_fuzzbench_weighted
libafl_fuzzbench_grimoire
libafl_fuzzbench_composition

The command would be
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-30-libafl-jemalloc --benchmarks assimp_assimp_fuzzer --fuzzers libafl_fuzzbench_cmplog libafl_fuzzbench_composition libafl_fuzzbench_cov_accounting libafl_fuzzbench_explore libafl_fuzzbench_fast libafl_fuzzbench_mopt libafl_fuzzbench_naive libafl_fuzzbench_naive_ctx libafl_fuzzbench_ngram4 libafl_fuzzbench_ngram8 libafl_fuzzbench_rand_scheduler libafl_fuzzbench_value_profile libafl_fuzzbench_weighted libafl_fuzzbench_grimoire

DonggeLiu · 2023-10-04T02:49:37Z

@alan32liu We found that with assimp_assimp_fuzzer sometimes our fuzzer can't catch the abort in the allocator, leading to the premature death of the fuzzer. so in the latest commit, we decided to use jemalloc instead to patch it.

Could you run the experiment on assimp_assimp_fuzzer for these fuzzers?

libafl_fuzzbench_cmplog

libafl_fuzzbench_cov_accounting

libafl_fuzzbench_explore

libafl_fuzzbench_fast

libafl_fuzzbench_mopt

libafl_fuzzbench_naive

libafl_fuzzbench_naive_ctx

libafl_fuzzbench_ngram4

libafl_fuzzbench_ngram8

libafl_fuzzbench_rand_scheduler

libafl_fuzzbench_value_profile

libafl_fuzzbench_weighted

libafl_fuzzbench_grimoire

libafl_fuzzbench_composition

The command would be /gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-30-libafl-jemalloc --benchmarks assimp_assimp_fuzzer --fuzzers libafl_fuzzbench_cmplog libafl_fuzzbench_composition libafl_fuzzbench_cov_accounting libafl_fuzzbench_explore libafl_fuzzbench_fast libafl_fuzzbench_mopt libafl_fuzzbench_naive libafl_fuzzbench_naive_ctx libafl_fuzzbench_ngram4 libafl_fuzzbench_ngram8 libafl_fuzzbench_rand_scheduler libafl_fuzzbench_value_profile libafl_fuzzbench_weighted libafl_fuzzbench_grimoire

No problem.
But I will split this into multiple smaller experiments because of FuzzBench's bottleneck in measurement.

DonggeLiu · 2023-10-04T02:50:41Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-30-libafl-jemalloc-1 --benchmarks assimp_assimp_fuzzer --fuzzers libafl_fuzzbench_cmplog libafl_fuzzbench_composition libafl_fuzzbench_cov_accounting libafl_fuzzbench_explore libafl_fuzzbench_fast

DonggeLiu · 2023-10-04T03:16:57Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-04-libafl-jemalloc-1 --benchmarks assimp_assimp_fuzzer --fuzzers libafl_fuzzbench_cmplog libafl_fuzzbench_composition libafl_fuzzbench_cov_accounting libafl_fuzzbench_explore libafl_fuzzbench_fast

DonggeLiu · 2023-10-04T03:29:08Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-04-libafl-jemalloc-2 --benchmarks assimp_assimp_fuzzer --fuzzers libafl_fuzzbench_mopt libafl_fuzzbench_naive libafl_fuzzbench_naive_ctx libafl_fuzzbench_ngram4 libafl_fuzzbench_ngram8

DonggeLiu · 2023-10-04T04:08:08Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-04-libafl-jemalloc-3 --benchmarks assimp_assimp_fuzzer --fuzzers libafl_fuzzbench_rand_scheduler libafl_fuzzbench_value_profile libafl_fuzzbench_weighted libafl_fuzzbench_grimoire

DonggeLiu · 2023-10-05T02:05:26Z

Experiment 2023-10-04-libafl-jemalloc-1 data and results will be available later at:
The experiment data.
The experiment report.

Experiment 2023-10-04-libafl-jemalloc-2 data and results will be available later at:
The experiment data.
The experiment report.

Experiment 2023-10-04-libafl-jemalloc-3 data and results will be available later at:
The experiment data.
The experiment report.

tokatoka · 2023-10-05T16:42:59Z

Hi @alan32liu
We now changed to use another metric to select the component of the composition fuzzer in the last commit. and want to see the updated result.

Could you run libafl_fuzzbench_composition on the previous 6 programs again?

The command would be,
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks assimp_assimp_fuzzer brotli_decode_fuzzer draco_draco_pc_decoder_fuzzer guetzli_guetzli_fuzzer libaom_av1_dec_fuzzer libcoap_pdu_parse_fuzzer --fuzzers libafl_fuzzbench_composition --experiment-name 2023-10-05-libafl-composition

DonggeLiu · 2023-10-05T23:50:21Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks assimp_assimp_fuzzer brotli_decode_fuzzer draco_draco_pc_decoder_fuzzer guetzli_guetzli_fuzzer libaom_av1_dec_fuzzer libcoap_pdu_parse_fuzzer --fuzzers libafl_fuzzbench_composition --experiment-name 2023-10-05-libafl-composition

DonggeLiu · 2023-10-05T23:55:52Z

Experiment 2023-10-05-libafl-composition data and results will be available later at:
The experiment data.
The experiment report.

tokatoka · 2023-10-13T14:46:27Z

Hi @alan32liu

We changed how we composed the fuzzer again, and this time, we would like to see the result on 11 programs.
Could you run the experiment again?

fuzzers are

libafl_fuzzbench_composition_2
libafl_fuzzbench_composition

benchmarks are

1. `assimp_assimp_fuzzer`
2. `astc-encoder_fuzz_astc_physical_to_symbolic`
3. `brotli_decode_fuzzer`
4. `double-conversion_string_to_double_fuzzer`
5. `draco_draco_pc_decoder_fuzzer`
6. `fmt_chrono-duration-fuzzer`
7. `icu_unicode_string_codepage_create_fuzzer`
8. `guetzli_guetzli_fuzzer`
9. `libaom_av1_dec_fuzzer`
10. `libcoap_pdu_parse_fuzzer`
11. `libhevc_hevc_dec_fuzzer`

The command would be
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks assimp_assimp_fuzzer astc-encoder_fuzz_astc_psssssssssssssssshysical_to_symbolic brotli_decode_fuzzer double-conversion_string_to_double_fuzzer draco_draco_pc_decoder_fuzzer fmt_chrono-duration-fuzzer guetzli_guetzli_fuzzer icu_unicode_string_codepage_create_fuzzer libaom_av1_dec_fuzzer libcoap_pdu_parse_fuzzer libhevc_hevc_dec_fuzzer --fuzzers libafl_fuzzbench_composition libafl_fuzzbench_composition_2 --experiment-name 2023-10-13-composition

DonggeLiu · 2023-10-14T01:50:15Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks assimp_assimp_fuzzer astc-encoder_fuzz_astc_physical_to_symbolic brotli_decode_fuzzer double-conversion_string_to_double_fuzzer draco_draco_pc_decoder_fuzzer fmt_chrono-duration-fuzzer guetzli_guetzli_fuzzer icu_unicode_string_codepage_create_fuzzer libaom_av1_dec_fuzzer libcoap_pdu_parse_fuzzer libhevc_hevc_dec_fuzzer --fuzzers libafl_fuzzbench_composition libafl_fuzzbench_composition_2 --experiment-name 2023-10-13-composition

DonggeLiu · 2023-10-14T01:53:25Z

Experiment 2023-10-13-composition data and results will be available later at:
The experiment data.
The experiment report.

tokatoka · 2023-10-23T15:14:05Z

Hi @alan32liu
We again adjusted the composition algorithm. could you run the experiment with the latest commit?

The commands would be
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks assimp_assimp_fuzzer astc-encoder_fuzz_astc_physical_to_symbolic brotli_decode_fuzzer double-conversion_string_to_double_fuzzer draco_draco_pc_decoder_fuzzer fmt_chrono-duration-fuzzer guetzli_guetzli_fuzzer icu_unicode_string_codepage_create_fuzzer libaom_av1_dec_fuzzer libcoap_pdu_parse_fuzzer libhevc_hevc_dec_fuzzer --fuzzers libafl_fuzzbench_composition libafl_fuzzbench_composition_3 --experiment-name 2023-10-23-composition

DonggeLiu · 2023-10-24T23:23:06Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks assimp_assimp_fuzzer astc-encoder_fuzz_astc_physical_to_symbolic brotli_decode_fuzzer double-conversion_string_to_double_fuzzer draco_draco_pc_decoder_fuzzer fmt_chrono-duration-fuzzer guetzli_guetzli_fuzzer icu_unicode_string_codepage_create_fuzzer libaom_av1_dec_fuzzer libcoap_pdu_parse_fuzzer libhevc_hevc_dec_fuzzer --fuzzers libafl_fuzzbench_composition libafl_fuzzbench_composition_3 --experiment-name 2023-10-23-compositio

DonggeLiu · 2023-10-24T23:29:49Z

Experiment 2023-10-23-compositio data and results will be available later at:
The experiment data.
The experiment report.

tokatoka and others added 2 commits September 2, 2023 14:31

add fuzzers

c3049f4

cherry pick

e26a282

delete eco, update commit

3b9e3fa

tokatoka closed this Sep 7, 2023

tokatoka reopened this Sep 18, 2023

tokatoka closed this Sep 28, 2023

jemalloc

e27c6b6

tokatoka reopened this Sep 30, 2023

use cohend to feed into the model

07cfa5e

tokatoka added 7 commits October 11, 2023 15:33

12 benchmakrs

8c86ae7

upd

f2e4e10

upd

db9cea8

typo

83f4b43

upd

a1dd4df

upd

8cb3988

add all

f4fa307

another different algorithm

b050e23

more

fe4d7b1

Another LibAFL compaison on new programs of SBFT'23 #1894

Are you sure you want to change the base?

Another LibAFL compaison on new programs of SBFT'23 #1894

Conversation

tokatoka commented Sep 2, 2023 • edited Loading

DonggeLiu commented Sep 2, 2023

DonggeLiu commented Sep 2, 2023

DonggeLiu commented Sep 2, 2023

DonggeLiu commented Sep 2, 2023

DonggeLiu commented Sep 2, 2023

tokatoka commented Sep 2, 2023

tokatoka commented Sep 3, 2023

DonggeLiu commented Sep 3, 2023

DonggeLiu commented Sep 3, 2023

DonggeLiu commented Sep 4, 2023

DonggeLiu commented Sep 4, 2023

DonggeLiu commented Sep 4, 2023

DonggeLiu commented Sep 4, 2023

DonggeLiu commented Sep 4, 2023

DonggeLiu commented Sep 4, 2023

DonggeLiu commented Sep 4, 2023

DonggeLiu commented Sep 4, 2023

tokatoka commented Sep 6, 2023 • edited Loading

Run1 (For missing values in libafl_fuzzbench_cmplog and libafl_fuzzbench_explore)

Run2 (For missing values in libafl_fuzzbench_fast, libafl_fuzzbench_cov_accounting, and libafl_fuzzbench_mopt)

Run3 (For missing values in libafl_fuzzbench_ngram8 and libafl_fuzzbench_naive_ctx)

tokatoka commented Sep 6, 2023 • edited Loading

tokatoka commented Sep 6, 2023 • edited Loading

DonggeLiu commented Sep 7, 2023

DonggeLiu commented Sep 7, 2023

tokatoka commented Sep 7, 2023

DonggeLiu commented Sep 7, 2023

tokatoka commented Sep 18, 2023 • edited Loading

tokatoka commented Sep 30, 2023

tokatoka commented Sep 30, 2023 • edited Loading

DonggeLiu commented Oct 4, 2023

DonggeLiu commented Oct 4, 2023

DonggeLiu commented Oct 4, 2023

DonggeLiu commented Oct 4, 2023

DonggeLiu commented Oct 4, 2023

DonggeLiu commented Oct 5, 2023

tokatoka commented Oct 5, 2023

DonggeLiu commented Oct 5, 2023

DonggeLiu commented Oct 5, 2023

tokatoka commented Oct 13, 2023

DonggeLiu commented Oct 14, 2023

DonggeLiu commented Oct 14, 2023

tokatoka commented Oct 23, 2023 • edited Loading

DonggeLiu commented Oct 24, 2023

DonggeLiu commented Oct 24, 2023

tokatoka commented Sep 2, 2023 •

edited

Loading

tokatoka commented Sep 6, 2023 •

edited

Loading

Run1 (For missing values in `libafl_fuzzbench_cmplog` and `libafl_fuzzbench_explore`)

Run2 (For missing values in `libafl_fuzzbench_fast`, `libafl_fuzzbench_cov_accounting`, and `libafl_fuzzbench_mopt`)

Run3 (For missing values in `libafl_fuzzbench_ngram8` and `libafl_fuzzbench_naive_ctx`)

tokatoka commented Sep 6, 2023 •

edited

Loading

tokatoka commented Sep 6, 2023 •

edited

Loading

tokatoka commented Sep 18, 2023 •

edited

Loading

tokatoka commented Sep 30, 2023 •

edited

Loading

tokatoka commented Oct 23, 2023 •

edited

Loading