LibAFL Saturation Experiment.

[tokatoka]

Hi @DonggeLiu
This is the longer fuzzer experiment that I was talking about last month.
For now can we check if this fuzzer stands the 24 hours run?

[tokatoka]

The command is

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-12-libafl --fuzzers libafl_saturation

[DonggeLiu]

Hi @DonggeLiu This is the longer fuzzer experiment that I was talking about last month. For now can we check if this fuzzer stands the 24 hours run?

Sure! It's actually 23 hours : )

Experiment 2024-05-12-libafl data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-12-libafl --fuzzers libafl_saturation

[tokatoka]

looks like it is not built. was something wrong?

[tokatoka]

nevermind 😅
i think i just forgot to refresh the webpage before checking the result.

[tokatoka]

Hello @DonggeLiu
I checked the log. I think the run was successful. Can we ask a 48 hour run that we discussed last month?

[DonggeLiu]

Hello @DonggeLiu I checked the log. I think the run was successful. Can we ask a 48 hour run that we discussed last month?

Sure!
Would you mind modify the experiment-config.yaml as discussed?
Change this to 2 days: https://github.com/google/fuzzbench/blob/master/service/experiment-config.yaml#L6
Change this to false: https://github.com/google/fuzzbench/blob/master/service/experiment-config.yaml#L14

@jonathanmetzman please let us know if I missed anything.
E.g., Shall we run a separate 48-hour exp for base fuzzers beforehand?
I reckon we only have their 24-hour results.

[tokatoka]

done

E.g., Shall we run a separate 48-hour exp for base fuzzers beforehand?

yeah i'm interested to see that too :)

[DonggeLiu]

done

E.g., Shall we run a separate 48-hour exp for base fuzzers beforehand?

yeah i'm interested to see that too :)

Sorry that this took so long, @jonathanmetzman and I were extremely busy last week.
We will start this tmr (if not today).

BTW, may I know which baseline fuzzers you are interested in comparing against?
Here are the options, but I presume not all of them are useful? (e.g., some were not updated in years.)

[tokatoka]

i'd like to see,

• afl
• aflfast
• aflplusplus
• centipede
• libafl
• libfuzzer

please!

[DonggeLiu]

Experiment 2024-05-22-bases data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).

[DonggeLiu]

Hi @tokatoka, while we are waiting for the base fuzzers experiment, would you like to run yours in parallel?
This can save some waiting time (particularly if some benchmarks fail), but it requires you to manually combine the two results together when both are ready. In addition, the report won't include the Unique code coverage plots section under each benchmark.

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-22-bases --fuzzers afl aflfast aflplusplus centipede libafl libfuzzer

[tokatoka]

Hi @tokatoka, while we are waiting for the base fuzzers experiment, would you like to run yours in parallel?
This can save some waiting time (particularly if some benchmarks fail), but it requires you to manually combine the two results together when both are ready. In addition, the report won't include the Unique code coverage plots section under each benchmark.

For me I can wait, and it's better for me to see the combined results

[tokatoka]

It seems they are stuck after 35 hours..?

[tokatoka]

but well it's fine..
can we start the experiment for our fuzzer too? @DonggeLiu

[DonggeLiu]

It seems they are stuck after 35 hours..?

Sorry I was traveling this week and did not check emails frequently.
@jonathanmetzman could you please have a look at this? It appears to be stuck at ~35-hour.

can we start the experiment for our fuzzer too? @DonggeLiu

We might have to understand why it stuck first.

[DonggeLiu]

I can see a lot of errors related to requesting metadata, maybe they are related?

network error when requesting metadata

[tokatoka]

Is there anything that I can help? 😃

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-06-04-bases --fuzzers afl

[DonggeLiu]

@jonathanmetzman, gentle ping : )

I suspect that this is the measurement bottleneck again, probably due to the experiment doubles the time?
Let me restart the experiment with afl only.
If that works, I will restart the experiments with one fuzzer for each.

Experiment 2024-06-04-bases data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).

---

For me to copy and paste later:

gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-22-bases --fuzzers aflfast 

gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-22-bases --fuzzers aflplusplus 

gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-22-bases --fuzzers centipede 

gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-22-bases --fuzzers libafl 

gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-22-bases --fuzzers libfuzzer

[DonggeLiu]

Hi @tokatoka Thanks for the waiting.
The report above confirms that the previous failure is caused by measurement bottleneck, I will only run one fuzzer per exp below.
Once they finish, I will run yours in another exp.

We can merge the statistics later manually, I don't think we can get unique coverage for each fuzzer this way, but it should be able to give us overall coverage info as usual.
Hope that's OK.

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-06-07-bases-aflfast --fuzzers aflfast

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-06-07-bases-aflpp --fuzzers aflplusplus

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-22-bases-centipede --fuzzers centipede

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-06-07-bases-libaf --fuzzers libafl

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-06-07-bases-libfuzzer --fuzzers libfuzzer

[DonggeLiu]

Experiment 2024-06-07-bases-aflfast data and results will be available later at:
The experiment data.
The experiment report.

---

Experiment 2024-06-07-bases-aflpp data and results will be available later at:
The experiment data.
The experiment report.

---

Experiment 2024-05-22-bases-centipede data and results will be available later at:
The experiment data.
The experiment report.

---

Experiment 2024-06-07-bases-libaf data and results will be available later at:
The experiment data.
The experiment report.

---

Experiment 2024-06-07-bases-libfuzzer data and results will be available later at:
The experiment data.
The experiment report.

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-06-07-bases-libfuzzer --fuzzers libfuzzer

[tokatoka]

@DonggeLiu
Thank you.
They seem to be working. Next can you run my fuzzer?

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-12-libafl --fuzzers libafl_saturation

Sure, I presume this is the right fuzzer?

gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-06-11-libafl-sat --fuzzers libafl_saturation

[DonggeLiu]

BTW, I noticed that the afl++ exp failed, likely due to measurement issues again.
Is that an important benchmark for you? I can further split the experiment by benchmarks if that's necessary.

[tokatoka]

Sure, I presume this is the right fuzzer?

Yes that's it.

BTW, I noticed that the afl++ exp failed, likely due to measurement issues again.
Is that an important benchmark for you? I can further split the experiment by benchmarks if that's necessary.

No that is not important so i don't need extra experiment for aflpp

Thank you @DonggeLiu

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-06-13-libafl-sat --fuzzers libafl_saturation

[DonggeLiu]

Thanks for the confirmation and waiting, @tokatoka.
Experiment 2024-06-13-libafl-sat data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).

[tokatoka]

Thank you 👍
We can close this