Add GreenBench implementation #1912
Conversation
| @@ -701,6 +701,7 @@ def __init__(self, trial): | |||
| self.time_ended = trial.time_ended | |||
| self.preemptible = trial.preemptible | |||
| self.cpuset = None | |||
| self.trial_group_num = trial.trial_group_num | |||
There was a problem hiding this comment.
I think this has been plumbed through properly but I'm not 100% sure (no reason to think it's wrong). Sorry this setup isn't great.
There was a problem hiding this comment.
Hi @jonathanmetzman the trial_group_num field is used as an identifier of the group of trials and this idea of trial grouping is used to make sure that we provide the same set of seeds to certain trails (within the same group).
For example, if in the config.yaml we set trials to 2 for the experiment so we have following trials in total (for fuzzer afl, libfuzzer and benchmark libjpeg):
- trial_id: 1, benchmark: libjpeg, fuzzer: afl
- trial_id: 2, benchmark: libjpeg, fuzzer: afl
- trial_id: 3, benchmark: libjpeg, fuzzer: libfuzzer
- trial_id: 4, benchmark: libjpeg, fuzzer: libfuzzer
so fuzzbench will generate following group (synthetically) and assign the same set initial seeds to trials within the same group:
- trial_id: 1, benchmark: libjpeg, fuzzer: afl --> Group A
- trial_id: 2, benchmark: libjpeg, fuzzer: afl --> Group B
- trial_id: 3, benchmark: libjpeg, fuzzer: libfuzzer --> Group A
- trial_id: 4, benchmark: libjpeg, fuzzer: libfuzzer --> Group B
Please let me know if it makes sense? also is there a way to only add/inject this trial_group_num field only when are running in MICRO_EXPERIMENT mode?
|
Also could you write short documentation explaining how to use the feature, I want to play with it. |
hi @jonathanmetzman, as this feature depends on oss-fuzz-corpus, so we need to specify in experiment-config.yml file. |
|
Can you fix the test failure please? |
|
Fixed all failed tests @jonathanmetzman can you please help to re-review? |
This reverts commit c734a74.
This PR ports implementation of https://github.com/Rigorous-Software-Engineering/greenbench into fuzzbench. Essentially, we introduce new experiment option to allow users to opt in micro benchmark fuzzing that described in the GreenBench paper.
This reverts commit c734a74.
This PR ports implementation of https://github.com/Rigorous-Software-Engineering/greenbench into fuzzbench. Essentially, we introduce new experiment option to allow users to opt in micro benchmark fuzzing that described in the GreenBench paper.