Launcher #48

vj-27 · 2021-04-08T08:31:42Z

issue #42
todo : Windows

libafl/src/utils.rs

fuzzers/libfuzzer_libpng/src/lib.rs

libafl/src/utils.rs

domenukk

Still fails on no_std, build with cargo build --no-default-features and remove all options that don't work.
I don't think launcher makes sense on no_std, so probably guard all related functions with #[cfg(feature = "std")]

domenukk · 2021-04-10T08:28:56Z

libafl/src/utils.rs

+    let num_cores = core_ids.len();
+    let (bind_to_core, cores) = parse_args(args, num_cores);
+
+    let _ = match unsafe { fork() }? {


We still need to handle this for Windows. It's not too hard, actually:
The first time, you reach this point, check if an env variable LIBAFL_IN_LAUNCHER (or something like this) exists.
If it does, see if we are supposed to go down the -> parent, or the -> child path.
If it doesn't, start up the parent etc. Set the env variable accordingly.
You can even test this on linux, if you simply don't use fork here.

I couldn't build this on windows. Any instructions for building ? the current implementation is tested only on linux :)

domenukk · 2021-04-10T08:30:41Z

fuzzers/libfuzzer_libpng/src/lib.rs

+}
+fn in_broker(broker_port: u16) -> Result<(), Error> {
+    let stats = SimpleStats::new(|s| println!("{}", s));
+    setup_new_llmp_broker::<


shouldn't this also be part of the launcher fn? The API doesn't look to clean right now...
We should provide sane and easy-to-use defaults, if people really want other behavior, they are free to code their own. :)

@vj-27 talking about this :)

domenukk · 2021-04-10T08:54:34Z

I've tried to add the no_std fixes, please test.
Also: If you add new pub functions, please add comments what they do (using ///) or make them non-pub, if they are ad-hoc internal functions.

domenukk · 2021-04-10T10:15:13Z

libafl/src/utils.rs

+            let sts = _handle.status();
+            #[cfg(feature = "std")]
+            println!("Exiting with status of broker {}", sts);
+            Ok(())


What happens if this function returns? It's one additional process that will run -> to where? No affinity set, not a broker.
I guess we can drop the nested fork alltogether, and handle this as broker?

Also, test if the broker will send an exit event to the clients or not (else, the broker may need to keep a handle of all children here and kill them on exit)

domenukk · 2021-04-10T18:02:58Z

Sweet! The only thing left to do is to clean up the api ;) Also, maybe win and linux launcher can be merged? Fork is "easier", but apart from that, it is roughly the same.

s1341 · 2021-04-10T18:26:53Z

Last thing... Maybe push the clap stuff into launcher? @domenukk wdyt?

domenukk · 2021-04-10T18:49:33Z

Last thing... Maybe push the clap stuff into launcher? @domenukk wdyt?

Not sure about that one:
LibAFL is a library, hence the launcher should probably not parse parameters itself. Other use cases may want to use different parameters, etc.

vj-27 · 2021-04-10T19:30:49Z

Sweet! The only thing left to do is to clean up the api ;) Also, maybe win and linux launcher can be merged? Fork is "easier", but apart from that, it is roughly the same.

so move the launcher from utils to each example?

s1341 · 2021-04-10T19:35:16Z

Ok. I accept what you're saying. Maybe the clap configuration can be done in code instead of a yaml, and then the launcher can provide a 'add arguments' function which anyone using the library can then use to add launcher-specific arguments to their fuzzer?

domenukk · 2021-04-10T19:38:18Z

so move the launcher from utils to each example?

No, simplify the launcher Api, I left a comment

domenukk · 2021-04-15T08:44:45Z

Recommend we have an option to save output to file(s) instead of just banishing it to /dev/null...

Is a file more useful than just printing to stdout here? Maybe, because we have multiple threads, right?

domenukk · 2021-04-15T08:45:58Z

@vj-27 are you still working on this pr or should I continue to clean it up? Mainly, the Api is still a bit convoluted. From an Api consumer perspective it should be as little work as possible to use :)

s1341 · 2021-04-15T10:22:03Z

Recommend we have an option to save output to file(s) instead of just banishing it to /dev/null...

Is a file more useful than just printing to stdout here? Maybe, because we have multiple threads, right?

Yup... It should either be a single file with a [$PID] line prefix, or separate files with a suffix of some sort.

andreafioraldi · 2021-05-17T12:15:54Z

And another small change before merging, maybe the launcher closure can take an index as parameter. In that way in the closure we can use different configuration, something like

if core_index == 0 {
    // enable CmpLog
}

domenukk · 2021-05-17T13:28:01Z

Pushed a CI script that at least tries to build them. Eventually, we need CI that runs the fuzzers for a while, too

andreafioraldi · 2021-05-18T11:05:10Z

The fuzzer works, but it is ugly. And, I still cannot use the cores at 100%, dunno why.

s1341 · 2021-05-18T16:56:18Z

I manage to get 100% with almost all green on an android device.

andreafioraldi

I manage to get 100% with almost all green on an android device.

andreafioraldi · 2021-04-15T08:04:22Z

fuzzers/libfuzzer_libpng/src/lib.rs

-/// The actual fuzzer
-fn fuzz(corpus_dirs: Vec<PathBuf>, objective_dir: PathBuf, broker_port: u16) -> Result<(), Error> {
-    // 'While the stats are state, they are usually used in the broker - which is likely never restarted
+struct FnArgs {


Instead of this struct, what about a closure?

andreafioraldi · 2021-04-15T08:04:53Z

fuzzers/libfuzzer_libpng/src/lib.rs

    // The restarting state will spawn the same process again as child, then restarted it each time it crashes.
    let (state, mut restarting_mgr) =
-        match setup_restarting_mgr::<_, _, StdShMem, _>(stats, broker_port) {
+        match setup_restarting_mgr_client::<_, _, StdShMem, _>(stats, fn_args.broker_port) {


Here I guess we can remove stats, it is used only in the broker

andreafioraldi · 2021-05-18T17:02:04Z

I manage to get 100% with almost all green on an android device.

Yes was my machine (13 days uptime with a fucked kernel setting), solved with a reboot.

I just figured out that I had an outdated pending review, don't read the last outdated comments.

s1341 · 2021-05-18T17:05:41Z

let me check it tomorrow on android, and then we can merge...

s1341 · 2021-05-19T04:37:33Z

@andreafioraldi Seems to work. There are still some bugs in android stuff, but I think they are unrelated to launcher. Go ahead and merge

domenukk · 2021-05-19T11:08:56Z

Good work everybody, thanks y'all ;)

andreafioraldi · 2021-05-19T11:20:38Z

I will update the book and release asap

* launcher in linux * silence stdout and stderr linux * arg parser and other changes * retry instead of sleep * no_std fixes * reordered includes * launcher for windows and kill clients when broker returns * cargo fmt * started launcher api cleanup * use closures instead of functions * small change * reordered launcher params * fixed clippy warnings * fixed no_std * moved launcher example to own folder * docu * cleanup launcher * more docs * Fix merge issues * Rework the launcher code to provide a cleaner API * Open file before spawning clients * launcher: fix merge issue, sleep for a different amount for each core * fixed no_std * Tcp Broker to Broker Communication (AFLplusplus#66) * initial b2b implementation * no_std and clippy fixes * b2b testcase added * more correct testcases * fixed b2b * typo * fixed unused warning * some clippy warning ignored * using clippy.sh * Update README.md * fixed clippy run in workflow * fixing clippy::match-same-arms * make clippy less pedantic * fixed some minor typos in the book * launcher: use s1341's fork of core_affinity * Build warning fix proposal, mostly about reference to packed fields. (AFLplusplus#79) * Observers refactor (AFLplusplus#84) * new observer structure with HasExecHooks * adapt libafl_frida to new observers * docstrings * Composing feedback (AFLplusplus#85) * composing feedbacks as logic operations and bump to 0.2 * adapt fuzzers and libafl_frida * fix windows build * fixed clippy warnings * Frida suppress instrumentation locations option (AFLplusplus#87) * Implement frida option * Format * add append/discard_metadata for and/or/not feedback (AFLplusplus#86) * add append/discard_metadata for and/or/not feedback * fix * Call append_metadata on crash (AFLplusplus#88) * Call append_metadata on crash * Formatting * Reachability example (AFLplusplus#65) * add reachability observer/feedback * add fuzzer exmaple * fmt * remove reachabilityobserver, use stdmapobserver instead * update diff.patch * update README * fix the clippy warning * Squashed commit of the following: commit f20524e Author: Andrea Fioraldi <[email protected]> Date: Tue May 4 16:00:39 2021 +0200 Composing feedback (AFLplusplus#85) * composing feedbacks as logic operations and bump to 0.2 * adapt fuzzers and libafl_frida * fix windows build commit e06efaa Author: Andrea Fioraldi <[email protected]> Date: Tue May 4 13:54:46 2021 +0200 Observers refactor (AFLplusplus#84) * new observer structure with HasExecHooks * adapt libafl_frida to new observers * docstrings commit 17c6fcd Merge: 08a2d43 a78a4b7 Author: Andrea Fioraldi <[email protected]> Date: Mon May 3 11:16:49 2021 +0200 Merge branch 'main' into dev commit 08a2d43 Author: David CARLIER <[email protected]> Date: Mon May 3 10:15:28 2021 +0100 Build warning fix proposal, mostly about reference to packed fields. (AFLplusplus#79) commit 88fe8fa Merge: d5d46ad d2e7719 Author: Andrea Fioraldi <[email protected]> Date: Mon May 3 11:05:42 2021 +0200 Merge pull request AFLplusplus#80 from marcograss/book-typos fixed some minor typos in the book commit a78a4b7 Author: s1341 <[email protected]> Date: Mon May 3 10:34:15 2021 +0300 frida-asan: Un-inline report funclet to reduce code bloat (AFLplusplus#81) * frida-asan: Outline report funclet to reduce code bloat * fmt commit d2e7719 Author: Marco Grassi <[email protected]> Date: Sun May 2 21:58:33 2021 +0800 fixed some minor typos in the book commit d5d46ad Author: Dominik Maier <[email protected]> Date: Sat May 1 23:09:10 2021 +0200 make clippy less pedantic commit 52d25e9 Author: Dominik Maier <[email protected]> Date: Sat May 1 22:23:59 2021 +0200 fixing clippy::match-same-arms commit cd66f88 Author: Dominik Maier <[email protected]> Date: Sat May 1 14:02:07 2021 +0200 fixed clippy run in workflow commit ddcf086 Author: Dominik Maier <[email protected]> Date: Sat May 1 13:53:29 2021 +0200 Update README.md commit c715f1f Author: Dominik Maier <[email protected]> Date: Sat May 1 13:48:38 2021 +0200 using clippy.sh commit 9374b26 Author: Dominik Maier <[email protected]> Date: Sat May 1 13:47:44 2021 +0200 some clippy warning ignored commit b9e75c0 Author: Dominik Maier <[email protected]> Date: Sat May 1 13:24:02 2021 +0200 Tcp Broker to Broker Communication (AFLplusplus#66) * initial b2b implementation * no_std and clippy fixes * b2b testcase added * more correct testcases * fixed b2b * typo * fixed unused warning * feedbacks now return a boolean value * use feedback_or, and modify Cargo.toml * fix diff between dev and this branch * fmt Co-authored-by: Dominik Maier <[email protected]> * clippy fixes * clippy fixes * clippy fixes, x86_64 warnings * more docs * Observers lifetime (AFLplusplus#89) * introduce MatchName and alow lifetimes in observers * adapt fuzzers to observers with lifetime * introduce type_eq when on nightly * fix no_std * fmt * Better docu (AFLplusplus#90) * more docs * more docs: * more docu * more docu * finished docs * cleaned up markup * must_use tags added * more docs * more docu, less clippy * more fixes * Clippy fixes (AFLplusplus#92) * more docs * more docs: * more docu * more docu * finished docs * cleaned up markup * must_use tags added * more docs * swapped if/else, as per clippy * more docu, less clippy * more fixes * Fix merge issues * Get rid of unneeded prints * Fix merge errors * added b2b to restarting interface * Setting SO_REUSEPORT * added b2b to launcher api * more windows launcher * Fix merge errors * Add b2b support to frida_libpng * make frida_libpng bind to a public address * Convert launcher into a builder LauncherBuilder * formatting * Convert setup_restarting_mgr to a builder RestartingMgrBuilder; leave setup_restarting_mgr_std as is, so that fuzzers work * RcShmem should be locked via a mutex * Wait at least 1 second between broker and first client, to avoid race * update frida_libpng README for cross-compiling to android (AFLplusplus#100) Co-authored-by: Ariel Zentner <[email protected]> * Fixed build for Windows * no_std fixes * reverted aa6773d & windows fixes * added pipes, moving to remove race conditions for rc shmem * fix unix build * fixed clippy: * fixed no_std once more * renamed b2b to remote_broker_addr * you get a pre_fork, and you get a post_fork, forks for everyone * switched to typed_builder * Fix merge isseu * Fix frida fuzzer with new Launcher builder * Introspection (AFLplusplus#97) * Rework to put `ClientPerfStats` in `State` and pass that along. Still need to work on getting granular information from `Feedback` and `Observer` * Add perf_stats feature to libafl/Cargo.toml * Update feedbacks to have with_perf * Remove unneeeded print statement * cargo fmt all the things * use local llvmint vs cpu specific asm for reading cycle counter * Remove debug testing code * Stats timeout to 3 seconds * Inline smallish functions for ClientPerfStats * Remove .libs/llvmint and have the correct conditional compilation of link_llvm_intrinsics on the perf_stats feature * pub(crate) the NUM_FEEDBACK and NUM_STAGES consts * Tcp Broker to Broker Communication (AFLplusplus#66) * initial b2b implementation * no_std and clippy fixes * b2b testcase added * more correct testcases * fixed b2b * typo * fixed unused warning * clippy fixes * fallback to systemtime on non-x86 * make clippy more strict * small fixes * bump 0.2.1 * readme Co-authored-by: ctfhacker <[email protected]> Co-authored-by: Dominik Maier <[email protected]> * typos (please review) * merged clippy.sh * utils * Add asan cores option (AFLplusplus#102) * added asan-cores option for frida fuzzer When asan is enabled (via LIBBAFL_FRIDA_OPTIONS enable-asan), you can filter exactly which of the cores asan should run on with the asan-cores variable. * add is_some check instead of !None Co-authored-by: Ariel Zentner <[email protected]> * moved utils to bolts * fixed typo * no_std fixes * unix fixes * fixed unix no_std build * fix llmp.rs * adapt libfuzzer_libpng_launcher * added all fuzzers to ci * fmt, improved ci * tests crate not ready for prime time * clippy fixes * make ci script executable * trying to fix example fuzzers * working libfuzzer_libpng_laucnher * frida_libpng builds * clippy * bump version * fix no_std * fix dep version * clippy fixes * more fies * clippy++ * warn again * clearer readme Co-authored-by: Vimal Joseph <[email protected]> Co-authored-by: Dominik Maier <[email protected]> Co-authored-by: s1341 <[email protected]> Co-authored-by: Marco Grassi <[email protected]> Co-authored-by: s1341 <[email protected]> Co-authored-by: Andrea Fioraldi <[email protected]> Co-authored-by: David CARLIER <[email protected]> Co-authored-by: Toka <[email protected]> Co-authored-by: r-e-l-z <[email protected]> Co-authored-by: Ariel Zentner <[email protected]> Co-authored-by: ctfhacker <[email protected]> Co-authored-by: hexcoder <[email protected]>

launcher in linux

0739306

vj-27 requested a review from domenukk April 8, 2021 08:31

silence stdout and stderr linux

710c28a

domenukk reviewed Apr 8, 2021

View reviewed changes

s1341 reviewed Apr 8, 2021

View reviewed changes

fuzzers/libfuzzer_libpng/src/lib.rs Outdated Show resolved Hide resolved

fuzzers/libfuzzer_libpng/src/lib.rs Outdated Show resolved Hide resolved

libafl/src/utils.rs Outdated Show resolved Hide resolved

libafl/src/utils.rs Outdated Show resolved Hide resolved

arg parser and other changes

b065bc3

vj-27 requested review from domenukk and s1341 April 9, 2021 19:43

retry instead of sleep

e1bce9c

domenukk reviewed Apr 10, 2021

View reviewed changes

domenukk added 3 commits April 10, 2021 10:44

Merge branch 'dev' into launcher

07b86a4

no_std fixes

afc53c8

no_std

5f14c14

domenukk added 2 commits April 10, 2021 11:15

reordered includes

fd35152

merged dev

f733bb1

domenukk reviewed Apr 10, 2021

View reviewed changes

Vimal Joseph and others added 4 commits April 11, 2021 04:38

launcher for windows and kill clients when broker returns

d3ec516

cargo fmt

3d345ed

Merge branch 'dev' into launcher

f911ce3

started launcher api cleanup

6df5236

domenukk added 3 commits May 17, 2021 14:57

added all fuzzers to ci

b8813cd

fmt, improved ci

149430b

tests crate not ready for prime time

94ace61

domenukk and others added 4 commits May 17, 2021 16:08

clippy fixes

ab5700a

make ci script executable

cd68db6

trying to fix example fuzzers

33e9166

working libfuzzer_libpng_laucnher

25b7a02

frida_libpng builds

b5a20a8

andreafioraldi approved these changes May 18, 2021

View reviewed changes

andreafioraldi and others added 9 commits May 19, 2021 10:21

clippy

8b42088

bump version

e627941

fix no_std

7b4481b

fix dep version

ace3c95

clippy fixes

7de7e70

more fies

c18adf5

clippy++

5bd1bbc

warn again

bc1b461

clearer readme

b4a1f75

domenukk merged commit d991395 into main May 19, 2021

domenukk deleted the launcher branch May 19, 2021 13:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Launcher #48

Launcher #48

vj-27 commented Apr 8, 2021 •

edited

Loading

domenukk left a comment

domenukk Apr 10, 2021

vj-27 Apr 10, 2021

domenukk Apr 10, 2021

domenukk Apr 10, 2021

domenukk commented Apr 10, 2021

domenukk Apr 10, 2021

domenukk Apr 10, 2021

domenukk commented Apr 10, 2021

s1341 commented Apr 10, 2021

domenukk commented Apr 10, 2021

vj-27 commented Apr 10, 2021

s1341 commented Apr 10, 2021

domenukk commented Apr 10, 2021

domenukk commented Apr 15, 2021

domenukk commented Apr 15, 2021

s1341 commented Apr 15, 2021

andreafioraldi commented May 17, 2021

domenukk commented May 17, 2021

andreafioraldi commented May 18, 2021

s1341 commented May 18, 2021

andreafioraldi left a comment

andreafioraldi Apr 15, 2021

andreafioraldi Apr 15, 2021

andreafioraldi commented May 18, 2021

s1341 commented May 18, 2021

s1341 commented May 19, 2021

domenukk commented May 19, 2021

andreafioraldi commented May 19, 2021

Launcher #48

Launcher #48

Conversation

vj-27 commented Apr 8, 2021 • edited Loading

domenukk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

domenukk commented Apr 10, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

domenukk commented Apr 10, 2021

s1341 commented Apr 10, 2021

domenukk commented Apr 10, 2021

vj-27 commented Apr 10, 2021

s1341 commented Apr 10, 2021

domenukk commented Apr 10, 2021

domenukk commented Apr 15, 2021

domenukk commented Apr 15, 2021

s1341 commented Apr 15, 2021

andreafioraldi commented May 17, 2021

domenukk commented May 17, 2021

andreafioraldi commented May 18, 2021

s1341 commented May 18, 2021

andreafioraldi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andreafioraldi commented May 18, 2021

s1341 commented May 18, 2021

s1341 commented May 19, 2021

domenukk commented May 19, 2021

andreafioraldi commented May 19, 2021

vj-27 commented Apr 8, 2021 •

edited

Loading