Native support for Go 1.18 corpus format

[elias-orijtech]

#7055 implements native Go 1.18 fuzzing but the corpus formats remain incompatible: OSS-Fuzz uses raw byte arrays, Go fuzzers support multiple arguments of various types (see https://go.dev/doc/fuzz/). #7055 solves some of this mismatch by implementing an encoding, https://github.com/AdaLogics/go-fuzz-headers/blob/main/consumer.go, to generate compatible parameters from OSS-Fuzz' byte array input.

However, several issues remain:

• Native Go corpus entries in testdata/fuzz/FuzzXXX have to be converted to OSS-Fuzz format during fuzz building. https://github.com/orijtech/otils/blob/master/corpus2ossfuzz/main.go implements this (see https://github.com/tendermint/tendermint/blob/cbae5f9f5336bd9882cb0b02912f7ae948fac5b4/test/fuzz/oss-fuzz-build.sh#L11 for example use), but it would be nice if OSS-Fuzz did this automatically.
• Testcases and corpus entries generated by OSS-Fuzz have to be converted to native Go format, through go-fuzz-headers or similar, before a crash can be reproduced.
• The byte array to multiple arguments encoding may not be optimal. For example, it encodes array length in a byte, making []byte slices longer than 255 bytes impossible: https://github.com/AdaLogics/go-fuzz-headers/blob/main/consumer.go#L422.

Maybe this is a duplicate of golang/go#50192. In any case, I'd love to know what plans there are for native Go corpus support.

[AdamKorcz]

This is in the process of being added. The first PR towards a fix is here: #7519. This will use the Go 1.18 runtime engine.

[elias-orijtech]

Great news! Sorry for not noticing your work in progress. Feel free to close this.

[AdamKorcz]

No problem. I am fine with keeping this issue open until it is actually fixed in OSS-fuzz.

[marten-seemann]

Sorry for the noise, I just wanted to check if there’s been any progress on Go native fuzzing. There are a couple of issues / PRs for that, but from the outside, it seems like all of them have stalled.

I’d love to rewrite the fuzz tests I have in quic-go to use Go native fuzzing, but doing so without the support for corpora doesn’t really make sense for some of the more complicated fuzz targets (e.g. fuzzing the handshake).

[AdamKorcz]

Sorry for the noise, I just wanted to check if there’s been any progress on Go native fuzzing. There are a couple of issues / PRs for that, but from the outside, it seems like all of them have stalled.

I’d love to rewrite the fuzz tests I have in quic-go to use Go native fuzzing, but doing so without the support for corpora doesn’t really make sense for some of the more complicated fuzz targets (e.g. fuzzing the handshake).

@marten-seemann Native fuzzing is supported. There are currently more than 100 native Go fuzzers running on OSS-Fuzz. They are run using libFuzzer and there are discrepancies in the corpora of the two engines. On that topic, are you manually generating the seed files?

Edit: Adding documentation on building native Go fuzzers: https://google.github.io/oss-fuzz/getting-started/new-project-guide/go-lang/#native-go-fuzzing-support. You need to add the github.com/AdamKorcz/go-118-fuzz-build/testing dependency to your go.mod at compile time. This is best done like this: https://github.com/istio/istio/blob/37bd292460f5b194eba749480f5f3b7c4daef9f5/tests/fuzz/oss_fuzz_build.sh#LL34C1-L34C99. You need to use the compile_native_go_fuzzer binary instead of the compile_go_fuzzer binary in your build file.

[marten-seemann]

@marten-seemann Native fuzzing is supported. There are currently more than 100 native Go fuzzers running on OSS-Fuzz. They are run using libFuzzer and there are discrepancies in the corpora of the two engines. On that topic, are you manually generating the seed files?

Currently my setup uses go-fuzz, and I'm manually generating seed files. I have a long-standing TODO to migrate to native Go fuzzing, and my understanding is that

• the only way to provide a corpus is using f.Add (for local development / fuzzing), but that those corpus entries will be ignored by OSS-Fuzz
• there's no way to export the entries added with f.Add to a format that OSS-Fuzz would understand
• there's no way to generate a corpus that OSS-Fuzz would understand, and that could somehow be transformed to a format that f.Add understands

Is that correct?

[elias-orijtech]

FWIW, corpus2ossfuzz converts native Go corpus files into OSS-Fuzz format. It doesn't handle f.Add'ed corpuses. Example use: cosmos-sdk.

[marten-seemann]

How do you get the native Go corpus file? Is this the corpus folder output by go test -fuzz? Is there any way to put specific inputs into that folder (kind of like f.Add does?)?

[elias-orijtech]

How do you get the native Go corpus file? Is this the corpus folder output by go test -fuzz?

Indeed. For example: https://github.com/cosmos/cosmos-sdk/tree/main/fuzz/tests/testdata/fuzz/FuzzCryptoHDDerivePrivateKeyForPath

Is there any way to put specific inputs into that folder (kind of like f.Add does?)?

Good question, I don't know of any way.

[AdamKorcz]

Is there any way to put specific inputs into that folder (kind of like f.Add does?)?

If you add it to the testdata/fuzz/FuzzerName folder, the Go fuzzing engine should use them.