Fix recent FuzzBench cloud experiment failures

[DonggeLiu]

1. Fix TypeError: expected str, bytes or os.PathLike object, not NoneType in 2024-08-10-test.

Traceback (most recent call last):
  File "/src/experiment/runner.py", line 468, in experiment_main
    runner.conduct_trial()
  File "/src/experiment/runner.py", line 290, in conduct_trial
    self.set_up_corpus_directories()
  File "/src/experiment/runner.py", line 275, in set_up_corpus_directories
    _unpack_clusterfuzz_seed_corpus(target_binary, input_corpus)
  File "/src/experiment/runner.py", line 144, in _unpack_clusterfuzz_seed_corpus
    seed_corpus_archive_path = get_clusterfuzz_seed_corpus_path(
  File "/src/experiment/runner.py", line 98, in get_clusterfuzz_seed_corpus_path
    fuzz_target_without_extension = os.path.splitext(fuzz_target_path)[0]
  File "/usr/local/lib/python3.10/posixpath.py", line 118, in splitext
    p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not NoneType

This happens on many benchmarks+fuzzers.
To be investigated later:

1. Why fuzz_target_path is None.

2. Why this did not happen in other recent experiments.

3. I thought I had seen this a long ago, Déjà vu?

4. Fixing No such file or directory: '/work/measurement-folders/<benchmark>-<fuzzer>/merged.json:

Traceback (most recent call last):
  File "/work/src/experiment/measurer/coverage_utils.py", line 74, in generate_coverage_report
    coverage_reporter.generate_coverage_summary_json()
  File "/work/src/experiment/measurer/coverage_utils.py", line 141, in generate_coverage_summary_json
    result = generate_json_summary(coverage_binary,
  File "/work/src/experiment/measurer/coverage_utils.py", line 269, in generate_json_summary
    with open(output_file, 'w', encoding='utf-8') as dst_file:
FileNotFoundError: [Errno 2] No such file or directory: '/work/measurement-folders/lcms_cms_transform_fuzzer-centipede/merged.json'

3. Remove incompatible benchmarks: openh264_decoder_fuzzer, stb_stbi_read_fuzzer

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-12-dg --fuzzers aflplusplus centipede honggfuzz libfuzzer --benchmarks stb_stbi_read_fuzzer openh264_decoder_fuzzer

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-12-2023 --fuzzers aflplusplus centipede honggfuzz libfuzzer --benchmarks stb_stbi_read_fuzzer openh264_decoder_fuzzer

[DonggeLiu]

Experiment 2024-08-12-2023 data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).

[DonggeLiu]

This failed likely because both fuzz targets failed to generate coverage repots, e.g.:

Not sure if this related: OSS-Fuzz's build status page shows openh264_decoder_fuzzer failed.

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-13-2023-libfuzzer-1 --fuzzers libfuzzer

[DonggeLiu]

Experiment 2024-08-13-2023-libfuzzer-1 data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).

[DonggeLiu]

Report is back : ) @addisoncrump
I will wait a bit longer before merging this to ensure the report stays alive.
Once I merge this to master, could you please update your PR and bring back the changes you added?
Thanks!

[addisoncrump]

Sure, I'll rebase.

[addisoncrump]

@DonggeLiu I am able to build both openh264 and stb_stbi fuzzers as in master locally with no issue. Like #2021, I think this is a cache issue.

[DonggeLiu]

@DonggeLiu I am able to build both openh264 and stb_stbi fuzzers as in master locally with no issue. Like #2021, I think this is a cache issue.

I see, thanks for the info!
Given that you are investigating this, is there any help I can provide?
For example, if you think some more cloud build logs can save you time debugging, please feel free to add them and request experiments.
I can run them for you and send you the related logs.

[DonggeLiu]

Report on this PR is still not ready, likely due to some VMs were preemptied.
I will give it one more day just to be 100% safe.

[addisoncrump]

Given that you are investigating this, is there any help I can provide?

Ah, I was investigating the specific issue with the bug benchmark. I don't think I can offer much help with the CI or the fuzzbench infra directly. I can say, however, that the coverage benchmarks you removed do work as expected locally with test-run. I need to check if the coverage measurer works as anticipated; maybe this needs to be updated instead.

[addisoncrump]

Ah, @DonggeLiu, try running make test-run-coverage-all. It complains that it can't find bloaty_fuzz_target on master 👀

[tokatoka]

@DonggeLiu I am able to build both openh264 and stb_stbi fuzzers as in master locally with no issue. Like #2021, I think this is a cache issue.

For me the same, they are working. I don't think they should be removed

[DonggeLiu]

I see, thanks @addisoncrump and @tokatoka .
I've brought them back.

The experiment is about to finish, I will merge this tmr morning.

[addisoncrump]

I confirmed the coverage measurers build locally as well. Will test when everything has finished building.

[addisoncrump]

Yup, I tested openh264 and stb benchmarks locally and they do perform measurements as anticipated. The issue is with the GCP runs, I would presume a build cache issue.

[DonggeLiu]

Yup, I tested openh264 and stb benchmarks locally and they do perform measurements as anticipated. The issue is with the GCP runs, I would presume a build cache issue.

I see, I reckon this could be due to impatible GCP vm environment and llvm?
I will look into this once I finish other tasks in hand.

Just to double-check @addisoncrump :
When you test them locally, did you remove their old local images beforehand?

[DonggeLiu]

Thanks for the information again, @addisoncrump!

[DonggeLiu]

TBR by @jonathanmetzman.

The experiment that proving this works:
#2023 (comment)

[addisoncrump]

When you test them locally, did you remove their old local images beforehand?

Yes, I do a docker system prune --all before every experiment.

[DonggeLiu]

When you test them locally, did you remove their old local images beforehand?

Yes, I do a docker system prune --all before every experiment.

I see, thanks for confirming.
I will merge this then.

[tokatoka]

I thought I had seen this a long ago, Déjà vu?

The same bug happened 1 year ago
#1886

[DonggeLiu]

The same bug happened 1 year ago #1886

Thanks for noticing this, let me see if @jonathanmetzman has more insight once he is back.

[DonggeLiu]

Add an error log here because this is unexpected.

[tokatoka]

One question, why is this function even called?

https://github.com/google/fuzzbench/blob/master/experiment/runner.py#L277
I think this is the line that eventually calls this line. But for example, when we observed the error for addison's experiment, the ossfuzz corpus was NOT used right? (unless they specified oss-fuzz-corpus: true)
then why we would unpack the clusterfuzz seed corpus at all?

aren't the seed corpus already prepared in build.sh or Dockerfile in each of the benchmarks?

[tokatoka]

Do you know if this env var CUSTOM_SEED_CORPUS_DIR set in normal(?) run or not?

[tokatoka]

To me these two lines seem wrong

        elif not environment.get('CUSTOM_SEED_CORPUS_DIR'):
            _unpack_clusterfuzz_seed_corpus(target_binary, input_corpus)

even if we don't use custom_seed_corpus_dir we don't necessarily need clusterfuzz seed corpus, do we??

although why this target_binary is None is another problem that needs investigation

[addisoncrump]

Just to reiterate, this is a major threat to validity -- especially when cached data is used. The cache completely overwrites the report, so the final report generated is simply showing only the last successful experiment. This effectively invalidates all future Fuzzbench reports until this issue is resolved.

I think the report generation issue indicates that safeguards should be put in place that simply terminate the experiment in such degenerative cases, since the results are effectively guaranteed to be invalid.