Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable incompatible benchmarks for cloud experiments #2030

Merged
merged 3 commits into from
Aug 16, 2024

Conversation

DonggeLiu
Copy link
Contributor

Temporarily disable benchmark stb_stbi_read_fuzzer and openh264_decoder_fuzzerfrom cloud experiments, becaue they are proven to be incompatible in cloud build/run environment.

@addisoncrump kindly confirmed that they work in local experiments.

@DonggeLiu
Copy link
Contributor Author

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-16-2030-1 --fuzzers libfuzzer

@DonggeLiu
Copy link
Contributor Author

Experiment 2024-08-16-2030-1 data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).

@jonathanmetzman
Copy link
Contributor

Temporarily disable benchmark stb_stbi_read_fuzzer and openh264_decoder_fuzzerfrom cloud experiments, becaue they are proven to be incompatible in cloud build/run environment.

@addisoncrump kindly confirmed that they work in local experiments.

COuld you explain why they don't work?

@DonggeLiu
Copy link
Contributor Author

COuld you explain why they don't work?

I did not have a chance to verify this, but it's likely due to llvm-cov failed to generate coverage summary json at here.
Maybe outdated llvm in cloud VMs?

@jonathanmetzman
Copy link
Contributor

COuld you explain why they don't work?

I did not have a chance to verify this, but it's likely due to llvm-cov failed to generate coverage summary json at here. Maybe outdated llvm in cloud VMs?

Eh then this should probably be fixed instead.

@jonathanmetzman
Copy link
Contributor

I don't object if you don't have the time though

@DonggeLiu
Copy link
Contributor Author

I don't object if you don't have the time though

I am a bit inclined to merge this temporary mitigation for now and fix it after we settle down the agent work, if you don't mind.

BTW, do we plan to update anything in FuzzBench recently?

@oliverchang
Copy link
Contributor

merge

+1, let's mitigate this first, and track this as a bug for when we can get to it.

@DonggeLiu
Copy link
Contributor Author

+1, let's mitigate this first, and track this as a bug for when we can get to it.

OK I've created an issue for this and will merge the PR once the experiment finishes without any error.

@DonggeLiu DonggeLiu merged commit adff65e into master Aug 16, 2024
5 checks passed
@DonggeLiu DonggeLiu deleted the disable-incompatible-benchmark-for-cloud branch August 16, 2024 08:46
@DonggeLiu DonggeLiu mentioned this pull request Aug 17, 2024
DonggeLiu added a commit that referenced this pull request Aug 17, 2024
I thought I reverted this in ##2030, but apparently I did not.
ardier pushed a commit to ardier/fuzzbench that referenced this pull request Nov 25, 2024
Temporarily disable benchmark `stb_stbi_read_fuzzer` and
`openh264_decoder_fuzzer`from cloud experiments, becaue they are
[proven](google#2023 (comment))
to be incompatible in cloud build/run environment.

@addisoncrump kindly confirmed that they [work in local
experiments](google#2023 (comment)).
ardier pushed a commit to ardier/fuzzbench that referenced this pull request Nov 25, 2024
I thought I reverted this in #google#2030, but apparently I did not.
ardier pushed a commit to ardier/fuzzbench that referenced this pull request Nov 25, 2024
Temporarily disable benchmark `stb_stbi_read_fuzzer` and
`openh264_decoder_fuzzer`from cloud experiments, becaue they are
[proven](google#2023 (comment))
to be incompatible in cloud build/run environment.

@addisoncrump kindly confirmed that they [work in local
experiments](google#2023 (comment)).
ardier pushed a commit to ardier/fuzzbench that referenced this pull request Nov 25, 2024
I thought I reverted this in #google#2030, but apparently I did not.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants