Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

There doesn't seem to be a way to embed ids into FI reports #468

Open
evverx opened this issue Aug 15, 2022 · 6 comments
Open

There doesn't seem to be a way to embed ids into FI reports #468

evverx opened this issue Aug 15, 2022 · 6 comments

Comments

@evverx
Copy link
Contributor

evverx commented Aug 15, 2022

I'm not sure how exactly FI is supposed to be used but it seems to me that in general building at least two reports and comparing them would be a reliable way to figure out whether anything has been improved. To make it easier to tell reports apart it would be great if there was a way to embed SHAs and/or dates into reports. Dates and SHAs would be useful on OSS-Fuzz too to make it clear that for example https://storage.googleapis.com/oss-fuzz-introspector/systemd/inspector-report/20220422/fuzz_report.html is out of date and shouldn't be relied on (The date is included in the link there but it took me a while to find it and I think it would be better if it was in the report itself).

More generally I think it would be ideal if FI could generate diffs of some kind to make it easier to compare two reports by analogy with how for example https://coveralls.io/github/systemd/systemd?branch=main works but I think for the time being SHAs/dates should at least make it easier to figure out where reports come from.

DavidKorczynski added a commit to DavidKorczynski/fuzz-introspector-2 that referenced this issue Aug 24, 2022
DavidKorczynski added a commit that referenced this issue Aug 24, 2022
* html_report: add report generation timestamp

Ref: #468

* fix flake
@MarahGamdou
Copy link

MarahGamdou commented Oct 24, 2022

Hi,
I'm wondering if the commit SHAs have been included in the reports or if it is possible to include them. If not, do you think the date of report generation can be reliable to deduce commit SHAs?
Thank you!

@DavidKorczynski
Copy link
Contributor

DavidKorczynski commented Oct 24, 2022

We have dates in the report now, which can be used for tracking.

We can add sha's of commits to e.g. the metadata section. Not all projects rely on git or version control systems as such (xpdf if I remember correctly) so I think it's smartest to keep it as an optional.

Diffing two reports is useful -- perhaps a way to start this is to include more summary and easy-to-digest data at the top of the report. A more thorough diffing is also interesting but will take some more effort.

@MarahGamdou
Copy link

MarahGamdou commented Oct 24, 2022

@DavidKorczynski Thanks!
I agree that this may not be useful for all projects, so it's better to keep it optional as you mentioned. I also think sha's of commits can be very useful to track the exact code that was used to generate the reports. Sha's of commits are also integrated into OSS-Fuzz reports now (in the fuzzer logs and bug reports).

@evverx
Copy link
Contributor Author

evverx commented Oct 24, 2022

@MarahGamdou I'm not sure if it helps but as a stopgap (assuming the projects you're interested in use git) it should be possible to embed SHAs into FI build logs by adding something like

printf 'SHA: %s\n' $(git rev-parse HEAD)

to OSS-Fuzz build scripts (they are usually named build.sh in the OSS-Fuzz repository). This way at least the last seven FI reports can be reliably mapped onto their source by going to the "Introspector Builds" tab at https://oss-fuzz-build-logs.storage.googleapis.com/index.html and looking for the "SHA" string in the logs. It can't be easily automated unfortunately so use cases like attesting that certain reports come from particular commits for example or keeping track of coverage automatically can't be covered by that stopgap.

@MarahGamdou
Copy link

@evverx Thanks for suggesting this stopgap! My use case is actually attesting that certain FI reports come from particular commits because I use the reports as fuzzing evidence.

@evverx
Copy link
Contributor Author

evverx commented Oct 25, 2022

I think to fully cover this use case OSS-Fuzz should provide provenance of some kind that can be processed automatically without having to parse HTML reports and apart from SHAs it should probably include links to repositories, raw data like #485 and some other metadata. Having said that it's an advanced feature and I don't think there are a lot testing services built with use cases like that in mind (and it's understandable because this level of quality assurance where testing artifacts are used like that can't be even reached for various reasons in most places).

I think at this point it's possible to semi-automatically gauge whether the latest snapshots of projects are fuzzed or not using links to the last successful builds but that's mostly useful in environments where packages always roll forward on a regular basis without any downstream patches.

AlexDev08 pushed a commit to AlexDev08/fuzz-introspector that referenced this issue Nov 20, 2024
* html_report: add report generation timestamp

Ref: ossf/fuzz-introspector#468

* fix flake
shovon58 added a commit to shovon58/oss-introspector that referenced this issue Nov 21, 2024
* html_report: add report generation timestamp

Ref: ossf/fuzz-introspector#468

* fix flake
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants