Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Umbrella issue: Projects failing #30

Open
DavidKorczynski opened this issue Jan 11, 2022 · 20 comments
Open

Umbrella issue: Projects failing #30

DavidKorczynski opened this issue Jan 11, 2022 · 20 comments
Labels
core feature help wanted Extra attention is needed

Comments

@DavidKorczynski
Copy link
Contributor

DavidKorczynski commented Jan 11, 2022

Umbrella issue for projects failing.

@DavidKorczynski
Copy link
Contributor Author

@Navidem I just tried nss and it killed my machine - most likely due to some resource exhaustion but am yet to isolate whether it's due to LTO or fuzz-introspector, or something third

@Navidem
Copy link
Contributor

Navidem commented Jan 11, 2022

@DavidKorczynski thanks for checking. I can say without LTO it finishes successfully within ~7 min on my side.

@DavidKorczynski
Copy link
Contributor Author

@DavidKorczynski thanks for checking. I can say without LTO it finishes successfully within ~7 min on my side.

I too compile it fine with python3 infra/helper.py build_fuzzers nss

@DavidKorczynski DavidKorczynski changed the title Projects failing Umbrella issue: Projects failing Jan 11, 2022
@DavidKorczynski
Copy link
Contributor Author

DavidKorczynski commented Jan 11, 2022

I can now build NSS with fuzz-introspector:

The issue is that NSS uses a lot of processes during compilation and each of these also uses a lot of memory. This can be solved by changing this line https://hg.mozilla.org/projects/nss/file/tip/build.sh#l276 from run_scanbuild "$ninja" -C "$target_dir" "${ninja_params[@]}" to run_scanbuild "$ninja" -C "$target_dir" -j 2 "${ninja_params[@]}"

I will explore if we can use some constraints on the Docker environment, e.g. https://docs.docker.com/config/containers/resource_constraints/#cpu This should work in a general way.

@oliverchang
Copy link
Contributor

@Navidem Let's create sub bugs and track them here for the ones you notice too!

@Navidem
Copy link
Contributor

Navidem commented Feb 3, 2022

freeimage fails to compile. Apparently it is not compatible with -flto.

@Navidem
Copy link
Contributor

Navidem commented Feb 3, 2022

Division by zero in post-processing/main.py
These projects encounter div by zero once the profile is not produced or fail to merge:

  • croaring [update: fixed]
  • nettle [update: fixed]
  • abseil-cpp [update: fixed]

Sample output for abseil-cpp:

Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:[+] Loading profiles
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader: - found 0 profiles to load
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:[+] Accummulating profiles
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:[+] Creating project profile
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Creating merged profile of 0 profiles
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Populating functions reached
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Populating functions unreached
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Creating all_functions dictionary
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Gathering complexity and incoming references of each function
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Completed creationg of merged profile
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:[+] Refining profiles
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:[+] Creating HTML report
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_html: - Creating HTML report
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_html: - Creating reachability overview table
Step #6 - "compile-libfuzzer-introspector-x86_64": Traceback (most recent call last):
Step #6 - "compile-libfuzzer-introspector-x86_64":   File "/fuzz-introspector/post-processing/main.py", line 73, in <module>
Step #6 - "compile-libfuzzer-introspector-x86_64":     run_analysis_on_dir(args.target_dir, args.git_repo_url, args.coverage_url)
Step #6 - "compile-libfuzzer-introspector-x86_64":   File "/fuzz-introspector/post-processing/main.py", line 44, in run_analysis_on_dir
Step #6 - "compile-libfuzzer-introspector-x86_64":     fuzz_html.create_html_report(
Step #6 - "compile-libfuzzer-introspector-x86_64":   File "/fuzz-introspector/post-processing/fuzz_html.py", line 526, in create_html_report
Step #6 - "compile-libfuzzer-introspector-x86_64":     html_string += create_top_summary_info(tables, project_profile)
Step #6 - "compile-libfuzzer-introspector-x86_64":   File "/fuzz-introspector/post-processing/fuzz_html.py", line 261, in create_top_summary_info
Step #6 - "compile-libfuzzer-introspector-x86_64":     functions_percentage = ((len(total_reached_functions)*1.0) / (len(total_reached_functions) + len(
Step #6 - "compile-libfuzzer-introspector-x86_64": ZeroDivisionError: float division by zero

Sample output for nettle:

Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:[+] Loading profiles
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_utils:f: fuzzerLogFile-1.data -- matches regex: fuzzerLogFile.*\.data$
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_utils:f: fuzzerLogFile-0.data -- matches regex: fuzzerLogFile.*\.data$
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader: - found 2 profiles to load
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader: - loading /src/inspector/fuzzerLogFile-1.data
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader: - loading /src/inspector/fuzzerLogFile-0.data
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:[+] Accummulating profiles
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:[+] Creating project profile
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Creating merged profile of 0 profiles
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Populating functions reached
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Populating functions unreached
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Creating all_functions dictionary
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Gathering complexity and incoming references of each function
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Completed creationg of merged profile
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:[+] Refining profiles
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:[+] Creating HTML report
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_html: - Creating HTML report
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_html: - Creating reachability overview table
Step #6 - "compile-libfuzzer-introspector-x86_64": Traceback (most recent call last):
Step #6 - "compile-libfuzzer-introspector-x86_64":   File "/fuzz-introspector/post-processing/main.py", line 73, in <module>
Step #6 - "compile-libfuzzer-introspector-x86_64":     run_analysis_on_dir(args.target_dir, args.git_repo_url, args.coverage_url)
Step #6 - "compile-libfuzzer-introspector-x86_64":   File "/fuzz-introspector/post-processing/main.py", line 44, in run_analysis_on_dir
Step #6 - "compile-libfuzzer-introspector-x86_64":     fuzz_html.create_html_report(
Step #6 - "compile-libfuzzer-introspector-x86_64":   File "/fuzz-introspector/post-processing/fuzz_html.py", line 526, in create_html_report
Step #6 - "compile-libfuzzer-introspector-x86_64":     html_string += create_top_summary_info(tables, project_profile)
Step #6 - "compile-libfuzzer-introspector-x86_64":   File "/fuzz-introspector/post-processing/fuzz_html.py", line 261, in create_top_summary_info
Step #6 - "compile-libfuzzer-introspector-x86_64":     functions_percentage = ((len(total_reached_functions)*1.0) / (len(total_reached_functions) + len(
Step #6 - "compile-libfuzzer-introspector-x86_64": ZeroDivisionError: float division by zero

@Navidem
Copy link
Contributor

Navidem commented Feb 3, 2022

The following projects fail to parse/merge profile because of encoding issues:

  • kamailio [update: fixed]

Sample output for kamailio:

Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:[+] Loading profiles
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_utils:f: fuzzerLogFile-1.data -- matches regex: fuzzerLogFile.*\.data$
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_utils:f: fuzzerLogFile-0.data -- matches regex: fuzzerLogFile.*\.data$
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader: - found 2 profiles to load
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader: - loading /src/inspector/fuzzerLogFile-1.data
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader: - loading /src/inspector/fuzzerLogFile-0.data
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:[+] Accummulating profiles
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_utils:f: fuzz_parse_msg.covreport -- matches regex: .*\.covreport$
Step #6 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_utils:f: fuzz_uri.covreport -- matches regex: .*\.covreport$
Step #6 - "compile-libfuzzer-introspector-x86_64": Traceback (most recent call last):
Step #6 - "compile-libfuzzer-introspector-x86_64":   File "/fuzz-introspector/post-processing/main.py", line 73, in <module>
Step #6 - "compile-libfuzzer-introspector-x86_64":     run_analysis_on_dir(args.target_dir, args.git_repo_url, args.coverage_url)
Step #6 - "compile-libfuzzer-introspector-x86_64":   File "/fuzz-introspector/post-processing/main.py", line 34, in run_analysis_on_dir
Step #6 - "compile-libfuzzer-introspector-x86_64":     profile.accummulate_profile(target_folder)
Step #6 - "compile-libfuzzer-introspector-x86_64":   File "/fuzz-introspector/post-processing/fuzz_data_loader.py", line 196, in accummulate_profile
Step #6 - "compile-libfuzzer-introspector-x86_64":     self.load_coverage(target_folder)
Step #6 - "compile-libfuzzer-introspector-x86_64":   File "/fuzz-introspector/post-processing/fuzz_data_loader.py", line 125, in load_coverage
Step #6 - "compile-libfuzzer-introspector-x86_64":     functions_hit, coverage_map = fuzz_cov_load.llvm_cov_load(target_folder, self.get_target_fuzzer_filename())
Step #6 - "compile-libfuzzer-introspector-x86_64":   File "/fuzz-introspector/post-processing/fuzz_cov_load.py", line 50, in llvm_cov_load
Step #6 - "compile-libfuzzer-introspector-x86_64":     for line in pf:
Step #6 - "compile-libfuzzer-introspector-x86_64":   File "/usr/local/lib/python3.8/codecs.py", line 322, in decode
Step #6 - "compile-libfuzzer-introspector-x86_64":     (result, consumed) = self._buffer_decode(data, self.errors, final)
Step #6 - "compile-libfuzzer-introspector-x86_64": UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 3854: invalid start byte

@DavidKorczynski
Copy link
Contributor Author

DavidKorczynski commented Feb 3, 2022

Division by zero in post-processing/main.py
These projects encounter div by zero once the profile is not produced or fail to merge:

Div by zero fixed in b03c4cf <-- This fixed another issue. The div-by-zero you highlight has not been fixed yet.

@DavidKorczynski
Copy link
Contributor Author

DavidKorczynski commented Feb 3, 2022

#30 (comment)

Kamailio should be fixed with 554ae74

have run locally where I verified the issue and also verified post-fix that it works now

@DavidKorczynski
Copy link
Contributor Author

DavidKorczynski commented Feb 3, 2022

Division by zero in post-processing/main.py These projects encounter div by zero once the profile is not produced or fail to merge:

  • croaring
  • nettle
  • abseil-cpp

This issue seem to have happened because these lines in the OSS-Fuzz integration:

https://github.com/google/oss-fuzz/blob/cb45dab05cdd76d34e2f16cdd96a1c1c1d12d701/infra/base-images/base-builder/compile#L210-L211

The problem is those find commands can overwrite fuzzer profiles if the fuzzer profiles have the same name. Fuzzer profiles will often have the same filename but in different locations.

I fixed this by adding a UUID to the fuzzer log files now: 4231c27 and I have verified that both croaring and nettle works now. As such, it shouldn't really happen now, but, the two lines I link to in the OSS-Fuzz integration should probably change considering fuzz introspector does not guarantee uniqueness (it almost does not) in output files. Fuzz introspector output files does guarantee {path, filename} to be unique though.

@Navidem
Copy link
Contributor

Navidem commented Feb 10, 2022

The following projects were failing because of keyError in all_functions dict, fixed by 65beb16 and 664f3bf

  • orbit
  • tinygltf
  • espeak-ng
  • snappy
  • netcdf
  • wuffs
  • dropbear

@Navidem
Copy link
Contributor

Navidem commented Feb 10, 2022

The following projects are failing with configure error Unknown float word ordering.

  • mercurial
  • python3-libraries
  • cpython3

A quick search shows people recommending use of -ffat-lto-objects when compiling for lto. Have not yet tested.

@DavidKorczynski
Copy link
Contributor Author

I added a small script to automate testing projects of oss-fuzz projects here https://github.com/ossf/fuzz-introspector/blob/main/oss_fuzz_integration/test_projects.sh

@Navidem
Copy link
Contributor

Navidem commented Feb 12, 2022

I did an investigation and extracted projects that are failing to link/build fuzzers and excluded those that are also failing with coverage build. Furthermore, I filtered those that fail when we only enable lto. These were projects that fail once -flegacy-pass-manager -flto flag is passed (without introspector pass):

bignum-fuzzer
freeradius
gnutls
cairo
ffmpeg
gdal
cpp-httplib
curl
freeimage
ghostscript
grok
hiredis
hoextdown
icu
jansson
lame
libidn2
libphonenumber
libpsl
librdkafka
libteken
libvips
llhttp
mdbtools
mpg123
ndpi
num-bigint
opencv
openh264
openssh
osquery
poppler
qpdf
rustcrypto
samba
sqlite3
tesseract-ocr
unrar
wget
wget2
wxwidgets
xnu
xvid

These are the project that we may give lower priority to debug.
On the other hand, the followings are the projects that introspector's pass is the main cause of build failure:

bloaty
dropbear
freetype2
gdk-pixbuf
glib
gpac
gstreamer
irssi
kcodecs
keystone
lcms
libprotobuf-mutator
mupdf
pidgin
quickjs
solidity
wuffs
xmlsec

@DavidKorczynski
Copy link
Contributor Author

DavidKorczynski commented Feb 25, 2022

introspector's pass is the main cause of build failure:

Do you mean here it's specifically to compiler pass or do some of these projects fail in the post-processing?

@DavidKorczynski
Copy link
Contributor Author

quickjs is working for me

@DavidKorczynski
Copy link
Contributor Author

wuffs is working for me, although the reports have fairly limited calltrees, and I think this is because of use of indirect pointers, e.g.: https://github.com/google/wuffs/blob/ccbec1833094b1d2ee18b969aaae356a46bc5b30/internal/cgen/base/pixconv-submodule.c#L5270-L5281

Ref #4

@Navidem Navidem added the help wanted Extra attention is needed label Feb 27, 2022
@Navidem Navidem added this to the v1 milestone Feb 27, 2022
@DavidKorczynski
Copy link
Contributor Author

DavidKorczynski commented Mar 9, 2022

Are we sure that all of these projects are related to fuzz-introspector? I am asking this because I debugged an issue over here #48 (comment) where it seems that the issue only happens when compiling with --sanitizer=introspector, however, the issue occurs even if FUZZ_INTROSPECTOR is unset, meaning that fuzz-introspector is essentially not run.

One of the potential issues is that --sanitizer=introspector also applies some other changes, such as specify

export LDFLAGS="-fuse-ld=gold"
export AR=llvm-ar
export RANLIB=llvm-ranlib

We should check if projects fail with these set, or, more specifically, if projects fail with --sanitizer=introspector but where FUZZ_INTROSPECTOR is unset. Because of they fail with that it's really not fuzz-introspector code as such.

I recon it would be nice to have some infrastructure to make it easy to isolate whether a given issue happens because of the fuzz-introspector llvm pass or e.g. the fact that we force use of -fuse-ld=gold. Also, it may seem that some issues only happens in the OSS-Fuzz environment, e.g. jsoncpp (see #48 (comment)).

I am currently looking at curl where I am suspecting the issue doesn't happen because of fuzz-introspector code but rather switchig to a linker that is not supported by curl. This infrastructure might be included here #41

@Navidem Navidem removed this from the v1 milestone Mar 30, 2022
@Navidem
Copy link
Contributor

Navidem commented Mar 30, 2022

cleared v1 milestone, as the debugging remaining failures has to continue afterwards.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core feature help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

3 participants