infra: print log data when Python fuzz_target fails #12502

correctmost · 2024-09-15T19:20:32Z

Description

This will make it easier to debug coverage failures that are not reproducible locally.

The failure that I am trying to debug:

https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=62231
- https://oss-fuzz-build-logs.storage.googleapis.com/log-c420cf0c-f073-4c42-b75c-422971ef272e.txt

Step #5: Already have image (with digest): gcr.io/oss-fuzz-base/base-runner
Step #5: Entering python fuzzing
Step #5: Error happened getting coverage of fuzz_parse
Step #5: This is likely because Atheris did not exit gracefully

Similar log data is displayed in other blocks:

oss-fuzz/infra/base-images/base-runner/coverage

Lines 101 to 105 in f716590

    
           timeout $TIMEOUT $OUT/$target $args &> $LOGS_DIR/$target.log 
        
           if (( $? != 0 )); then 
        
             echo "Error occured while running $target:" 
        
             cat $LOGS_DIR/$target.log 
        
           fi

oss-fuzz/infra/base-images/base-runner/coverage

Lines 149 to 153 in f716590

    
           timeout $TIMEOUT $OUT/$target -test.coverprofile $DUMPS_DIR/$target.profdata &> $LOGS_DIR/$target.log 
        
           if (( $? != 0 )); then 
        
             echo "Error occured while running $target:" 
        
             cat $LOGS_DIR/$target.log 
        
           fi

oss-fuzz/infra/base-images/base-runner/coverage

Lines 206 to 210 in f716590

    
           timeout $TIMEOUT $OUT/$target $args &> $LOGS_DIR/$target.log 
        
           if (( $? != 0 )); then 
        
             echo "Error occured while running $target:" 
        
             cat $LOGS_DIR/$target.log 
        
           fi

oss-fuzz/infra/base-images/base-runner/coverage

Lines 255 to 260 in f716590

    
           JAZZERJS_EXTRA_ARGS=$jazzerjs_args $OUT/$target $args &> $LOGS_DIR/$target.log 
        
           if (( $? != 0 )); then 
        
             echo "Error occured while running $target:" 
        
             cat $LOGS_DIR/$target.log 
        
           fi

This PR is a continuation of #12405 with a renamed branch to avoid trial-build errors:

ERROR: (gcloud.builds.submit) INVALID_ARGUMENT: invalid build: invalid build tag "testing-cm/display-coverage-log": must match format "^[\\w][\\w.-]{0,127}$"

correctmost · 2024-09-27T22:32:01Z

@jonathanmetzman can you trigger a trial build? I think this change should be safe.

Thanks!

jonathanmetzman · 2024-09-30T15:58:02Z

/gcbrun trial_build.py python

correctmost · 2024-10-02T09:01:39Z

/gcbrun trial_build.py python

I verified that log data is now printed for Python coverage failures:

Project: bleach

master
- https://oss-fuzz-build-logs.storage.googleapis.com/log-52209599-6262-4e08-b6f0-53b2be4b9829.txt

Step #5: Entering python fuzzing
Step #5: Error happened getting coverage of linkify_fuzzer
Step #5: This is likely because Atheris did not exit gracefully
Step #5: Archive:  /workspace/out/libfuzzer-coverage-x86_64/sanitize_fuzzer.pkg.deps.zip

This PR branch
- https://oss-fuzz-gcb-logs.storage.googleapis.com/log-3ad59941-5472-4759-97d5-a4060803d137.txt

Step #25: Entering python fuzzing
Step #25: Error happened getting coverage of linkify_fuzzer
Step #25: This is likely because Atheris did not exit gracefully
Step #25: ERROR: ld.so: object '/workspace/out/libfuzzer-coverage-x86_64/sanitizer_with_fuzzer.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
Step #25: ERROR: ld.so: object '/workspace/out/libfuzzer-coverage-x86_64/sanitizer_with_fuzzer.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
Step #25: INFO: Using built-in libfuzzer
Step #25: WARNING: Failed to find function "__sanitizer_acquire_crash_state".
Step #25: WARNING: Failed to find function "__sanitizer_print_stack_trace".
Step #25: WARNING: Failed to find function "__sanitizer_set_death_callback".
Step #25: INFO: Running with entropic power schedule (0xFF, 100).
Step #25: INFO: Seed: 1424109555
Step #25: INFO:     5194 files found in /corpus/linkify_fuzzer
Step #25: INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1048576 bytes
Step #25: INFO: seed corpus: files: 5194 min: 1b max: 5222430b total: 17229845b rss: 51Mb
Step #25: #2048	pulse  cov: 1511 ft: 5202 corp: 1463/18Kb exec/s: 1024 rss: 53Mb
Step #25: 
Step #25:  === Uncaught Python exception: ===
Step #25: ValueError: list.remove(x): x not in list
Step #25: Traceback (most recent call last):
Step #25:   File "linkify_fuzzer.py", line 38, in TestOneInput
Step #25:   File "bleach/__init__.py", line 125, in linkify
Step #25:   File "bleach/linkifier.py", line 183, in linkify
Step #25:   File "bleach/_vendor/html5lib/html5parser.py", line 312, in parseFragment
Step #25:   File "bleach/html5lib_shim.py", line 521, in _parse
Step #25:   File "bleach/_vendor/html5lib/html5parser.py", line 242, in mainLoop
Step #25:   File "bleach/_vendor/html5lib/html5parser.py", line 496, in processEndTag
Step #25:   File "bleach/_vendor/html5lib/html5parser.py", line 1819, in endTagOther
Step #25:   File "bleach/_vendor/html5lib/html5parser.py", line 496, in processEndTag
Step #25:   File "bleach/_vendor/html5lib/html5parser.py", line 1541, in endTagFormatting
Step #25:   File "bleach/_vendor/html5lib/treebuilders/etree.py", line 106, in removeChild
Step #25: ValueError: list.remove(x): x not in list
Step #25: 
Step #25: INFO: Instrumenting bleach
Step #25: WARNING: It looks like this module is imported by a custom loader. Atheris has experimental support for this. However, it may be incompatible with certain libraries. If you experience unusual errors or poor coverage collection, try atheris.instrument_all() instead, add enable_loader_override=False to instrument_imports(), or file an issue on GitHub.
Step #25: INFO: Instrumenting bleach.linkifier
Step #25: INFO: Instrumenting urllib
Step #25: INFO: Instrumenting urllib.parse
Step #25: INFO: Instrumenting bleach.callbacks
Step #25: INFO: Instrumenting bleach.html5lib_shim
Step #25: INFO: Instrumenting bleach._vendor
Step #25: INFO: Instrumenting bleach._vendor.html5lib
Step #25: INFO: Instrumenting __future__
Step #25: INFO: Instrumenting bleach._vendor.html5lib.html5parser
Step #25: INFO: Instrumenting six
Step #25: INFO: Instrumenting bleach._vendor.html5lib._inputstream
Step #25: INFO: Instrumenting http
Step #25: INFO: Instrumenting http.client
Step #25: INFO: Instrumenting email
Step #25: INFO: Instrumenting email.parser
Step #25: INFO: Instrumenting email.feedparser
Step #25: INFO: Instrumenting email.errors
Step #25: INFO: Instrumenting email._policybase
Step #25: INFO: Instrumenting email.header
Step #25: INFO: Instrumenting email.quoprimime
Step #25: INFO: Instrumenting email.base64mime
Step #25: INFO: Instrumenting email.charset
Step #25: INFO: Instrumenting email.encoders
Step #25: INFO: Instrumenting quopri
Step #25: INFO: Instrumenting email.utils
Step #25: INFO: Instrumenting email._parseaddr
Step #25: INFO: Instrumenting calendar
Step #25: INFO: Instrumenting email.message
Step #25: INFO: Instrumenting uu
Step #25: INFO: Instrumenting email._encoded_words
Step #25: INFO: Instrumenting email.iterators
Step #25: INFO: Instrumenting ssl
Step #25: INFO: Instrumenting webencodings
Step #25: INFO: Instrumenting webencodings.labels
Step #25: INFO: Instrumenting encodings.utf_16_le
Step #25: INFO: Instrumenting encodings.utf_16_be
Step #25: INFO: Instrumenting bleach._vendor.html5lib.constants
Step #25: INFO: Instrumenting bleach._vendor.html5lib._utils
Step #25: INFO: Instrumenting xml.etree
Step #25: INFO: Instrumenting xml.etree.ElementTree
Step #25: INFO: Instrumenting xml.etree.ElementPath
Step #25: INFO: Instrumenting bleach._vendor.html5lib._tokenizer
Step #25: INFO: Instrumenting bleach._vendor.html5lib._trie
Step #25: INFO: Instrumenting bleach._vendor.html5lib._trie.py
Step #25: INFO: Instrumenting bleach._vendor.html5lib._trie._base
Step #25: INFO: Instrumenting bleach._vendor.html5lib.treebuilders
Step #25: INFO: Instrumenting bleach._vendor.html5lib.treebuilders.base
Step #25: INFO: Instrumenting bleach._vendor.html5lib.treewalkers
Step #25: INFO: Instrumenting bleach._vendor.html5lib.serializer
Step #25: INFO: Instrumenting xml.sax
Step #25: INFO: Instrumenting xml.sax.xmlreader
Step #25: INFO: Instrumenting xml.sax.handler
Step #25: INFO: Instrumenting xml.sax._exceptions
Step #25: INFO: Instrumenting xml.sax.saxutils
Step #25: INFO: Instrumenting urllib.request
Step #25: INFO: Instrumenting tempfile
Step #25: INFO: Instrumenting urllib.error
Step #25: INFO: Instrumenting urllib.response
Step #25: INFO: Instrumenting bleach._vendor.html5lib.filters
Step #25: INFO: Instrumenting bleach._vendor.html5lib.filters.base
Step #25: INFO: Instrumenting bleach._vendor.html5lib.filters.sanitizer
Step #25: INFO: Instrumenting bleach.sanitizer
Step #25: INFO: Instrumenting bleach.parse_shim
Step #25: INFO: Instrumenting bleach._vendor.parse
Step #25: ==52== ERROR: libFuzzer: fuzz target exited
Step #25: SUMMARY: libFuzzer: fuzz target exited
Step #25: MS: 0 ; base unit: 0000000000000000000000000000000000000000
Step #25: 0xff,0x3c,0xf3,0x20,0xff,0xbe,0x3c,0x74,0x61,0x62,0x6c,0x65,0xbe,0xbc,0x6c,0x69,0xbe,0x3c,0x73,0x3e,0x3c,0x73,0xbe,0x3c,0x73,0x3e,0x3c,0x73,0x3e,0x3c,0x2f,0x73,0x3e,0x3c,0x2f,0x73,0x3e,0x3c,0x2f,0x73,0x3e,0x3c,0x2f,0x73,0x3e,
Step #25: \377<\363 \377\276<table\276\274li\276<s><s\276<s><s></s></s></s></s>
Step #25: artifact_prefix='./'; Test unit written to ./crash-5dd3b269ae81fa9a611a1c074fc0a23d52736810
Step #25: Base64: /zzzIP++PHRhYmxlvrxsab48cz48c748cz48cz48L3M+PC9zPjwvcz48L3M+
Step #25: Archive:  /workspace/out/libfuzzer-coverage-x86_64/sanitize_fuzzer.pkg.deps.zip

This will make it easier to debug coverage failures that are not reproducible locally.

vitorguidi · 2024-10-07T17:18:24Z

/gcbrun trial_build.py python

vitorguidi

lgtm

This was referenced Sep 15, 2024

infra: print log data when Python fuzz_target fails #12405

Closed

Update OSS-Fuzz integration pylint-dev/astroid#2511

Closed

correctmost force-pushed the display-coverage-log branch 2 times, most recently from defafdb to 8ccd721 Compare September 27, 2024 21:22

infra: print log data when Python fuzz_target fails

ce01782

This will make it easier to debug coverage failures that are not reproducible locally.

correctmost force-pushed the display-coverage-log branch from 8ccd721 to ce01782 Compare October 3, 2024 00:56

Merge branch 'master' into display-coverage-log

12c3dc2

vitorguidi approved these changes Oct 7, 2024

View reviewed changes

vitorguidi merged commit 0fe8dae into google:master Oct 7, 2024
18 of 19 checks passed

correctmost deleted the display-coverage-log branch October 7, 2024 21:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

infra: print log data when Python fuzz_target fails #12502

infra: print log data when Python fuzz_target fails #12502

correctmost commented Sep 15, 2024

correctmost commented Sep 27, 2024

jonathanmetzman commented Sep 30, 2024

correctmost commented Oct 2, 2024

vitorguidi commented Oct 7, 2024

vitorguidi left a comment

	timeout $TIMEOUT $OUT/$target $args &> $LOGS_DIR/$target.log
	if (( $? != 0 )); then
	echo "Error occured while running $target:"
	cat $LOGS_DIR/$target.log
	fi

	timeout $TIMEOUT $OUT/$target -test.coverprofile $DUMPS_DIR/$target.profdata &> $LOGS_DIR/$target.log
	if (( $? != 0 )); then
	echo "Error occured while running $target:"
	cat $LOGS_DIR/$target.log
	fi

	JAZZERJS_EXTRA_ARGS=$jazzerjs_args $OUT/$target $args &> $LOGS_DIR/$target.log

	if (( $? != 0 )); then
	echo "Error occured while running $target:"
	cat $LOGS_DIR/$target.log
	fi

infra: print log data when Python fuzz_target fails #12502

infra: print log data when Python fuzz_target fails #12502

Conversation

correctmost commented Sep 15, 2024

Description

correctmost commented Sep 27, 2024

jonathanmetzman commented Sep 30, 2024

correctmost commented Oct 2, 2024

vitorguidi commented Oct 7, 2024

vitorguidi left a comment

Choose a reason for hiding this comment