Unicode: write fuzzing tests for normalization #576

the-moisrex · 2024-11-13T12:07:33Z

Normalization algorithms (specially UTF-8 encoded one because of utf_reducer) really needs a fuzz testing because there are too many asserts in the there.

The text was updated successfully, but these errors were encountered:

the-moisrex · 2024-12-27T11:50:47Z

INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 2765359094
INFO: A corpus is not provided, starting from an empty corpus
#2	INITED exec/s: 0 rss: 35Mb
WARNING: no interesting inputs were found so far. Is the code instrumented for coverage?
This may also happen if the target rejected all inputs we tried so far
#1048576	pulse  corp: 1/1b lim: 4 exec/s: 524288 rss: 81Mb
#2097152	pulse  corp: 1/1b lim: 4 exec/s: 524288 rss: 129Mb
#4194304	pulse  corp: 1/1b lim: 4 exec/s: 466033 rss: 224Mb
#8388608	pulse  corp: 1/1b lim: 4 exec/s: 441505 rss: 415Mb
#16777216	pulse  corp: 1/1b lim: 4 exec/s: 453438 rss: 798Mb
#33554432	pulse  corp: 1/1b lim: 4 exec/s: 447392 rss: 1134Mb
#67108864	pulse  corp: 1/1b lim: 4 exec/s: 444429 rss: 1134Mb
/webpp/tests/./unicode_fuzz.hpp:46: Failure
Expected: (res16.size()) != (0), actual: 0 vs 0
\x0\xffffffdb\x0\x3

/webpp/tests/./unicode_fuzz.hpp:46: Failure
Expected: (res16.size()) != (0), actual: 0 vs 0
\x0\xffffffdb\x0\x3

#134217728	pulse  corp: 1/1b lim: 4 exec/s: 437191 rss: 1134Mb
AddressSanitizer:DEADLYSIGNAL
=================================================================
==56701==ERROR: AddressSanitizer: stack-overflow on address 0x7ffcf546e000 (pc 0x62b14e08f2ad bp 0x7ffcf546b530 sp 0x7ffcf546b530 T0)
    #0 0x62b14e08f2ad in void webpp::unicode::unchecked::next_char<char16_t*>(char16_t*&) /webpp/tests/./../webpp/unicode/././unicode.hpp:607:23
    #1 0x62b14e0a11f0 in webpp::unicode::pin_type<3ul, 4ul, char16_t*, char32_t>::operator++() /webpp/tests/./../webpp/unicode/utf_reducer.hpp:588:17
    #2 0x62b14e08f07c in unsigned long webpp::unicode::canonical_compose<unsigned long, char16_t*, char16_t const*>(char16_t*&, char16_t const*) /webpp/tests/./../webpp/unicode/normalization.hpp:606:13
    #3 0x62b14e08df2a in void webpp::unicode::canonical_compose<std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t>>, true>(std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t>>&) /webpp/tests/./../webpp/unicode/normalization.hpp:658:20
    #4 0x62b14e08dd24 in void webpp::unicode::normalize<(webpp::unicode::normalization_form)1, std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t>>>(std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t>>&) /webpp/tests/./../webpp/unicode/normalization.hpp:700:13
    #5 0x62b14e08989a in std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t>> webpp::unicode::toNFC<std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t>>>(std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t>>) /webpp/tests/./../webpp/unicode/normalization.hpp:710:9
    #6 0x62b14e088508 in webpp::tests::unicode_fuzz(std::basic_string_view<char, std::char_traits<char>>) /webpp/tests/./unicode_fuzz.hpp:40:28
    #7 0x62b14e0a88b5 in void std::__invoke_impl<void, void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&>(std::__invoke_other, void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&) /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../include/c++/14.2.1/bits/invoke.h:61:14
    #8 0x62b14e0a886c in std::__invoke_result<void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&>::type std::__invoke<void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&>(void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&) /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../include/c++/14.2.1/bits/invoke.h:96:14
    #9 0x62b14e0a883c in std::invoke_result<void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&>::type std::invoke<void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&>(void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&) /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../include/c++/14.2.1/functional:120:14
    #10 0x62b14e0902e2 in void fuzz_passer<void (&)(std::basic_string_view<char, std::char_traits<char>>)>(void (&)(std::basic_string_view<char, std::char_traits<char>>), unsigned char const*, unsigned long) /webpp/tests/common/fuzz_common.hpp:19:27
    #11 0x62b14e0881d3 in LLVMFuzzerTestOneInput /webpp/tests/unicode_fuzz.cpp:6:1
    #12 0x62b14df2c253 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/webpp/build-dev-clang/fuzz-unicode+0x68253) (BuildId: b0c80d65d65933ba7d486a8bc1479478c071fd16)
    #13 0x62b14df2cf00 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) (/webpp/build-dev-clang/fuzz-unicode+0x68f00) (BuildId: b0c80d65d65933ba7d486a8bc1479478c071fd16)
    #14 0x62b14df2df78 in fuzzer::Fuzzer::MutateAndTestOne() (/webpp/build-dev-clang/fuzz-unicode+0x69f78) (BuildId: b0c80d65d65933ba7d486a8bc1479478c071fd16)
    #15 0x62b14df2f267 in fuzzer::Fuzzer::Loop(std::vector<fuzzer::SizedFile, std::allocator<fuzzer::SizedFile>>&) (/webpp/build-dev-clang/fuzz-unicode+0x6b267) (BuildId: b0c80d65d65933ba7d486a8bc1479478c071fd16)
    #16 0x62b14df11394 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/webpp/build-dev-clang/fuzz-unicode+0x4d394) (BuildId: b0c80d65d65933ba7d486a8bc1479478c071fd16)
    #17 0x62b14defabc7 in main (/webpp/build-dev-clang/fuzz-unicode+0x36bc7) (BuildId: b0c80d65d65933ba7d486a8bc1479478c071fd16)
    #18 0x7ee6a6d8ee07 in __libc_start_call_main /usr/src/debug/glibc/glibc/csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #19 0x7ee6a6d8eecb in __libc_start_main /usr/src/debug/glibc/glibc/csu/../csu/libc-start.c:360:3
    #20 0x62b14defac04 in _start (/webpp/build-dev-clang/fuzz-unicode+0x36c04) (BuildId: b0c80d65d65933ba7d486a8bc1479478c071fd16)

SUMMARY: AddressSanitizer: stack-overflow /webpp/tests/./../webpp/unicode/././unicode.hpp:607:23 in void webpp::unicode::unchecked::next_char<char16_t*>(char16_t*&)
==56701==ABORTING
MS: 3 InsertByte-InsertByte-InsertByte-; base unit: adc83b19e793491b1c6ea0fd8b46cd9f32e592fc
0xe0,0xde,0xbc,0xa,
\340\336\274\012
artifact_prefix='./'; Test unit written to ./crash-fbadb5c8ae00196a4797621bc0d94c5c385637f1
Base64: 4N68Cg==

the-moisrex · 2024-12-27T11:59:39Z

fuzz-unicode -max_len=5
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 624740413
INFO: A corpus is not provided, starting from an empty corpus
#2	INITED exec/s: 0 rss: 33Mb
WARNING: no interesting inputs were found so far. Is the code instrumented for coverage?
This may also happen if the target rejected all inputs we tried so far
#1048576	pulse  corp: 1/1b lim: 5 exec/s: 524288 rss: 83Mb
#2097152	pulse  corp: 1/1b lim: 5 exec/s: 419430 rss: 133Mb
#4194304	pulse  corp: 1/1b lim: 5 exec/s: 419430 rss: 234Mb
#8388608	pulse  corp: 1/1b lim: 5 exec/s: 399457 rss: 436Mb
#16777216	pulse  corp: 1/1b lim: 5 exec/s: 390167 rss: 841Mb
AddressSanitizer:DEADLYSIGNAL
=================================================================
==58199==ERROR: AddressSanitizer: SEGV on unknown address 0x00fdb1116b01 (pc 0x5ab9014b7480 bp 0x7ffdb11162c0 sp 0x7ffdb11162c0 T0)
==58199==The signal is caused by a READ memory access.
fish: Job 1, 'fuzz-unicode -max_len=5' terminated by signal SIGSEGV (Address boundary error)

Well, how the heck am I supposed to debug this when you don't give me the input?

the-moisrex · 2024-12-27T12:05:03Z

INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1781691294
INFO: A corpus is not provided, starting from an empty corpus
#2	INITED exec/s: 0 rss: 33Mb
WARNING: no interesting inputs were found so far. Is the code instrumented for coverage?
This may also happen if the target rejected all inputs we tried so far
#1048576	pulse  corp: 1/1b lim: 6 exec/s: 524288 rss: 109Mb
#2097152	pulse  corp: 1/1b lim: 6 exec/s: 349525 rss: 184Mb
#4194304	pulse  corp: 1/1b lim: 6 exec/s: 349525 rss: 336Mb
#8388608	pulse  corp: 1/1b lim: 6 exec/s: 349525 rss: 639Mb
#16777216	pulse  corp: 1/1b lim: 6 exec/s: 349525 rss: 693Mb
/webpp/tests/./unicode_fuzz.hpp:46: Failure
Expected: (res16.size()) != (0), actual: 0 vs 0
\x0\xffffffdb\x0\x3

/webpp/tests/./unicode_fuzz.hpp:46: Failure
Expected: (res16.size()) != (0), actual: 0 vs 0
\x0\xffffffdb\x0\x3

/webpp/tests/./unicode_fuzz.hpp:46: Failure
Expected: (res16.size()) != (0), actual: 0 vs 0
\x0\xffffffdb\x3\x3

/webpp/tests/./unicode_fuzz.hpp:46: Failure
Expected: (res16.size()) != (0), actual: 0 vs 0
\x0\xffffffdb\x3\x3

/webpp/tests/./unicode_fuzz.hpp:46: Failure
Expected: (res16.size()) != (0), actual: 0 vs 0
\x0\xffffffda\x0\x3\xa

/webpp/tests/./unicode_fuzz.hpp:46: Failure
Expected: (res16.size()) != (0), actual: 0 vs 0
\x0\xffffffda\x0\x3\xa

#33554432	pulse  corp: 1/1b lim: 6 exec/s: 345921 rss: 694Mb
/webpp/tests/./unicode_fuzz.hpp:46: Failure
Expected: (res16.size()) != (0), actual: 0 vs 0
\x0\xffffffd9\x8\x3

/webpp/tests/./unicode_fuzz.hpp:46: Failure
Expected: (res16.size()) != (0), actual: 0 vs 0
\x0\xffffffd9\x8\x3

/webpp/tests/./unicode_fuzz.hpp:46: Failure
Expected: (res16.size()) != (0), actual: 0 vs 0
\x2c\xffffffd8\x3\x3\x3

/webpp/tests/./unicode_fuzz.hpp:46: Failure
Expected: (res16.size()) != (0), actual: 0 vs 0
\x2c\xffffffd8\x3\x3\x3

#67108864	pulse  corp: 1/1b lim: 6 exec/s: 342392 rss: 695Mb
AddressSanitizer:DEADLYSIGNAL
=================================================================
==59875==ERROR: AddressSanitizer: stack-overflow on address 0x7ffdce06c000 (pc 0x5f668c8892ad bp 0x7ffdce069f30 sp 0x7ffdce069f30 T0)
    #0 0x5f668c8892ad in void webpp::unicode::unchecked::next_char<char16_t*>(char16_t*&) /webpp/tests/./../webpp/unicode/././unicode.hpp:607:23
    #1 0x5f668c89b1f0 in webpp::unicode::pin_type<3ul, 4ul, char16_t*, char32_t>::operator++() /webpp/tests/./../webpp/unicode/utf_reducer.hpp:588:17
    #2 0x5f668c88907c in unsigned long webpp::unicode::canonical_compose<unsigned long, char16_t*, char16_t const*>(char16_t*&, char16_t const*) /webpp/tests/./../webpp/unicode/normalization.hpp:606:13
    #3 0x5f668c887f2a in void webpp::unicode::canonical_compose<std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t>>, true>(std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t>>&) /webpp/tests/./../webpp/unicode/normalization.hpp:658:20
    #4 0x5f668c887d24 in void webpp::unicode::normalize<(webpp::unicode::normalization_form)1, std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t>>>(std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t>>&) /webpp/tests/./../webpp/unicode/normalization.hpp:700:13
    #5 0x5f668c88389a in std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t>> webpp::unicode::toNFC<std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t>>>(std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t>>) /webpp/tests/./../webpp/unicode/normalization.hpp:710:9
    #6 0x5f668c882508 in webpp::tests::unicode_fuzz(std::basic_string_view<char, std::char_traits<char>>) /webpp/tests/./unicode_fuzz.hpp:40:28
    #7 0x5f668c8a28b5 in void std::__invoke_impl<void, void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&>(std::__invoke_other, void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&) /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../include/c++/14.2.1/bits/invoke.h:61:14
    #8 0x5f668c8a286c in std::__invoke_result<void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&>::type std::__invoke<void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&>(void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&) /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../include/c++/14.2.1/bits/invoke.h:96:14
    #9 0x5f668c8a283c in std::invoke_result<void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&>::type std::invoke<void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&>(void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&) /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../include/c++/14.2.1/functional:120:14
    #10 0x5f668c88a2e2 in void fuzz_passer<void (&)(std::basic_string_view<char, std::char_traits<char>>)>(void (&)(std::basic_string_view<char, std::char_traits<char>>), unsigned char const*, unsigned long) /webpp/tests/common/fuzz_common.hpp:19:27
    #11 0x5f668c8821d3 in LLVMFuzzerTestOneInput /webpp/tests/unicode_fuzz.cpp:6:1
    #12 0x5f668c726253 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/webpp/build-dev-clang/fuzz-unicode+0x68253) (BuildId: b0c80d65d65933ba7d486a8bc1479478c071fd16)
    #13 0x5f668c726f00 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) (/webpp/build-dev-clang/fuzz-unicode+0x68f00) (BuildId: b0c80d65d65933ba7d486a8bc1479478c071fd16)
    #14 0x5f668c727f78 in fuzzer::Fuzzer::MutateAndTestOne() (/webpp/build-dev-clang/fuzz-unicode+0x69f78) (BuildId: b0c80d65d65933ba7d486a8bc1479478c071fd16)
    #15 0x5f668c729267 in fuzzer::Fuzzer::Loop(std::vector<fuzzer::SizedFile, std::allocator<fuzzer::SizedFile>>&) (/webpp/build-dev-clang/fuzz-unicode+0x6b267) (BuildId: b0c80d65d65933ba7d486a8bc1479478c071fd16)
    #16 0x5f668c70b394 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/webpp/build-dev-clang/fuzz-unicode+0x4d394) (BuildId: b0c80d65d65933ba7d486a8bc1479478c071fd16)
    #17 0x5f668c6f4bc7 in main (/webpp/build-dev-clang/fuzz-unicode+0x36bc7) (BuildId: b0c80d65d65933ba7d486a8bc1479478c071fd16)
    #18 0x7b6b20225e07 in __libc_start_call_main /usr/src/debug/glibc/glibc/csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #19 0x7b6b20225ecb in __libc_start_main /usr/src/debug/glibc/glibc/csu/../csu/libc-start.c:360:3
    #20 0x5f668c6f4c04 in _start (/webpp/build-dev-clang/fuzz-unicode+0x36c04) (BuildId: b0c80d65d65933ba7d486a8bc1479478c071fd16)

SUMMARY: AddressSanitizer: stack-overflow /webpp/tests/./../webpp/unicode/././unicode.hpp:607:23 in void webpp::unicode::unchecked::next_char<char16_t*>(char16_t*&)
==59875==ABORTING
MS: 5 CrossOver-InsertByte-InsertByte-ChangeBinInt-ChangeBit-; base unit: adc83b19e793491b1c6ea0fd8b46cd9f32e592fc
0x7a,0xdf,0xef,0x8,
z\337\357\010
artifact_prefix='./'; Test unit written to ./crash-bf6cae9f3b3c761abe4f70c1cd8bee2ac07725d1
Base64: et/vCA==

the-moisrex · 2024-12-27T14:27:41Z

ninja fuzz-unicode; ./fuzz-unicode -max_len=4

[3/4] Building CXX object tests/CMakeFiles/fuzz-unicode.dir/unicode_fuzz.cpp.o
In file included from /webpp/tests/unicode_fuzz.cpp:2:
In file included from /webpp/tests/./unicode_fuzz.hpp:6:
/webpp/tests/./../webpp/unicode/normalization.hpp:723:82: warning: unused parameter 'start' [-Wunused-parameter]
  723 |     [[nodiscard]] static constexpr normalization_form normalization_form_of(Iter start, EIter end) noexcept {
      |                                                                                  ^
/webpp/tests/./../webpp/unicode/normalization.hpp:723:95: warning: unused parameter 'end' [-Wunused-parameter]
  723 |     [[nodiscard]] static constexpr normalization_form normalization_form_of(Iter start, EIter end) noexcept {
      |                                                                                               ^
2 warnings generated.
[4/4] Linking CXX executable fuzz-unicode
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 431125217
INFO: A corpus is not provided, starting from an empty corpus
#2	INITED exec/s: 0 rss: 33Mb
WARNING: no interesting inputs were found so far. Is the code instrumented for coverage?
This may also happen if the target rejected all inputs we tried so far
#1048576	pulse  corp: 1/1b lim: 4 exec/s: 524288 rss: 82Mb
#2097152	pulse  corp: 1/1b lim: 4 exec/s: 524288 rss: 129Mb
#4194304	pulse  corp: 1/1b lim: 4 exec/s: 466033 rss: 224Mb
#8388608	pulse  corp: 1/1b lim: 4 exec/s: 466033 rss: 415Mb
#16777216	pulse  corp: 1/1b lim: 4 exec/s: 453438 rss: 796Mb
#33554432	pulse  corp: 1/1b lim: 4 exec/s: 447392 rss: 1124Mb
#67108864	pulse  corp: 1/1b lim: 4 exec/s: 444429 rss: 1124Mb
/webpp/tests/./unicode_fuzz.hpp:47: Failure
Expected: (res16.size()) != (0), actual: 0 vs 0
\x\0\x�x\0\x

/webpp/tests/./unicode_fuzz.hpp:47: Failure
Expected: (res16.size()) != (0), actual: 0 vs 0
\x\0\x�x\0\x

#134217728	pulse  corp: 1/1b lim: 4 exec/s: 445906 rss: 1124Mb
AddressSanitizer:DEADLYSIGNAL
=================================================================
==83768==ERROR: AddressSanitizer: stack-overflow on address 0x7ffec3f17000 (pc 0x5a810ccde2ad bp 0x7ffec3f14580 sp 0x7ffec3f14580 T0)
    #0 0x5a810ccde2ad in void webpp::unicode::unchecked::next_char<char16_t*>(char16_t*&) /webpp/tests/./../webpp/unicode/././unicode.hpp:607:23
    #1 0x5a810ccf01f0 in webpp::unicode::pin_type<3ul, 4ul, char16_t*, char32_t>::operator++() /webpp/tests/./../webpp/unicode/utf_reducer.hpp:588:17
    #2 0x5a810ccde07c in unsigned long webpp::unicode::canonical_compose<unsigned long, char16_t*, char16_t const*>(char16_t*&, char16_t const*) /webpp/tests/./../webpp/unicode/normalization.hpp:606:13
    #3 0x5a810ccdcf2a in void webpp::unicode::canonical_compose<std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t>>, true>(std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t>>&) /webpp/tests/./../webpp/unicode/normalization.hpp:658:20
    #4 0x5a810ccdcd24 in void webpp::unicode::normalize<(webpp::unicode::normalization_form)1, std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t>>>(std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t>>&) /webpp/tests/./../webpp/unicode/normalization.hpp:700:13
    #5 0x5a810ccd889a in std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t>> webpp::unicode::toNFC<std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t>>>(std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t>>) /webpp/tests/./../webpp/unicode/normalization.hpp:710:9
    #6 0x5a810ccd7508 in webpp::tests::unicode_fuzz(std::basic_string_view<char, std::char_traits<char>>) /webpp/tests/./unicode_fuzz.hpp:41:28
    #7 0x5a810ccf78b5 in void std::__invoke_impl<void, void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&>(std::__invoke_other, void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&) /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../include/c++/14.2.1/bits/invoke.h:61:14
    #8 0x5a810ccf786c in std::__invoke_result<void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&>::type std::__invoke<void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&>(void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&) /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../include/c++/14.2.1/bits/invoke.h:96:14
    #9 0x5a810ccf783c in std::invoke_result<void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&>::type std::invoke<void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&>(void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&) /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../include/c++/14.2.1/functional:120:14
    #10 0x5a810ccdf2e2 in void fuzz_passer<void (&)(std::basic_string_view<char, std::char_traits<char>>)>(void (&)(std::basic_string_view<char, std::char_traits<char>>), unsigned char const*, unsigned long) /webpp/tests/common/fuzz_common.hpp:19:27
    #11 0x5a810ccd71d3 in LLVMFuzzerTestOneInput /webpp/tests/unicode_fuzz.cpp:6:1
    #12 0x5a810cb7b253 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/webpp/build-dev-clang/fuzz-unicode+0x68253) (BuildId: e93aaa9f5b94f5779902e4fe472a70dd11c7f8ba)
    #13 0x5a810cb7bf00 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) (/webpp/build-dev-clang/fuzz-unicode+0x68f00) (BuildId: e93aaa9f5b94f5779902e4fe472a70dd11c7f8ba)
    #14 0x5a810cb7cf78 in fuzzer::Fuzzer::MutateAndTestOne() (/webpp/build-dev-clang/fuzz-unicode+0x69f78) (BuildId: e93aaa9f5b94f5779902e4fe472a70dd11c7f8ba)
    #15 0x5a810cb7e267 in fuzzer::Fuzzer::Loop(std::vector<fuzzer::SizedFile, std::allocator<fuzzer::SizedFile>>&) (/webpp/build-dev-clang/fuzz-unicode+0x6b267) (BuildId: e93aaa9f5b94f5779902e4fe472a70dd11c7f8ba)
    #16 0x5a810cb60394 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/webpp/build-dev-clang/fuzz-unicode+0x4d394) (BuildId: e93aaa9f5b94f5779902e4fe472a70dd11c7f8ba)
    #17 0x5a810cb49bc7 in main (/webpp/build-dev-clang/fuzz-unicode+0x36bc7) (BuildId: e93aaa9f5b94f5779902e4fe472a70dd11c7f8ba)
    #18 0x70b052225e07 in __libc_start_call_main /usr/src/debug/glibc/glibc/csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #19 0x70b052225ecb in __libc_start_main /usr/src/debug/glibc/glibc/csu/../csu/libc-start.c:360:3
    #20 0x5a810cb49c04 in _start (/webpp/build-dev-clang/fuzz-unicode+0x36c04) (BuildId: e93aaa9f5b94f5779902e4fe472a70dd11c7f8ba)

SUMMARY: AddressSanitizer: stack-overflow /webpp/tests/./../webpp/unicode/././unicode.hpp:607:23 in void webpp::unicode::unchecked::next_char<char16_t*>(char16_t*&)
==83768==ABORTING
MS: 3 InsertByte-InsertByte-InsertByte-; base unit: adc83b19e793491b1c6ea0fd8b46cd9f32e592fc
0xe0,0xde,0xbc,0xa,
\340\336\274\012
artifact_prefix='./'; Test unit written to ./crash-fbadb5c8ae00196a4797621bc0d94c5c385637f1
Base64: 4N68Cg==

the-moisrex · 2024-12-30T08:27:56Z

This one does not blow, but address sanitizer does:

INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 2444696733
INFO: A corpus is not provided, starting from an empty corpus
#2	INITED exec/s: 0 rss: 37Mb
WARNING: no interesting inputs were found so far. Is the code instrumented for coverage?
This may also happen if the target rejected all inputs we tried so far
#1048576	pulse  corp: 1/1b lim: 5 exec/s: 524288 rss: 88Mb
#2097152	pulse  corp: 1/1b lim: 5 exec/s: 419430 rss: 137Mb
#4194304	pulse  corp: 1/1b lim: 5 exec/s: 381300 rss: 238Mb
#8388608	pulse  corp: 1/1b lim: 5 exec/s: 364722 rss: 440Mb
#16777216	pulse  corp: 1/1b lim: 5 exec/s: 364722 rss: 845Mb
#33554432	pulse  corp: 1/1b lim: 5 exec/s: 353204 rss: 982Mb
#67108864	pulse  corp: 1/1b lim: 5 exec/s: 353204 rss: 983Mb
#134217728	pulse  corp: 1/1b lim: 5 exec/s: 355073 rss: 984Mb
AddressSanitizer: CHECK failed: asan_allocator.cpp:239 "((old_chunk_state)) == ((CHUNK_QUARANTINE))" (0xff, 0x3) (tid=171611)
    #0 0x6069ff14595b in __asan::CheckUnwind() asan_rtl.cpp.o
    #1 0x6069ff1647b0 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) (/webpp/build-dev-clang/fuzz-unicode+0x1a17b0) (BuildId: 6e63114e5a7ed3034b19c8857e237254fba1d2a4)
    #2 0x6069ff05abe9 in __sanitizer::Quarantine<__asan::QuarantineCallback, __asan::AsanChunk>::DoRecycle(__sanitizer::QuarantineCache<__asan::QuarantineCallback>*, __asan::QuarantineCallback) (.isra.0) asan_allocator.cpp.o
    #3 0x6069ff05ae1e in __sanitizer::Quarantine<__asan::QuarantineCallback, __asan::AsanChunk>::Recycle(unsigned long, __asan::QuarantineCallback) (/webpp/build-dev-clang/fuzz-unicode+0x97e1e) (BuildId: 6e63114e5a7ed3034b19c8857e237254fba1d2a4)
    #4 0x6069ff0574b1 in __asan::asan_delete(void*, unsigned long, unsigned long, __sanitizer::BufferedStackTrace*, __asan::AllocType) (/webpp/build-dev-clang/fuzz-unicode+0x944b1) (BuildId: 6e63114e5a7ed3034b19c8857e237254fba1d2a4)
    #5 0x6069ff1850cb in operator delete(void*) (/webpp/build-dev-clang/fuzz-unicode+0x1c20cb) (BuildId: 6e63114e5a7ed3034b19c8857e237254fba1d2a4)
    #6 0x6069ff19492c in std::__new_allocator<char16_t>::deallocate(char16_t*, unsigned long) /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../include/c++/14.2.1/bits/new_allocator.h:172:2
    #7 0x6069ff1948cd in std::allocator<char16_t>::deallocate(char16_t*, unsigned long) /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../include/c++/14.2.1/bits/allocator.h:208:25
    #8 0x6069ff1948cd in std::allocator_traits<std::allocator<char16_t>>::deallocate(std::allocator<char16_t>&, char16_t*, unsigned long) /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../include/c++/14.2.1/bits/alloc_traits.h:513:13
    #9 0x6069ff1948cd in std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t>>::_M_destroy(unsigned long) /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../include/c++/14.2.1/bits/basic_string.h:294:9
    #10 0x6069ff1947ed in std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t>>::_M_dispose() /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../include/c++/14.2.1/bits/basic_string.h:288:4
    #11 0x6069ff1922d8 in std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t>>::~basic_string() /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../include/c++/14.2.1/bits/basic_string.h:809:9
    #12 0x6069ff1886d6 in webpp::tests::unicode_fuzz(std::basic_string_view<char, std::char_traits<char>>) /webpp/tests/./unicode_fuzz.hpp:56:5
    #13 0x6069ff1aa7a5 in void std::__invoke_impl<void, void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&>(std::__invoke_other, void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&) /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../include/c++/14.2.1/bits/invoke.h:61:14
    #14 0x6069ff1aa75c in std::__invoke_result<void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&>::type std::__invoke<void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&>(void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&) /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../include/c++/14.2.1/bits/invoke.h:96:14
    #15 0x6069ff1aa72c in std::invoke_result<void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&>::type std::invoke<void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&>(void (&)(std::basic_string_view<char, std::char_traits<char>>), std::basic_string_view<char, std::char_traits<char>> const&) /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../include/c++/14.2.1/functional:120:14
    #16 0x6069ff191cc2 in void fuzz_passer<void (&)(std::basic_string_view<char, std::char_traits<char>>)>(void (&)(std::basic_string_view<char, std::char_traits<char>>), unsigned char const*, unsigned long) /webpp/tests/common/fuzz_common.hpp:19:27
    #17 0x6069ff1871d3 in LLVMFuzzerTestOneInput /webpp/tests/unicode_fuzz.cpp:6:1
    #18 0x6069ff02b253 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/webpp/build-dev-clang/fuzz-unicode+0x68253) (BuildId: 6e63114e5a7ed3034b19c8857e237254fba1d2a4)
    #19 0x6069ff02bf00 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) (/webpp/build-dev-clang/fuzz-unicode+0x68f00) (BuildId: 6e63114e5a7ed3034b19c8857e237254fba1d2a4)
    #20 0x6069ff02cf78 in fuzzer::Fuzzer::MutateAndTestOne() (/webpp/build-dev-clang/fuzz-unicode+0x69f78) (BuildId: 6e63114e5a7ed3034b19c8857e237254fba1d2a4)
    #21 0x6069ff02e267 in fuzzer::Fuzzer::Loop(std::vector<fuzzer::SizedFile, std::allocator<fuzzer::SizedFile>>&) (/webpp/build-dev-clang/fuzz-unicode+0x6b267) (BuildId: 6e63114e5a7ed3034b19c8857e237254fba1d2a4)
    #22 0x6069ff010394 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/webpp/build-dev-clang/fuzz-unicode+0x4d394) (BuildId: 6e63114e5a7ed3034b19c8857e237254fba1d2a4)
    #23 0x6069feff9bc7 in main (/webpp/build-dev-clang/fuzz-unicode+0x36bc7) (BuildId: 6e63114e5a7ed3034b19c8857e237254fba1d2a4)
    #24 0x717f8f57ce07 in __libc_start_call_main /usr/src/debug/glibc/glibc/csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #25 0x717f8f57cecb in __libc_start_main /usr/src/debug/glibc/glibc/csu/../csu/libc-start.c:360:3
    #26 0x6069feff9c04 in _start (/webpp/build-dev-clang/fuzz-unicode+0x36c04) (BuildId: 6e63114e5a7ed3034b19c8857e237254fba1d2a4)

MS: 2 InsertRepeatedBytes-EraseBytes-; base unit: adc83b19e793491b1c6ea0fd8b46cd9f32e592fc
0xa,0xae,0xae,0xae,
\012\256\256\256
artifact_prefix='./'; Test unit written to ./crash-f6cee298d4f4bc3a327b450b2d13f092b63d09cb
Base64: Cq6urg==

the-moisrex added Research Security Security Bug Warning A warnings / not errors labels Nov 13, 2024

the-moisrex added this to Web++ Unicode Nov 13, 2024

the-moisrex added this to Web++ Security Dec 18, 2024

the-moisrex added a commit that referenced this issue Dec 18, 2024

Adding fuzzer for Normalization; #576

f9fc84f

the-moisrex self-assigned this Dec 18, 2024

the-moisrex added a commit that referenced this issue Dec 19, 2024

Fixing parsing of negative code points; #576

fb80518

the-moisrex added a commit that referenced this issue Dec 20, 2024

Adding test; #576

07943a5

the-moisrex added a commit that referenced this issue Dec 20, 2024

Checked::next_code_point; #576

f9806d7

the-moisrex added a commit that referenced this issue Dec 21, 2024

Fixing Checked::next_code_point; #576

0695d6a

the-moisrex added a commit that referenced this issue Dec 21, 2024

Benchmarking next_code_point; #576

9c4a925

the-moisrex added a commit that referenced this issue Dec 21, 2024

Updating Benchmark; #576

eb2e31f

the-moisrex added a commit that referenced this issue Dec 22, 2024

Fixing utf-16; #576

cae43fa

the-moisrex added a commit that referenced this issue Dec 23, 2024

Add new breaking tests; #576

06b9ad9

the-moisrex added a commit that referenced this issue Dec 23, 2024

Fixing canonical reorder algo; #576

590a444

the-moisrex added a commit that referenced this issue Dec 23, 2024

Fixing next_char; #576

9d0bd6d

the-moisrex added a commit that referenced this issue Dec 24, 2024

Unifying fuzz tests; #576

31e713a

the-moisrex added a commit that referenced this issue Dec 24, 2024

Fix next_code_point<char16_t>; #576

03e0297

the-moisrex added a commit that referenced this issue Dec 25, 2024

Making canonical_decompose safe; #576

a064df7

the-moisrex added a commit that referenced this issue Dec 25, 2024

String may contain zeros; #576

f8edf41

the-moisrex added a commit that referenced this issue Dec 26, 2024

Chains + new fuzz tests; #576

66d7ca6

the-moisrex added a commit that referenced this issue Dec 27, 2024

Fixing char16_t next code point; #576

d5ae420

the-moisrex moved this to In Progress in Web++ Unicode Dec 27, 2024

the-moisrex moved this to In Progress in Web++ Security Dec 27, 2024

the-moisrex added a commit that referenced this issue Dec 27, 2024

Making next_char safe; #576

627b094

the-moisrex added a commit that referenced this issue Dec 27, 2024

Removing const-pins; #576

7c2d9f1

the-moisrex added a commit that referenced this issue Dec 28, 2024

Implement safe checked::prev_code_point; #584 #576

4533b91

the-moisrex added a commit that referenced this issue Dec 30, 2024

Moving holes conditionally; #576

c3950ce

the-moisrex added a commit that referenced this issue Dec 30, 2024

Adding tests; #576

63138c4

the-moisrex added a commit that referenced this issue Dec 30, 2024

Securing canonical_composed; #576

9d3c324

the-moisrex added a commit that referenced this issue Dec 30, 2024

Making the hole bigger; #576

d77214a

the-moisrex closed this as completed Dec 30, 2024

github-project-automation bot moved this from In Progress to Done in Web++ Security Dec 30, 2024

github-project-automation bot moved this from In Progress to Done in Web++ Unicode Dec 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unicode: write fuzzing tests for normalization #576

Unicode: write fuzzing tests for normalization #576

the-moisrex commented Nov 13, 2024 •

edited

Loading

the-moisrex commented Dec 27, 2024

the-moisrex commented Dec 27, 2024

the-moisrex commented Dec 27, 2024

the-moisrex commented Dec 27, 2024

the-moisrex commented Dec 30, 2024

Unicode: write fuzzing tests for normalization #576

Unicode: write fuzzing tests for normalization #576

Comments

the-moisrex commented Nov 13, 2024 • edited Loading

the-moisrex commented Dec 27, 2024

the-moisrex commented Dec 27, 2024

the-moisrex commented Dec 27, 2024

the-moisrex commented Dec 27, 2024

the-moisrex commented Dec 30, 2024

the-moisrex commented Nov 13, 2024 •

edited

Loading