Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-enable ASAN when building with clang #4013

Merged
merged 2 commits into from
Sep 24, 2020

Conversation

darosior
Copy link
Collaborator

@darosior darosior commented Sep 7, 2020

I'm looking at including some fuzz targets, and will probably use clang along with its address sanitizer (and hopefully many others!).

This removes the restriction on only using gcc with ASAN, by using suppressions instead when using clang (see #2285 for the reason).
This also enables ASAN on non-already-under-valgrind Travis jobs.

@darosior
Copy link
Collaborator Author

darosior commented Sep 7, 2020

Detected a use-after-free :-)

See:

==20190==ERROR: AddressSanitizer: heap-use-after-free on address 0x60b0000f4e78 at pc 0x00000043adc3 bp 0x7ffda57c70c0 sp 0x7ffda57c6848
READ of size 33 at 0x60b0000f4e78 thread T0
    #0 0x43adc2 in memcmp (/home/darosior/projects/lightning/lightningd/lightningd+0x43adc2)
    #1 0x52e953 in node_id_eq /home/darosior/projects/lightning/./common/node_id.h:14:9
    #2 0x52e8d2 in peer_by_id /home/darosior/projects/lightning/lightningd/peer_control.c:153:7
    #3 0x50134c in process_check_funding_broadcast /home/darosior/projects/lightning/lightningd/channel_control.c:748:9
    #4 0x4f8133 in getutxout_callback /home/darosior/projects/lightning/lightningd/bitcoind.c:551:3
    #5 0x541bcc in plugin_response_handle /home/darosior/projects/lightning/lightningd/plugin.c:366:2
    #6 0x5418c0 in plugin_read_json_one /home/darosior/projects/lightning/lightningd/plugin.c:472:9
    #7 0x53f247 in plugin_read_json /home/darosior/projects/lightning/lightningd/plugin.c:517:8
    #8 0x5c1480 in next_plan /home/darosior/projects/lightning/ccan/ccan/io/io.c:59:9
    #9 0x5c21c8 in do_plan /home/darosior/projects/lightning/ccan/ccan/io/io.c:407:10
    #10 0x5c2033 in io_ready /home/darosior/projects/lightning/ccan/ccan/io/io.c:417:8
    #11 0x5c414a in io_loop /home/darosior/projects/lightning/ccan/ccan/io/poll.c:445:5
    #12 0x51153e in io_loop_with_timers /home/darosior/projects/lightning/lightningd/io_loop_with_timers.c:24:12
    #13 0x516332 in main /home/darosior/projects/lightning/lightningd/lightningd.c:1018:22
    #14 0x7f3faff4bbba in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26bba)
    #15 0x41ffd9 in _start (/home/darosior/projects/lightning/lightningd/lightningd+0x41ffd9)

0x60b0000f4e78 is located 40 bytes inside of 105-byte region [0x60b0000f4e50,0x60b0000f4eb9)
freed by thread T0 here:
    #0 0x4c7aa2 in free (/home/darosior/projects/lightning/lightningd/lightningd+0x4c7aa2)
    #1 0x5d40d3 in del_tree /home/darosior/projects/lightning/ccan/ccan/tal/tal.c:421:9
    #2 0x5d4052 in del_tree /home/darosior/projects/lightning/ccan/ccan/tal/tal.c:412:4
    #3 0x5d3e39 in tal_free /home/darosior/projects/lightning/ccan/ccan/tal/tal.c:486:3
    #4 0x511ba2 in command_raw_complete /home/darosior/projects/lightning/lightningd/jsonrpc.c:455:2
    #5 0x511c65 in command_success /home/darosior/projects/lightning/lightningd/jsonrpc.c:467:9
    #6 0x4ffbc1 in forget /home/darosior/projects/lightning/lightningd/channel_control.c:305:15
    #7 0x501b92 in handle_error_channel /home/darosior/projects/lightning/lightningd/channel_control.c:320:2
    #8 0x500910 in channel_msg /home/darosior/projects/lightning/lightningd/channel_control.c:373:3
    #9 0x5477a2 in sd_msg_read /home/darosior/projects/lightning/lightningd/subd.c:480:6
    #10 0x5c1480 in next_plan /home/darosior/projects/lightning/ccan/ccan/io/io.c:59:9
    #11 0x5c21c8 in do_plan /home/darosior/projects/lightning/ccan/ccan/io/io.c:407:10
    #12 0x5c2033 in io_ready /home/darosior/projects/lightning/ccan/ccan/io/io.c:417:8
    #13 0x5c414a in io_loop /home/darosior/projects/lightning/ccan/ccan/io/poll.c:445:5
    #14 0x51153e in io_loop_with_timers /home/darosior/projects/lightning/lightningd/io_loop_with_timers.c:24:12
    #15 0x516332 in main /home/darosior/projects/lightning/lightningd/lightningd.c:1018:22
    #16 0x7f3faff4bbba in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26bba)

previously allocated by thread T0 here:
    #0 0x4c7e23 in __interceptor_malloc (/home/darosior/projects/lightning/lightningd/lightningd+0x4c7e23)
    #1 0x5d39f4 in allocate /home/darosior/projects/lightning/ccan/ccan/tal/tal.c:250:14
    #2 0x5d387f in tal_alloc_ /home/darosior/projects/lightning/ccan/ccan/tal/tal.c:428:17
    #3 0x501014 in cancel_channel_before_broadcast /home/darosior/projects/lightning/lightningd/channel_control.c:776:33
    #4 0x522170 in cancel_after_fundchannel_complete_success /home/darosior/projects/lightning/lightningd/opening_control.c:328:14
    #5 0x5220aa in funding_success /home/darosior/projects/lightning/lightningd/opening_control.c:339:3
    #6 0x520e65 in opening_funder_finished /home/darosior/projects/lightning/lightningd/opening_control.c:490:2
    #7 0x520249 in openingd_msg /home/darosior/projects/lightning/lightningd/opening_control.c:937:3
    #8 0x5477a2 in sd_msg_read /home/darosior/projects/lightning/lightningd/subd.c:480:6
    #9 0x548224 in read_fds /home/darosior/projects/lightning/lightningd/subd.c:308:10
    #10 0x5c1480 in next_plan /home/darosior/projects/lightning/ccan/ccan/io/io.c:59:9
    #11 0x5c21c8 in do_plan /home/darosior/projects/lightning/ccan/ccan/io/io.c:407:10
    #12 0x5c2033 in io_ready /home/darosior/projects/lightning/ccan/ccan/io/io.c:417:8
    #13 0x5c414a in io_loop /home/darosior/projects/lightning/ccan/ccan/io/poll.c:445:5
    #14 0x51153e in io_loop_with_timers /home/darosior/projects/lightning/lightningd/io_loop_with_timers.c:24:12
    #15 0x516332 in main /home/darosior/projects/lightning/lightningd/lightningd.c:1018:22
    #16 0x7f3faff4bbba in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26bba)

@darosior
Copy link
Collaborator Author

darosior commented Sep 7, 2020

Special-cased it too as (it's a well-known race and) i don't see how we could prevent it.

@cdecker cdecker self-requested a review September 7, 2020 19:19
@rustyrussell
Copy link
Contributor

Woah, a well-known use-after free? That's not good!

What's happening here? Looks like the cc should be owned by NULL, so that freeing the cmd doesn't destroy it. (we'll still get a memleak if the bcli plugin crashes, but hard to care).

@darosior
Copy link
Collaborator Author

darosior commented Sep 8, 2020

Spent half an hour writing about how i already tried this and this doesn't work, but actually you are right. Pushed a fix and removed the suppression.

@darosior
Copy link
Collaborator Author

darosior commented Sep 8, 2020

(Also for the record by "well-known" i meant the functionality race, so we might have expected the data race: that was nonsensical)

@darosior
Copy link
Collaborator Author

darosior commented Sep 8, 2020

Travis really does not like the 100 nodes of test_funding_cancel_race under ASAN..
For the gcc + ASAN builds i can't spot the issue but it's likely some kind of oom too.

@rustyrussell
Copy link
Contributor

Hmm, if ASAN isn't going in, can you open a PR for the others? Be nice to have them in 0.9.1...

@darosior
Copy link
Collaborator Author

darosior commented Sep 9, 2020 via email

@darosior darosior mentioned this pull request Sep 9, 2020
@darosior darosior force-pushed the clang_asan branch 3 times, most recently from d137f36 to 3307d7f Compare September 10, 2020 12:17
@darosior
Copy link
Collaborator Author

Trying only on the clang job now (for some reason the gcc ones produce errors..)

@cdecker
Copy link
Member

cdecker commented Sep 19, 2020

Very nice, it'd be a major improvement to be able to run lightweight sanitizers in production (valgrind has ~20x performance impact).

However I wonder why the suppressions are located in tests/ I'd expect them to be in contrib/ or similar.

clang did the hard work here:

openingd/dualopend.c:958:42: error: result of comparison of constant 'WIRE_DUAL_OPEN_FAIL' (7003) with expression of type 'u8' (aka 'unsigned char') is always false
      [-Werror,-Wtautological-constant-out-of-range-compare]
        if ((msg_type = fromwire_peektype(msg)) == WIRE_DUAL_OPEN_FAIL) {
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^  ~~~~~~~~~~~~~~~~~~~

Co-Authored-by: the clang compiler
Signed-off-by: Antoine Poinsot <[email protected]>
@darosior
Copy link
Collaborator Author

darosior commented Sep 22, 2020

Rebased on #4076 and moved the sanitizers suppressions in contrib/
EDIT: hoping for the ASAN job to not oom

@rustyrussell
Copy link
Contributor

Oh, can we have a Changelog-Added: build: clang build now supports --enable-address-sanitizer.?

@rustyrussell
Copy link
Contributor

Ack 9910e5f

@darosior
Copy link
Collaborator Author

Sure, but i doubt that Travis is going to pass anyways :/ (going to wait for it then amend)

@darosior
Copy link
Collaborator Author

Travis still ooms on test_funding_cancel_race.. Introducing ASAN flags seems overkill, i'll just remove the last commit.

@darosior
Copy link
Collaborator Author

darosior commented Sep 22, 2020

Added the Changelog line.
Removed the last commit, was:

diff --git a/.travis.yml b/.travis.yml
index 6fdf5d1b5..3ac97a873 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -16,7 +16,7 @@ env:
   - ARCH=64 TEST_CMD="make check-source check-units installcheck" DEVELOPER=1 EXPERIMENTAL_FEATURES=1
 
   # All of the following will just run `make pytest`
-  - VALGRIND=0 ARCH=64 DEVELOPER=1 COMPILER=clang
+  - VALGRIND=0 ARCH=64 DEVELOPER=1 COMPILER=clang ASAN=1 LSAN_OPTIONS="suppressions=$PWD/contrib/sanitizer_suppressions/lsan"
   - VALGRIND=0 ARCH=64 DEVELOPER=1 COMPILER=gcc
   - VALGRIND=0 ARCH=64 DEVELOPER=0 COMPILER=gcc COMPAT=0 TEST_GROUP=1 TEST_GROUP_COUNT=2
   - VALGRIND=0 ARCH=64 DEVELOPER=0 COMPILER=gcc COMPAT=0 TEST_GROUP=2 TEST_GROUP_COUNT=2

Prefer adding LSAN_OPTIONS="suppressions=$PWD/tests/sanitizer_suppressions/lsan" when CC=clang instead.

Changelog-Added: build: clang build now supports --enable-address-sanitizer .
Signed-off-by: Antoine Poinsot <[email protected]>
Copy link
Contributor

@rustyrussell rustyrussell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack ad06763

@rustyrussell rustyrussell merged commit 3382daf into ElementsProject:master Sep 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants