Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

proton raw connection event leak #469

Closed
jiridanek opened this issue May 14, 2022 · 9 comments
Closed

proton raw connection event leak #469

jiridanek opened this issue May 14, 2022 · 9 comments
Assignees
@astitcher

Comments

@jiridanek
Copy link
Contributor

https://github.com/jiridanek/skupper-router/runs/6430169530?check_suite_focus=true#step:26:9200

I will have to check if maybe I broke this (the CI link goes to my PR), but I think that is unlikely and this is caused by Proton change.

E           ==4413==ERROR: LeakSanitizer: detected memory leaks
E           
E           Direct leak of 40 byte(s) in 1 object(s) allocated from:
E               #0 0x7f511b131ad7 in calloc (/lib64/libasan.so.6+0xaead7)
E               #1 0x7f511aba802d in pni_mem_zallocate /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/memory.c:271
E               #2 0x7f511aba802d in pni_default_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/object/object.c:74
E               #3 0x7f511aba802d in pni_class_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/object/object.c:116
E               #4 0x7f511aba802d in pn_class_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/object/object.c:155
E               #5 0x7f511aba81b9 in pn_handler_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/reactor/handler.c:60
E               #6 0x4351fd in main /home/runner/work/skupper-router/skupper-router/skupper-router/tests/test-sender.c:420
E               #7 0x7f5119333f1f in __libc_start_call_main (/lib64/libc.so.6+0x40f1f)
E           
E           Indirect leak of 40 byte(s) in 1 object(s) allocated from:
E               #0 0x7f511b131ad7 in calloc (/lib64/libasan.so.6+0xaead7)
E               #1 0x7f511aba802d in pni_mem_zallocate /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/memory.c:271
E               #2 0x7f511aba802d in pni_default_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/object/object.c:74
E               #3 0x7f511aba802d in pni_class_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/object/object.c:116
E               #4 0x7f511aba802d in pn_class_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/object/object.c:155
E               #5 0x7f511aba81b9 in pn_handler_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/reactor/handler.c:60
E               #6 0x435205 in main /home/runner/work/skupper-router/skupper-router/skupper-router/tests/test-sender.c:421
E               #7 0x7f5119333f1f in __libc_start_call_main (/lib64/libc.so.6+0x40f1f)
E           
E           -----------------------------------------------------
E           Suppressions used:
E             count      bytes template
E                 2        176 ^pn_list$
E           -----------------------------------------------------
E           
E           SUMMARY: AddressSanitizer: 80 byte(s) leaked in 2 allocation(s).
@jiridanek
Copy link
Contributor Author

jiridanek commented May 14, 2022 •
edited
Loading

@astitcher Happened again on main, so probably caused by Proton change.

https://github.com/skupperproject/skupper-router/runs/6431188026?check_suite_focus=true#step:26:8516

The suppressions for Proton types in the router currently are

skupper-router/tests/lsan.supp

Lines 21 to 27 in e43f50f

# to be triaged; system_tests_http1_adaptor
leak:^pn_condition$
leak:^pn_raw_connection$
leak:^pgetaddrinfo$
# to be triaged; system_tests_link_routes
leak:^pni_init_default_logger$

skupper-router/tests/lsan.supp

Lines 78 to 89 in e43f50f

### Qpid Proton
# Proton suppressions taken from Proton's lsan.supp
# this appears in system_tests_open_properties:
leak:^pni_data_grow$
leak:^pn_buffer$
leak:^pn_buffer_ensure$
# this appears in system_tests_http1_adaptor:
leak:^pn_string_grow$
leak:^pn_object_new$
leak:^pn_list$
leak:^pni_record_create$

@astitcher
Copy link
Contributor

@jiridanek Hmm, that does look lik e it's caused by that change but I can't see anything different about this use compared to others that do work.

@kgiusti kgiusti added this to the 2.1.0 milestone May 17, 2022
@kgiusti
Copy link
Contributor

kgiusti commented May 17, 2022

@jiridanek @astitcher if this is likely a proton bug is there a proton JIRA we can link to?

@kgiusti
Copy link
Contributor

kgiusti commented May 18, 2022

This backtrace is coming from the router - it is not a reactor-based traceback:

https://github.com/skupperproject/skupper-router/runs/6489086985?check_suite_focus=true#step:26:1757

55: =================================================================
55: ==3274==ERROR: LeakSanitizer: detected memory leaks
55:
55: Indirect leak of 64 byte(s) in 1 object(s) allocated from:
55: #0 0x7f4cd7202ad7 in calloc (/lib64/libasan.so.6+0xaead7)
55: #1 0x7f4cd6c4e04d in pni_mem_zallocate /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/memory.c:271
55: #2 0x7f4cd6c4e04d in pni_default_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/object/object.c:74
55: #3 0x7f4cd6c4e04d in pni_class_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/object/object.c:116
55: #4 0x7f4cd6c4e04d in pn_class_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/object/object.c:155
55: #5 0x7f4cd6cadb98 in pn_event /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/event.c:287
55: #6 0x7f4cd6cadb98 in pn_collector_put /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/event.c:156
55: #7 0x7f4cd709ea6b in pni_raw_put_event /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/proactor/raw_connection.c:420
55: #8 0x7f4cd709ea6b in pni_raw_connected /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/proactor/raw_connection.c:468
55: #9 0x7f4cd709ea6b in praw_connection_connected_lh /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/proactor/epoll_raw_connection.c:82
55: #10 0x7f4cd70bae4c in pni_raw_connection_process /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/proactor/epoll_raw_connection.c:376
55: #11 0x7f4cd70bae4c in process /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/proactor/epoll.c:2244
55: #12 0x7f4cd70bae4c in next_event_batch /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/proactor/epoll.c:2449
55: #13 0x683080 in thread_run /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:1080
55: #14 0x7f4cd5bc08c9 in start_thread (/lib64/libc.so.6+0xa08c9)
55:
55: Indirect leak of 64 byte(s) in 1 object(s) allocated from:
55: #0 0x7f4cd7202ad7 in calloc (/lib64/libasan.so.6+0xaead7)
55: #1 0x7f4cd6c4e04d in pni_mem_zallocate /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/memory.c:271
55: #2 0x7f4cd6c4e04d in pni_default_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/object/object.c:74
55: #3 0x7f4cd6c4e04d in pni_class_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/object/object.c:116
55: #4 0x7f4cd6c4e04d in pn_class_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/object/object.c:155
55: #5 0x7f4cd6cadb98 in pn_event /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/event.c:287
55: #6 0x7f4cd6cadb98 in pn_collector_put /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/event.c:156
55: #7 0x7f4cd709ea6b in pni_raw_put_event /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/proactor/raw_connection.c:420
55: #8 0x7f4cd709ea6b in pni_raw_connected /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/proactor/raw_connection.c:468
55: #9 0x7f4cd709ea6b in praw_connection_connected_lh /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/proactor/epoll_raw_connection.c:82
55: #10 0x7f4cd70cb237 in pn_listener_raw_accept /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/proactor/epoll_raw_connection.c:240
55: #11 0x6be3e9 in qdr_http_connection_ingress /home/runner/work/skupper-router/skupper-router/skupper-router/src/adaptors/http2/http2_adaptor.c:1801
55: #12 0x67bb64 in handle_event_with_context /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:780
55: #13 0x67bb64 in do_handle_raw_connection_event /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:786
55: #14 0x67bb64 in handle /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:1063
55: #15 0x683235 in thread_run /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:1095
55: #16 0x7f4cd5bc08c9 in start_thread (/lib64/libc.so.6+0xa08c9)
55:
55: Indirect leak of 40 byte(s) in 1 object(s) allocated from:
55: #0 0x7f4cd7202ad7 in calloc (/lib64/libasan.so.6+0xaead7)
55: #1 0x7f4cd6c4e04d in pni_mem_zallocate /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/memory.c:271
55: #2 0x7f4cd6c4e04d in pni_default_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/object/object.c:74
55: #3 0x7f4cd6c4e04d in pni_class_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/object/object.c:116
55: #4 0x7f4cd6c4e04d in pn_class_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/object/object.c:155
55: #5 0x7f4cd6c5c78a in pn_record /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/object/record.c:66
55: #6 0x7f4cd6ca32e6 in pn_event_initialize /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/event.c:235
55: #7 0x7f4cd6c4e013 in pn_class_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/object/object.c:157
55: #8 0x7f4cd6cadb98 in pn_event /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/event.c:287
55: #9 0x7f4cd6cadb98 in pn_collector_put /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/event.c:156
55: #10 0x7f4cd709ea6b in pni_raw_put_event /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/proactor/raw_connection.c:420
55: #11 0x7f4cd709ea6b in pni_raw_connected /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/proactor/raw_connection.c:468
55: #12 0x7f4cd709ea6b in praw_connection_connected_lh /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/proactor/epoll_raw_connection.c:82
55: #13 0x7f4cd70cb237 in pn_listener_raw_accept /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/proactor/epoll_raw_connection.c:240
55: #14 0x6be3e9 in qdr_http_connection_ingress /home/runner/work/skupper-router/skupper-router/skupper-router/src/adaptors/http2/http2_adaptor.c:1801
55: #15 0x67bb64 in handle_event_with_context /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:780
55: #16 0x67bb64 in do_handle_raw_connection_event /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:786
55: #17 0x67bb64 in handle /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:1063
55: #18 0x683235 in thread_run /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:1095
55: #19 0x7f4cd5bc08c9 in start_thread (/lib64/libc.so.6+0xa08c9)
55:
55: Indirect leak of 40 byte(s) in 1 object(s) allocated from:
55: #0 0x7f4cd7202ad7 in calloc (/lib64/libasan.so.6+0xaead7)
55: #1 0x7f4cd6c4e04d in pni_mem_zallocate /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/memory.c:271
55: #2 0x7f4cd6c4e04d in pni_default_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/object/object.c:74
55: #3 0x7f4cd6c4e04d in pni_class_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/object/object.c:116
55: #4 0x7f4cd6c4e04d in pn_class_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/object/object.c:155
55: #5 0x7f4cd6c5c78a in pn_record /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/object/record.c:66
55: #6 0x7f4cd6ca32e6 in pn_event_initialize /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/event.c:235
55: #7 0x7f4cd6c4e013 in pn_class_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/object/object.c:157
55: #8 0x7f4cd6cadb98 in pn_event /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/event.c:287
55: #9 0x7f4cd6cadb98 in pn_collector_put /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/event.c:156
55: #10 0x7f4cd709ea6b in pni_raw_put_event /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/proactor/raw_connection.c:420
55: #11 0x7f4cd709ea6b in pni_raw_connected /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/proactor/raw_connection.c:468
55: #12 0x7f4cd709ea6b in praw_connection_connected_lh /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/proactor/epoll_raw_connection.c:82
55: #13 0x7f4cd70bae4c in pni_raw_connection_process /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/proactor/epoll_raw_connection.c:376
55: #14 0x7f4cd70bae4c in process /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/proactor/epoll.c:2244
55: #15 0x7f4cd70bae4c in next_event_batch /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/proactor/epoll.c:2449
55: #16 0x683080 in thread_run /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:1080
55: #17 0x7f4cd5bc08c9 in start_thread (/lib64/libc.so.6+0xa08c9)

@kgiusti
Copy link
Contributor

kgiusti commented May 18, 2022

Proton JIRA: https://issues.apache.org/jira/browse/PROTON-2546

@kgiusti kgiusti changed the title leak from pni_class_new with proton main proton raw connection event leak May 18, 2022
@kgiusti
Copy link
Contributor

kgiusti commented May 18, 2022

https://github.com/jiridanek/skupper-router/runs/6430169530?check_suite_focus=true#step:26:9200

I will have to check if maybe I broke this (the CI link goes to my PR), but I think that is unlikely and this is caused by Proton change.

E           ==4413==ERROR: LeakSanitizer: detected memory leaks
E           
E           Direct leak of 40 byte(s) in 1 object(s) allocated from:
E               #0 0x7f511b131ad7 in calloc (/lib64/libasan.so.6+0xaead7)
E               #1 0x7f511aba802d in pni_mem_zallocate /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/memory.c:271
E               #2 0x7f511aba802d in pni_default_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/object/object.c:74
E               #3 0x7f511aba802d in pni_class_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/object/object.c:116
E               #4 0x7f511aba802d in pn_class_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/object/object.c:155
E               #5 0x7f511aba81b9 in pn_handler_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/reactor/handler.c:60
E               #6 0x4351fd in main /home/runner/work/skupper-router/skupper-router/skupper-router/tests/test-sender.c:420
E               #7 0x7f5119333f1f in __libc_start_call_main (/lib64/libc.so.6+0x40f1f)
E           
E           Indirect leak of 40 byte(s) in 1 object(s) allocated from:
E               #0 0x7f511b131ad7 in calloc (/lib64/libasan.so.6+0xaead7)
E               #1 0x7f511aba802d in pni_mem_zallocate /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/memory.c:271
E               #2 0x7f511aba802d in pni_default_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/object/object.c:74
E               #3 0x7f511aba802d in pni_class_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/object/object.c:116
E               #4 0x7f511aba802d in pn_class_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/core/object/object.c:155
E               #5 0x7f511aba81b9 in pn_handler_new /home/runner/work/skupper-router/skupper-router/qpid-proton/c/src/reactor/handler.c:60
E               #6 0x435205 in main /home/runner/work/skupper-router/skupper-router/skupper-router/tests/test-sender.c:421
E               #7 0x7f5119333f1f in __libc_start_call_main (/lib64/libc.so.6+0x40f1f)
E           
E           -----------------------------------------------------
E           Suppressions used:
E             count      bytes template
E                 2        176 ^pn_list$
E           -----------------------------------------------------
E           
E           SUMMARY: AddressSanitizer: 80 byte(s) leaked in 2 allocation(s).

Actually, this leak is coming from the reactor code, which is deprecated but still used by a few test clients. The other leak (see #469 (comment)) is actually the one for concern.

I'll open a separate bug to track the reactor deprecation issue.

@kgiusti kgiusti mentioned this issue May 18, 2022
Closed
@kgiusti
Copy link
Contributor

kgiusti commented May 18, 2022

FYI #482 will track removal of the leaky reactor stuff

@jiridanek
Copy link
Contributor Author

This should be fixed by apache/qpid-proton@e2fa36f

@ganeshmurthy ganeshmurthy modified the milestones: 2.1.0, 2.2.0 Sep 29, 2022
@ganeshmurthy ganeshmurthy modified the milestones: 2.2.0, 2.3.0 Nov 11, 2022
@ganeshmurthy ganeshmurthy modified the milestones: 2.3.0, 2.4.0 Feb 6, 2023
@ganeshmurthy ganeshmurthy modified the milestones: 2.4.0, 2.5.0 Apr 12, 2023
@ganeshmurthy
Copy link
Contributor

This should be fixed by apache/qpid-proton@e2fa36f

Closing this issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@astitcher astitcher

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@astitcher @kgiusti @jiridanek @ganeshmurthy