Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

lua: src/zuuid.c:82: zuuid_new: Assertion `0' failed. #1407

Closed
grondo opened this issue Mar 31, 2018 · 7 comments
Closed

lua: src/zuuid.c:82: zuuid_new: Assertion `0' failed. #1407

grondo opened this issue Mar 31, 2018 · 7 comments

Comments

@grondo
Copy link
Contributor

grondo commented Mar 31, 2018

Related to #1406, when running 1600 task job, flux-wreckrun dies with

lua: src/zuuid.c:82: zuuid_new: Assertion `0' failed.

I'm assuming that is a malloc failure internal to czmq. I'm not sure why with only 1600 tasks we'd be running out of malloc space, but possibly a memory leak in the lua bindings.
@grondo
Copy link
Contributor Author

grondo commented Mar 31, 2018

Ran flux wreckrun -n240 under valgrind and got the following, to be investigated-- only the largest leak seems to scale with job size.

==79585== HEAP SUMMARY:
==79585==     in use at exit: 9,415,853 bytes in 60,210 blocks
==79585==   total heap usage: 2,192,269 allocs, 2,132,059 frees, 128,648,120 bytes allocated
==79585== 
==79585== 44 bytes in 44 blocks are definitely lost in loss record 649 of 1,210
==79585==    at 0x4C2B975: calloc (vg_replace_malloc.c:711)
==79585==    by 0x6404518: zio_json_decode (zio.c:995)
==79585==    by 0x6401A58: kz_get (kz.c:329)
==79585==    by 0x63DCD85: iowatcher_kz_ready_cb (flux-lua.c:1303)
==79585==    by 0x640161D: kvswatch_cb (kz.c:396)
==79585==    by 0x6641FB2: dispatch_watch (kvs_watch.c:197)
==79585==    by 0x66421E9: watch_response_cb (kvs_watch.c:230)
==79585==    by 0x6636513: call_handler (msg_handler.c:302)
==79585==    by 0x663668E: dispatch_message (msg_handler.c:326)
==79585==    by 0x663668E: handle_cb (msg_handler.c:392)
==79585==    by 0x665BCC4: ev_invoke_pending (ev.c:3314)
==79585==    by 0x665EE26: ev_run (ev.c:3717)
==79585==    by 0x66359B2: flux_reactor_run (reactor.c:128)
==79585==    by 0x63DB78C: l_flux_reactor_start (flux-lua.c:1981)
==79585==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E4CE56: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E4274C: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E41A6D: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E428D9: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E3E44C: lua_pcall (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x40174B: ??? (in /usr/bin/lua)
==79585==    by 0x40211D: ??? (in /usr/bin/lua)
==79585==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E42703: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E41A6D: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E428D9: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E3E4D6: lua_cpcall (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x401403: ??? (in /usr/bin/lua)
==79585==    by 0x5C22C04: (below main) (libc-start.c:274)
==79585== 
==79585== 81 (72 direct, 9 indirect) bytes in 1 blocks are definitely lost in loss record 828 of 1,210
==79585==    at 0x4C2B975: calloc (vg_replace_malloc.c:711)
==79585==    by 0x6405DC1: json_object_new (json_object.c:184)
==79585==    by 0x6406838: json_c_object_new_string_len (json_object.c:720)
==79585==    by 0x64072EF: json_c_tokener_parse_ex (json_tokener.c:484)
==79585==    by 0x6408CE6: json_c_tokener_parse_verbose (json_tokener.c:175)
==79585==    by 0x6408D3D: json_c_tokener_parse (json_tokener.c:163)
==79585==    by 0x63DD249: l_kvswatcher (flux-lua.c:1141)
==79585==    by 0x6642A56: flux_kvs_watch (kvs_watch.c:450)
==79585==    by 0x63DF6E4: l_kvswatcher_add (flux-lua.c:1192)
==79585==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E4CE56: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E4274C: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E41A6D: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E428D9: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E3E44C: lua_pcall (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x40174B: ??? (in /usr/bin/lua)
==79585==    by 0x40211D: ??? (in /usr/bin/lua)
==79585==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E42703: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E41A6D: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E428D9: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E3E4D6: lua_cpcall (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x401403: ??? (in /usr/bin/lua)
==79585==    by 0x5C22C04: (below main) (libc-start.c:274)
==79585== 
==79585== 97 bytes in 1 blocks are definitely lost in loss record 855 of 1,210
==79585==    at 0x4C29BE3: malloc (vg_replace_malloc.c:299)
==79585==    by 0x7399620: ??? (in /usr/lib64/libzmq.so.5.0.1)
==79585==    by 0x70EAEEC: zframe_new (in /usr/lib64/libczmq.so.3.0.0)
==79585==    by 0x70F5675: zmsg_addmem (in /usr/lib64/libczmq.so.3.0.0)
==79585==    by 0x663AFD9: flux_msg_copy (message.c:1341)
==79585==    by 0x663C9AE: response_cb (rpc.c:166)
==79585==    by 0x6636513: call_handler (msg_handler.c:302)
==79585==    by 0x663676E: dispatch_message (msg_handler.c:316)
==79585==    by 0x663676E: handle_cb (msg_handler.c:392)
==79585==    by 0x665BCC4: ev_invoke_pending (ev.c:3314)
==79585==    by 0x665EE26: ev_run (ev.c:3717)
==79585==    by 0x66359B2: flux_reactor_run (reactor.c:128)
==79585==    by 0x663F852: flux_future_wait_for (future.c:340)
==79585==    by 0x663F968: flux_future_get (future.c:357)
==79585==    by 0x663CB08: flux_rpc_get_vunpack (rpc.c:122)
==79585==    by 0x663CB08: flux_rpc_get_unpack (rpc.c:137)
==79585==    by 0x6642251: kvs_watch_rpc_get (kvs_watch.c:312)
==79585==    by 0x6642A11: flux_kvs_watch (kvs_watch.c:444)
==79585==    by 0x63DF6E4: l_kvswatcher_add (flux-lua.c:1192)
==79585==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E4CE56: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E4274C: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E41A6D: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E428D9: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E3E44C: lua_pcall (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x40174B: ??? (in /usr/bin/lua)
==79585==    by 0x40211D: ??? (in /usr/bin/lua)
==79585==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E42703: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E41A6D: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E428D9: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E3E4D6: lua_cpcall (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x401403: ??? (in /usr/bin/lua)
==79585==    by 0x5C22C04: (below main) (libc-start.c:274)
==79585== 
==79585== 97 bytes in 1 blocks are definitely lost in loss record 856 of 1,210
==79585==    at 0x4C29BE3: malloc (vg_replace_malloc.c:299)
==79585==    by 0x7399620: ??? (in /usr/lib64/libzmq.so.5.0.1)
==79585==    by 0x70EAEEC: zframe_new (in /usr/lib64/libczmq.so.3.0.0)
==79585==    by 0x70F5675: zmsg_addmem (in /usr/lib64/libczmq.so.3.0.0)
==79585==    by 0x663AFD9: flux_msg_copy (message.c:1341)
==79585==    by 0x6634B72: flux_requeue (handle.c:730)
==79585==    by 0x6636911: dispatch_requeue.isra.5 (msg_handler.c:98)
==79585==    by 0x66371F8: flux_dispatch_requeue (msg_handler.c:629)
==79585==    by 0x663F80C: flux_future_wait_for (future.c:344)
==79585==    by 0x663F968: flux_future_get (future.c:357)
==79585==    by 0x663CB08: flux_rpc_get_vunpack (rpc.c:122)
==79585==    by 0x663CB08: flux_rpc_get_unpack (rpc.c:137)
==79585==    by 0x6640581: decode_treeobj (kvs_lookup.c:183)
==79585==    by 0x6640AC3: flux_kvs_lookup_get (kvs_lookup.c:202)
==79585==    by 0x64014C1: getnext (kz.c:261)
==79585==    by 0x6401A84: kz_get (kz.c:324)
==79585==    by 0x63DCD85: iowatcher_kz_ready_cb (flux-lua.c:1303)
==79585==    by 0x640161D: kvswatch_cb (kz.c:396)
==79585==    by 0x6641FB2: dispatch_watch (kvs_watch.c:197)
==79585==    by 0x66421E9: watch_response_cb (kvs_watch.c:230)
==79585==    by 0x6636513: call_handler (msg_handler.c:302)
==79585==    by 0x663668E: dispatch_message (msg_handler.c:326)
==79585==    by 0x663668E: handle_cb (msg_handler.c:392)
==79585==    by 0x665BCC4: ev_invoke_pending (ev.c:3314)
==79585==    by 0x665EE26: ev_run (ev.c:3717)
==79585==    by 0x66359B2: flux_reactor_run (reactor.c:128)
==79585==    by 0x63DB78C: l_flux_reactor_start (flux-lua.c:1981)
==79585==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E4CE56: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E4274C: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E41A6D: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E428D9: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E3E44C: lua_pcall (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x40174B: ??? (in /usr/bin/lua)
==79585==    by 0x40211D: ??? (in /usr/bin/lua)
==79585==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E42703: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E41A6D: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E428D9: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E3E4D6: lua_cpcall (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x401403: ??? (in /usr/bin/lua)
==79585==    by 0x5C22C04: (below main) (libc-start.c:274)
==79585== 
==79585== 161 (144 direct, 17 indirect) bytes in 2 blocks are definitely lost in loss record 938 of 1,210
==79585==    at 0x4C2B975: calloc (vg_replace_malloc.c:711)
==79585==    by 0x6405DC1: json_object_new (json_object.c:184)
==79585==    by 0x6406838: json_c_object_new_string_len (json_object.c:720)
==79585==    by 0x64072EF: json_c_tokener_parse_ex (json_tokener.c:484)
==79585==    by 0x6408CE6: json_c_tokener_parse_verbose (json_tokener.c:175)
==79585==    by 0x6408D3D: json_c_tokener_parse (json_tokener.c:163)
==79585==    by 0x63DD249: l_kvswatcher (flux-lua.c:1141)
==79585==    by 0x66421E9: watch_response_cb (kvs_watch.c:230)
==79585==    by 0x6636513: call_handler (msg_handler.c:302)
==79585==    by 0x663668E: dispatch_message (msg_handler.c:326)
==79585==    by 0x663668E: handle_cb (msg_handler.c:392)
==79585==    by 0x665BCC4: ev_invoke_pending (ev.c:3314)
==79585==    by 0x665EE26: ev_run (ev.c:3717)
==79585==    by 0x66359B2: flux_reactor_run (reactor.c:128)
==79585==    by 0x63DB78C: l_flux_reactor_start (flux-lua.c:1981)
==79585==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E4CE56: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E4274C: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E41A6D: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E428D9: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E3E44C: lua_pcall (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x40174B: ??? (in /usr/bin/lua)
==79585==    by 0x40211D: ??? (in /usr/bin/lua)
==79585==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E42703: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E41A6D: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E428D9: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E3E4D6: lua_cpcall (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x401403: ??? (in /usr/bin/lua)
==79585==    by 0x5C22C04: (below main) (libc-start.c:274)
==79585== 
==79585== 436 bytes in 436 blocks are definitely lost in loss record 1,004 of 1,210
==79585==    at 0x4C2B975: calloc (vg_replace_malloc.c:711)
==79585==    by 0x6404518: zio_json_decode (zio.c:995)
==79585==    by 0x6401A58: kz_get (kz.c:329)
==79585==    by 0x63DCD85: iowatcher_kz_ready_cb (flux-lua.c:1303)
==79585==    by 0x640161D: kvswatch_cb (kz.c:396)
==79585==    by 0x6641FB2: dispatch_watch (kvs_watch.c:197)
==79585==    by 0x6642BBF: watch_dir (kvs_watch.c:474)
==79585==    by 0x6642BBF: flux_kvs_watch_dir (kvs_watch.c:501)
==79585==    by 0x6401C7F: kz_set_ready_cb (kz.c:409)
==79585==    by 0x63DF43A: l_iowatcher_add (flux-lua.c:1375)
==79585==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E4CE56: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E4274C: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E41A6D: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E428D9: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E3E44C: lua_pcall (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x40174B: ??? (in /usr/bin/lua)
==79585==    by 0x40211D: ??? (in /usr/bin/lua)
==79585==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E42703: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E41A6D: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E428D9: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E3E4D6: lua_cpcall (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x401403: ??? (in /usr/bin/lua)
==79585==    by 0x5C22C04: (below main) (libc-start.c:274)
==79585== 
==79585== 4,113 bytes in 1 blocks are possibly lost in loss record 1,104 of 1,210
==79585==    at 0x4C29BE3: malloc (vg_replace_malloc.c:299)
==79585==    by 0x63E2E98: cbuf_create (cbuf.c:242)
==79585==    by 0x64036A4: zio_init_buffer (zio.c:230)
==79585==    by 0x6403DEB: zio_allocate (zio.c:267)
==79585==    by 0x64042C1: zio_reader_create (zio.c:897)
==79585==    by 0x63DF344: l_iowatcher_add (flux-lua.c:1356)
==79585==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E4CE56: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E4274C: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E41A6D: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E428D9: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E3E44C: lua_pcall (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x40174B: ??? (in /usr/bin/lua)
==79585==    by 0x40211D: ??? (in /usr/bin/lua)
==79585==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E42703: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E41A6D: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E428D9: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E3E4D6: lua_cpcall (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x401403: ??? (in /usr/bin/lua)
==79585==    by 0x5C22C04: (below main) (libc-start.c:274)
==79585== 
==79585== 5,456 bytes in 22 blocks are definitely lost in loss record 1,112 of 1,210
==79585==    at 0x4C29BE3: malloc (vg_replace_malloc.c:299)
==79585==    by 0x7399620: ??? (in /usr/lib64/libzmq.so.5.0.1)
==79585==    by 0x70EAEEC: zframe_new (in /usr/lib64/libczmq.so.3.0.0)
==79585==    by 0x70F5675: zmsg_addmem (in /usr/lib64/libczmq.so.3.0.0)
==79585==    by 0x663AFD9: flux_msg_copy (message.c:1341)
==79585==    by 0x6634B72: flux_requeue (handle.c:730)
==79585==    by 0x6636911: dispatch_requeue.isra.5 (msg_handler.c:98)
==79585==    by 0x66371F8: flux_dispatch_requeue (msg_handler.c:629)
==79585==    by 0x663F80C: flux_future_wait_for (future.c:344)
==79585==    by 0x663F968: flux_future_get (future.c:357)
==79585==    by 0x663CB08: flux_rpc_get_vunpack (rpc.c:122)
==79585==    by 0x663CB08: flux_rpc_get_unpack (rpc.c:137)
==79585==    by 0x6640581: decode_treeobj (kvs_lookup.c:183)
==79585==    by 0x6640AC3: flux_kvs_lookup_get (kvs_lookup.c:202)
==79585==    by 0x66415F7: flux_kvs_get (kvs_classic.c:43)
==79585==    by 0x63E050A: l_kvsdir_index (kvs-lua.c:294)
==79585==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E42703: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E4B4A5: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E4CB98: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E4274C: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E41A6D: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E428D9: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E3E44C: lua_pcall (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x63DD123: kvswatch_cb_common.isra.4 (flux-lua.c:1122)
==79585==    by 0x66421E9: watch_response_cb (kvs_watch.c:230)
==79585==    by 0x6636513: call_handler (msg_handler.c:302)
==79585==    by 0x663668E: dispatch_message (msg_handler.c:326)
==79585==    by 0x663668E: handle_cb (msg_handler.c:392)
==79585==    by 0x665BCC4: ev_invoke_pending (ev.c:3314)
==79585==    by 0x665EE26: ev_run (ev.c:3717)
==79585==    by 0x66359B2: flux_reactor_run (reactor.c:128)
==79585==    by 0x63DB78C: l_flux_reactor_start (flux-lua.c:1981)
==79585==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E4CE56: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E4274C: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E41A6D: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E428D9: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E3E44C: lua_pcall (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x40174B: ??? (in /usr/bin/lua)
==79585==    by 0x40211D: ??? (in /usr/bin/lua)
==79585==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E42703: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E41A6D: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E428D9: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E3E4D6: lua_cpcall (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x401403: ??? (in /usr/bin/lua)
==79585==    by 0x5C22C04: (below main) (libc-start.c:274)
==79585== 
==79585== 88,430 bytes in 436 blocks are definitely lost in loss record 1,199 of 1,210
==79585==    at 0x4C29BE3: malloc (vg_replace_malloc.c:299)
==79585==    by 0x7399620: ??? (in /usr/lib64/libzmq.so.5.0.1)
==79585==    by 0x70EAEEC: zframe_new (in /usr/lib64/libczmq.so.3.0.0)
==79585==    by 0x70F5675: zmsg_addmem (in /usr/lib64/libczmq.so.3.0.0)
==79585==    by 0x663AFD9: flux_msg_copy (message.c:1341)
==79585==    by 0x663C9AE: response_cb (rpc.c:166)
==79585==    by 0x6636513: call_handler (msg_handler.c:302)
==79585==    by 0x663676E: dispatch_message (msg_handler.c:316)
==79585==    by 0x663676E: handle_cb (msg_handler.c:392)
==79585==    by 0x665BCC4: ev_invoke_pending (ev.c:3314)
==79585==    by 0x665EE26: ev_run (ev.c:3717)
==79585==    by 0x66359B2: flux_reactor_run (reactor.c:128)
==79585==    by 0x663F852: flux_future_wait_for (future.c:340)
==79585==    by 0x663F968: flux_future_get (future.c:357)
==79585==    by 0x663CB08: flux_rpc_get_vunpack (rpc.c:122)
==79585==    by 0x663CB08: flux_rpc_get_unpack (rpc.c:137)
==79585==    by 0x6642251: kvs_watch_rpc_get (kvs_watch.c:312)
==79585==    by 0x6642B75: watch_dir (kvs_watch.c:468)
==79585==    by 0x6642B75: flux_kvs_watch_dir (kvs_watch.c:501)
==79585==    by 0x6401C7F: kz_set_ready_cb (kz.c:409)
==79585==    by 0x63DF43A: l_iowatcher_add (flux-lua.c:1375)
==79585==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E4CE56: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E4274C: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E41A6D: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E428D9: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E3E44C: lua_pcall (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x40174B: ??? (in /usr/bin/lua)
==79585==    by 0x40211D: ??? (in /usr/bin/lua)
==79585==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E42703: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E41A6D: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E428D9: ??? (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x4E3E4D6: lua_cpcall (in /usr/lib64/liblua-5.1.so)
==79585==    by 0x401403: ??? (in /usr/bin/lua)
==79585==    by 0x5C22C04: (below main) (libc-start.c:274)
==79585== 
==79585== LEAK SUMMARY:
==79585==    definitely lost: 94,776 bytes in 943 blocks
==79585==    indirectly lost: 26 bytes in 3 blocks
==79585==      possibly lost: 4,113 bytes in 1 blocks
==79585==    still reachable: 9,316,938 bytes in 59,263 blocks
==79585==         suppressed: 0 bytes in 0 blocks

@grondo
Copy link
Contributor Author

grondo commented Mar 31, 2018 •
edited
Loading

Ok, found the big leak. (Edit: spoke too soon, that was not at all correct sorry! crash hidden by valgrind output)

@grondo
Copy link
Contributor Author

grondo commented Mar 31, 2018

Sorry, didn't really get anywhere with this. Maybe the kvs watcher futures aren't really a leak, or if they are leaked fixing won't even help the current problem so this might be low priority.

@grondo grondo mentioned this issue Apr 2, 2018
Merged
@grondo
Copy link
Contributor Author

grondo commented Apr 2, 2018

As noted in #1411, I found that raising the fd limit works around this issue to some unknown limit (so there may be some latent fd leak in the lua code)

@grondo
Copy link
Contributor Author

grondo commented Apr 2, 2018

Haven't investigated further, but in case it is informative, running valgrind with --track-fds=yes for a flux wreckrun -n4 hostname

==111064== 
==111064== FILE DESCRIPTORS: 27 open at exit.
==111064== Open file descriptor 26:
==111064==    at 0x5CF95D2: signalfd (signalfd.c:29)
==111064==    by 0x63DEB40: l_signal_handler_add (flux-lua.c:1861)
==111064==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x4E4CE56: ??? (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x4E4274C: ??? (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x4E41A6D: ??? (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x4E428D9: ??? (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x4E3E44C: lua_pcall (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x40174B: ??? (in /usr/bin/lua)
==111064==    by 0x40211D: ??? (in /usr/bin/lua)
==111064==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x4E42703: ??? (in /usr/lib64/liblua-5.1.so)
==111064== 
==111064== Open file descriptor 25:
==111064==    at 0x5CF95D2: signalfd (signalfd.c:29)
==111064==    by 0x63DEB40: l_signal_handler_add (flux-lua.c:1861)
==111064==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x4E4CE56: ??? (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x4E4274C: ??? (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x4E41A6D: ??? (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x4E428D9: ??? (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x4E3E44C: lua_pcall (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x40174B: ??? (in /usr/bin/lua)
==111064==    by 0x40211D: ??? (in /usr/bin/lua)
==111064==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x4E42703: ??? (in /usr/lib64/liblua-5.1.so)
==111064== 
==111064== Open file descriptor 24:
==111064==    at 0x5CF9897: epoll_create1 (syscall-template.S:81)
==111064==    by 0x66331F5: epoll_init (ev_epoll.c:243)
==111064==    by 0x66331F5: loop_init (ev.c:2923)
==111064==    by 0x663344A: ev_loop_new (ev.c:3102)
==111064==    by 0x663689D: flux_reactor_create (reactor.c:87)
==111064==    by 0x66408E5: now_context_create (future.c:101)
==111064==    by 0x66408E5: flux_future_wait_for (future.c:327)
==111064==    by 0x6640988: flux_future_get (future.c:357)
==111064==    by 0x663DB28: flux_rpc_get_vunpack (rpc.c:122)
==111064==    by 0x663DB28: flux_rpc_get_unpack (rpc.c:137)
==111064==    by 0x6643271: kvs_watch_rpc_get (kvs_watch.c:312)
==111064==    by 0x6643B95: watch_dir (kvs_watch.c:468)
==111064==    by 0x6643B95: flux_kvs_watch_dir (kvs_watch.c:501)
==111064==    by 0x6401D5E: kz_set_ready_cb (kz.c:455)
==111064==    by 0x63DF49A: l_iowatcher_add (flux-lua.c:1363)
==111064==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==111064== 
==111064== Open file descriptor 23:
==111064==    at 0x5CF9897: epoll_create1 (syscall-template.S:81)
==111064==    by 0x66331F5: epoll_init (ev_epoll.c:243)
==111064==    by 0x66331F5: loop_init (ev.c:2923)
==111064==    by 0x663344A: ev_loop_new (ev.c:3102)
==111064==    by 0x663689D: flux_reactor_create (reactor.c:87)
==111064==    by 0x66408E5: now_context_create (future.c:101)
==111064==    by 0x66408E5: flux_future_wait_for (future.c:327)
==111064==    by 0x6640988: flux_future_get (future.c:357)
==111064==    by 0x663DB28: flux_rpc_get_vunpack (rpc.c:122)
==111064==    by 0x663DB28: flux_rpc_get_unpack (rpc.c:137)
==111064==    by 0x6643271: kvs_watch_rpc_get (kvs_watch.c:312)
==111064==    by 0x6643B95: watch_dir (kvs_watch.c:468)
==111064==    by 0x6643B95: flux_kvs_watch_dir (kvs_watch.c:501)
==111064==    by 0x6401D5E: kz_set_ready_cb (kz.c:455)
==111064==    by 0x63DF49A: l_iowatcher_add (flux-lua.c:1363)
==111064==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==111064== 
==111064== Open file descriptor 22:
==111064==    at 0x5CF9897: epoll_create1 (syscall-template.S:81)
==111064==    by 0x66331F5: epoll_init (ev_epoll.c:243)
==111064==    by 0x66331F5: loop_init (ev.c:2923)
==111064==    by 0x663344A: ev_loop_new (ev.c:3102)
==111064==    by 0x663689D: flux_reactor_create (reactor.c:87)
==111064==    by 0x66408E5: now_context_create (future.c:101)
==111064==    by 0x66408E5: flux_future_wait_for (future.c:327)
==111064==    by 0x6640988: flux_future_get (future.c:357)
==111064==    by 0x663DB28: flux_rpc_get_vunpack (rpc.c:122)
==111064==    by 0x663DB28: flux_rpc_get_unpack (rpc.c:137)
==111064==    by 0x6643271: kvs_watch_rpc_get (kvs_watch.c:312)
==111064==    by 0x6643B95: watch_dir (kvs_watch.c:468)
==111064==    by 0x6643B95: flux_kvs_watch_dir (kvs_watch.c:501)
==111064==    by 0x6401D5E: kz_set_ready_cb (kz.c:455)
==111064==    by 0x63DF49A: l_iowatcher_add (flux-lua.c:1363)
==111064==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==111064== 
==111064== Open file descriptor 21:
==111064==    at 0x5CF9897: epoll_create1 (syscall-template.S:81)
==111064==    by 0x66331F5: epoll_init (ev_epoll.c:243)
==111064==    by 0x66331F5: loop_init (ev.c:2923)
==111064==    by 0x663344A: ev_loop_new (ev.c:3102)
==111064==    by 0x663689D: flux_reactor_create (reactor.c:87)
==111064==    by 0x66408E5: now_context_create (future.c:101)
==111064==    by 0x66408E5: flux_future_wait_for (future.c:327)
==111064==    by 0x6640988: flux_future_get (future.c:357)
==111064==    by 0x663DB28: flux_rpc_get_vunpack (rpc.c:122)
==111064==    by 0x663DB28: flux_rpc_get_unpack (rpc.c:137)
==111064==    by 0x6643271: kvs_watch_rpc_get (kvs_watch.c:312)
==111064==    by 0x6643B95: watch_dir (kvs_watch.c:468)
==111064==    by 0x6643B95: flux_kvs_watch_dir (kvs_watch.c:501)
==111064==    by 0x6401D5E: kz_set_ready_cb (kz.c:455)
==111064==    by 0x63DF49A: l_iowatcher_add (flux-lua.c:1363)
==111064==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==111064== 
==111064== Open file descriptor 20: /dev/pts/0
==111064==    at 0x5CEAEE7: dup (syscall-template.S:81)
==111064==    by 0x63DF36D: l_iowatcher_add (flux-lua.c:1342)
==111064==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x4E4CE56: ??? (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x4E4274C: ??? (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x4E41A6D: ??? (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x4E428D9: ??? (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x4E3E44C: lua_pcall (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x40174B: ??? (in /usr/bin/lua)
==111064==    by 0x40211D: ??? (in /usr/bin/lua)
==111064==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x4E42703: ??? (in /usr/lib64/liblua-5.1.so)
==111064== 
==111064== Open file descriptor 19:
==111064==    at 0x5CF9897: epoll_create1 (syscall-template.S:81)
==111064==    by 0x66331F5: epoll_init (ev_epoll.c:243)
==111064==    by 0x66331F5: loop_init (ev.c:2923)
==111064==    by 0x663344A: ev_loop_new (ev.c:3102)
==111064==    by 0x663689D: flux_reactor_create (reactor.c:87)
==111064==    by 0x66408E5: now_context_create (future.c:101)
==111064==    by 0x66408E5: flux_future_wait_for (future.c:327)
==111064==    by 0x6640988: flux_future_get (future.c:357)
==111064==    by 0x663DB28: flux_rpc_get_vunpack (rpc.c:122)
==111064==    by 0x663DB28: flux_rpc_get_unpack (rpc.c:137)
==111064==    by 0x6643271: kvs_watch_rpc_get (kvs_watch.c:312)
==111064==    by 0x6643B95: watch_dir (kvs_watch.c:468)
==111064==    by 0x6643B95: flux_kvs_watch_dir (kvs_watch.c:501)
==111064==    by 0x6401D5E: kz_set_ready_cb (kz.c:455)
==111064==    by 0x63DF49A: l_iowatcher_add (flux-lua.c:1363)
==111064==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==111064== 
==111064== Open file descriptor 18:
==111064==    at 0x5CF9897: epoll_create1 (syscall-template.S:81)
==111064==    by 0x66331F5: epoll_init (ev_epoll.c:243)
==111064==    by 0x66331F5: loop_init (ev.c:2923)
==111064==    by 0x663344A: ev_loop_new (ev.c:3102)
==111064==    by 0x663689D: flux_reactor_create (reactor.c:87)
==111064==    by 0x66408E5: now_context_create (future.c:101)
==111064==    by 0x66408E5: flux_future_wait_for (future.c:327)
==111064==    by 0x6640988: flux_future_get (future.c:357)
==111064==    by 0x663DB28: flux_rpc_get_vunpack (rpc.c:122)
==111064==    by 0x663DB28: flux_rpc_get_unpack (rpc.c:137)
==111064==    by 0x6643271: kvs_watch_rpc_get (kvs_watch.c:312)
==111064==    by 0x6643B95: watch_dir (kvs_watch.c:468)
==111064==    by 0x6643B95: flux_kvs_watch_dir (kvs_watch.c:501)
==111064==    by 0x6401D5E: kz_set_ready_cb (kz.c:455)
==111064==    by 0x63DF49A: l_iowatcher_add (flux-lua.c:1363)
==111064==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==111064== 
==111064== Open file descriptor 17:
==111064==    at 0x5CF9897: epoll_create1 (syscall-template.S:81)
==111064==    by 0x66331F5: epoll_init (ev_epoll.c:243)
==111064==    by 0x66331F5: loop_init (ev.c:2923)
==111064==    by 0x663344A: ev_loop_new (ev.c:3102)
==111064==    by 0x663689D: flux_reactor_create (reactor.c:87)
==111064==    by 0x66408E5: now_context_create (future.c:101)
==111064==    by 0x66408E5: flux_future_wait_for (future.c:327)
==111064==    by 0x6640988: flux_future_get (future.c:357)
==111064==    by 0x663DB28: flux_rpc_get_vunpack (rpc.c:122)
==111064==    by 0x663DB28: flux_rpc_get_unpack (rpc.c:137)
==111064==    by 0x6643271: kvs_watch_rpc_get (kvs_watch.c:312)
==111064==    by 0x6643B95: watch_dir (kvs_watch.c:468)
==111064==    by 0x6643B95: flux_kvs_watch_dir (kvs_watch.c:501)
==111064==    by 0x6401D5E: kz_set_ready_cb (kz.c:455)
==111064==    by 0x63DF49A: l_iowatcher_add (flux-lua.c:1363)
==111064==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==111064== 
==111064== Open file descriptor 16:
==111064==    at 0x5CF9897: epoll_create1 (syscall-template.S:81)
==111064==    by 0x66331F5: epoll_init (ev_epoll.c:243)
==111064==    by 0x66331F5: loop_init (ev.c:2923)
==111064==    by 0x663344A: ev_loop_new (ev.c:3102)
==111064==    by 0x663689D: flux_reactor_create (reactor.c:87)
==111064==    by 0x66408E5: now_context_create (future.c:101)
==111064==    by 0x66408E5: flux_future_wait_for (future.c:327)
==111064==    by 0x6640988: flux_future_get (future.c:357)
==111064==    by 0x663DB28: flux_rpc_get_vunpack (rpc.c:122)
==111064==    by 0x663DB28: flux_rpc_get_unpack (rpc.c:137)
==111064==    by 0x6643271: kvs_watch_rpc_get (kvs_watch.c:312)
==111064==    by 0x6643B95: watch_dir (kvs_watch.c:468)
==111064==    by 0x6643B95: flux_kvs_watch_dir (kvs_watch.c:501)
==111064==    by 0x6401D5E: kz_set_ready_cb (kz.c:455)
==111064==    by 0x63DF49A: l_iowatcher_add (flux-lua.c:1363)
==111064==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==111064== 
==111064== Open file descriptor 15:
==111064==    at 0x5CF9897: epoll_create1 (syscall-template.S:81)
==111064==    by 0x66331F5: epoll_init (ev_epoll.c:243)
==111064==    by 0x66331F5: loop_init (ev.c:2923)
==111064==    by 0x663344A: ev_loop_new (ev.c:3102)
==111064==    by 0x663689D: flux_reactor_create (reactor.c:87)
==111064==    by 0x66408E5: now_context_create (future.c:101)
==111064==    by 0x66408E5: flux_future_wait_for (future.c:327)
==111064==    by 0x6640988: flux_future_get (future.c:357)
==111064==    by 0x663DB28: flux_rpc_get_vunpack (rpc.c:122)
==111064==    by 0x663DB28: flux_rpc_get_unpack (rpc.c:137)
==111064==    by 0x6643271: kvs_watch_rpc_get (kvs_watch.c:312)
==111064==    by 0x6643B95: watch_dir (kvs_watch.c:468)
==111064==    by 0x6643B95: flux_kvs_watch_dir (kvs_watch.c:501)
==111064==    by 0x6401D5E: kz_set_ready_cb (kz.c:455)
==111064==    by 0x63DF49A: l_iowatcher_add (flux-lua.c:1363)
==111064==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==111064== 
==111064== Open file descriptor 14:
==111064==    at 0x5CF9897: epoll_create1 (syscall-template.S:81)
==111064==    by 0x66331F5: epoll_init (ev_epoll.c:243)
==111064==    by 0x66331F5: loop_init (ev.c:2923)
==111064==    by 0x663344A: ev_loop_new (ev.c:3102)
==111064==    by 0x663689D: flux_reactor_create (reactor.c:87)
==111064==    by 0x66408E5: now_context_create (future.c:101)
==111064==    by 0x66408E5: flux_future_wait_for (future.c:327)
==111064==    by 0x6640988: flux_future_get (future.c:357)
==111064==    by 0x663DB28: flux_rpc_get_vunpack (rpc.c:122)
==111064==    by 0x663DB28: flux_rpc_get_unpack (rpc.c:137)
==111064==    by 0x6643271: kvs_watch_rpc_get (kvs_watch.c:312)
==111064==    by 0x6643B95: watch_dir (kvs_watch.c:468)
==111064==    by 0x6643B95: flux_kvs_watch_dir (kvs_watch.c:501)
==111064==    by 0x6401D5E: kz_set_ready_cb (kz.c:455)
==111064==    by 0x63DF49A: l_iowatcher_add (flux-lua.c:1363)
==111064==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==111064== 
==111064== Open file descriptor 13:
==111064==    at 0x5CF9897: epoll_create1 (syscall-template.S:81)
==111064==    by 0x66331F5: epoll_init (ev_epoll.c:243)
==111064==    by 0x66331F5: loop_init (ev.c:2923)
==111064==    by 0x663344A: ev_loop_new (ev.c:3102)
==111064==    by 0x663689D: flux_reactor_create (reactor.c:87)
==111064==    by 0x66408E5: now_context_create (future.c:101)
==111064==    by 0x66408E5: flux_future_wait_for (future.c:327)
==111064==    by 0x6640988: flux_future_get (future.c:357)
==111064==    by 0x663DB28: flux_rpc_get_vunpack (rpc.c:122)
==111064==    by 0x663DB28: flux_rpc_get_unpack (rpc.c:137)
==111064==    by 0x6643271: kvs_watch_rpc_get (kvs_watch.c:312)
==111064==    by 0x6643B95: watch_dir (kvs_watch.c:468)
==111064==    by 0x6643B95: flux_kvs_watch_dir (kvs_watch.c:501)
==111064==    by 0x6401D5E: kz_set_ready_cb (kz.c:455)
==111064==    by 0x63DF49A: l_iowatcher_add (flux-lua.c:1363)
==111064==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==111064== 
==111064== Open file descriptor 12:
==111064==    at 0x5CF9897: epoll_create1 (syscall-template.S:81)
==111064==    by 0x66331F5: epoll_init (ev_epoll.c:243)
==111064==    by 0x66331F5: loop_init (ev.c:2923)
==111064==    by 0x663344A: ev_loop_new (ev.c:3102)
==111064==    by 0x663689D: flux_reactor_create (reactor.c:87)
==111064==    by 0x66408E5: now_context_create (future.c:101)
==111064==    by 0x66408E5: flux_future_wait_for (future.c:327)
==111064==    by 0x6640988: flux_future_get (future.c:357)
==111064==    by 0x663DB28: flux_rpc_get_vunpack (rpc.c:122)
==111064==    by 0x663DB28: flux_rpc_get_unpack (rpc.c:137)
==111064==    by 0x6643271: kvs_watch_rpc_get (kvs_watch.c:312)
==111064==    by 0x6643B95: watch_dir (kvs_watch.c:468)
==111064==    by 0x6643B95: flux_kvs_watch_dir (kvs_watch.c:501)
==111064==    by 0x6401D5E: kz_set_ready_cb (kz.c:455)
==111064==    by 0x63DF49A: l_iowatcher_add (flux-lua.c:1363)
==111064==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==111064== 
==111064== Open file descriptor 11:
==111064==    at 0x5CF95FD: eventfd (eventfd.c:28)
==111064==    by 0x66545E7: msglist_pollfd (msglist.c:180)
==111064==    by 0x6636040: flux_pollfd (handle.c:782)
==111064==    by 0x664005F: ev_flux_init (ev_flux.c:80)
==111064==    by 0x6636C1D: flux_handle_watcher_create (reactor.c:307)
==111064==    by 0x6637B40: dispatch_get (msg_handler.c:153)
==111064==    by 0x6637EAE: flux_msg_handler_create (msg_handler.c:522)
==111064==    by 0x663D73E: initialize_cb (rpc.c:186)
==111064==    by 0x6640858: flux_future_wait_for (future.c:336)
==111064==    by 0x6640988: flux_future_get (future.c:357)
==111064==    by 0x663DB28: flux_rpc_get_vunpack (rpc.c:122)
==111064==    by 0x663DB28: flux_rpc_get_unpack (rpc.c:137)
==111064==    by 0x6643271: kvs_watch_rpc_get (kvs_watch.c:312)
==111064== 
==111064== Open file descriptor 10:
==111064==    at 0x5CF9897: epoll_create1 (syscall-template.S:81)
==111064==    by 0x6636030: flux_pollfd (handle.c:779)
==111064==    by 0x664005F: ev_flux_init (ev_flux.c:80)
==111064==    by 0x6636C1D: flux_handle_watcher_create (reactor.c:307)
==111064==    by 0x6637B40: dispatch_get (msg_handler.c:153)
==111064==    by 0x6637EAE: flux_msg_handler_create (msg_handler.c:522)
==111064==    by 0x663D73E: initialize_cb (rpc.c:186)
==111064==    by 0x6640858: flux_future_wait_for (future.c:336)
==111064==    by 0x6640988: flux_future_get (future.c:357)
==111064==    by 0x663DB28: flux_rpc_get_vunpack (rpc.c:122)
==111064==    by 0x663DB28: flux_rpc_get_unpack (rpc.c:137)
==111064==    by 0x6643271: kvs_watch_rpc_get (kvs_watch.c:312)
==111064==    by 0x6643A31: flux_kvs_watch (kvs_watch.c:444)
==111064== 
==111064== Open file descriptor 9:
==111064==    at 0x5CF9897: epoll_create1 (syscall-template.S:81)
==111064==    by 0x66331F5: epoll_init (ev_epoll.c:243)
==111064==    by 0x66331F5: loop_init (ev.c:2923)
==111064==    by 0x663344A: ev_loop_new (ev.c:3102)
==111064==    by 0x663689D: flux_reactor_create (reactor.c:87)
==111064==    by 0x66408E5: now_context_create (future.c:101)
==111064==    by 0x66408E5: flux_future_wait_for (future.c:327)
==111064==    by 0x6640988: flux_future_get (future.c:357)
==111064==    by 0x663DB28: flux_rpc_get_vunpack (rpc.c:122)
==111064==    by 0x663DB28: flux_rpc_get_unpack (rpc.c:137)
==111064==    by 0x6643271: kvs_watch_rpc_get (kvs_watch.c:312)
==111064==    by 0x6643A31: flux_kvs_watch (kvs_watch.c:444)
==111064==    by 0x63DF714: l_kvswatcher_add (flux-lua.c:1192)
==111064==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x4E4CE56: ??? (in /usr/lib64/liblua-5.1.so)
==111064== 
==111064== Open file descriptor 6:
==111064==    at 0x5CF9897: epoll_create1 (syscall-template.S:81)
==111064==    by 0x66331F5: epoll_init (ev_epoll.c:243)
==111064==    by 0x66331F5: loop_init (ev.c:2923)
==111064==    by 0x663344A: ev_loop_new (ev.c:3102)
==111064==    by 0x663689D: flux_reactor_create (reactor.c:87)
==111064==    by 0x6636986: flux_get_reactor (reactor.c:112)
==111064==    by 0x66406EF: flux_future_set_flux (future.c:273)
==111064==    by 0x663D884: flux_rpc_msg (rpc.c:220)
==111064==    by 0x663DCD8: flux_rpc_vpack (rpc.c:302)
==111064==    by 0x663DCD8: flux_rpc_pack (rpc.c:315)
==111064==    by 0x664334A: kvs_watch_rpc (kvs_watch.c:252)
==111064==    by 0x6643A1C: flux_kvs_watch (kvs_watch.c:442)
==111064==    by 0x63DF714: l_kvswatcher_add (flux-lua.c:1192)
==111064==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==111064== 
==111064== Open file descriptor 8:
==111064==    at 0x5CF95FD: eventfd (eventfd.c:28)
==111064==    by 0x66545E7: msglist_pollfd (msglist.c:180)
==111064==    by 0x6636040: flux_pollfd (handle.c:782)
==111064==    by 0x664005F: ev_flux_init (ev_flux.c:80)
==111064==    by 0x6636C1D: flux_handle_watcher_create (reactor.c:307)
==111064==    by 0x6637B40: dispatch_get (msg_handler.c:153)
==111064==    by 0x6637EAE: flux_msg_handler_create (msg_handler.c:522)
==111064==    by 0x663D73E: initialize_cb (rpc.c:186)
==111064==    by 0x6640858: flux_future_wait_for (future.c:336)
==111064==    by 0x6640988: flux_future_get (future.c:357)
==111064==    by 0x663DB28: flux_rpc_get_vunpack (rpc.c:122)
==111064==    by 0x663DB28: flux_rpc_get_unpack (rpc.c:137)
==111064==    by 0x6634D6F: attr_get_rpc (attr.c:107)
==111064==    by 0x6634D6F: flux_attr_get (attr.c:207)
==111064== 
==111064== Open file descriptor 7:
==111064==    at 0x5CF9897: epoll_create1 (syscall-template.S:81)
==111064==    by 0x6636030: flux_pollfd (handle.c:779)
==111064==    by 0x664005F: ev_flux_init (ev_flux.c:80)
==111064==    by 0x6636C1D: flux_handle_watcher_create (reactor.c:307)
==111064==    by 0x6637B40: dispatch_get (msg_handler.c:153)
==111064==    by 0x6637EAE: flux_msg_handler_create (msg_handler.c:522)
==111064==    by 0x663D73E: initialize_cb (rpc.c:186)
==111064==    by 0x6640858: flux_future_wait_for (future.c:336)
==111064==    by 0x6640988: flux_future_get (future.c:357)
==111064==    by 0x663DB28: flux_rpc_get_vunpack (rpc.c:122)
==111064==    by 0x663DB28: flux_rpc_get_unpack (rpc.c:137)
==111064==    by 0x6634D6F: attr_get_rpc (attr.c:107)
==111064==    by 0x6634D6F: flux_attr_get (attr.c:207)
==111064==    by 0x6634811: flux_get_size (info.c:42)
==111064== 
==111064== Open file descriptor 5:
==111064==    at 0x5CF9897: epoll_create1 (syscall-template.S:81)
==111064==    by 0x66331F5: epoll_init (ev_epoll.c:243)
==111064==    by 0x66331F5: loop_init (ev.c:2923)
==111064==    by 0x663344A: ev_loop_new (ev.c:3102)
==111064==    by 0x663689D: flux_reactor_create (reactor.c:87)
==111064==    by 0x6636986: flux_get_reactor (reactor.c:112)
==111064==    by 0x66406EF: flux_future_set_flux (future.c:273)
==111064==    by 0x663D884: flux_rpc_msg (rpc.c:220)
==111064==    by 0x663DCD8: flux_rpc_vpack (rpc.c:302)
==111064==    by 0x663DCD8: flux_rpc_pack (rpc.c:315)
==111064==    by 0x6634D3A: attr_get_rpc (attr.c:104)
==111064==    by 0x6634D3A: flux_attr_get (attr.c:207)
==111064==    by 0x6634811: flux_get_size (info.c:42)
==111064==    by 0x63DC740: l_flux_size (flux-lua.c:456)
==111064==    by 0x63DC740: l_flux_index (flux-lua.c:481)
==111064==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==111064== 
==111064== Open AF_UNIX socket 4: <unknown>
==111064==    at 0x5CFA4C7: socket (syscall-template.S:81)
==111064==    by 0xA23A737: connector_init (local.c:284)
==111064==    by 0x6635638: flux_open (handle.c:291)
==111064==    by 0x63DDCFC: l_flux_new (flux-lua.c:248)
==111064==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x4E4CE56: ??? (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x4E4274C: ??? (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x4E41A6D: ??? (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x4E428D9: ??? (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x4E3E44C: lua_pcall (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x40174B: ??? (in /usr/bin/lua)
==111064==    by 0x40211D: ??? (in /usr/bin/lua)
==111064== 
==111064== Open AF_UNIX socket 3: <unknown>
==111064==    at 0x5CFA4C7: socket (syscall-template.S:81)
==111064==    by 0xA23A737: connector_init (local.c:284)
==111064==    by 0x6635638: flux_open (handle.c:291)
==111064==    by 0x63DDCFC: l_flux_new (flux-lua.c:248)
==111064==    by 0x4E42323: ??? (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x4E4CE56: ??? (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x4E4274C: ??? (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x4E41A6D: ??? (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x4E428D9: ??? (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x4E3E44C: lua_pcall (in /usr/lib64/liblua-5.1.so)
==111064==    by 0x40174B: ??? (in /usr/bin/lua)
==111064==    by 0x40211D: ??? (in /usr/bin/lua)
==111064==

@garlick
Copy link
Member

garlick commented Apr 2, 2018

zuuid_new() calls assert(false) if it fails to open /dev/urandom (line 64 in czmq 4.0.2), so that is a good clue!

@grondo grondo mentioned this issue Apr 10, 2018
Closed
grondo added a commit to grondo/flux-core that referenced this issue Feb 5, 2019
@grondo
remove wreck, jsc
1c0af46 
The wreck exec system is worthless, remove it along with associated
commands, tests, and support code.

Since libjsc doesn't work without wreck, it is removed as well.

Fixes flux-framework#1984

Closes flux-framework#1947
Closes flux-framework#1618
Closes flux-framework#1595
Closes flux-framework#1593
Closes flux-framework#1534
Closes flux-framework#1468
Closes flux-framework#1443
Closes flux-framework#1438
Closes flux-framework#1419
Closes flux-framework#1410
Closes flux-framework#1407
Closes flux-framework#1393
Closes flux-framework#915
Closes flux-framework#894
Closes flux-framework#866
Closes flux-framework#833
Closes flux-framework#774
Closes flux-framework#772
Closes flux-framework#335
Closes flux-framework#249
grondo added a commit to grondo/flux-core that referenced this issue Feb 5, 2019
@grondo
remove wreck, jsc
432e7ae 
The wreck exec system is worthless, remove it along with associated
commands, tests, and support code.

Since libjsc doesn't work without wreck, it is removed as well.

Fixes flux-framework#1984

Closes flux-framework#1947
Closes flux-framework#1618
Closes flux-framework#1595
Closes flux-framework#1593
Closes flux-framework#1534
Closes flux-framework#1468
Closes flux-framework#1443
Closes flux-framework#1438
Closes flux-framework#1419
Closes flux-framework#1410
Closes flux-framework#1407
Closes flux-framework#1393
Closes flux-framework#915
Closes flux-framework#894
Closes flux-framework#866
Closes flux-framework#833
Closes flux-framework#774
Closes flux-framework#772
Closes flux-framework#335
Closes flux-framework#249
grondo added a commit to grondo/flux-core that referenced this issue Feb 9, 2019
@grondo
remove wreck, jsc
6a54da1 
The wreck exec system is worthless, remove it along with associated
commands, tests, and support code.

Since libjsc doesn't work without wreck, it is removed as well.

Fixes flux-framework#1984

Closes flux-framework#1947
Closes flux-framework#1618
Closes flux-framework#1595
Closes flux-framework#1593
Closes flux-framework#1534
Closes flux-framework#1468
Closes flux-framework#1443
Closes flux-framework#1438
Closes flux-framework#1419
Closes flux-framework#1410
Closes flux-framework#1407
Closes flux-framework#1393
Closes flux-framework#915
Closes flux-framework#894
Closes flux-framework#866
Closes flux-framework#833
Closes flux-framework#774
Closes flux-framework#772
Closes flux-framework#335
Closes flux-framework#249
@grondo
Copy link
Contributor Author

grondo commented Feb 13, 2019

closed by #1988

@grondo grondo closed this as completed Feb 13, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@garlick @grondo