Assertion `max_iov > 1

[skypexu]

Describe the bug

Thread 67 "multi_threaded_" received signal SIGABRT, Aborted.

[Switching to Thread 0x7fff097fa700 (LWP 1186942)]
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007ffff71cf537 in __GI_abort () at abort.c:79
#2 0x00007ffff7da4c85 in ucs_fatal_error_message (file=0x7ffff7f76ec8 "/home/yfxu/github/ucx/src/ucp/proto/proto_am.inl", line=256,
function=0x7ffff7f78b70 <FUNCTION.53> "ucp_am_zcopy_common", message_buf=0x7fff097f8b30 "Assertion max_iov > 1' failed") at debug/assert.c:38 #3 0x00007ffff7da4dee in ucs_fatal_error_format (file=0x7ffff7f76ec8 "/home/yfxu/github/ucx/src/ucp/proto/proto_am.inl", line=256, function=0x7ffff7f78b70 <__FUNCTION__.53> "ucp_am_zcopy_common", format=0x7ffff7f763d8 "Assertion %s' failed") at debug/assert.c:53
#4 0x00007ffff7e1dae7 in ucp_am_zcopy_common (is_middle=0, state=0x7fff097f9070, am_id=26 '\032', max_length=16459, max_iov=0, iov_count=5, iov=0x7fff097f9050, user_hdr_offset=0,
user_hdr_size=24, user_hdr_desc=0x7fff703f1ff0, hdr_size=8, hdr=0x7fff097f90b0, req=0x7fffd4023900) at /home/yfxu/github/ucx/src/ucp/proto/proto_am.inl:256
#5 ucp_do_am_zcopy_single (complete=0x7ffff7e1b455 <ucp_am_zcopy_req_complete>, user_hdr_size=24, user_hdr_desc=0x7fff703f1ff0, hdr_size=8, hdr=0x7fff097f90b0, am_id=26 '\032',
self=0x7fffd40239e0) at /home/yfxu/github/ucx/src/ucp/proto/proto_am.inl:312
#6 ucp_am_zcopy_single_reply (self=0x0) at core/ucp_am.c:704
#7 0x00007ffff7f645f0 in ucp_request_try_send (req=0x7fffd4023900) at /home/yfxu/github/ucx/src/ucp/core/ucp_request.inl:334
#8 ucp_request_send (req=0x7fffd4023900) at /home/yfxu/github/ucx/src/ucp/core/ucp_request.inl:357
#9 ucp_wireup_replay_pending_request (self=0x7fffd40239e0, arg=0x7fff8f623000) at wireup/wireup.c:872
#10 0x00007ffff7f64693 in ucp_wireup_replay_pending_requests (ucp_ep=0x7fff8f623000, tmp_pending_queue=0x7fff097f94d0) at wireup/wireup.c:882
#11 0x00007ffff7f670b5 in ucp_wireup_init_lanes (ep=0x7fff8f623000, ep_init_flags=6, local_tl_bitmap=0x7ffff7f752e0 <ucp_tl_bitmap_max>, remote_address=0x7fff097f9710,
addr_indices=0x7fff097f96a0) at wireup/wireup.c:1417
#12 0x00007ffff7f62c02 in ucp_wireup_process_pre_request (worker=0x7fff98040a00, ep=0x7fff8f623000, msg=0x7fff705360e7, remote_address=0x7fff097f9710) at wireup/wireup.c:488

Version and config

/usr/local/ucx/bin/ucx_info -v

Version 1.13.0

Git branch 'v1.13.x', revision a1f2220

Configured with: --prefix=/usr/local/ucx --enable-debug

UCX_TLS=tcp,sysv

Debug

It is inspected that shared memory communication is being used, and ucx crashed.

[brminich]

@skypexu, is it possible to provide a reproducer?