ConnectX-3 with RoCEv1: Problems with UCX 1.15.0 #9430
Comments
|
@eakneller could you please share |
Here is the information you requested: $ sudo apt install rdma-core Reading package lists... Done $ ib_send_bw -d rocep68s0 -i 1 -F -z -R --report_gbits Waiting for client to connect... Dual-port : OFF Device : rocep68s0 Waiting for client rdma_cm QP to connect local address: LID 0000 QPN 0x0263 PSN 0x12b556 #bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps] $ ib_send_bw -d rocep68s0 -i 1 -F -z -R --report_gbits 10.13.13.30 Dual-port : OFF Device : rocep68s0 local address: LID 0000 QPN 0x0263 PSN 0xf3741b #bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps] |
|
@eakneller can you check if it works with some older UCX version? this is an old NIC and we could try to figure out if the root cause is change in UCX. |
Will do. Any recommendations on which version or should I just go back as far as possible? |
|
I would suggest go back one by one released versions then bisect to identify exact regression. |
ConnectX-3 with RoCEv1: Problems with UCX 1.15.0
Hello All,
I have four Supermicro dual socket AMD Rome servers connected with dual-port ConnectX-3 NIC’s without a switch (NICs are directly connected with 56GbIB/40Gbe DAC’s) and each node has two NICs. The ConnectX-3 cards are configured to Ethernet mode using mstconfig and I am using inbox drivers since vendor OFED software cannot be installed on recent Linux kernels. I have also installed several programs and libraries to enable IB/RDMA/RoCEv1 and have performed successful tests using ib_send_bw using the “-R” option in addition to a test using udaddy for a two-node case, which returns 0 (i.e. RDMA is working). I can also successfully run openmpi without UCX across all nodes in Ethernet mode. However, I cannot get ucx 1.15.0 (from tar ball) to work. Details are provided below about steps to reproduce, setup, tests results and error messages. My ultimate goal is to use openmpi and ucx in RoCEv1 mode to take advantage of RDMA. Any insights you can provide to resolve these issues would be appreciated.
OS: PopOS 22.04 (based on Ubuntu 22.04)
Software installed via “sudo apt install” for configuring ConnectX-3 in Ethernet and RoCEv1 modes: mstconfig, rdma-core, opensm, ibutils, infiniband-diags, ibverbs-utils, rdmacm-utils, libibverbs-dev, librdmacm-dev
Failed UCX Tests
ucp_client_server test between machine myserver1 with IP 10.12.12.30 to machine myclient1 with IP 10.12.12.31
$ ./ucp_client_server &
$ server is listening on IP 0.0.0.0 port 13337
Waiting for connection...
Server received a connection request from client at address 10.12.12.31:50329
error handling callback was invoked with status -25 (Connection reset by remote peer)
unable to receive UCX message (Connection reset by remote peer)
server failed on iteration #1
$ ./ucp_client_server -a 10.12.12.30
[1697942476.562276] [myclient1:36632:0] rc_iface.c:905 UCX ERROR error modifying QP to RTR: Connection timed out
error handling callback was invoked with status -3 (Input/output error)
unable to send UCX message (Input/output error)
client failed on iteration #1
$ ucx_perftest -c 0
Waiting for connection...
Accepted connection from 10.15.15.51:51158
| API: protocol layer |
| Test: tag match latency |
| Data layout: (automatic) |
| Send memory: host |
| Recv memory: host |
| Message size: 8
$ ucx_perftest 10.12.12.30 -t tag_lat -c 1
| | | latency (usec) | bandwidth (MB/s) | message rate (msg/s) |
| Stage | # iterations | 50.0%ile | average | overall | average | overall | average | overall |
[1697942829.711159] [myserver1:36653:0] ib_device.c:1163 UCX ERROR ibv_create_ah(dlid=49152 sl=0 port=1 src_path_bits=0 dgid=::ffff:10.15.15.50 sgid_index=1 traffic_class=0) for UD verbs connect on rocep68s0 failed: Operation not permitted
[1697942829.711566] [myserver1:36653:0] libperf.c:1085 UCX ERROR ucp_ep_create() failed: Address not valid
[1697942829.711616] [myserver1:36653:0] libperf.c:1690 UCX ERROR Failed to setup endpoints: Address not valid
[1697942829.732146] [myserver1:36653:0] ucp_worker.c:2764 UCX WARN worker 0x558eb135b0b0: 2 pending operations were not flushed
[1697942829.734862] [myserver1:36653:0] rc_iface.c:801 UCX WARN some eps were not destroyed
[1697942829.734883] [myserver1:36653:0] ib_verbs.h:134 UCX WARN ibv_destroy_srq() failed: Device or resource busy
[1697942829.735245] [myserver1:36653:0] ib_iface.c:1073 UCX WARN ibv_destroy_cq(RX) returned 16: Device or resource busy
[1697942829.735255] [myserver1:36653:0] ib_iface.c:1073 UCX WARN ibv_destroy_cq(TX) returned 16: Device or resource busy
[1697942829.735262] [myserver1:36653:0] ib_iface.c:1460 UCX WARN ibv_destroy_comp_channel(comp_channel) returned 16: Device or resource busy
[1697942829.745574] [myserver1:36653:0] ib_device.c:608 UCX WARN async_events_hash not empty
[1697942829.745599] [myserver1:36653:0] ib_md.c:1739 UCX WARN ibv_dealloc_pd() failed: Device or resource busy
[1697942829.752861] [myserver1:36653:0] perftest_run.c:334 UCX ERROR Failed to run test: Address not valid
Successful IB/RDMA tests
$ ib_send_bw -d rocep68s0 -i 1 -F -R --report_gbits
Waiting for client to connect...
Send BW Test
Dual-port : OFF Device : rocep68s0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: ON
ibv_wr* API : OFF
RX depth : 512
CQ Moderation : 1
Mtu : 1024[B]
Link type : Ethernet
GID index : 1
Max inline data : 0[B]
rdma_cm QPs : ON
Data ex. method : rdma_cm
Waiting for client rdma_cm QP to connect
Please run the same command with the IB/RoCE interface IP
local address: LID 0000 QPN 0x0386 PSN 0x19c8ea
GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:15:15:50
remote address: LID 0000 QPN 0x0371 PSN 0xd506d0
GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:15:15:51
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
65536 1000 0.00 36.16 0.068968
$ ib_send_bw -d rocep68s0 -i 2 -F -R --report_gbits 10.13.13.30
Dual-port : OFF Device : rocep68s0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: ON
ibv_wr* API : OFF
TX depth : 128
CQ Moderation : 1
Mtu : 1024[B]
Link type : Ethernet
GID index : 1
Max inline data : 0[B]
rdma_cm QPs : ON
Data ex. method : rdma_cm
local address: LID 0000 QPN 0x0371 PSN 0xd506d0
GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:15:15:51
remote address: LID 0000 QPN 0x0386 PSN 0x19c8ea
GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:15:15:50
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
65536 1000 36.43 36.07 0.068798
Setup Information
$ ucx_info -v
Library version: 1.15.0
Library path: …./ucx-1.15.0/lib/libucs.so.0
API headers version: 1.15.0
Git branch '', revision 348d14f
Configured with: --disable-logging --disable-debug --disable-assertions --disable-params-check –prefix=…./ucx-1.15.0
$ ucx_info -c
UCX_LOG_LEVEL=WARN
UCX_LOG_FILE_FILTER=*
UCX_LOG_BUFFER=1K
UCX_LOG_DATA_SIZE=0
UCX_LOG_PRINT_ENABLE=n
UCX_HANDLE_ERRORS=bt
UCX_ERROR_MAIL_TO=
UCX_ERROR_MAIL_FOOTER=
UCX_GDB_$ ucx_info -c
UCX_LOG_LEVEL=WARN
UCX_LOG_FILE_FILTER=*
UCX_LOG_BUFFER=1K
UCX_LOG_DATA_SIZE=0
UCX_LOG_PRINT_ENABLE=n
UCX_HANDLE_ERRORS=bt
UCX_ERROR_MAIL_TO=
UCX_ERROR_MAIL_FOOTER=
UCX_GDB_COMMAND=gdb -quiet
UCX_DEBUG_SIGNO=HUP
UCX_LOG_LEVEL_TRIGGER=FATAL
UCX_WARN_UNUSED_ENV_VARS=y
UCX_MEMTYPE_CACHE=try
UCX_ASYNC_SIGNO=ALRM
UCX_MEMTRACK_LIMIT=inf
UCX_RCACHE_CHECK_PFN=0
UCX_MODULE_DIR=/mnt/extradrive1/nfs/apps/ucx-1.15.0/lib/ucx
UCX_MODULE_LOG_LEVEL=TRACE
UCX_MODULES=all
UCX_TOPO_PRIO=sysfs,default
UCX_LOG_FILE=
UCX_LOG_FILE_SIZE=inf
UCX_LOG_FILE_ROTATE=0
UCX_ERROR_SIGNALS=ILL,SEGV,BUS,FPE
UCX_VFS_ENABLE=y
UCX_VFS_THREAD_AFFINITY=n
UCX_MEMTRACK_DEST=
UCX_PROFILE_MODE=
UCX_PROFILE_FILE=ucx_%h_%p.prof
UCX_PROFILE_LOG_SIZE=4M
UCX_RCACHE_STAT_MIN=4K
UCX_RCACHE_STAT_MAX=1M
UCX_BUILTIN_MEMCPY_MIN=auto
UCX_BUILTIN_MEMCPY_MAX=auto
UCX_MEM_LOG_LEVEL=WARN
UCX_MEM_ALLOC_ALIGN=16
UCX_MEM_EVENTS=y
UCX_MEM_MMAP_HOOK_MODE=bistro
UCX_MEM_MALLOC_HOOKS=y
UCX_MEM_MALLOC_RELOC=y
UCX_MEM_CUDA_HOOK_MODE=bistro
UCX_MEM_DYNAMIC_MMAP_THRESH=y
UCX_MEM_DLOPEN_PROCESS_RPATH=y
UCX_MEM_MODULE_UNLOAD_PREVENT_MODE=lazy
UCX_SELF_NUM_DEVICES=1
UCX_SELF_ALLOC=huge,thp,md,mmap,heap
UCX_SELF_$ ucx_info -c
UCX_LOG_LEVEL=WARN
UCX_LOG_FILE_FILTER=*
UCX_LOG_BUFFER=1K
UCX_LOG_DATA_SIZE=0
UCX_LOG_PRINT_ENABLE=n
UCX_HANDLE_ERRORS=bt
UCX_ERROR_MAIL_TO=
UCX_ERROR_MAIL_FOOTER=
UCX_GDB_COMMAND=gdb -quiet
UCX_DEBUG_SIGNO=HUP
UCX_LOG_LEVEL_TRIGGER=FATAL
UCX_WARN_UNUSED_ENV_VARS=y
UCX_MEMTYPE_CACHE=try
UCX_ASYNC_SIGNO=ALRM
UCX_MEMTRACK_LIMIT=inf
UCX_RCACHE_CHECK_PFN=0
UCX_MODULE_DIR=/mnt/extradrive1/nfs/apps/ucx-1.15.0/lib/ucx
UCX_MODULE_LOG_LEVEL=TRACE
UCX_MODULES=all
UCX_TOPO_PRIO=sysfs,default
UCX_LOG_FILE=
UCX_LOG_FILE_SIZE=inf
UCX_LOG_FILE_ROTATE=0
UCX_ERROR_SIGNALS=ILL,SEGV,BUS,FPE
UCX_VFS_ENABLE=y
UCX_VFS_THREAD_AFFINITY=n
UCX_MEMTRACK_DEST=
UCX_PROFILE_MODE=
UCX_PROFILE_FILE=ucx_%h_%p.prof
UCX_PROFILE_LOG_SIZE=4M
UCX_RCACHE_STAT_MIN=4K
UCX_RCACHE_STAT_MAX=1M
UCX_BUILTIN_MEMCPY_MIN=auto
UCX_BUILTIN_MEMCPY_MAX=auto
UCX_MEM_LOG_LEVEL=WARN
UCX_MEM_ALLOC_ALIGN=16
UCX_MEM_EVENTS=y
UCX_MEM_MMAP_HOOK_MODE=bistro
UCX_MEM_MALLOC_HOOKS=y
UCX_MEM_MALLOC_RELOC=y
UCX_MEM_CUDA_HOOK_MODE=bistro
UCX_MEM_DYNAMIC_MMAP_THRESH=y
UCX_MEM_DLOPEN_PROCESS_RPATH=y
UCX_MEM_MODULE_UNLOAD_PREVENT_MODE=lazy
UCX_SELF_NUM_DEVICES=1
UCX_SELF_ALLOC=huge,thp,md,mmap,heap
UCX_SELF_FAILURE=DIAG
UCX_SELF_MAX_NUM_EPS=inf
UCX_SELF_SEG_SIZE=8K
UCX_TCP_AF_PRIO=inet,inet6
UCX_TCP_CM_FAILURE=DIAG
UCX_TCP_CM_REUSEADDR=n
UCX_TCP_CM_PRIV_DATA_LEN=2K
UCX_TCP_CM_SNDBUF=auto
UCX_TCP_CM_RCVBUF=auto
UCX_TCP_CM_SYN_CNT=auto
UCX_TCP_ALLOC=huge,thp,md,mmap,heap
UCX_TCP_FAILURE=DIAG
UCX_TCP_MAX_NUM_EPS=256
UCX_TCP_TX_SEG_SIZE=8K
UCX_TCP_RX_SEG_SIZE=64K
UCX_TCP_MAX_IOV=6
UCX_TCP_SENDV_THRESH=2K
UCX_TCP_PREFER_DEFAULT=y
UCX_TCP_PUT_ENABLE=y
UCX_TCP_CONN_NB=n
UCX_TCP_MAX_POLL=16
UCX_TCP_MAX_CONN_RETRIES=25
UCX_TCP_NODELAY=y
UCX_TCP_SNDBUF=auto
UCX_TCP_RCVBUF=auto
UCX_TCP_SYN_CNT=auto
UCX_TCP_TX_MAX_BUFS=-1
UCX_TCP_TX_BUFS_GROW=8
UCX_TCP_TX_MAX_CHUNK_SIZE=128M
UCX_TCP_TX_GROW_FACTOR=1.000
UCX_TCP_RX_MAX_BUFS=-1
UCX_TCP_RX_BUFS_GROW=8
UCX_TCP_RX_MAX_CHUNK_SIZE=128M
UCX_TCP_RX_GROW_FACTOR=1.000
UCX_TCP_PORT_RANGE=0
UCX_TCP_MAX_BW=2200.00MBps
UCX_TCP_KEEPIDLE=10000000.00us
UCX_TCP_KEEPCNT=auto
UCX_TCP_KEEPINTVL=2000000.00us
UCX_SYSV_HUGETLB_MODE=try
UCX_SYSV_ALLOC=md,mmap,heap
UCX_SYSV_FAILURE=DIAG
UCX_SYSV_MAX_NUM_EPS=inf
UCX_SYSV_BW=15360.00MBps
UCX_SYSV_FIFO_SIZE=64
UCX_SYSV_SEG_SIZE=8256
UCX_SYSV_FIFO_RELEASE_FACTOR=0.500
UCX_SYSV_RX_MAX_BUFS=-1
UCX_SYSV_RX_BUFS_GROW=512
UCX_SYSV_RX_MAX_CHUNK_SIZE=128M
UCX_SYSV_RX_GROW_FACTOR=2.000
UCX_SYSV_FIFO_HUGETLB=n
UCX_SYSV_FIFO_ELEM_SIZE=128
UCX_SYSV_FIFO_MAX_POLL=16
UCX_SYSV_ERROR_HANDLING=n
UCX_POSIX_HUGETLB_MODE=try
UCX_POSIX_DIR=/dev/shm
UCX_POSIX_SHM_MIN_SIZE=16M
UCX_POSIX_USE_PROC_LINK=y
UCX_POSIX_ALLOC=md,mmap,heap
UCX_POSIX_FAILURE=DIAG
UCX_POSIX_MAX_NUM_EPS=inf
UCX_POSIX_BW=15360.00MBps
UCX_POSIX_FIFO_SIZE=64
UCX_POSIX_SEG_SIZE=8256
UCX_POSIX_FIFO_RELEASE_FACTOR=0.500
UCX_POSIX_RX_MAX_BUFS=-1
UCX_POSIX_RX_BUFS_GROW=512
UCX_POSIX_RX_MAX_CHUNK_SIZE=128M
UCX_POSIX_RX_GROW_FACTOR=1.000
UCX_POSIX_FIFO_HUGETLB=n
UCX_POSIX_FIFO_ELEM_SIZE=128
UCX_POSIX_FIFO_MAX_POLL=16
UCX_POSIX_ERROR_HANDLING=n
UCX_NET_DEVICES=all
UCX_SHM_DEVICES=all
UCX_ACC_DEVICES=all
UCX_SELF_DEVICES=all
UCX_TLS=all
UCX_PROTOS=all
UCX_ALLOC_PRIO=md:sysv,md:posix,huge,thp,md:,mmap,heap
UCX_RNDV_FRAG_SIZE=host:512K,cuda:4M
UCX_RNDV_FRAG_ALLOC_COUNT=host:128,cuda:128
UCX_SOCKADDR_TLS_PRIORITY=rdmacm,tcp,sockcm
UCX_WARN_INVALID_CONFIG=y
UCX_RX_MPOOL_SIZES=64,1K
UCX_RCACHE_ENABLE=try
UCX_SELECT_DISTANCE_MD=cuda_cpy
UCX_MEMTYPE_REG_WHOLE_ALLOC_TYPES=cuda
UCX_RNDV_MEMTYPE_DIRECT_SIZE=inf
UCX_BCOPY_THRESH=auto
UCX_RNDV_THRESH=auto
UCX_RNDV_SEND_NBR_THRESH=256K
UCX_RNDV_THRESH_FALLBACK=inf
UCX_RNDV_PERF_DIFF=1.000
UCX_MULTI_LANE_MAX_RATIO=4.000
UCX_MULTI_PATH_RATIO=auto
UCX_MAX_EAGER_RAILS=1
UCX_MAX_RNDV_RAILS=2
UCX_MAX_RMA_RAILS=1
UCX_MIN_RNDV_CHUNK_SIZE=16K
UCX_RNDV_SCHEME=auto
UCX_RKEY_PTR_SEG_SIZE=512K
UCX_ZCOPY_THRESH=auto
UCX_BCOPY_BW=auto
UCX_ATOMIC_MODE=guess
UCX_ADDRESS_DEBUG_INFO=n
UCX_MAX_WORKER_ADDRESS_NAME=32
UCX_USE_MT_MUTEX=n
UCX_ADAPTIVE_PROGRESS=y
UCX_SEG_SIZE=8K
UCX_TM_THRESH=1K
UCX_TM_MAX_BB_SIZE=1K
UCX_TM_FORCE_THRESH=8K
UCX_TM_SW_RNDV=n
UCX_NUM_EPS=auto
UCX_NUM_PPN=auto
UCX_RNDV_FRAG_MEM_TYPE=host
UCX_RNDV_PIPELINE_SEND_THRESH=inf
UCX_RNDV_PIPELINE_SHM_ENABLE=y
UCX_FLUSH_WORKER_EPS=y
UCX_FENCE_MODE=auto
UCX_UNIFIED_MODE=n
UCX_CM_USE_ALL_DEVICES=y
UCX_LISTENER_BACKLOG=auto
UCX_PROTO_ENABLE=n
UCX_PROTO_REQUEST_RESET=n
UCX_KEEPALIVE_INTERVAL=20000000.00us
UCX_KEEPALIVE_NUM_EPS=128
UCX_RESOLVE_REMOTE_EP_ID=off
UCX_PROTO_INDIRECT_ID=auto
UCX_RNDV_PUT_FORCE_FLUSH=n
UCX_SA_DATA_VERSION=v1
UCX_RKEY_MPOOL_MAX_MD=2
UCX_ADDRESS_VERSION=v1
UCX_PROTO_INFO=n
UCX_RNDV_ALIGN_THRESH=64K
UCX_PROTO_INFO_DIR=
UCX_REG_NONBLOCK_MEM_TYPES=
UCX_IB_REG_METHODS=rcache,direct
UCX_IB_RCACHE_MEM_PRIO=1000
UCX_IB_RCACHE_OVERHEAD=auto
UCX_IB_RCACHE_ADDR_ALIGN=16
UCX_IB_RCACHE_MAX_REGIONS=inf
UCX_IB_RCACHE_MAX_SIZE=inf
UCX_IB_RCACHE_MAX_UNRELEASED=512M
UCX_IB_RCACHE_PURGE_ON_FORK=y
UCX_IB_MEM_REG_OVERHEAD=16.00us
UCX_IB_MEM_REG_GROWTH=0.00us
UCX_IB_FORK_INIT=try
UCX_IB_ASYNC_EVENTS=y
UCX_IB_ETH_PAUSE_ON=y
UCX_IB_ODP_PREFETCH=n
UCX_IB_DEVICE_SPECS=
UCX_IB_PREFER_NEAREST_DEVICE=y
UCX_IB_INDIRECT_ATOMIC=y
UCX_IB_GID_INDEX=auto
UCX_IB_SUBNET_PREFIX=
UCX_IB_GPU_DIRECT_RDMA=try
UCX_IB_PCI_BW=
UCX_IB_MLX5_DEVX=try
UCX_IB_MLX5_DEVX_OBJECTS=rcqp,rcsrq,dct,dcsrq,dci,cq
UCX_IB_REG_MT_THRESH=4G
UCX_IB_REG_MT_CHUNK=2G
UCX_IB_REG_MT_BIND=n
UCX_IB_PCI_RELAXED_ORDERING=auto
UCX_IB_MAX_IDLE_RKEY_COUNT=16
UCX_DC_MLX5_ALLOC=huge,thp,md,mmap,heap
UCX_DC_MLX5_FAILURE=DIAG
UCX_DC_MLX5_MAX_NUM_EPS=inf
UCX_DC_MLX5_SEG_SIZE=8256
UCX_DC_MLX5_TX_QUEUE_LEN=128
UCX_DC_MLX5_TX_MAX_BATCH=16
UCX_DC_MLX5_TX_MAX_POLL=16
UCX_DC_MLX5_TX_MIN_INLINE=64
UCX_DC_MLX5_TX_INLINE_RESP=64
UCX_DC_MLX5_TX_MIN_SGE=5
UCX_DC_MLX5_TX_MAX_BUFS=-1
UCX_DC_MLX5_TX_BUFS_GROW=1024
UCX_DC_MLX5_TX_MAX_CHUNK_SIZE=128M
UCX_DC_MLX5_TX_GROW_FACTOR=1.000
UCX_DC_MLX5_RX_QUEUE_LEN=4095
UCX_DC_MLX5_RX_MAX_BATCH=16
UCX_DC_MLX5_RX_MAX_POLL=16
UCX_DC_MLX5_RX_INLINE=64
UCX_DC_MLX5_RX_MAX_BUFS=-1
UCX_DC_MLX5_RX_BUFS_GROW=0
UCX_DC_MLX5_RX_MAX_CHUNK_SIZE=128M
UCX_DC_MLX5_RX_GROW_FACTOR=1.000
UCX_DC_MLX5_ADDR_TYPE=auto
UCX_DC_MLX5_IS_GLOBAL=n
UCX_DC_MLX5_SL=auto
UCX_DC_MLX5_TRAFFIC_CLASS=auto
UCX_DC_MLX5_HOP_LIMIT=255
UCX_DC_MLX5_NUM_PATHS=auto
UCX_DC_MLX5_ROCE_LOCAL_SUBNET=n
UCX_DC_MLX5_ROCE_SUBNET_PREFIX_LEN=auto
UCX_DC_MLX5_ROCE_PATH_FACTOR=1
UCX_DC_MLX5_LID_PATH_BITS=0
UCX_DC_MLX5_PKEY=auto
UCX_DC_MLX5_PATH_MTU=default
UCX_DC_MLX5_COUNTER_SET_ID=auto
UCX_DC_MLX5_MAX_RD_ATOMIC=auto
UCX_DC_MLX5_TIMEOUT=1000000.00us
UCX_DC_MLX5_RETRY_COUNT=7
UCX_DC_MLX5_RNR_TIMEOUT=1000.00us
UCX_DC_MLX5_RNR_RETRY_COUNT=7
UCX_DC_MLX5_FC_ENABLE=y
UCX_DC_MLX5_FC_WND_SIZE=512
UCX_DC_MLX5_FC_HARD_THRESH=0.250
UCX_DC_MLX5_FENCE=auto
UCX_DC_MLX5_MAX_GET_ZCOPY=auto
UCX_DC_MLX5_TX_NUM_GET_BYTES=inf
UCX_DC_MLX5_TX_POLL_ALWAYS=n
UCX_DC_MLX5_ECE=0
UCX_DC_MLX5_DM_SIZE=2K
UCX_DC_MLX5_DM_COUNT=1
UCX_DC_MLX5_MMIO_MODE=auto
UCX_DC_MLX5_AR_ENABLE=auto
UCX_DC_MLX5_TX_CQE_ZIP_ENABLE=n
UCX_DC_MLX5_RX_CQE_ZIP_ENABLE=n
UCX_DC_MLX5_TX_MAX_BB=inf
UCX_DC_MLX5_TM_ENABLE=n
UCX_DC_MLX5_TM_LIST_SIZE=1024
UCX_DC_MLX5_TM_SEG_SIZE=48K
UCX_DC_MLX5_TM_MP_SRQ_ENABLE=try
UCX_DC_MLX5_TM_MP_NUM_STRIDES=8
UCX_DC_MLX5_EXP_BACKOFF=0
UCX_DC_MLX5_SRQ_TOPO=list
UCX_DC_MLX5_LOG_ACK_REQ_FREQ=8
UCX_DC_MLX5_RX_QUEUE_LEN_INIT=128
UCX_DC_MLX5_NUM_DCI=8
UCX_DC_MLX5_TX_POLICY=dcs_quota
UCX_DC_MLX5_LAG_PORT_AFFINITY=auto
UCX_DC_MLX5_DCI_FULL_HANDSHAKE=n
UCX_DC_MLX5_DCT_PORT_AFFINITY=default
UCX_DC_MLX5_DCI_KA_FULL_HANDSHAKE=n
UCX_DC_MLX5_DCT_FULL_HANDSHAKE=n
UCX_DC_MLX5_RAND_DCI_SEED=0
UCX_DC_MLX5_QUOTA=32
UCX_DC_MLX5_FC_HARD_REQ_TIMEOUT=5000000.00us
UCX_DC_MLX5_NUM_DCI_CHANNELS=8
UCX_DC_MLX5_COMPACT_AV=y
UCX_RC_VERBS_ALLOC=huge,thp,md,mmap,heap
UCX_RC_VERBS_FAILURE=DIAG
UCX_RC_VERBS_MAX_NUM_EPS=256
UCX_RC_VERBS_SEG_SIZE=8256
UCX_RC_VERBS_TX_QUEUE_LEN=256
UCX_RC_VERBS_TX_MAX_BATCH=16
UCX_RC_VERBS_TX_MAX_POLL=16
UCX_RC_VERBS_TX_MIN_INLINE=64
UCX_RC_VERBS_TX_INLINE_RESP=64
UCX_RC_VERBS_TX_MIN_SGE=5
UCX_RC_VERBS_TX_MAX_BUFS=-1
UCX_RC_VERBS_TX_BUFS_GROW=1024
UCX_RC_VERBS_TX_MAX_CHUNK_SIZE=128M
UCX_RC_VERBS_TX_GROW_FACTOR=1.000
UCX_RC_VERBS_RX_QUEUE_LEN=4095
UCX_RC_VERBS_RX_MAX_BATCH=16
UCX_RC_VERBS_RX_MAX_POLL=16
UCX_RC_VERBS_RX_INLINE=64
UCX_RC_VERBS_RX_MAX_BUFS=-1
UCX_RC_VERBS_RX_BUFS_GROW=0
UCX_RC_VERBS_RX_MAX_CHUNK_SIZE=128M
UCX_RC_VERBS_RX_GROW_FACTOR=1.000
UCX_RC_VERBS_ADDR_TYPE=auto
UCX_RC_VERBS_IS_GLOBAL=n
UCX_RC_VERBS_SL=auto
UCX_RC_VERBS_TRAFFIC_CLASS=auto
UCX_RC_VERBS_HOP_LIMIT=255
UCX_RC_VERBS_NUM_PATHS=auto
UCX_RC_VERBS_ROCE_LOCAL_SUBNET=n
UCX_RC_VERBS_ROCE_SUBNET_PREFIX_LEN=auto
UCX_RC_VERBS_ROCE_PATH_FACTOR=1
UCX_RC_VERBS_LID_PATH_BITS=0
UCX_RC_VERBS_PKEY=auto
UCX_RC_VERBS_PATH_MTU=default
UCX_RC_VERBS_COUNTER_SET_ID=auto
UCX_RC_VERBS_MAX_RD_ATOMIC=auto
UCX_RC_VERBS_TIMEOUT=1000000.00us
UCX_RC_VERBS_RETRY_COUNT=7
UCX_RC_VERBS_RNR_TIMEOUT=1000.00us
UCX_RC_VERBS_RNR_RETRY_COUNT=7
UCX_RC_VERBS_FC_ENABLE=y
UCX_RC_VERBS_FC_WND_SIZE=512
UCX_RC_VERBS_FC_HARD_THRESH=0.250
UCX_RC_VERBS_FENCE=auto
UCX_RC_VERBS_MAX_GET_ZCOPY=auto
UCX_RC_VERBS_TX_NUM_GET_BYTES=inf
UCX_RC_VERBS_TX_POLL_ALWAYS=n
UCX_RC_VERBS_ECE=0
UCX_RC_VERBS_FC_SOFT_THRESH=0.500
UCX_RC_VERBS_TX_CQ_MODERATION=64
UCX_RC_VERBS_TX_CQ_LEN=4096
UCX_RC_VERBS_MAX_AM_HDR=128
UCX_RC_VERBS_TX_MAX_WR=inf
UCX_RC_VERBS_FLUSH_MODE=auto
UCX_RC_MLX5_ALLOC=huge,thp,md,mmap,heap
UCX_RC_MLX5_FAILURE=DIAG
UCX_RC_MLX5_MAX_NUM_EPS=256
UCX_RC_MLX5_SEG_SIZE=8256
UCX_RC_MLX5_TX_QUEUE_LEN=256
UCX_RC_MLX5_TX_MAX_BATCH=16
UCX_RC_MLX5_TX_MAX_POLL=16
UCX_RC_MLX5_TX_MIN_INLINE=64
UCX_RC_MLX5_TX_INLINE_RESP=64
UCX_RC_MLX5_TX_MIN_SGE=5
UCX_RC_MLX5_TX_MAX_BUFS=-1
UCX_RC_MLX5_TX_BUFS_GROW=1024
UCX_RC_MLX5_TX_MAX_CHUNK_SIZE=128M
UCX_RC_MLX5_TX_GROW_FACTOR=1.000
UCX_RC_MLX5_RX_QUEUE_LEN=4095
UCX_RC_MLX5_RX_MAX_BATCH=16
UCX_RC_MLX5_RX_MAX_POLL=16
UCX_RC_MLX5_RX_INLINE=64
UCX_RC_MLX5_RX_MAX_BUFS=-1
UCX_RC_MLX5_RX_BUFS_GROW=0
UCX_RC_MLX5_RX_MAX_CHUNK_SIZE=128M
UCX_RC_MLX5_RX_GROW_FACTOR=1.000
UCX_RC_MLX5_ADDR_TYPE=auto
UCX_RC_MLX5_IS_GLOBAL=n
UCX_RC_MLX5_SL=auto
UCX_RC_MLX5_TRAFFIC_CLASS=auto
UCX_RC_MLX5_HOP_LIMIT=255
UCX_RC_MLX5_NUM_PATHS=auto
UCX_RC_MLX5_ROCE_LOCAL_SUBNET=n
UCX_RC_MLX5_ROCE_SUBNET_PREFIX_LEN=auto
UCX_RC_MLX5_ROCE_PATH_FACTOR=1
UCX_RC_MLX5_LID_PATH_BITS=0
UCX_RC_MLX5_PKEY=auto
UCX_RC_MLX5_PATH_MTU=default
UCX_RC_MLX5_COUNTER_SET_ID=auto
UCX_RC_MLX5_MAX_RD_ATOMIC=auto
UCX_RC_MLX5_TIMEOUT=1000000.00us
UCX_RC_MLX5_RETRY_COUNT=7
UCX_RC_MLX5_RNR_TIMEOUT=1000.00us
UCX_RC_MLX5_RNR_RETRY_COUNT=7
UCX_RC_MLX5_FC_ENABLE=y
UCX_RC_MLX5_FC_WND_SIZE=512
UCX_RC_MLX5_FC_HARD_THRESH=0.250
UCX_RC_MLX5_FENCE=auto
UCX_RC_MLX5_MAX_GET_ZCOPY=auto
UCX_RC_MLX5_TX_NUM_GET_BYTES=inf
UCX_RC_MLX5_TX_POLL_ALWAYS=n
UCX_RC_MLX5_ECE=0
UCX_RC_MLX5_FC_SOFT_THRESH=0.500
UCX_RC_MLX5_TX_CQ_MODERATION=64
UCX_RC_MLX5_TX_CQ_LEN=4096
UCX_RC_MLX5_DM_SIZE=2K
UCX_RC_MLX5_DM_COUNT=1
UCX_RC_MLX5_MMIO_MODE=auto
UCX_RC_MLX5_AR_ENABLE=auto
UCX_RC_MLX5_TX_CQE_ZIP_ENABLE=n
UCX_RC_MLX5_RX_CQE_ZIP_ENABLE=n
UCX_RC_MLX5_TX_MAX_BB=inf
UCX_RC_MLX5_TM_ENABLE=n
UCX_RC_MLX5_TM_LIST_SIZE=1024
UCX_RC_MLX5_TM_SEG_SIZE=48K
UCX_RC_MLX5_TM_MP_SRQ_ENABLE=try
UCX_RC_MLX5_TM_MP_NUM_STRIDES=8
UCX_RC_MLX5_EXP_BACKOFF=0
UCX_RC_MLX5_SRQ_TOPO=cyclic,cyclic_emulated
UCX_RC_MLX5_LOG_ACK_REQ_FREQ=8
UCX_UD_VERBS_ALLOC=huge,thp,md,mmap,heap
UCX_UD_VERBS_FAILURE=DIAG
UCX_UD_VERBS_MAX_NUM_EPS=inf
UCX_UD_VERBS_SEG_SIZE=8K
UCX_UD_VERBS_TX_QUEUE_LEN=256
UCX_UD_VERBS_TX_MAX_BATCH=16
UCX_UD_VERBS_TX_MAX_POLL=16
UCX_UD_VERBS_TX_MIN_INLINE=64
UCX_UD_VERBS_TX_INLINE_RESP=0
UCX_UD_VERBS_TX_MIN_SGE=5
UCX_UD_VERBS_TX_MAX_BUFS=-1
UCX_UD_VERBS_TX_BUFS_GROW=1024
UCX_UD_VERBS_TX_MAX_CHUNK_SIZE=128M
UCX_UD_VERBS_TX_GROW_FACTOR=1.000
UCX_UD_VERBS_RX_QUEUE_LEN=4096
UCX_UD_VERBS_RX_MAX_BATCH=16
UCX_UD_VERBS_RX_MAX_POLL=16
UCX_UD_VERBS_RX_INLINE=0
UCX_UD_VERBS_RX_MAX_BUFS=-1
UCX_UD_VERBS_RX_BUFS_GROW=0
UCX_UD_VERBS_RX_MAX_CHUNK_SIZE=128M
UCX_UD_VERBS_RX_GROW_FACTOR=1.000
UCX_UD_VERBS_ADDR_TYPE=auto
UCX_UD_VERBS_IS_GLOBAL=n
UCX_UD_VERBS_SL=auto
UCX_UD_VERBS_TRAFFIC_CLASS=auto
UCX_UD_VERBS_HOP_LIMIT=255
UCX_UD_VERBS_NUM_PATHS=auto
UCX_UD_VERBS_ROCE_LOCAL_SUBNET=n
UCX_UD_VERBS_ROCE_SUBNET_PREFIX_LEN=auto
UCX_UD_VERBS_ROCE_PATH_FACTOR=1
UCX_UD_VERBS_LID_PATH_BITS=0
UCX_UD_VERBS_PKEY=auto
UCX_UD_VERBS_PATH_MTU=default
UCX_UD_VERBS_COUNTER_SET_ID=auto
UCX_UD_VERBS_RX_QUEUE_LEN_INIT=128
UCX_UD_VERBS_LINGER_TIMEOUT=300000000.00us
UCX_UD_VERBS_TIMEOUT=30000000.00us
UCX_UD_VERBS_TIMER_TICK=10000.00us
UCX_UD_VERBS_TIMER_BACKOFF=2.000
UCX_UD_VERBS_ASYNC_TIMER_TICK=100000.00us
UCX_UD_VERBS_MIN_POKE_TIME=250000.00us
UCX_UD_VERBS_ETH_DGID_CHECK=y
UCX_UD_VERBS_MAX_WINDOW=1025
UCX_UD_VERBS_RX_ASYNC_MAX_POLL=64
UCX_UD_MLX5_ALLOC=huge,thp,md,mmap,heap
UCX_UD_MLX5_FAILURE=DIAG
UCX_UD_MLX5_MAX_NUM_EPS=inf
UCX_UD_MLX5_SEG_SIZE=8K
UCX_UD_MLX5_TX_QUEUE_LEN=256
UCX_UD_MLX5_TX_MAX_BATCH=16
UCX_UD_MLX5_TX_MAX_POLL=16
UCX_UD_MLX5_TX_MIN_INLINE=64
UCX_UD_MLX5_TX_INLINE_RESP=0
UCX_UD_MLX5_TX_MIN_SGE=5
UCX_UD_MLX5_TX_MAX_BUFS=-1
UCX_UD_MLX5_TX_BUFS_GROW=1024
UCX_UD_MLX5_TX_MAX_CHUNK_SIZE=128M
UCX_UD_MLX5_TX_GROW_FACTOR=1.000
UCX_UD_MLX5_RX_QUEUE_LEN=4096
UCX_UD_MLX5_RX_MAX_BATCH=16
UCX_UD_MLX5_RX_MAX_POLL=16
UCX_UD_MLX5_RX_INLINE=0
UCX_UD_MLX5_RX_MAX_BUFS=-1
UCX_UD_MLX5_RX_BUFS_GROW=0
UCX_UD_MLX5_RX_MAX_CHUNK_SIZE=128M
UCX_UD_MLX5_RX_GROW_FACTOR=1.000
UCX_UD_MLX5_ADDR_TYPE=auto
UCX_UD_MLX5_IS_GLOBAL=n
UCX_UD_MLX5_SL=auto
UCX_UD_MLX5_TRAFFIC_CLASS=auto
UCX_UD_MLX5_HOP_LIMIT=255
UCX_UD_MLX5_NUM_PATHS=auto
UCX_UD_MLX5_ROCE_LOCAL_SUBNET=n
UCX_UD_MLX5_ROCE_SUBNET_PREFIX_LEN=auto
UCX_UD_MLX5_ROCE_PATH_FACTOR=1
UCX_UD_MLX5_LID_PATH_BITS=0
UCX_UD_MLX5_PKEY=auto
UCX_UD_MLX5_PATH_MTU=default
UCX_UD_MLX5_COUNTER_SET_ID=auto
UCX_UD_MLX5_RX_QUEUE_LEN_INIT=128
UCX_UD_MLX5_LINGER_TIMEOUT=300000000.00us
UCX_UD_MLX5_TIMEOUT=30000000.00us
UCX_UD_MLX5_TIMER_TICK=10000.00us
UCX_UD_MLX5_TIMER_BACKOFF=2.000
UCX_UD_MLX5_ASYNC_TIMER_TICK=100000.00us
UCX_UD_MLX5_MIN_POKE_TIME=250000.00us
UCX_UD_MLX5_ETH_DGID_CHECK=y
UCX_UD_MLX5_MAX_WINDOW=1025
UCX_UD_MLX5_RX_ASYNC_MAX_POLL=64
UCX_UD_MLX5_DM_SIZE=2K
UCX_UD_MLX5_DM_COUNT=1
UCX_UD_MLX5_MMIO_MODE=auto
UCX_UD_MLX5_AR_ENABLE=auto
UCX_UD_MLX5_TX_CQE_ZIP_ENABLE=n
UCX_UD_MLX5_RX_CQE_ZIP_ENABLE=n
UCX_UD_MLX5_COMPACT_AV=y
UCX_RDMA_CM_FAILURE=DIAG
UCX_RDMA_CM_REUSEADDR=n
UCX_RDMA_CM_SOURCE_ADDRESS=
UCX_RDMA_CM_TIMEOUT=10000000.00us
UCX_RDMA_CM_RESERVED_QPN=try
UCX_CMA_MEMORY_INVALIDATE=n
UCX_CMA_ALLOC=huge,thp,mmap,heap
UCX_CMA_FAILURE=DIAG
UCX_CMA_MAX_NUM_EPS=inf
UCX_CMA_BW=11145.00MBps
UCX_CMA_MAX_IOV=16
UCX_CMA_SEG_SIZE=512K
UCX_CMA_TX_QUOTA=1
UCX_CMA_TX_MAX_BUFS=-1
UCX_CMA_TX_BUFS_GROW=8
UCX_CMA_TX_MAX_CHUNK_SIZE=128M
UCX_CMA_TX_GROW_FACTOR=1.000FAILURE=DIAG
UCX_SELF_MAX_NUM_EPS=inf
UCX_SELF_SEG_SIZE=8K
UCX_TCP_AF_PRIO=inet,inet6
UCX_TCP_CM_FAILURE=DIAG
UCX_TCP_CM_REUSEADDR=n
UCX_TCP_CM_PRIV_DATA_LEN=2K
UCX_TCP_CM_SNDBUF=auto
UCX_TCP_CM_RCVBUF=auto
UCX_TCP_CM_SYN_CNT=auto
UCX_TCP_ALLOC=huge,thp,md,mmap,heap
UCX_TCP_FAILURE=DIAG
UCX_TCP_MAX_NUM_EPS=256
UCX_TCP_TX_SEG_SIZE=8K
UCX_TCP_RX_SEG_SIZE=64K
UCX_TCP_MAX_IOV=6
UCX_TCP_SENDV_THRESH=2K
UCX_TCP_PREFER_DEFAULT=y
UCX_TCP_PUT_ENABLE=y
UCX_TCP_CONN_NB=n
UCX_TCP_MAX_POLL=16
UCX_TCP_MAX_CONN_RETRIES=25
UCX_TCP_NODELAY=y
UCX_TCP_SNDBUF=auto
UCX_TCP_RCVBUF=auto
UCX_TCP_SYN_CNT=auto
UCX_TCP_TX_MAX_BUFS=-1
UCX_TCP_TX_BUFS_GROW=8
UCX_TCP_TX_MAX_CHUNK_SIZE=128M
UCX_TCP_TX_GROW_FACTOR=1.000
UCX_TCP_RX_MAX_BUFS=-1
UCX_TCP_RX_BUFS_GROW=8
UCX_TCP_RX_MAX_CHUNK_SIZE=128M
UCX_TCP_RX_GROW_FACTOR=1.000
UCX_TCP_PORT_RANGE=0
UCX_TCP_MAX_BW=2200.00MBps
UCX_TCP_KEEPIDLE=10000000.00us
UCX_TCP_KEEPCNT=auto
UCX_TCP_KEEPINTVL=2000000.00us
UCX_SYSV_HUGETLB_MODE=try
UCX_SYSV_ALLOC=md,mmap,heap
UCX_SYSV_FAILURE=DIAG
UCX_SYSV_MAX_NUM_EPS=inf
UCX_SYSV_BW=15360.00MBps
UCX_SYSV_FIFO_SIZE=64
UCX_SYSV_SEG_SIZE=8256
UCX_SYSV_FIFO_RELEASE_FACTOR=0.500
UCX_SYSV_RX_MAX_BUFS=-1
UCX_SYSV_RX_BUFS_GROW=512
UCX_SYSV_RX_MAX_CHUNK_SIZE=128M
UCX_SYSV_RX_GROW_FACTOR=2.000
UCX_SYSV_FIFO_HUGETLB=n
UCX_SYSV_FIFO_ELEM_SIZE=128
UCX_SYSV_FIFO_MAX_POLL=16
UCX_SYSV_ERROR_HANDLING=n
UCX_POSIX_HUGETLB_MODE=try
UCX_POSIX_DIR=/dev/shm
UCX_POSIX_SHM_MIN_SIZE=16M
UCX_POSIX_USE_PROC_LINK=y
UCX_POSIX_ALLOC=md,mmap,heap
UCX_POSIX_FAILURE=DIAG
UCX_POSIX_MAX_NUM_EPS=inf
UCX_POSIX_BW=15360.00MBps
UCX_POSIX_FIFO_SIZE=64
UCX_POSIX_SEG_SIZE=8256
UCX_POSIX_FIFO_RELEASE_FACTOR=0.500
UCX_POSIX_RX_MAX_BUFS=-1
UCX_POSIX_RX_BUFS_GROW=512
UCX_POSIX_RX_MAX_CHUNK_SIZE=128M
UCX_POSIX_RX_GROW_FACTOR=1.000
UCX_POSIX_FIFO_HUGETLB=n
UCX_POSIX_FIFO_ELEM_SIZE=128
UCX_POSIX_FIFO_MAX_POLL=16
UCX_POSIX_ERROR_HANDLING=n
UCX_NET_DEVICES=all
UCX_SHM_DEVICES=all
UCX_ACC_DEVICES=all
UCX_SELF_DEVICES=all
UCX_TLS=all
UCX_PROTOS=all
UCX_ALLOC_PRIO=md:sysv,md:posix,huge,thp,md:,mmap,heap
UCX_RNDV_FRAG_SIZE=host:512K,cuda:4M
UCX_RNDV_FRAG_ALLOC_COUNT=host:128,cuda:128
UCX_SOCKADDR_TLS_PRIORITY=rdmacm,tcp,sockcm
UCX_WARN_INVALID_CONFIG=y
UCX_RX_MPOOL_SIZES=64,1K
UCX_RCACHE_ENABLE=try
UCX_SELECT_DISTANCE_MD=cuda_cpy
UCX_MEMTYPE_REG_WHOLE_ALLOC_TYPES=cuda
UCX_RNDV_MEMTYPE_DIRECT_SIZE=inf
UCX_BCOPY_THRESH=auto
UCX_RNDV_THRESH=auto
UCX_RNDV_SEND_NBR_THRESH=256K
UCX_RNDV_THRESH_FALLBACK=inf
UCX_RNDV_PERF_DIFF=1.000
UCX_MULTI_LANE_MAX_RATIO=4.000
UCX_MULTI_PATH_RATIO=auto
UCX_MAX_EAGER_RAILS=1
UCX_MAX_RNDV_RAILS=2
UCX_MAX_RMA_RAILS=1
UCX_MIN_RNDV_CHUNK_SIZE=16K
UCX_RNDV_SCHEME=auto
UCX_RKEY_PTR_SEG_SIZE=512K
UCX_ZCOPY_THRESH=auto
UCX_BCOPY_BW=auto
UCX_ATOMIC_MODE=guess
UCX_ADDRESS_DEBUG_INFO=n
UCX_MAX_WORKER_ADDRESS_NAME=32
UCX_USE_MT_MUTEX=n
UCX_ADAPTIVE_PROGRESS=y
UCX_SEG_SIZE=8K
UCX_TM_THRESH=1K
UCX_TM_MAX_BB_SIZE=1K
UCX_TM_FORCE_THRESH=8K
UCX_TM_SW_RNDV=n
UCX_NUM_EPS=auto
UCX_NUM_PPN=auto
UCX_RNDV_FRAG_MEM_TYPE=host
UCX_RNDV_PIPELINE_SEND_THRESH=inf
UCX_RNDV_PIPELINE_SHM_ENABLE=y
UCX_FLUSH_WORKER_EPS=y
UCX_FENCE_MODE=auto
UCX_UNIFIED_MODE=n
UCX_CM_USE_ALL_DEVICES=y
UCX_LISTENER_BACKLOG=auto
UCX_PROTO_ENABLE=n
UCX_PROTO_REQUEST_RESET=n
UCX_KEEPALIVE_INTERVAL=20000000.00us
UCX_KEEPALIVE_NUM_EPS=128
UCX_RESOLVE_REMOTE_EP_ID=off
UCX_PROTO_INDIRECT_ID=auto
UCX_RNDV_PUT_FORCE_FLUSH=n
UCX_SA_DATA_VERSION=v1
UCX_RKEY_MPOOL_MAX_MD=2
UCX_ADDRESS_VERSION=v1
UCX_PROTO_INFO=n
UCX_RNDV_ALIGN_THRESH=64K
UCX_PROTO_INFO_DIR=
UCX_REG_NONBLOCK_MEM_TYPES=
UCX_IB_REG_METHODS=rcache,direct
UCX_IB_RCACHE_MEM_PRIO=1000
UCX_IB_RCACHE_OVERHEAD=auto
UCX_IB_RCACHE_ADDR_ALIGN=16
UCX_IB_RCACHE_MAX_REGIONS=inf
UCX_IB_RCACHE_MAX_SIZE=inf
UCX_IB_RCACHE_MAX_UNRELEASED=512M
UCX_IB_RCACHE_PURGE_ON_FORK=y
UCX_IB_MEM_REG_OVERHEAD=16.00us
UCX_IB_MEM_REG_GROWTH=0.00us
UCX_IB_FORK_INIT=try
UCX_IB_ASYNC_EVENTS=y
UCX_IB_ETH_PAUSE_ON=y
UCX_IB_ODP_PREFETCH=n
UCX_IB_DEVICE_SPECS=
UCX_IB_PREFER_NEAREST_DEVICE=y
UCX_IB_INDIRECT_ATOMIC=y
UCX_IB_GID_INDEX=auto
UCX_IB_SUBNET_PREFIX=
UCX_IB_GPU_DIRECT_RDMA=try
UCX_IB_PCI_BW=
UCX_IB_MLX5_DEVX=try
UCX_IB_MLX5_DEVX_OBJECTS=rcqp,rcsrq,dct,dcsrq,dci,cq
UCX_IB_REG_MT_THRESH=4G
UCX_IB_REG_MT_CHUNK=2G
UCX_IB_REG_MT_BIND=n
UCX_IB_PCI_RELAXED_ORDERING=auto
UCX_IB_MAX_IDLE_RKEY_COUNT=16
UCX_DC_MLX5_ALLOC=huge,thp,md,mmap,heap
UCX_DC_MLX5_FAILURE=DIAG
UCX_DC_MLX5_MAX_NUM_EPS=inf
UCX_DC_MLX5_SEG_SIZE=8256
UCX_DC_MLX5_TX_QUEUE_LEN=128
UCX_DC_MLX5_TX_MAX_BATCH=16
UCX_DC_MLX5_TX_MAX_POLL=16
UCX_DC_MLX5_TX_MIN_INLINE=64
UCX_DC_MLX5_TX_INLINE_RESP=64
UCX_DC_MLX5_TX_MIN_SGE=5
UCX_DC_MLX5_TX_MAX_BUFS=-1
UCX_DC_MLX5_TX_BUFS_GROW=1024
UCX_DC_MLX5_TX_MAX_CHUNK_SIZE=128M
UCX_DC_MLX5_TX_GROW_FACTOR=1.000
UCX_DC_MLX5_RX_QUEUE_LEN=4095
UCX_DC_MLX5_RX_MAX_BATCH=16
UCX_DC_MLX5_RX_MAX_POLL=16
UCX_DC_MLX5_RX_INLINE=64
UCX_DC_MLX5_RX_MAX_BUFS=-1
UCX_DC_MLX5_RX_BUFS_GROW=0
UCX_DC_MLX5_RX_MAX_CHUNK_SIZE=128M
UCX_DC_MLX5_RX_GROW_FACTOR=1.000
UCX_DC_MLX5_ADDR_TYPE=auto
UCX_DC_MLX5_IS_GLOBAL=n
UCX_DC_MLX5_SL=auto
UCX_DC_MLX5_TRAFFIC_CLASS=auto
UCX_DC_MLX5_HOP_LIMIT=255
UCX_DC_MLX5_NUM_PATHS=auto
UCX_DC_MLX5_ROCE_LOCAL_SUBNET=n
UCX_DC_MLX5_ROCE_SUBNET_PREFIX_LEN=auto
UCX_DC_MLX5_ROCE_PATH_FACTOR=1
UCX_DC_MLX5_LID_PATH_BITS=0
UCX_DC_MLX5_PKEY=auto
UCX_DC_MLX5_PATH_MTU=default
UCX_DC_MLX5_COUNTER_SET_ID=auto
UCX_DC_MLX5_MAX_RD_ATOMIC=auto
UCX_DC_MLX5_TIMEOUT=1000000.00us
UCX_DC_MLX5_RETRY_COUNT=7
UCX_DC_MLX5_RNR_TIMEOUT=1000.00us
UCX_DC_MLX5_RNR_RETRY_COUNT=7
UCX_DC_MLX5_FC_ENABLE=y
UCX_DC_MLX5_FC_WND_SIZE=512
UCX_DC_MLX5_FC_HARD_THRESH=0.250
UCX_DC_MLX5_FENCE=auto
UCX_DC_MLX5_MAX_GET_ZCOPY=auto
UCX_DC_MLX5_TX_NUM_GET_BYTES=inf
UCX_DC_MLX5_TX_POLL_ALWAYS=n
UCX_DC_MLX5_ECE=0
UCX_DC_MLX5_DM_SIZE=2K
UCX_DC_MLX5_DM_COUNT=1
UCX_DC_MLX5_MMIO_MODE=auto
UCX_DC_MLX5_AR_ENABLE=auto
UCX_DC_MLX5_TX_CQE_ZIP_ENABLE=n
UCX_DC_MLX5_RX_CQE_ZIP_ENABLE=n
UCX_DC_MLX5_TX_MAX_BB=inf
UCX_DC_MLX5_TM_ENABLE=n
UCX_DC_MLX5_TM_LIST_SIZE=1024
UCX_DC_MLX5_TM_SEG_SIZE=48K
UCX_DC_MLX5_TM_MP_SRQ_ENABLE=try
UCX_DC_MLX5_TM_MP_NUM_STRIDES=8
UCX_DC_MLX5_EXP_BACKOFF=0
UCX_DC_MLX5_SRQ_TOPO=list
UCX_DC_MLX5_LOG_ACK_REQ_FREQ=8
UCX_DC_MLX5_RX_QUEUE_LEN_INIT=128
UCX_DC_MLX5_NUM_DCI=8
UCX_DC_MLX5_TX_POLICY=dcs_quota
UCX_DC_MLX5_LAG_PORT_AFFINITY=auto
UCX_DC_MLX5_DCI_FULL_HANDSHAKE=n
UCX_DC_MLX5_DCT_PORT_AFFINITY=default
UCX_DC_MLX5_DCI_KA_FULL_HANDSHAKE=n
UCX_DC_MLX5_DCT_FULL_HANDSHAKE=n
UCX_DC_MLX5_RAND_DCI_SEED=0
UCX_DC_MLX5_QUOTA=32
UCX_DC_MLX5_FC_HARD_REQ_TIMEOUT=5000000.00us
UCX_DC_MLX5_NUM_DCI_CHANNELS=8
UCX_DC_MLX5_COMPACT_AV=y
UCX_RC_VERBS_ALLOC=huge,thp,md,mmap,heap
UCX_RC_VERBS_FAILURE=DIAG
UCX_RC_VERBS_MAX_NUM_EPS=256
UCX_RC_VERBS_SEG_SIZE=8256
UCX_RC_VERBS_TX_QUEUE_LEN=256
UCX_RC_VERBS_TX_MAX_BATCH=16
UCX_RC_VERBS_TX_MAX_POLL=16
UCX_RC_VERBS_TX_MIN_INLINE=64
UCX_RC_VERBS_TX_INLINE_RESP=64
UCX_RC_VERBS_TX_MIN_SGE=5
UCX_RC_VERBS_TX_MAX_BUFS=-1
UCX_RC_VERBS_TX_BUFS_GROW=1024
UCX_RC_VERBS_TX_MAX_CHUNK_SIZE=128M
UCX_RC_VERBS_TX_GROW_FACTOR=1.000
UCX_RC_VERBS_RX_QUEUE_LEN=4095
UCX_RC_VERBS_RX_MAX_BATCH=16
UCX_RC_VERBS_RX_MAX_POLL=16
UCX_RC_VERBS_RX_INLINE=64
UCX_RC_VERBS_RX_MAX_BUFS=-1
UCX_RC_VERBS_RX_BUFS_GROW=0
UCX_RC_VERBS_RX_MAX_CHUNK_SIZE=128M
UCX_RC_VERBS_RX_GROW_FACTOR=1.000
UCX_RC_VERBS_ADDR_TYPE=auto
UCX_RC_VERBS_IS_GLOBAL=n
UCX_RC_VERBS_SL=auto
UCX_RC_VERBS_TRAFFIC_CLASS=auto
UCX_RC_VERBS_HOP_LIMIT=255
UCX_RC_VERBS_NUM_PATHS=auto
UCX_RC_VERBS_ROCE_LOCAL_SUBNET=n
UCX_RC_VERBS_ROCE_SUBNET_PREFIX_LEN=auto
UCX_RC_VERBS_ROCE_PATH_FACTOR=1
UCX_RC_VERBS_LID_PATH_BITS=0
UCX_RC_VERBS_PKEY=auto
UCX_RC_VERBS_PATH_MTU=default
UCX_RC_VERBS_COUNTER_SET_ID=auto
UCX_RC_VERBS_MAX_RD_ATOMIC=auto
UCX_RC_VERBS_TIMEOUT=1000000.00us
UCX_RC_VERBS_RETRY_COUNT=7
UCX_RC_VERBS_RNR_TIMEOUT=1000.00us
UCX_RC_VERBS_RNR_RETRY_COUNT=7
UCX_RC_VERBS_FC_ENABLE=y
UCX_RC_VERBS_FC_WND_SIZE=512
UCX_RC_VERBS_FC_HARD_THRESH=0.250
UCX_RC_VERBS_FENCE=auto
UCX_RC_VERBS_MAX_GET_ZCOPY=auto
UCX_RC_VERBS_TX_NUM_GET_BYTES=inf
UCX_RC_VERBS_TX_POLL_ALWAYS=n
UCX_RC_VERBS_ECE=0
UCX_RC_VERBS_FC_SOFT_THRESH=0.500
UCX_RC_VERBS_TX_CQ_MODERATION=64
UCX_RC_VERBS_TX_CQ_LEN=4096
UCX_RC_VERBS_MAX_AM_HDR=128
UCX_RC_VERBS_TX_MAX_WR=inf
UCX_RC_VERBS_FLUSH_MODE=auto
UCX_RC_MLX5_ALLOC=huge,thp,md,mmap,heap
UCX_RC_MLX5_FAILURE=DIAG
UCX_RC_MLX5_MAX_NUM_EPS=256
UCX_RC_MLX5_SEG_SIZE=8256
UCX_RC_MLX5_TX_QUEUE_LEN=256
UCX_RC_MLX5_TX_MAX_BATCH=16
UCX_RC_MLX5_TX_MAX_POLL=16
UCX_RC_MLX5_TX_MIN_INLINE=64
UCX_RC_MLX5_TX_INLINE_RESP=64
UCX_RC_MLX5_TX_MIN_SGE=5
UCX_RC_MLX5_TX_MAX_BUFS=-1
UCX_RC_MLX5_TX_BUFS_GROW=1024
UCX_RC_MLX5_TX_MAX_CHUNK_SIZE=128M
UCX_RC_MLX5_TX_GROW_FACTOR=1.000
UCX_RC_MLX5_RX_QUEUE_LEN=4095
UCX_RC_MLX5_RX_MAX_BATCH=16
UCX_RC_MLX5_RX_MAX_POLL=16
UCX_RC_MLX5_RX_INLINE=64
UCX_RC_MLX5_RX_MAX_BUFS=-1
UCX_RC_MLX5_RX_BUFS_GROW=0
UCX_RC_MLX5_RX_MAX_CHUNK_SIZE=128M
UCX_RC_MLX5_RX_GROW_FACTOR=1.000
UCX_RC_MLX5_ADDR_TYPE=auto
UCX_RC_MLX5_IS_GLOBAL=n
UCX_RC_MLX5_SL=auto
UCX_RC_MLX5_TRAFFIC_CLASS=auto
UCX_RC_MLX5_HOP_LIMIT=255
UCX_RC_MLX5_NUM_PATHS=auto
UCX_RC_MLX5_ROCE_LOCAL_SUBNET=n
UCX_RC_MLX5_ROCE_SUBNET_PREFIX_LEN=auto
UCX_RC_MLX5_ROCE_PATH_FACTOR=1
UCX_RC_MLX5_LID_PATH_BITS=0
UCX_RC_MLX5_PKEY=auto
UCX_RC_MLX5_PATH_MTU=default
UCX_RC_MLX5_COUNTER_SET_ID=auto
UCX_RC_MLX5_MAX_RD_ATOMIC=auto
UCX_RC_MLX5_TIMEOUT=1000000.00us
UCX_RC_MLX5_RETRY_COUNT=7
UCX_RC_MLX5_RNR_TIMEOUT=1000.00us
UCX_RC_MLX5_RNR_RETRY_COUNT=7
UCX_RC_MLX5_FC_ENABLE=y
UCX_RC_MLX5_FC_WND_SIZE=512
UCX_RC_MLX5_FC_HARD_THRESH=0.250
UCX_RC_MLX5_FENCE=auto
UCX_RC_MLX5_MAX_GET_ZCOPY=auto
UCX_RC_MLX5_TX_NUM_GET_BYTES=inf
UCX_RC_MLX5_TX_POLL_ALWAYS=n
UCX_RC_MLX5_ECE=0
UCX_RC_MLX5_FC_SOFT_THRESH=0.500
UCX_RC_MLX5_TX_CQ_MODERATION=64
UCX_RC_MLX5_TX_CQ_LEN=4096
UCX_RC_MLX5_DM_SIZE=2K
UCX_RC_MLX5_DM_COUNT=1
UCX_RC_MLX5_MMIO_MODE=auto
UCX_RC_MLX5_AR_ENABLE=auto
UCX_RC_MLX5_TX_CQE_ZIP_ENABLE=n
UCX_RC_MLX5_RX_CQE_ZIP_ENABLE=n
UCX_RC_MLX5_TX_MAX_BB=inf
UCX_RC_MLX5_TM_ENABLE=n
UCX_RC_MLX5_TM_LIST_SIZE=1024
UCX_RC_MLX5_TM_SEG_SIZE=48K
UCX_RC_MLX5_TM_MP_SRQ_ENABLE=try
UCX_RC_MLX5_TM_MP_NUM_STRIDES=8
UCX_RC_MLX5_EXP_BACKOFF=0
UCX_RC_MLX5_SRQ_TOPO=cyclic,cyclic_emulated
UCX_RC_MLX5_LOG_ACK_REQ_FREQ=8
UCX_UD_VERBS_ALLOC=huge,thp,md,mmap,heap
UCX_UD_VERBS_FAILURE=DIAG
UCX_UD_VERBS_MAX_NUM_EPS=inf
UCX_UD_VERBS_SEG_SIZE=8K
UCX_UD_VERBS_TX_QUEUE_LEN=256
UCX_UD_VERBS_TX_MAX_BATCH=16
UCX_UD_VERBS_TX_MAX_POLL=16
UCX_UD_VERBS_TX_MIN_INLINE=64
UCX_UD_VERBS_TX_INLINE_RESP=0
UCX_UD_VERBS_TX_MIN_SGE=5
UCX_UD_VERBS_TX_MAX_BUFS=-1
UCX_UD_VERBS_TX_BUFS_GROW=1024
UCX_UD_VERBS_TX_MAX_CHUNK_SIZE=128M
UCX_UD_VERBS_TX_GROW_FACTOR=1.000
UCX_UD_VERBS_RX_QUEUE_LEN=4096
UCX_UD_VERBS_RX_MAX_BATCH=16
UCX_UD_VERBS_RX_MAX_POLL=16
UCX_UD_VERBS_RX_INLINE=0
UCX_UD_VERBS_RX_MAX_BUFS=-1
UCX_UD_VERBS_RX_BUFS_GROW=0
UCX_UD_VERBS_RX_MAX_CHUNK_SIZE=128M
UCX_UD_VERBS_RX_GROW_FACTOR=1.000
UCX_UD_VERBS_ADDR_TYPE=auto
UCX_UD_VERBS_IS_GLOBAL=n
UCX_UD_VERBS_SL=auto
UCX_UD_VERBS_TRAFFIC_CLASS=auto
UCX_UD_VERBS_HOP_LIMIT=255
UCX_UD_VERBS_NUM_PATHS=auto
UCX_UD_VERBS_ROCE_LOCAL_SUBNET=n
UCX_UD_VERBS_ROCE_SUBNET_PREFIX_LEN=auto
UCX_UD_VERBS_ROCE_PATH_FACTOR=1
UCX_UD_VERBS_LID_PATH_BITS=0
UCX_UD_VERBS_PKEY=auto
UCX_UD_VERBS_PATH_MTU=default
UCX_UD_VERBS_COUNTER_SET_ID=auto
UCX_UD_VERBS_RX_QUEUE_LEN_INIT=128
UCX_UD_VERBS_LINGER_TIMEOUT=300000000.00us
UCX_UD_VERBS_TIMEOUT=30000000.00us
UCX_UD_VERBS_TIMER_TICK=10000.00us
UCX_UD_VERBS_TIMER_BACKOFF=2.000
UCX_UD_VERBS_ASYNC_TIMER_TICK=100000.00us
UCX_UD_VERBS_MIN_POKE_TIME=250000.00us
UCX_UD_VERBS_ETH_DGID_CHECK=y
UCX_UD_VERBS_MAX_WINDOW=1025
UCX_UD_VERBS_RX_ASYNC_MAX_POLL=64
UCX_UD_MLX5_ALLOC=huge,thp,md,mmap,heap
UCX_UD_MLX5_FAILURE=DIAG
UCX_UD_MLX5_MAX_NUM_EPS=inf
UCX_UD_MLX5_SEG_SIZE=8K
UCX_UD_MLX5_TX_QUEUE_LEN=256
UCX_UD_MLX5_TX_MAX_BATCH=16
UCX_UD_MLX5_TX_MAX_POLL=16
UCX_UD_MLX5_TX_MIN_INLINE=64
UCX_UD_MLX5_TX_INLINE_RESP=0
UCX_UD_MLX5_TX_MIN_SGE=5
UCX_UD_MLX5_TX_MAX_BUFS=-1
UCX_UD_MLX5_TX_BUFS_GROW=1024
UCX_UD_MLX5_TX_MAX_CHUNK_SIZE=128M
UCX_UD_MLX5_TX_GROW_FACTOR=1.000
UCX_UD_MLX5_RX_QUEUE_LEN=4096
UCX_UD_MLX5_RX_MAX_BATCH=16
UCX_UD_MLX5_RX_MAX_POLL=16
UCX_UD_MLX5_RX_INLINE=0
UCX_UD_MLX5_RX_MAX_BUFS=-1
UCX_UD_MLX5_RX_BUFS_GROW=0
UCX_UD_MLX5_RX_MAX_CHUNK_SIZE=128M
UCX_UD_MLX5_RX_GROW_FACTOR=1.000
UCX_UD_MLX5_ADDR_TYPE=auto
UCX_UD_MLX5_IS_GLOBAL=n
UCX_UD_MLX5_SL=auto
UCX_UD_MLX5_TRAFFIC_CLASS=auto
UCX_UD_MLX5_HOP_LIMIT=255
UCX_UD_MLX5_NUM_PATHS=auto
UCX_UD_MLX5_ROCE_LOCAL_SUBNET=n
UCX_UD_MLX5_ROCE_SUBNET_PREFIX_LEN=auto
UCX_UD_MLX5_ROCE_PATH_FACTOR=1
UCX_UD_MLX5_LID_PATH_BITS=0
UCX_UD_MLX5_PKEY=auto
UCX_UD_MLX5_PATH_MTU=default
UCX_UD_MLX5_COUNTER_SET_ID=auto
UCX_UD_MLX5_RX_QUEUE_LEN_INIT=128
UCX_UD_MLX5_LINGER_TIMEOUT=300000000.00us
UCX_UD_MLX5_TIMEOUT=30000000.00us
UCX_UD_MLX5_TIMER_TICK=10000.00us
UCX_UD_MLX5_TIMER_BACKOFF=2.000
UCX_UD_MLX5_ASYNC_TIMER_TICK=100000.00us
UCX_UD_MLX5_MIN_POKE_TIME=250000.00us
UCX_UD_MLX5_ETH_DGID_CHECK=y
UCX_UD_MLX5_MAX_WINDOW=1025
UCX_UD_MLX5_RX_ASYNC_MAX_POLL=64
UCX_UD_MLX5_DM_SIZE=2K
UCX_UD_MLX5_DM_COUNT=1
UCX_UD_MLX5_MMIO_MODE=auto
UCX_UD_MLX5_AR_ENABLE=auto
UCX_UD_MLX5_TX_CQE_ZIP_ENABLE=n
UCX_UD_MLX5_RX_CQE_ZIP_ENABLE=n
UCX_UD_MLX5_COMPACT_AV=y
UCX_RDMA_CM_FAILURE=DIAG
UCX_RDMA_CM_REUSEADDR=n
UCX_RDMA_CM_SOURCE_ADDRESS=
UCX_RDMA_CM_TIMEOUT=10000000.00us
UCX_RDMA_CM_RESERVED_QPN=try
UCX_CMA_MEMORY_INVALIDATE=n
UCX_CMA_ALLOC=huge,thp,mmap,heap
UCX_CMA_FAILURE=DIAG
UCX_CMA_MAX_NUM_EPS=inf
UCX_CMA_BW=11145.00MBps
UCX_CMA_MAX_IOV=16
UCX_CMA_SEG_SIZE=512K
UCX_CMA_TX_QUOTA=1
UCX_CMA_TX_MAX_BUFS=-1
UCX_CMA_TX_BUFS_GROW=8
UCX_CMA_TX_MAX_CHUNK_SIZE=128M
UCX_CMA_TX_GROW_FACTOR=1.000COMMAND=gdb -quiet
UCX_DEBUG_SIGNO=HUP
UCX_LOG_LEVEL_TRIGGER=FATAL
UCX_WARN_UNUSED_ENV_VARS=y
UCX_MEMTYPE_CACHE=try
UCX_ASYNC_SIGNO=ALRM
UCX_MEMTRACK_LIMIT=inf
UCX_RCACHE_CHECK_PFN=0
UCX_MODULE_DIR=/mnt/extradrive1/nfs/apps/ucx-1.15.0/lib/ucx
UCX_MODULE_LOG_LEVEL=TRACE
UCX_MODULES=all
UCX_TOPO_PRIO=sysfs,default
UCX_LOG_FILE=
UCX_LOG_FILE_SIZE=inf
UCX_LOG_FILE_ROTATE=0
UCX_ERROR_SIGNALS=ILL,SEGV,BUS,FPE
UCX_VFS_ENABLE=y
UCX_VFS_THREAD_AFFINITY=n
UCX_MEMTRACK_DEST=
UCX_PROFILE_MODE=
UCX_PROFILE_FILE=ucx_%h_%p.prof
UCX_PROFILE_LOG_SIZE=4M
UCX_RCACHE_STAT_MIN=4K
UCX_RCACHE_STAT_MAX=1M
UCX_BUILTIN_MEMCPY_MIN=auto
UCX_BUILTIN_MEMCPY_MAX=auto
UCX_MEM_LOG_LEVEL=WARN
UCX_MEM_ALLOC_ALIGN=16
UCX_MEM_EVENTS=y
UCX_MEM_MMAP_HOOK_MODE=bistro
UCX_MEM_MALLOC_HOOKS=y
UCX_MEM_MALLOC_RELOC=y
UCX_MEM_CUDA_HOOK_MODE=bistro
UCX_MEM_DYNAMIC_MMAP_THRESH=y
UCX_MEM_DLOPEN_PROCESS_RPATH=y
UCX_MEM_MODULE_UNLOAD_PREVENT_MODE=lazy
UCX_SELF_NUM_DEVICES=1
UCX_SELF_ALLOC=huge,thp,md,mmap,heap
UCX_SELF_FAILURE=DIAG
UCX_SELF_MAX_NUM_EPS=inf
UCX_SELF_SEG_SIZE=8K
UCX_TCP_AF_PRIO=inet,inet6
UCX_TCP_CM_FAILURE=DIAG
UCX_TCP_CM_REUSEADDR=n
UCX_TCP_CM_PRIV_DATA_LEN=2K
UCX_TCP_CM_SNDBUF=auto
UCX_TCP_CM_RCVBUF=auto
UCX_TCP_CM_SYN_CNT=auto
UCX_TCP_ALLOC=huge,thp,md,mmap,heap
UCX_TCP_FAILURE=DIAG
UCX_TCP_MAX_NUM_EPS=256
UCX_TCP_TX_SEG_SIZE=8K
UCX_TCP_RX_SEG_SIZE=64K
UCX_TCP_MAX_IOV=6
UCX_TCP_SENDV_THRESH=2K
UCX_TCP_PREFER_DEFAULT=y
UCX_TCP_PUT_ENABLE=y
UCX_TCP_CONN_NB=n
UCX_TCP_MAX_POLL=16
UCX_TCP_MAX_CONN_RETRIES=25
UCX_TCP_NODELAY=y
UCX_TCP_SNDBUF=auto
UCX_TCP_RCVBUF=auto
UCX_TCP_SYN_CNT=auto
UCX_TCP_TX_MAX_BUFS=-1
UCX_TCP_TX_BUFS_GROW=8
UCX_TCP_TX_MAX_CHUNK_SIZE=128M
UCX_TCP_TX_GROW_FACTOR=1.000
UCX_TCP_RX_MAX_BUFS=-1
UCX_TCP_RX_BUFS_GROW=8
UCX_TCP_RX_MAX_CHUNK_SIZE=128M
UCX_TCP_RX_GROW_FACTOR=1.000
UCX_TCP_PORT_RANGE=0
UCX_TCP_MAX_BW=2200.00MBps
UCX_TCP_KEEPIDLE=10000000.00us
UCX_TCP_KEEPCNT=auto
UCX_TCP_KEEPINTVL=2000000.00us
UCX_SYSV_HUGETLB_MODE=try
UCX_SYSV_ALLOC=md,mmap,heap
UCX_SYSV_FAILURE=DIAG
UCX_SYSV_MAX_NUM_EPS=inf
UCX_SYSV_BW=15360.00MBps
UCX_SYSV_FIFO_SIZE=64
UCX_SYSV_SEG_SIZE=8256
UCX_SYSV_FIFO_RELEASE_FACTOR=0.500
UCX_SYSV_RX_MAX_BUFS=-1
UCX_SYSV_RX_BUFS_GROW=512
UCX_SYSV_RX_MAX_CHUNK_SIZE=128M
UCX_SYSV_RX_GROW_FACTOR=2.000
UCX_SYSV_FIFO_HUGETLB=n
UCX_SYSV_FIFO_ELEM_SIZE=128
UCX_SYSV_FIFO_MAX_POLL=16
UCX_SYSV_ERROR_HANDLING=n
UCX_POSIX_HUGETLB_MODE=try
UCX_POSIX_DIR=/dev/shm
UCX_POSIX_SHM_MIN_SIZE=16M
UCX_POSIX_USE_PROC_LINK=y
UCX_POSIX_ALLOC=md,mmap,heap
UCX_POSIX_FAILURE=DIAG
UCX_POSIX_MAX_NUM_EPS=inf
UCX_POSIX_BW=15360.00MBps
UCX_POSIX_FIFO_SIZE=64
UCX_POSIX_SEG_SIZE=8256
UCX_POSIX_FIFO_RELEASE_FACTOR=0.500
UCX_POSIX_RX_MAX_BUFS=-1
UCX_POSIX_RX_BUFS_GROW=512
UCX_POSIX_RX_MAX_CHUNK_SIZE=128M
UCX_POSIX_RX_GROW_FACTOR=1.000
UCX_POSIX_FIFO_HUGETLB=n
UCX_POSIX_FIFO_ELEM_SIZE=128
UCX_POSIX_FIFO_MAX_POLL=16
UCX_POSIX_ERROR_HANDLING=n
UCX_NET_DEVICES=all
UCX_SHM_DEVICES=all
UCX_ACC_DEVICES=all
UCX_SELF_DEVICES=all
UCX_TLS=all
UCX_PROTOS=all
UCX_ALLOC_PRIO=md:sysv,md:posix,huge,thp,md:*,mmap,heap
UCX_RNDV_FRAG_SIZE=host:512K,cuda:4M
UCX_RNDV_FRAG_ALLOC_COUNT=host:128,cuda:128
UCX_SOCKADDR_TLS_PRIORITY=rdmacm,tcp,sockcm
UCX_WARN_INVALID_CONFIG=y
UCX_RX_MPOOL_SIZES=64,1K
UCX_RCACHE_ENABLE=try
UCX_SELECT_DISTANCE_MD=cuda_cpy
UCX_MEMTYPE_REG_WHOLE_ALLOC_TYPES=cuda
UCX_RNDV_MEMTYPE_DIRECT_SIZE=inf
UCX_BCOPY_THRESH=auto
UCX_RNDV_THRESH=auto
UCX_RNDV_SEND_NBR_THRESH=256K
UCX_RNDV_THRESH_FALLBACK=inf
UCX_RNDV_PERF_DIFF=1.000
UCX_MULTI_LANE_MAX_RATIO=4.000
UCX_MULTI_PATH_RATIO=auto
UCX_MAX_EAGER_RAILS=1
UCX_MAX_RNDV_RAILS=2
UCX_MAX_RMA_RAILS=1
UCX_MIN_RNDV_CHUNK_SIZE=16K
UCX_RNDV_SCHEME=auto
UCX_RKEY_PTR_SEG_SIZE=512K
UCX_ZCOPY_THRESH=auto
UCX_BCOPY_BW=auto
UCX_ATOMIC_MODE=guess
UCX_ADDRESS_DEBUG_INFO=n
UCX_MAX_WORKER_ADDRESS_NAME=32
UCX_USE_MT_MUTEX=n
UCX_ADAPTIVE_PROGRESS=y
UCX_SEG_SIZE=8K
UCX_TM_THRESH=1K
UCX_TM_MAX_BB_SIZE=1K
UCX_TM_FORCE_THRESH=8K
UCX_TM_SW_RNDV=n
UCX_NUM_EPS=auto
UCX_NUM_PPN=auto
UCX_RNDV_FRAG_MEM_TYPE=host
UCX_RNDV_PIPELINE_SEND_THRESH=inf
UCX_RNDV_PIPELINE_SHM_ENABLE=y
UCX_FLUSH_WORKER_EPS=y
UCX_FENCE_MODE=auto
UCX_UNIFIED_MODE=n
UCX_CM_USE_ALL_DEVICES=y
UCX_LISTENER_BACKLOG=auto
UCX_PROTO_ENABLE=n
UCX_PROTO_REQUEST_RESET=n
UCX_KEEPALIVE_INTERVAL=20000000.00us
UCX_KEEPALIVE_NUM_EPS=128
UCX_RESOLVE_REMOTE_EP_ID=off
UCX_PROTO_INDIRECT_ID=auto
UCX_RNDV_PUT_FORCE_FLUSH=n
UCX_SA_DATA_VERSION=v1
UCX_RKEY_MPOOL_MAX_MD=2
UCX_ADDRESS_VERSION=v1
UCX_PROTO_INFO=n
UCX_RNDV_ALIGN_THRESH=64K
UCX_PROTO_INFO_DIR=
UCX_REG_NONBLOCK_MEM_TYPES=
UCX_IB_REG_METHODS=rcache,direct
UCX_IB_RCACHE_MEM_PRIO=1000
UCX_IB_RCACHE_OVERHEAD=auto
UCX_IB_RCACHE_ADDR_ALIGN=16
UCX_IB_RCACHE_MAX_REGIONS=inf
UCX_IB_RCACHE_MAX_SIZE=inf
UCX_IB_RCACHE_MAX_UNRELEASED=512M
UCX_IB_RCACHE_PURGE_ON_FORK=y
UCX_IB_MEM_REG_OVERHEAD=16.00us
UCX_IB_MEM_REG_GROWTH=0.00us
UCX_IB_FORK_INIT=try
UCX_IB_ASYNC_EVENTS=y
UCX_IB_ETH_PAUSE_ON=y
UCX_IB_ODP_PREFETCH=n
UCX_IB_DEVICE_SPECS=
UCX_IB_PREFER_NEAREST_DEVICE=y
UCX_IB_INDIRECT_ATOMIC=y
UCX_IB_GID_INDEX=auto
UCX_IB_SUBNET_PREFIX=
UCX_IB_GPU_DIRECT_RDMA=try
UCX_IB_PCI_BW=
UCX_IB_MLX5_DEVX=try
UCX_IB_MLX5_DEVX_OBJECTS=rcqp,rcsrq,dct,dcsrq,dci,cq
UCX_IB_REG_MT_THRESH=4G
UCX_IB_REG_MT_CHUNK=2G
UCX_IB_REG_MT_BIND=n
UCX_IB_PCI_RELAXED_ORDERING=auto
UCX_IB_MAX_IDLE_RKEY_COUNT=16
UCX_DC_MLX5_ALLOC=huge,thp,md,mmap,heap
UCX_DC_MLX5_FAILURE=DIAG
UCX_DC_MLX5_MAX_NUM_EPS=inf
UCX_DC_MLX5_SEG_SIZE=8256
UCX_DC_MLX5_TX_QUEUE_LEN=128
UCX_DC_MLX5_TX_MAX_BATCH=16
UCX_DC_MLX5_TX_MAX_POLL=16
UCX_DC_MLX5_TX_MIN_INLINE=64
UCX_DC_MLX5_TX_INLINE_RESP=64
UCX_DC_MLX5_TX_MIN_SGE=5
UCX_DC_MLX5_TX_MAX_BUFS=-1
UCX_DC_MLX5_TX_BUFS_GROW=1024
UCX_DC_MLX5_TX_MAX_CHUNK_SIZE=128M
UCX_DC_MLX5_TX_GROW_FACTOR=1.000
UCX_DC_MLX5_RX_QUEUE_LEN=4095
UCX_DC_MLX5_RX_MAX_BATCH=16
UCX_DC_MLX5_RX_MAX_POLL=16
UCX_DC_MLX5_RX_INLINE=64
UCX_DC_MLX5_RX_MAX_BUFS=-1
UCX_DC_MLX5_RX_BUFS_GROW=0
UCX_DC_MLX5_RX_MAX_CHUNK_SIZE=128M
UCX_DC_MLX5_RX_GROW_FACTOR=1.000
UCX_DC_MLX5_ADDR_TYPE=auto
UCX_DC_MLX5_IS_GLOBAL=n
UCX_DC_MLX5_SL=auto
UCX_DC_MLX5_TRAFFIC_CLASS=auto
UCX_DC_MLX5_HOP_LIMIT=255
UCX_DC_MLX5_NUM_PATHS=auto
UCX_DC_MLX5_ROCE_LOCAL_SUBNET=n
UCX_DC_MLX5_ROCE_SUBNET_PREFIX_LEN=auto
UCX_DC_MLX5_ROCE_PATH_FACTOR=1
UCX_DC_MLX5_LID_PATH_BITS=0
UCX_DC_MLX5_PKEY=auto
UCX_DC_MLX5_PATH_MTU=default
UCX_DC_MLX5_COUNTER_SET_ID=auto
UCX_DC_MLX5_MAX_RD_ATOMIC=auto
UCX_DC_MLX5_TIMEOUT=1000000.00us
UCX_DC_MLX5_RETRY_COUNT=7
UCX_DC_MLX5_RNR_TIMEOUT=1000.00us
UCX_DC_MLX5_RNR_RETRY_COUNT=7
UCX_DC_MLX5_FC_ENABLE=y
UCX_DC_MLX5_FC_WND_SIZE=512
UCX_DC_MLX5_FC_HARD_THRESH=0.250
UCX_DC_MLX5_FENCE=auto
UCX_DC_MLX5_MAX_GET_ZCOPY=auto
UCX_DC_MLX5_TX_NUM_GET_BYTES=inf
UCX_DC_MLX5_TX_POLL_ALWAYS=n
UCX_DC_MLX5_ECE=0
UCX_DC_MLX5_DM_SIZE=2K
UCX_DC_MLX5_DM_COUNT=1
UCX_DC_MLX5_MMIO_MODE=auto
UCX_DC_MLX5_AR_ENABLE=auto
UCX_DC_MLX5_TX_CQE_ZIP_ENABLE=n
UCX_DC_MLX5_RX_CQE_ZIP_ENABLE=n
UCX_DC_MLX5_TX_MAX_BB=inf
UCX_DC_MLX5_TM_ENABLE=n
UCX_DC_MLX5_TM_LIST_SIZE=1024
UCX_DC_MLX5_TM_SEG_SIZE=48K
UCX_DC_MLX5_TM_MP_SRQ_ENABLE=try
UCX_DC_MLX5_TM_MP_NUM_STRIDES=8
UCX_DC_MLX5_EXP_BACKOFF=0
UCX_DC_MLX5_SRQ_TOPO=list
UCX_DC_MLX5_LOG_ACK_REQ_FREQ=8
UCX_DC_MLX5_RX_QUEUE_LEN_INIT=128
UCX_DC_MLX5_NUM_DCI=8
UCX_DC_MLX5_TX_POLICY=dcs_quota
UCX_DC_MLX5_LAG_PORT_AFFINITY=auto
UCX_DC_MLX5_DCI_FULL_HANDSHAKE=n
UCX_DC_MLX5_DCT_PORT_AFFINITY=default
UCX_DC_MLX5_DCI_KA_FULL_HANDSHAKE=n
UCX_DC_MLX5_DCT_FULL_HANDSHAKE=n
UCX_DC_MLX5_RAND_DCI_SEED=0
UCX_DC_MLX5_QUOTA=32
UCX_DC_MLX5_FC_HARD_REQ_TIMEOUT=5000000.00us
UCX_DC_MLX5_NUM_DCI_CHANNELS=8
UCX_DC_MLX5_COMPACT_AV=y
UCX_RC_VERBS_ALLOC=huge,thp,md,mmap,heap
UCX_RC_VERBS_FAILURE=DIAG
UCX_RC_VERBS_MAX_NUM_EPS=256
UCX_RC_VERBS_SEG_SIZE=8256
UCX_RC_VERBS_TX_QUEUE_LEN=256
UCX_RC_VERBS_TX_MAX_BATCH=16
UCX_RC_VERBS_TX_MAX_POLL=16
UCX_RC_VERBS_TX_MIN_INLINE=64
UCX_RC_VERBS_TX_INLINE_RESP=64
UCX_RC_VERBS_TX_MIN_SGE=5
UCX_RC_VERBS_TX_MAX_BUFS=-1
UCX_RC_VERBS_TX_BUFS_GROW=1024
UCX_RC_VERBS_TX_MAX_CHUNK_SIZE=128M
UCX_RC_VERBS_TX_GROW_FACTOR=1.000
UCX_RC_VERBS_RX_QUEUE_LEN=4095
UCX_RC_VERBS_RX_MAX_BATCH=16
UCX_RC_VERBS_RX_MAX_POLL=16
UCX_RC_VERBS_RX_INLINE=64
UCX_RC_VERBS_RX_MAX_BUFS=-1
UCX_RC_VERBS_RX_BUFS_GROW=0
UCX_RC_VERBS_RX_MAX_CHUNK_SIZE=128M
UCX_RC_VERBS_RX_GROW_FACTOR=1.000
UCX_RC_VERBS_ADDR_TYPE=auto
UCX_RC_VERBS_IS_GLOBAL=n
UCX_RC_VERBS_SL=auto
UCX_RC_VERBS_TRAFFIC_CLASS=auto
UCX_RC_VERBS_HOP_LIMIT=255
UCX_RC_VERBS_NUM_PATHS=auto
UCX_RC_VERBS_ROCE_LOCAL_SUBNET=n
UCX_RC_VERBS_ROCE_SUBNET_PREFIX_LEN=auto
UCX_RC_VERBS_ROCE_PATH_FACTOR=1
UCX_RC_VERBS_LID_PATH_BITS=0
UCX_RC_VERBS_PKEY=auto
UCX_RC_VERBS_PATH_MTU=default
UCX_RC_VERBS_COUNTER_SET_ID=auto
UCX_RC_VERBS_MAX_RD_ATOMIC=auto
UCX_RC_VERBS_TIMEOUT=1000000.00us
UCX_RC_VERBS_RETRY_COUNT=7
UCX_RC_VERBS_RNR_TIMEOUT=1000.00us
UCX_RC_VERBS_RNR_RETRY_COUNT=7
UCX_RC_VERBS_FC_ENABLE=y
UCX_RC_VERBS_FC_WND_SIZE=512
UCX_RC_VERBS_FC_HARD_THRESH=0.250
UCX_RC_VERBS_FENCE=auto
UCX_RC_VERBS_MAX_GET_ZCOPY=auto
UCX_RC_VERBS_TX_NUM_GET_BYTES=inf
UCX_RC_VERBS_TX_POLL_ALWAYS=n
UCX_RC_VERBS_ECE=0
UCX_RC_VERBS_FC_SOFT_THRESH=0.500
UCX_RC_VERBS_TX_CQ_MODERATION=64
UCX_RC_VERBS_TX_CQ_LEN=4096
UCX_RC_VERBS_MAX_AM_HDR=128
UCX_RC_VERBS_TX_MAX_WR=inf
UCX_RC_VERBS_FLUSH_MODE=auto
UCX_RC_MLX5_ALLOC=huge,thp,md,mmap,heap
UCX_RC_MLX5_FAILURE=DIAG
UCX_RC_MLX5_MAX_NUM_EPS=256
UCX_RC_MLX5_SEG_SIZE=8256
UCX_RC_MLX5_TX_QUEUE_LEN=256
UCX_RC_MLX5_TX_MAX_BATCH=16
UCX_RC_MLX5_TX_MAX_POLL=16
UCX_RC_MLX5_TX_MIN_INLINE=64
UCX_RC_MLX5_TX_INLINE_RESP=64
UCX_RC_MLX5_TX_MIN_SGE=5
UCX_RC_MLX5_TX_MAX_BUFS=-1
UCX_RC_MLX5_TX_BUFS_GROW=1024
UCX_RC_MLX5_TX_MAX_CHUNK_SIZE=128M
UCX_RC_MLX5_TX_GROW_FACTOR=1.000
UCX_RC_MLX5_RX_QUEUE_LEN=4095
UCX_RC_MLX5_RX_MAX_BATCH=16
UCX_RC_MLX5_RX_MAX_POLL=16
UCX_RC_MLX5_RX_INLINE=64
UCX_RC_MLX5_RX_MAX_BUFS=-1
UCX_RC_MLX5_RX_BUFS_GROW=0
UCX_RC_MLX5_RX_MAX_CHUNK_SIZE=128M
UCX_RC_MLX5_RX_GROW_FACTOR=1.000
UCX_RC_MLX5_ADDR_TYPE=auto
UCX_RC_MLX5_IS_GLOBAL=n
UCX_RC_MLX5_SL=auto
UCX_RC_MLX5_TRAFFIC_CLASS=auto
UCX_RC_MLX5_HOP_LIMIT=255
UCX_RC_MLX5_NUM_PATHS=auto
UCX_RC_MLX5_ROCE_LOCAL_SUBNET=n
UCX_RC_MLX5_ROCE_SUBNET_PREFIX_LEN=auto
UCX_RC_MLX5_ROCE_PATH_FACTOR=1
UCX_RC_MLX5_LID_PATH_BITS=0
UCX_RC_MLX5_PKEY=auto
UCX_RC_MLX5_PATH_MTU=default
UCX_RC_MLX5_COUNTER_SET_ID=auto
UCX_RC_MLX5_MAX_RD_ATOMIC=auto
UCX_RC_MLX5_TIMEOUT=1000000.00us
UCX_RC_MLX5_RETRY_COUNT=7
UCX_RC_MLX5_RNR_TIMEOUT=1000.00us
UCX_RC_MLX5_RNR_RETRY_COUNT=7
UCX_RC_MLX5_FC_ENABLE=y
UCX_RC_MLX5_FC_WND_SIZE=512
UCX_RC_MLX5_FC_HARD_THRESH=0.250
UCX_RC_MLX5_FENCE=auto
UCX_RC_MLX5_MAX_GET_ZCOPY=auto
UCX_RC_MLX5_TX_NUM_GET_BYTES=inf
UCX_RC_MLX5_TX_POLL_ALWAYS=n
UCX_RC_MLX5_ECE=0
UCX_RC_MLX5_FC_SOFT_THRESH=0.500
UCX_RC_MLX5_TX_CQ_MODERATION=64
UCX_RC_MLX5_TX_CQ_LEN=4096
UCX_RC_MLX5_DM_SIZE=2K
UCX_RC_MLX5_DM_COUNT=1
UCX_RC_MLX5_MMIO_MODE=auto
UCX_RC_MLX5_AR_ENABLE=auto
UCX_RC_MLX5_TX_CQE_ZIP_ENABLE=n
UCX_RC_MLX5_RX_CQE_ZIP_ENABLE=n
UCX_RC_MLX5_TX_MAX_BB=inf
UCX_RC_MLX5_TM_ENABLE=n
UCX_RC_MLX5_TM_LIST_SIZE=1024
UCX_RC_MLX5_TM_SEG_SIZE=48K
UCX_RC_MLX5_TM_MP_SRQ_ENABLE=try
UCX_RC_MLX5_TM_MP_NUM_STRIDES=8
UCX_RC_MLX5_EXP_BACKOFF=0
UCX_RC_MLX5_SRQ_TOPO=cyclic,cyclic_emulated
UCX_RC_MLX5_LOG_ACK_REQ_FREQ=8
UCX_UD_VERBS_ALLOC=huge,thp,md,mmap,heap
UCX_UD_VERBS_FAILURE=DIAG
UCX_UD_VERBS_MAX_NUM_EPS=inf
UCX_UD_VERBS_SEG_SIZE=8K
UCX_UD_VERBS_TX_QUEUE_LEN=256
UCX_UD_VERBS_TX_MAX_BATCH=16
UCX_UD_VERBS_TX_MAX_POLL=16
UCX_UD_VERBS_TX_MIN_INLINE=64
UCX_UD_VERBS_TX_INLINE_RESP=0
UCX_UD_VERBS_TX_MIN_SGE=5
UCX_UD_VERBS_TX_MAX_BUFS=-1
UCX_UD_VERBS_TX_BUFS_GROW=1024
UCX_UD_VERBS_TX_MAX_CHUNK_SIZE=128M
UCX_UD_VERBS_TX_GROW_FACTOR=1.000
UCX_UD_VERBS_RX_QUEUE_LEN=4096
UCX_UD_VERBS_RX_MAX_BATCH=16
UCX_UD_VERBS_RX_MAX_POLL=16
UCX_UD_VERBS_RX_INLINE=0
UCX_UD_VERBS_RX_MAX_BUFS=-1
UCX_UD_VERBS_RX_BUFS_GROW=0
UCX_UD_VERBS_RX_MAX_CHUNK_SIZE=128M
UCX_UD_VERBS_RX_GROW_FACTOR=1.000
UCX_UD_VERBS_ADDR_TYPE=auto
UCX_UD_VERBS_IS_GLOBAL=n
UCX_UD_VERBS_SL=auto
UCX_UD_VERBS_TRAFFIC_CLASS=auto
UCX_UD_VERBS_HOP_LIMIT=255
UCX_UD_VERBS_NUM_PATHS=auto
UCX_UD_VERBS_ROCE_LOCAL_SUBNET=n
UCX_UD_VERBS_ROCE_SUBNET_PREFIX_LEN=auto
UCX_UD_VERBS_ROCE_PATH_FACTOR=1
UCX_UD_VERBS_LID_PATH_BITS=0
UCX_UD_VERBS_PKEY=auto
UCX_UD_VERBS_PATH_MTU=default
UCX_UD_VERBS_COUNTER_SET_ID=auto
UCX_UD_VERBS_RX_QUEUE_LEN_INIT=128
UCX_UD_VERBS_LINGER_TIMEOUT=300000000.00us
UCX_UD_VERBS_TIMEOUT=30000000.00us
UCX_UD_VERBS_TIMER_TICK=10000.00us
UCX_UD_VERBS_TIMER_BACKOFF=2.000
UCX_UD_VERBS_ASYNC_TIMER_TICK=100000.00us
UCX_UD_VERBS_MIN_POKE_TIME=250000.00us
UCX_UD_VERBS_ETH_DGID_CHECK=y
UCX_UD_VERBS_MAX_WINDOW=1025
UCX_UD_VERBS_RX_ASYNC_MAX_POLL=64
UCX_UD_MLX5_ALLOC=huge,thp,md,mmap,heap
UCX_UD_MLX5_FAILURE=DIAG
UCX_UD_MLX5_MAX_NUM_EPS=inf
UCX_UD_MLX5_SEG_SIZE=8K
UCX_UD_MLX5_TX_QUEUE_LEN=256
UCX_UD_MLX5_TX_MAX_BATCH=16
UCX_UD_MLX5_TX_MAX_POLL=16
UCX_UD_MLX5_TX_MIN_INLINE=64
UCX_UD_MLX5_TX_INLINE_RESP=0
UCX_UD_MLX5_TX_MIN_SGE=5
UCX_UD_MLX5_TX_MAX_BUFS=-1
UCX_UD_MLX5_TX_BUFS_GROW=1024
UCX_UD_MLX5_TX_MAX_CHUNK_SIZE=128M
UCX_UD_MLX5_TX_GROW_FACTOR=1.000
UCX_UD_MLX5_RX_QUEUE_LEN=4096
UCX_UD_MLX5_RX_MAX_BATCH=16
UCX_UD_MLX5_RX_MAX_POLL=16
UCX_UD_MLX5_RX_INLINE=0
UCX_UD_MLX5_RX_MAX_BUFS=-1
UCX_UD_MLX5_RX_BUFS_GROW=0
UCX_UD_MLX5_RX_MAX_CHUNK_SIZE=128M
UCX_UD_MLX5_RX_GROW_FACTOR=1.000
UCX_UD_MLX5_ADDR_TYPE=auto
UCX_UD_MLX5_IS_GLOBAL=n
UCX_UD_MLX5_SL=auto
UCX_UD_MLX5_TRAFFIC_CLASS=auto
UCX_UD_MLX5_HOP_LIMIT=255
UCX_UD_MLX5_NUM_PATHS=auto
UCX_UD_MLX5_ROCE_LOCAL_SUBNET=n
UCX_UD_MLX5_ROCE_SUBNET_PREFIX_LEN=auto
UCX_UD_MLX5_ROCE_PATH_FACTOR=1
UCX_UD_MLX5_LID_PATH_BITS=0
UCX_UD_MLX5_PKEY=auto
UCX_UD_MLX5_PATH_MTU=default
UCX_UD_MLX5_COUNTER_SET_ID=auto
UCX_UD_MLX5_RX_QUEUE_LEN_INIT=128
UCX_UD_MLX5_LINGER_TIMEOUT=300000000.00us
UCX_UD_MLX5_TIMEOUT=30000000.00us
UCX_UD_MLX5_TIMER_TICK=10000.00us
UCX_UD_MLX5_TIMER_BACKOFF=2.000
UCX_UD_MLX5_ASYNC_TIMER_TICK=100000.00us
UCX_UD_MLX5_MIN_POKE_TIME=250000.00us
UCX_UD_MLX5_ETH_DGID_CHECK=y
UCX_UD_MLX5_MAX_WINDOW=1025
UCX_UD_MLX5_RX_ASYNC_MAX_POLL=64
UCX_UD_MLX5_DM_SIZE=2K
UCX_UD_MLX5_DM_COUNT=1
UCX_UD_MLX5_MMIO_MODE=auto
UCX_UD_MLX5_AR_ENABLE=auto
UCX_UD_MLX5_TX_CQE_ZIP_ENABLE=n
UCX_UD_MLX5_RX_CQE_ZIP_ENABLE=n
UCX_UD_MLX5_COMPACT_AV=y
UCX_RDMA_CM_FAILURE=DIAG
UCX_RDMA_CM_REUSEADDR=n
UCX_RDMA_CM_SOURCE_ADDRESS=
UCX_RDMA_CM_TIMEOUT=10000000.00us
UCX_RDMA_CM_RESERVED_QPN=try
UCX_CMA_MEMORY_INVALIDATE=n
UCX_CMA_ALLOC=huge,thp,mmap,heap
UCX_CMA_FAILURE=DIAG
UCX_CMA_MAX_NUM_EPS=inf
UCX_CMA_BW=11145.00MBps
UCX_CMA_MAX_IOV=16
UCX_CMA_SEG_SIZE=512K
UCX_CMA_TX_QUOTA=1
UCX_CMA_TX_MAX_BUFS=-1
UCX_CMA_TX_BUFS_GROW=8
UCX_CMA_TX_MAX_CHUNK_SIZE=128M
UCX_CMA_TX_GROW_FACTOR=1.000
The text was updated successfully, but these errors were encountered: