Home

SparkRDMA is a high-performance, scalable and efficient ShuffleManager plugin for Apache Spark. It utilizes RDMA (Remote Direct Memory Access) technology to reduce CPU cycles needed for Shuffle data transfers. It reduces memory usage by reusing memory for transfers instead of copying data multiple times down the traditional TCP-stack.

RDMA is supported on various types of networks, such as traditional Ethernet with RoCE (RDMA over Converged Ethernet), Infiniband and more.

SparkRDMA is build to provide the best performance out of the box. However, if one wishes to squeeze the most out of its installation, we provide multiple configuration properties to precisely tune SparkRDMA on a per-job basis. For more information on how to tune a system, please refer to guides offered in this wiki:

• Description of SparkRDMA's configuration properties
• Recommended performance tuning steps for Mellanox Network Adapters
• General performance tips
• Troubleshooting guide