HERD

A Highly Efficient key-value system for RDMA. The original paper describing HERD appeared in SIGCOMM'14.

Important: This code is not maintained anymore. Please use the new and improved (up to 83% faster) version of HERD from our USENIX ATC 16 paper. The code is available here.

This version of HERD has been tested for the following configuration:

1. Software

• OS: Ubuntu 12.04 (kernel 3.2.0)
• RDMA drivers: mlx4 from MLNX OFED 2.2. I suggest using the MLNX OFED version for Ubuntu 12.04.

2. Hardware

• RNICs:
  • ConnectX-3 353A (InfiniBand)
  • ConnectX-3 313A (RoCE)
  • ConnectX-3 354A (InfiniBand)

Initial setup:

• I assume that the machines are named: node-i.RDMA.fawn.apt.emulab.net starting from i = 1.

  • The experiment requires at least (1 + (NUM_CLIENTS / num_processes)) machines. node-1 is the server machine. NUM_CLIENTS is the total number of client processes, defined in common.h. num_processes is the number of client processes per machine, defined in run-machine.sh.

  • To modify HERD for your machine names:

    • Make appropriate changes in kill-remote.sh and run-servers.sh.
    • Change the server's machine name in the servers file. Clients use this file to connect to server processes.

  • Make sure that ports 5500 to 5515 are available on the server machine. Server process i listens for clients on port 5500 + i.

• Execute the following commands at the server machine:

cd ~
git clone https://github.com/anujkaliaiitd/HERD.git
export PATH=~/HERD/scripts:$PATH
cd HERD
sudo ./shm-init.sh	# Increase shmmax and shmall
sudo hugepages-create.sh 0 4096		# Create hugepages on socket 0. Do for all sockets.

• Mount the HERD folder on all client machines via NFS.

Quick start:

• Run make on the server machine to build the executables.

• To run the clients automatically along with the server:

# At node-1 (server)
./run-servers.sh

• If you do not want to run clients automatically from the server, delete the 2nd loop from run-servers.sh. Then:

# At node-1 (server)
./run-servers.sh
# At node-2 (client 0)
./run-machine.sh 0
# At node-i (client i - 2)
./run-machine.sh (i - 2)

• To kill the server processes, run local-kill.sh at the server machine. To kill the client processes remotely, run kill-remote.sh at the server machine.

License

Copyright 2014 Carnegie Mellon University

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.