[Z Notebook] Performance testing for REST

##Performance testing for REST

QBit is similar to an actor model. QBit is actually an active object model. The QBit services currently sit behind a queue. Calls are queued to the service, and return messages are sent on a return queue. The programming model looks similar to Spring MVC, but it is actually a different model. However, it is still important to show how QBit does against Spring MVC / Spring Boot. This comparison is not a true apples to apples comparison. It is not meant to upset anyone in the Spring MVC community.

Server specifications

For this test, we are using a c3.8xlarge, which is an Amazon EC2 instance with 32 CPUs and 60 GB of memory.

Steps to reproduce this test

Step 1: Create a server instance in Amazon EC2

Download the pem file for the server. For this experiment use Amazon Linux which is a CentOS fork.

Step 2: Change the file permissions on your pem file

$ chmod 400 loadtestingqbit.pem 

Step 3: Login to new server instance

$ ssh -i loadtestingqbit.pem [email protected]

Step 4: Install Java 8

$ sudo yum install  java-1.8.0-openjdk-devel -y

Edit .bashrc and set Java home to Java 8

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk

Then

source ~/.bashrc

Step 5: Install git

$  sudo yum install -y git

Step 6: Install maven

$ wget http://mirror.symnds.com/software/Apache/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.tar.gz
$ tar zxvf apache-maven-3.2.5-bin.tar.gz 
$ sudo mv apache-maven-3.2.5 /usr/local/
$ sudo ln -s /usr/local/apache-maven-3.2.5/ /usr/local/maven

Next add the env variables to your ~/.bashrc file

export M2_HOME=/usr/local/maven
export M2=$M2_HOME/bin 
export PATH=$M2:$PATH

Then

source ~/.bashrc

Lastly

$ mvn -version
Apache Maven 3.2.5 (12a6b3acb947671f09b81f49094c53f426d8cea1; 2014-12-14T17:29:23+00:00)
Maven home: /usr/local/maven
Java version: 1.8.0_31, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.31-2.b13.5.amzn1.x86_64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.14.27-25.47.amzn1.x86_64", arch: "amd64", family: "unix"

If you have done everything correctly, maven should be pointing to JDK 1.8.

Step 7: Build boon

QBit relies on Boon. Both QBit and Boon are in the maven public repo, but currently the latest versions for this perf test are snapshots.

$ git clone https://github.com/boonproject/boon.git
$ cd boon
$ mvn clean install

Since this is the first time we are using maven on this box, it has to download the world. Welcome to Sparta.

With success you will see:

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Boon and subprojects ............................... SUCCESS [  9.871 s]
[INFO] boon ............................................... SUCCESS [ 22.579 s]
[INFO] bnsf-core .......................................... SUCCESS [  0.484 s]
[INFO] etcd-bundle ........................................ SUCCESS [  0.006 s]
[INFO] ETCD Client ........................................ SUCCESS [ 44.705 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:18 min
[INFO] Finished at: 2015-02-02T09:34:37+00:00
[INFO] Final Memory: 33M/592M

If there are build failures, try running with -DskipTests and then email me a nasty gram.

Now the boon snapshot jar will be in the local maven repo.

Step 8 Download and install gradle

$ cd ~
$ wget https://services.gradle.org/distributions/gradle-2.2.1-all.zip
$ unzip gradle-2.2.1-all.zip 
$ sudo mv gradle-2.2.1 /usr/local/
$ sudo ln -s /usr/local/gradle-2.2.1/ /usr/local/gradle

Next add the env variables to your ~/.bashrc file

export GRADLE_HOME=/usr/local/gradle
export GRADLE=$GRADLE_HOME/bin 
export PATH=$GRADLE:$PATH

Then

source ~/.bashrc

Lastly

$ gradle -version

------------------------------------------------------------
Gradle 2.2.1
------------------------------------------------------------

Build time:   2014-11-24 09:45:35 UTC
Build number: none
Revision:     6fcb59c06f43a4e6b1bcb401f7686a8601a1fb4a

Groovy:       2.3.6
Ant:          Apache Ant(TM) version 1.9.3 compiled on December 23 2013
JVM:          1.8.0_31 (Oracle Corporation 25.31-b07)
OS:           Linux 3.14.27-25.47.amzn1.x86_64 amd64

Step 9 Build qbit

$ git clone https://github.com/advantageous/qbit.git
$ cd qbit
$ gradle clean build install -Dskip.tests=true

There are quite a few integration tests so it is best to skip them.

Test the client and server are working

$ cd qbit-examples/qbit-examples-standalone/
$ gradle runRestServer

In another terminal

$ ssh -i loadtestingqbit.pem [email protected]
$ cd qbit/qbit-examples/qbit-examples-standalone/
$ gradle runRestClient

The rest client sends 500K requests.

#Step 11 Installing wrk

Follow the instructions here

https://github.com/wg/wrk/wiki/Installing-Wrk-on-Linux/_history

sudo  yum groupinstall -y 'Development Tools'
sudo yum install  -y openssl-devel
sudo yum install -y git
sudo yum install -y lua-devel
git clone https://github.com/wg/wrk.git
cd wrk
make

#Step 12 Use wrk to test spring boot example

$ pwd
/home/ec2-user/qbit

$ cd qbit-examples/spring/spring-boot/

$ gradle run

In another terminal.

$ ./wrk -c 3000 -d 10s http://localhost:8080/services/myservice/ping -H "X_USER_ID: RICK"   --timeout 100000s -t 8
Running 10s test @ http://localhost:8080/services/myservice/ping

#Step 13 Install wrk on another instance. Run it against port 8080 and port 6060.

Very preliminary results.

I have not optimized the TCP stacks yet.

From one EC2 instance to another

Spring Boot

$ ./wrk -c 1000 -d 30s http://10.37.1.103:8080/services/myservice/ping -H "X_USER_ID: RICK"   --timeout 100000s -t 8
Running 30s test @ http://10.37.1.103:8080/services/myservice/ping
 8 threads and 1000 connections
 Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency    19.97ms   35.74ms   1.88s    97.43%
   Req/Sec     7.01k     1.86k   10.27k    65.19%
 1659865 requests in 30.00s, 300.81MB read
Requests/sec:  55332.94
Transfer/sec:     10.03MB

QBit

 ./wrk -c 1000 -d 30s http://10.37.1.103:6060/services/myservice/ping -H "X_USER_ID: RICK"   --timeout 100000s -t 8
Running 30s test @ http://10.37.1.103:6060/services/myservice/ping
 8 threads and 1000 connections
 Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency    14.59ms   15.81ms   1.74s    99.67%
   Req/Sec     9.13k     1.61k   12.52k    76.18%
 2138507 requests in 30.00s, 177.43MB read
Requests/sec:  71289.16
Transfer/sec:      5.91MB

This is not a real test. This is a test that qbit should lose. It wins but not by much. More tests are coming.

Code

To run the examples and see the code go here:

https://github.com/advantageous/qbit/tree/master/qbit-examples/spring/spring-and-qbit-cpu-bench
https://github.com/advantageous/qbit/tree/master/qbit-examples/qbit-examples-standalone

Follow the instructions for running a Java app from gradle. There are extra tasks to run Qbit clients, servers, etc.

To run HTTP pipeline tests see:

To run a 5x pipeline you need this Lua script.

init = function(args)
   wrk.init(args)

   local r = {}
   r[1] = wrk.format(nil, "/services/myservice/ping")
   r[2] = wrk.format(nil, "/services/myservice/ping")
   r[3] = wrk.format(nil, "/services/myservice/ping")
   r[4] = wrk.format(nil, "/services/myservice/ping")
   r[5] = wrk.format(nil, "/services/myservice/ping")

   req = table.concat(r)
end

request = function()
   return req
end

Scripts to lua are passed with -s option

 ./wrk -d 60s -c 1000 -s pipleine2.lua  "http://54.184.83.95:8080" --timeout 100s -t 8 --timeout 10s

To run the CPU intensive test use the following pipeline file

init = function(args)
   wrk.init(args)

   local r = {}

   r[1] = wrk.format("GET", "/services/myservice/addkey/?key=1&value=mom")
   r[2] = wrk.format("GET", "/services/myservice/addkey/?key=2&value=mom")
   r[3] = wrk.format("GET", "/services/myservice/addkey/?key=3&value=mom")
   r[4] = wrk.format("GET", "/services/myservice/addkey/?key=4&value=mom")
   r[5] = wrk.format("GET", "/services/myservice/addkey/?key=5&value=mom")
   r[6] = wrk.format("GET", "/services/myservice/addkey/?key=6&value=mom")
   r[7] = wrk.format("GET", "/services/myservice/addkey/?key=7&value=mom")
   r[8] = wrk.format("GET", "/services/myservice/addkey/?key=8&value=mom")
   r[9] = wrk.format("GET", "/services/myservice/addkey/?key=9&value=mom")
   r[10] = wrk.format("GET", "/services/myservice/addkey/?key=0&value=mom")

   req = table.concat(r)
end

request = function()
   return req
end

##Notes on perf testing

Testing is tier to tier between two EC2 c3.8xlarge 10 GBE (1.7 GBE effective) running Amazon EC2.

###Summary: QBit HTTP is 2x faster than Spring Boot / Jetty for the CPU intensive benchmark.

QBit WebSocket is 10x faster than Spring Boot / Jetty for the CPU intensive benchmark.

#Test description PING: The non CPU intensive test with/without HTTP pipeline and HTTP Websocket. The code is a simple REST ping implemented with Spring REST style annotations in both Spring Boot and QBit. QBit servers all requests from one service thread. Spring Boot is using a thread pool to handle requests managed by Jetty, Tomcat or Undertow. We used Undertow, Jetty, and Tomcat plugins for Spring Boot. Since this is not CPU intensive test and there is no thread sync, Spring Boot should win easily, but does not. All implementations were warmed up before the test.

#Test description CPU Intensive: This is for a CPU intensive test that simulates a periodic I/O operation.

QBit was twice as fast for this test due to the queue programming model (similar to Rust channels, Go channels, Akka messaging, etc.) versus multi-thread for apple to apples test communication, and 10x faster for WebSocket communication.

From earlier tests, one could tell that Jetty has a faster HTTP pipeline implementation than Vertx, which was negated by this test due to the CPU intensive nature and the periodic IO simulation. QBit will continue to enqueue requests in batches, and do protocol parsing while it is doing IO and CPU operations in other threads.

The IO simulation was every five requests, we check to see if a five second time period has past every five alls and if it has we sleep for a period of 200 ms. The QBit version does the check on queue limit and queue empty. A more fair test to QBit would have been to set a timer to call the IO operation every five seconds in addition to the check every 5 which is what QBit is doing. But the overhead of thread contention on a five second timer, would be so small that I left it out.