Benchmark Cuba

[kathgironpe]

I bet the results won't differ greatly from Sinatra but Cuba's worth benchmarking for those who are interested to know.

Like many, I hesitate to use Elixir. I think Golang is worth a try for most projects.

[chrismccord]

@mroth accepts pull requests with code and clear instructions on how to run the code/benchmarks. I'm sure he'd be happy to include if you'd like to put something together.

Like many, I hesitate to use Elixir. I think Golang is worth a try for most projects.

In this case we started out comparing web "frameworks" - Rails and Phoenix, and people started submitting their web library of choice. Go can be a fine choice, but it has no solution that competes on features which is what this repo was started to compare. In any case, I'd love to convince you to let go of that hesitation :)

[kathgironpe]

@chrismccord I've created a PR.

Re: Elixir & Phoenix. I am convinced it's a good option based on the benchmarks but I am uncertain if it will work for current needs. It's not exactly a simple web app. I am learning both Go and Elixir. My hesitation to use Elixir for a project (I need to ship quickly) has a lot to do with the fact that I got so used to doing everything Ruby. It's been my go-to language since 2008. Unlike most, I am using Cuba and Sinatra. I am greatly unhappy with Ruby on Rails.

While I am aware Ruby is one of the worst languages for tasks like networking, I can make non-trivial sites fast with it. Ruby's not so bad but as this repo shows, it's incredibly slow compared to Elixir and Go. Been learning Go for over a year now. I've yet to find time to read Elixir books. Found out you wrote one.

[mroth]

The code for this is merged in #15 but keeping this issue open as a reminder to self to benchmark with it during the next found.

[jaydanielian]

I just wanted to add that I did run the benchmark locally with Cuba, and I ran it in a few different combinations. I remember a previous benchmark mentioning that Cuba and jruby/torqbox was actually the fastest ruby/rack based framework. We have been running Cuba and jruby-1.7.17 in production for a REST base API and it has held up very well.

TL;DR => Cuba + new jruby-9.0.4.0 is the fastest benchmark I executed for ruby/rack implementations. There were some socket errors with each jRuby execution however.

My test machine locally is a mid 2012 2.6 Ghz MBP with 16GB. Here is my test output (I ran each test in each configuration about 5 times, so these are the results of the output after some warmup and leveling off). The biggest difference was in the jRuby executions, as the JVM really needs time to warm up. I wanted to benchmark Cuba and MRI 2.2 as that is what was output in the other benchmarks. Selfishly, I wanted to run my current configuration which is cuba/jruby-1.17 and torqbox. Lastly, I wanted to see what the latest version of jRuby and the latest torquebox server could do.

***** running against ruby-2.2.2 with cuba and puma
$ wrk -t4 -c100 -d30S --timeout 2000 "http://127.0.0.1:9292/showdown"
Running 30s test @ http://127.0.0.1:9292/showdown
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 7.26ms 2.66ms 43.83ms 74.05%
Req/Sec 2.05k 554.05 4.47k 63.58%
239994 requests in 30.00s, 473.54MB read
Requests/sec: 8000.51
Transfer/sec: 15.79MB

***** running against jruby-1.7.17 with cuba and torqbox
$ wrk -t4 -c100 -d30S --timeout 2000 "http://127.0.0.1:9292/showdown"
Running 30s test @ http://127.0.0.1:9292/showdown
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 8.49ms 9.16ms 148.83ms 97.87%
Req/Sec 3.26k 602.99 8.16k 74.98%
381547 requests in 30.00s, 781.60MB read
Socket errors: connect 0, read 48, write 0, timeout 0
Requests/sec: 12717.95
Transfer/sec: 26.05MB

***** running against jruby-9.0.4.0 with cuba and torquebox-4.0.0.beta2
$ wrk -t4 -c100 -d30S --timeout 2000 "http://127.0.0.1:9292/showdown"
Running 30s test @ http://127.0.0.1:9292/showdown
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 7.65ms 9.31ms 128.14ms 99.18%
Req/Sec 3.66k 466.81 5.15k 82.50%
427357 requests in 30.00s, 0.85GB read
Socket errors: connect 0, read 43, write 3, timeout 0
Requests/sec: 14245.39
Transfer/sec: 29.18MB