Performance tests

[benlangfeld]

Executing this on my laptop I get this:

docker-compose run tests rake performance
Starting message_bus_postgres_1 ... done
Starting message_bus_redis_1    ... done
/usr/local/bin/ruby -e "ARGV.each{|f| load f}" spec/performance/publish.rb
Running publication benchmark with 10000 iterations on backends: [:postgres, :redis, :memory]

                                                                   user     system      total        real
postgres - publication only                                    2.880000   1.640000   4.520000 ( 26.512851)
redis - publication only                                       0.870000   0.690000   1.560000 (  3.890986)
memory - publication only                                      1.130000   0.000000   1.130000 (  1.142721)

postgres - subscription no trimming                            3.000000   1.970000   4.970000 ( 28.369772)
redis - subscription no trimming                               2.510000   1.180000   3.690000 (  6.072367)
memory - subscription no trimming                              2.900000   0.000000   2.900000 (  2.909922)

postgres - subscription with trimming                          3.620000   2.030000   5.650000 ( 28.796871)
redis - subscription with trimming                             2.050000   0.940000   2.990000 (  5.231705)
memory - subscription with trimming                            1.340000   0.000000   1.340000 (  1.346209)

[postgres - subscription no trimming]: 10000 messages sent, 10000 received, rate of 100.0%
[redis - subscription no trimming]: 10000 messages sent, 10000 received, rate of 100.0%
[memory - subscription no trimming]: 10000 messages sent, 10000 received, rate of 100.0%

[postgres - subscription with trimming]: 10000 messages sent, 10000 received, rate of 100.0%
[redis - subscription with trimming]: 10000 messages sent, 10000 received, rate of 100.0%
[memory - subscription with trimming]: 10000 messages sent, 10000 received, rate of 100.0%

[benlangfeld]

The last build (on Travis) on this branch on Ruby 2.5 looked like this:

/home/travis/.rvm/rubies/ruby-2.5.1/bin/ruby -e "ARGV.each{|f| load f}" spec/performance/publish.rb
Running publication benchmark with 10000 iterations on backends: [:memory, :redis, :postgres]
                                          user     system      total        real
memory - publication only             1.160000   0.000000   1.160000 (  1.158841)
redis - publication only              0.944000   0.168000   1.112000 (  1.537435)
postgres - publication only           2.068000   0.856000   2.924000 ( 10.642180)
memory - subscription no trimming     3.076000   0.000000   3.076000 (  3.073760)
redis - subscription no trimming      1.708000   0.232000   1.940000 (  2.242161)
postgres - subscription no trimming   2.552000   0.780000   3.332000 ( 15.507882)
memory - subscription with trimming   1.356000   0.000000   1.356000 (  1.356482)
redis - subscription with trimming    1.920000   0.300000   2.220000 (  2.531762)
postgres - subscription with trimming 2.860000   0.864000   3.724000 ( 11.353771)
[memory - subscription no trimming]: 10000 messages sent, 10000 received, rate of 100.0%
[redis - subscription no trimming]: 10000 messages sent, 10000 received, rate of 100.0%
[postgres - subscription no trimming]: 10000 messages sent, 10000 received, rate of 100.0%
[memory - subscription with trimming]: 10000 messages sent, 10000 received, rate of 100.0%
[redis - subscription with trimming]: 10000 messages sent, 10000 received, rate of 100.0%
[postgres - subscription with trimming]: 10000 messages sent, 10000 received, rate of 100.0%

Building #173 rebased on this branch yields this result:

/home/travis/.rvm/rubies/ruby-2.4.4/bin/ruby -e "ARGV.each{|f| load f}" spec/performance/publish.rb
Running publication benchmark with 10000 iterations on backends: [:memory, :redis_streams, :redis, :postgres]
                                               user     system      total        real
memory - publication only                  1.270000   0.000000   1.270000 (  1.273156)
redis_streams - publication only           1.220000   0.220000   1.440000 (  2.148001)
redis - publication only                   1.180000   0.260000   1.440000 (  1.947661)
postgres - publication only                2.380000   0.780000   3.160000 ( 10.946480)
memory - subscription no trimming          3.260000   0.000000   3.260000 (  3.265381)
redis_streams - subscription no trimming   3.550000   0.670000   4.220000 (  4.318782)
redis - subscription no trimming           2.110000   0.390000   2.500000 (  2.622871)
postgres - subscription no trimming        2.980000   0.780000   3.760000 ( 16.510002)
memory - subscription with trimming        1.450000   0.000000   1.450000 (  1.453217)
redis_streams - subscription with trimming 3.460000   0.760000   4.220000 (  4.366596)
redis - subscription with trimming         2.260000   0.480000   2.740000 (  2.897497)
postgres - subscription with trimming      3.230000   1.040000   4.270000 ( 11.139309)
[memory - subscription no trimming]: 10000 messages sent, 10000 received, rate of 100.0%
[redis_streams - subscription no trimming]: 10000 messages sent, 10000 received, rate of 100.0%
[redis - subscription no trimming]: 10000 messages sent, 10000 received, rate of 100.0%
[postgres - subscription no trimming]: 10000 messages sent, 10000 received, rate of 100.0%
[memory - subscription with trimming]: 10000 messages sent, 10000 received, rate of 100.0%
[redis_streams - subscription with trimming]: 10000 messages sent, 10000 received, rate of 100.0%
[redis - subscription with trimming]: 10000 messages sent, 10000 received, rate of 100.0%
[postgres - subscription with trimming]: 10000 messages sent, 10000 received, rate of 100.0%

This shows that Redis Streams are marginally slower to publish to than Sorted Sets (though this result may not be statistically significant), while they are currently clearly slower to subscribe to, though it doesn't seem to matter if trimming happens or not. I suspect the main issue is the overhead in executing XREAD vs the streaming of results from Redis PubSub. I'll see if I can run the benchmarks against the version which uses Streams + Redis PubSub to evaluate this possibility.

[benlangfeld]

@SamSaffron What do you think of these benchmarks for their own sake, regardless of the Streams implementation?

[benlangfeld]

Here are the results for Streams + Redis PubSub:

/home/travis/.rvm/rubies/ruby-2.5.1/bin/ruby -e "ARGV.each{|f| load f}" spec/performance/publish.rb
Running publication benchmark with 10000 iterations on backends: [:memory, :redis_streams, :redis, :postgres]
                                                                   user     system      total        real
memory - publication only                                      1.208000   0.008000   1.216000 (  1.216110)
redis_streams - publication only                               1.064000   0.208000   1.272000 (  1.696576)
redis - publication only                                       1.028000   0.180000   1.208000 (  1.589538)
postgres - publication only                                    2.296000   0.820000   3.116000 ( 11.218943)
memory - subscription no trimming                              3.236000   0.000000   3.236000 (  3.233522)
redis_streams - subscription no trimming                       1.716000   0.216000   1.932000 (  2.328350)
redis - subscription no trimming                               1.696000   0.284000   1.980000 (  2.301549)
postgres - subscription no trimming                            2.964000   0.888000   3.852000 ( 16.724460)
memory - subscription with trimming                            1.408000   0.000000   1.408000 (  1.404389)
redis_streams - subscription with trimming                     1.724000   0.232000   1.956000 (  2.374443)
redis - subscription with trimming                             1.780000   0.324000   2.104000 (  2.450212)
postgres - subscription with trimming                          3.032000   0.896000   3.928000 ( 10.525489)
[memory - subscription no trimming]: 10000 messages sent, 10000 received, rate of 100.0%
[redis_streams - subscription no trimming]: 10000 messages sent, 10000 received, rate of 100.0%
[redis - subscription no trimming]: 10000 messages sent, 10000 received, rate of 100.0%
[postgres - subscription no trimming]: 10000 messages sent, 10000 received, rate of 100.0%
[memory - subscription with trimming]: 10000 messages sent, 10000 received, rate of 100.0%
[redis_streams - subscription with trimming]: 10000 messages sent, 10000 received, rate of 100.0%
[redis - subscription with trimming]: 10000 messages sent, 10000 received, rate of 100.0%
[postgres - subscription with trimming]: 10000 messages sent, 10000 received, rate of 100.0%

This appears to confirm that the overhead is in usage of XREAD.

[jeremyevans]

These changes look good to me, and I think it's definitely valuable to have performance benchmarks. I don't know that I would go so far as to running them in the default rake task, but I don't have strong feelings either way.

[benlangfeld]

I included them in the default rake task for visibility. If they don’t get shoved in one’s face regularly, they’ll be very easy to forget about.

[SamSaffron]

Thanks I love having the perf tests but lets not have them as our default rake task. Feel free to merge the tests once you shift it to a dedicated task and add info to the readme. I think calling this out nicely in the readme is a way of showing this off. We can also look at adding this to: https://rubybench.org/ which would be super nice.

Regarding redis streams, I kind of feel it does not have the numbers to justify large changes to message bus at the moment, if it was 30% faster or somehow more reliable then I would be far more behind it but it is not faster so I am not sure what this adds at the moment outside of code churn that is very risky.

I think it is far better to focus on nicer diagnostics, seeing if we can squeeze more perf out of our current implementations a nice public web page and so on.

Stuff like service workers are also interesting cause a bunch of tabs can be multiplexed across a single connection to the db, but I just don't feel the amount of boat rocking is justified for a slower provider.

[SamSaffron]

also curious if @antirez has any feedback here. regardless I think it is pretty amazing how redis is so close performance wise to a pure memory provider.