A more robust JMH benchmarking set-up #963

gvsmirnov · 2014-03-13T20:31:52Z

Usage:

$ gradle benchmarks
$ java -jar rxjava-core/build/distributions/rxjava-core-0.17.2-SNAPSHOT-benchmarks.jar [...]

For example:

$ java -jar rxjava-core/build/distributions/rxjava-core-0.17.2-SNAPSHOT-benchmarks.jar -f 1 -wi 10 -i 10 -tu us -bm avgt

Benchmark                                  (size)   Mode   Samples         Mean   Mean error    Units
r.o.ObservableBenchmark.measureBaseline         1   avgt        10        0.003        0.000    us/op
r.o.ObservableBenchmark.measureBaseline      1024   avgt        10        2.764        0.051    us/op
r.o.ObservableBenchmark.measureBaseline   1048576   avgt        10     3104.088       49.586    us/op
r.o.ObservableBenchmark.measureMap              1   avgt        10        0.100        0.003    us/op
r.o.ObservableBenchmark.measureMap           1024   avgt        10        5.036        0.059    us/op
r.o.ObservableBenchmark.measureMap        1048576   avgt        10     6693.271      277.604    us/op

See #776 for more details.

benjchristensen · 2014-03-13T20:35:22Z

This looks awesome. I look forward to using this for all our perf testing.

A more robust JMH benchmarking set-up

headinthebox · 2014-03-13T21:02:28Z

+1

gvsmirnov · 2014-03-13T21:52:26Z

Thanks! I'm a bit worried about the failing build, though. I don't think my changes should have caused those tests to fail.

benjchristensen · 2014-03-13T22:44:40Z

I don't think my changes should have caused those tests to fail.

It doesn't look related.

benjchristensen · 2014-03-15T23:13:57Z

It looks like this change to build.gradle broke the Eclipse project import. Refreshing dependencies in Eclipse now loads the root project as a source folder and causes other dependency issues. Not yet sure why or how to fix it.

gvsmirnov · 2014-03-17T19:52:24Z

Ah. What a shame :( I Admit to not having actually tested it in Eclipse. Works fine in Command Line and in IDEA. Would you kindly point out the steps to reproduce this? Eclipse and gradle plugin versions and such?

benjchristensen · 2014-03-24T19:45:23Z

I've reproduced it with multiple versions of Eclipse (Juno, Kepler, Luna) using the Gradle IDE plugin available in the Eclipse Marketplace.

benjchristensen · 2014-03-26T21:41:07Z

benjchristensen · 2014-03-26T21:41:39Z

Version: Kepler Service Release 2
Build id: 20140224-0627

Gradle IDE 3.4.0.201310051517-RELEASE org.springsource.ide.eclipse.gradle.feature.feature.group GoPivotal, Inc.

benjchristensen · 2014-03-26T22:32:22Z

With the help of @quidryan I think we've got this solved. Pull request coming ...

benjchristensen · 2014-03-27T05:20:32Z

This is excellent ... thank you @gvsmirnov for your help on this!

Is there a way to measure object allocation in the benchmarks?

The best I've found is the HS_GC profiler, but it's not obvious how to use that to determine object allocations.

# Run progress: 0.00% complete, ETA 00:01:00
# VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/Home/jre/bin/java
# VM options: -Xmx512m -Dfile.encoding=UTF-8
# Fork: 1 of 1
# Warmup: 5 iterations, 1 s each
# Measurement: 5 iterations, 5 s each
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: rx.operators.OperatorSerializePerf.serializedSingleStream
# Parameters: (size = 1048576)
# Warmup Iteration   1: 91.687 ns/op
# Warmup Iteration   2: 72.425 ns/op
# Warmup Iteration   3: 66.230 ns/op
# Warmup Iteration   4: 66.658 ns/op
# Warmup Iteration   5: 67.470 ns/op
Iteration   1: 65.993 ns/op
      HS(GC) | difference: {sun.gc.collector.0.invocations=122, sun.gc.collector.0.lastEntryTime=4972697000, sun.gc.collector.0.lastExitTime=4972719000, sun.gc.collector.0.time=109162000, sun.gc.compressedclassspace.used=568, sun.gc.generation.0.space.0.used=50104784, sun.gc.generation.1.space.0.used=32768, sun.gc.metaspace.used=5232, sun.gc.policy.avgMinorIntervalTime=-1, sun.gc.policy.avgPromotedAvg=-439, sun.gc.policy.avgPromotedDev=-1478, sun.gc.policy.avgPromotedPaddedAvg=-4873, sun.gc.policy.avgSurvivedAvg=-3923, sun.gc.policy.avgSurvivedDev=2660, sun.gc.policy.avgSurvivedPaddedAvg=4056, sun.gc.policy.avgYoungLive=-3920, sun.gc.policy.liveSpace=-3936}
             |

Iteration   2: 65.861 ns/op
      HS(GC) | difference: {sun.gc.collector.0.invocations=123, sun.gc.collector.0.lastEntryTime=5002277000, sun.gc.collector.0.lastExitTime=5002144000, sun.gc.collector.0.time=105387000, sun.gc.compressedclassspace.used=568, sun.gc.generation.0.space.0.used=-50104784, sun.gc.generation.0.space.1.used=-65536, sun.gc.generation.0.space.2.used=65536, sun.gc.generation.1.space.0.used=46608, sun.gc.metaspace.used=6840, sun.gc.policy.avgSurvivedAvg=4832, sun.gc.policy.avgSurvivedDev=345, sun.gc.policy.avgSurvivedPaddedAvg=5868, sun.gc.policy.avgYoungLive=4832, sun.gc.policy.liveSpace=4832, sun.gc.tlab.maxSlowWaste=2, sun.gc.tlab.slowWaste=2}
             |

Iteration   3: 66.177 ns/op
      HS(GC) | difference: {sun.gc.collector.0.invocations=123, sun.gc.collector.0.lastEntryTime=5023959000, sun.gc.collector.0.lastExitTime=5023980000, sun.gc.collector.0.time=104882000, sun.gc.generation.0.space.1.used=65536, sun.gc.generation.0.space.2.used=-65536, sun.gc.generation.1.space.0.used=24576, sun.gc.metaspace.used=648, sun.gc.policy.avgMinorIntervalTime=2, sun.gc.policy.avgSurvivedAvg=-1588, sun.gc.policy.avgSurvivedDev=-3708, sun.gc.policy.avgSurvivedPaddedAvg=-12713, sun.gc.policy.avgYoungLive=-1588, sun.gc.policy.liveSpace=-1600, sun.gc.policy.minorGcCost=1, sun.gc.policy.mutatorCost=-1, sun.gc.tlab.maxSlowWaste=-2, sun.gc.tlab.slowWaste=-2}
             |

Iteration   4: 66.783 ns/op
      HS(GC) | difference: {sun.gc.collector.0.invocations=121, sun.gc.collector.0.lastEntryTime=4988853000, sun.gc.collector.0.lastExitTime=4988935000, sun.gc.collector.0.time=109285000, sun.gc.compressedclassspace.used=568, sun.gc.generation.0.space.0.used=107367384, sun.gc.generation.0.space.1.used=-65536, sun.gc.generation.0.space.2.used=65536, sun.gc.generation.1.space.0.used=32768, sun.gc.metaspace.used=3976, sun.gc.policy.avgMinorIntervalTime=-1, sun.gc.policy.avgSurvivedAvg=-2029, sun.gc.policy.avgSurvivedDev=2406, sun.gc.policy.avgSurvivedPaddedAvg=5189, sun.gc.policy.avgYoungLive=-2029, sun.gc.policy.liveSpace=-2016, sun.gc.tlab.maxSlowWaste=-1, sun.gc.tlab.slowWaste=-1}
             |

Iteration   5: 66.107 ns/op
      HS(GC) | difference: {sun.gc.collector.0.invocations=123, sun.gc.collector.0.lastEntryTime=5019505000, sun.gc.collector.0.lastExitTime=5019477000, sun.gc.collector.0.time=108837000, sun.gc.generation.0.space.0.used=-107367384, sun.gc.generation.0.space.1.used=65536, sun.gc.generation.0.space.2.used=-65536, sun.gc.generation.1.space.0.used=32768, sun.gc.metaspace.used=2736, sun.gc.policy.avgMinorIntervalTime=-2, sun.gc.policy.avgSurvivedAvg=4671, sun.gc.policy.avgSurvivedDev=-3890, sun.gc.policy.avgSurvivedPaddedAvg=-6998, sun.gc.policy.avgYoungLive=4671, sun.gc.policy.liveSpace=4672, sun.gc.policy.minorGcCost=-1, sun.gc.policy.mutatorCost=1, sun.gc.tlab.maxSlowWaste=1, sun.gc.tlab.slowWaste=1}
             |


Result : 66.184 ±(99.9%) 1.370 ns/op
  Statistics: (min, avg, max) = (65.861, 66.184, 66.783), stdev = 0.356
  Confidence interval (99.9%): [64.814, 67.554]


# Run progress: 50.00% complete, ETA 00:00:30
# VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/Home/jre/bin/java
# VM options: -Xmx512m -Dfile.encoding=UTF-8
# Fork: 1 of 1
# Warmup: 5 iterations, 1 s each
# Measurement: 5 iterations, 5 s each
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: rx.operators.OperatorSerializePerf.synchronizedSingleStream
# Parameters: (size = 1048576)
# Warmup Iteration   1: 85.609 ns/op
# Warmup Iteration   2: 65.273 ns/op
# Warmup Iteration   3: 67.605 ns/op
# Warmup Iteration   4: 70.206 ns/op
# Warmup Iteration   5: 67.573 ns/op
Iteration   1: 67.817 ns/op
      HS(GC) | difference: {sun.gc.collector.0.invocations=113, sun.gc.collector.0.lastEntryTime=5009256000, sun.gc.collector.0.lastExitTime=5009219000, sun.gc.collector.0.time=115343000, sun.gc.compressedclassspace.used=568, sun.gc.generation.0.space.0.used=-89328184, sun.gc.generation.0.space.1.used=65536, sun.gc.generation.0.space.2.used=-65536, sun.gc.generation.1.space.0.used=57344, sun.gc.metaspace.used=5232, sun.gc.policy.avgMinorIntervalTime=-2, sun.gc.policy.avgMinorPauseTime=1, sun.gc.policy.avgPromotedAvg=-489, sun.gc.policy.avgPromotedDev=-1604, sun.gc.policy.avgPromotedPaddedAvg=-5301, sun.gc.policy.avgSurvivedAvg=-5555, sun.gc.policy.avgSurvivedDev=352, sun.gc.policy.avgSurvivedPaddedAvg=-4500, sun.gc.policy.avgYoungLive=-5551, sun.gc.policy.liveSpace=-5536, sun.gc.policy.minorPauseYoungSlope=1}
             |

Iteration   2: 65.182 ns/op
      HS(GC) | difference: {sun.gc.collector.0.invocations=117, sun.gc.collector.0.lastEntryTime=4985950000, sun.gc.collector.0.lastExitTime=4985991000, sun.gc.collector.0.time=116225000, sun.gc.compressedclassspace.used=568, sun.gc.generation.0.space.0.used=14292528, sun.gc.generation.0.space.1.used=-65536, sun.gc.generation.0.space.2.used=65536, sun.gc.generation.1.space.0.used=40960, sun.gc.metaspace.used=9880, sun.gc.policy.avgMinorIntervalTime=1, sun.gc.policy.avgMinorPauseTime=-1, sun.gc.policy.avgPromotedPaddedAvg=-1, sun.gc.policy.avgSurvivedAvg=-3828, sun.gc.policy.avgSurvivedDev=285, sun.gc.policy.avgSurvivedPaddedAvg=-2975, sun.gc.policy.avgYoungLive=-3828, sun.gc.policy.liveSpace=-3840, sun.gc.tlab.maxSlowWaste=-3, sun.gc.tlab.slowWaste=-3}
             |

Iteration   3: 64.883 ns/op
      HS(GC) | difference: {sun.gc.collector.0.invocations=118, sun.gc.collector.0.lastEntryTime=5006972000, sun.gc.collector.0.lastExitTime=5007203000, sun.gc.collector.0.time=116378000, sun.gc.generation.0.space.0.used=-14292528, sun.gc.generation.0.space.2.used=32768, sun.gc.generation.1.space.0.used=40960, sun.gc.metaspace.used=648, sun.gc.policy.avgMinorIntervalTime=-1, sun.gc.policy.avgMinorPauseTime=1, sun.gc.policy.avgSurvivedAvg=3869, sun.gc.policy.avgSurvivedDev=2754, sun.gc.policy.avgSurvivedPaddedAvg=12131, sun.gc.policy.avgYoungLive=3869, sun.gc.policy.liveSpace=3872, sun.gc.policy.minorPauseTime=1, sun.gc.policy.survived=32768}
             |

Iteration   4: 67.096 ns/op
      HS(GC) | difference: {sun.gc.collector.0.invocations=114, sun.gc.collector.0.lastEntryTime=5001540000, sun.gc.collector.0.lastExitTime=5001264000, sun.gc.collector.0.time=114907000, sun.gc.compressedclassspace.used=568, sun.gc.generation.0.space.0.used=125059448, sun.gc.generation.0.space.2.used=-32768, sun.gc.generation.1.space.0.used=32768, sun.gc.metaspace.used=3976, sun.gc.policy.avgMinorIntervalTime=4, sun.gc.policy.avgMinorPauseTime=-1, sun.gc.policy.avgSurvivedAvg=-7072, sun.gc.policy.avgSurvivedDev=-2906, sun.gc.policy.avgSurvivedPaddedAvg=-15789, sun.gc.policy.avgYoungLive=-7072, sun.gc.policy.liveSpace=-7072, sun.gc.policy.minorPauseTime=-1, sun.gc.policy.survived=-32768, sun.gc.tlab.maxSlowWaste=1, sun.gc.tlab.slowWaste=1}
             |

Iteration   5: 66.292 ns/op
      HS(GC) | difference: {sun.gc.collector.0.invocations=116, sun.gc.collector.0.lastEntryTime=5025068000, sun.gc.collector.0.lastExitTime=5025017000, sun.gc.collector.0.time=116565000, sun.gc.generation.0.space.0.used=-125059448, sun.gc.generation.1.space.0.used=49152, sun.gc.metaspace.used=2736, sun.gc.policy.avgMinorIntervalTime=-2, sun.gc.policy.avgPromotedPaddedAvg=1, sun.gc.policy.avgSurvivedAvg=6102, sun.gc.policy.avgSurvivedDev=-3403, sun.gc.policy.avgSurvivedPaddedAvg=-4108, sun.gc.policy.avgYoungLive=6102, sun.gc.policy.liveSpace=6112, sun.gc.tlab.maxSlowWaste=2, sun.gc.tlab.slowWaste=2}
             |


Result : 66.254 ±(99.9%) 4.788 ns/op
  Statistics: (min, avg, max) = (64.883, 66.254, 67.817), stdev = 1.243
  Confidence interval (99.9%): [61.466, 71.041]


Benchmark                                             (size)   Mode   Samples         Mean   Mean error    Units
r.o.OperatorSerializePerf.serializedSingleStream     1048576   avgt         5       66.184        1.370    ns/op
r.o.OperatorSerializePerf.synchronizedSingleStream   1048576   avgt         5       66.254        4.788    ns/op

These two implementations should have very different object allocation characteristics.

The time/throughput is basically the same, but one of them in our production environment works, the other causes GC pressure and intermittent problems. I'm trying to prove that with JMH so I can then test alternate implementations.

benjchristensen · 2014-03-27T05:37:12Z

I am unable to get the shadowJar functionality working:

    task perfJar(type: Jar, dependsOn: perfClasses) {
        from sourceSets.perf.output + sourceSets.main.output
    }


    // not working
    task benchmarksJar(dependsOn: perfJar) {
        apply plugin: 'shadow'
        shadow {
            classifier = "benchmarks"
            includeDependenciesFor = ["runtime", "perfRuntime"]

            transformer(com.github.jengelman.gradle.plugins.shadow.transformers.ManifestResourceTransformer) {
                mainClass = "org.openjdk.jmh.Main"
            }
        }
    }

- this allows running benchmarks - config is hardcoded right now - wasn't able to get uberjar/shadowjar functionality working (ReactiveX#963 (comment))

gvsmirnov · 2014-03-27T21:05:43Z

@benjchristensen Uh. #991 pretty much undid what I had done.

One of the main points in using shadowJar was to avoid exactly what you see now: the params passed to JMH being hardcoded in the build script.

So, what has happened here is alright for a temporary solution, just so it works in Eclipse, but we'll have to do it properly. I hope I can look into it this weekend.

gvsmirnov · 2014-03-27T21:06:39Z

@benjchristensen what do you mean exactly by measuring object allocation?

quidryan · 2014-03-27T21:20:08Z

Which params are being hardcoded which weren't before?

gvsmirnov · 2014-03-27T21:25:25Z

@quidryan Please see here: benjchristensen@ab7f408

There are two problems with using a type: JavaExec task to run benchmarks:

Passing arguments to JMH is a pain in the ass, see the link above for an example
You would need to have the source code in all the environments you want to execute the benchmarks in. You would have to build the project in all the environments.

These two are sufficient to prefer packing everything into a single runnable jar "with batteries".

quidryan · 2014-03-27T21:48:57Z

Concerning args (which I didn't realize would part of JMH), we can pass in args, e.g gradlew benchmark -PjmhArgs="something" after adding this to the JavaExec configuration:

if (project.hasProperty('jmhArgs') {
    args jmhArgs
}

I'll defer to @benjchristensen to what the goal of the benchmarks are for, which is pertinent for #2. IIRC, it's primary purpose to compare the performance of a builds (via PRs). In which case, I'd imagine we'd want a static set of args to keep things standard across runs. If necessary to run on other hosts, then the shadow approach makes sense. Though I might recommend OneJar instead.

benjchristensen · 2014-03-27T21:51:08Z

Yes, I just made that change, it's pretty straight forward:

../gradlew benchmarks '-Pjmh=-f 1 -tu ns -bm avgt -wi 5 -i 5 -r 1 .*OperatorSerializePerf.*'

benjchristensen · 2014-03-27T21:52:25Z

I want 3 things

The benchmarks task to default to running all benchmarks with reasonable defaults
The benchmarks task to support custom arguments as shown above
A benchmarksJar task that creates the uber/fat/shadow Jar that can be used independently of Gradle and taken to other environments.

benjchristensen · 2014-03-27T21:56:50Z

I just merged #995 which allows custom args.

I would still like a solution for generating the fat jar for execution away from Gradle.

benjchristensen · 2014-03-27T22:00:09Z

what do you mean exactly by measuring object allocation?

@gvsmirnov I would like to measure the impact on GC between two different class implementations and thus want to see things like number of objects allocated, pause times, young gen, time going to synchronization, etc.

Here is the code being tested if it’s somehow relevant:

SerializedObserver: https://github.com/Netflix/RxJava/blob/master/rxjava-core/src/main/java/rx/observers/SerializedObserver.java
SynchronizedObserver: https://github.com/Netflix/RxJava/blob/master/rxjava-core/src/main/java/rx/observers/SynchronizedObserver.java

I’m testing with JMH and a custom test:

JMH: The benchmark is here: https://github.com/Netflix/RxJava/blob/master/rxjava-core/src/perf/java/rx/operators/OperatorSerializePerf.java
Custom: https://github.com/Netflix/RxJava/blob/master/rxjava-core/src/perf/java/rx/archive/operators/OperatorSerializePerformance.java

When I run this code in production we see GC issues when using SerializedObserver but benchmarking is not showing this (which may mean this class is not actually the issue, but signs in prod suggest so as when I switch usage to SynchronizedObserver GC is better).

I’m trying to use JMH to measure performance and memory impact. Interestingly, the JMH tests show serialized and synchronized as basically performing the same, even though my custom tests and production testing show a difference. When I turn on HS_GC profiling, I get data, but am not seeing how it answers my questions.

Below are some of the results, including the HS_GC output.

Tests run with https://github.com/Netflix/RxJava/blob/master/build.gradle#L81

#### JMH Benchmarks

0.17.3

Benchmark                                                          (size)   Mode   Samples         Mean   Mean error    Units
r.operators.OperatorSerializePerf.noSerializationSingleThreaded      1024   avgt         5       45.504        1.710    ns/op
r.operators.OperatorSerializePerf.noSerializationSingleThreaded   1048576   avgt         5       58.600        5.647    ns/op 
r.operators.OperatorSerializePerf.serializedSingleStream             1024   avgt         5       68.610        4.596    ns/op
r.operators.OperatorSerializePerf.serializedSingleStream          1048576   avgt         5       71.313        2.318    ns/op 
r.operators.OperatorSerializePerf.synchronizedSingleStream           1024   avgt         5       73.322        3.666    ns/op
r.operators.OperatorSerializePerf.synchronizedSingleStream        1048576   avgt         5       76.518        1.355    ns/op

0.17.2

Benchmark                                                          (size)   Mode   Samples         Mean   Mean error    Units
r.operators.OperatorSerializePerf.noSerializationSingleThreaded      1024   avgt         5       45.790        1.184    ns/op
r.operators.OperatorSerializePerf.noSerializationSingleThreaded   1048576   avgt         5       58.518        3.788    ns/op 
r.operators.OperatorSerializePerf.serializedSingleStream             1024   avgt         5       72.665        7.851    ns/op
r.operators.OperatorSerializePerf.serializedSingleStream          1048576   avgt         5       74.788        2.946    ns/op
r.operators.OperatorSerializePerf.synchronizedSingleStream           1024   avgt         5       73.661        3.499    ns/op
r.operators.OperatorSerializePerf.synchronizedSingleStream        1048576   avgt         5       78.386        5.036    ns/op

#### Manual Benchmarks

/**
 * 0.17.3:
 * 
 * Run: 10 - 9,746,505 ops/sec
 * Run: 11 - 9,956,019 ops/sec
 * Run: 12 - 10,053,770 ops/sec
 * Run: 13 - 10,076,958 ops/sec
 * Run: 14 - 9,983,319 ops/sec
 * 
 * 0.17.2:
 * 
 * Run: 10 - 9,851,999 ops/sec
 * Run: 11 - 9,726,975 ops/sec
 * Run: 12 - 9,719,762 ops/sec
 * Run: 13 - 9,668,141 ops/sec
 * Run: 14 - 9,799,700 ops/sec
 * 
 * @param input
 */
public void serializedSingleStream(Input input) {
    for (int i = 0; i < reps; i++) {
        input.observable.serialize().subscribe(input.subscriber);
    }
}



#### GC Profiling

###### 0.17.3

rx.operators.OperatorSerializePerf.noSerializationSingleThreaded

      HS(GC) | difference: {sun.gc.collector.0.invocations=101, sun.gc.collector.0.lastEntryTime=4996826, sun.gc.collector.0.lastExitTime=4996824, sun.gc.collector.0.time=48345, sun.gc.generation.0.space.0.capacity=37748736, sun.gc.generation.0.space.1.used=-65536, sun.gc.generation.0.space.2.used=65536, sun.gc.generation.1.space.0.used=49152, sun.gc.generation.2.space.0.used=536, sun.gc.policy.avgBaseFootprint=544, sun.gc.policy.avgPromotedDev=-3040, sun.gc.policy.avgPromotedPaddedAvg=-9120, sun.gc.policy.avgSurvivedAvg=-3293, sun.gc.policy.avgSurvivedDev=-2172, sun.gc.policy.avgSurvivedPaddedAvg=-9810, sun.gc.policy.avgYoungLive=-3293, sun.gc.policy.edenSize=37748736, sun.gc.policy.freeSpace=37748736, sun.gc.policy.liveSpace=-2752, sun.gc.policy.oldEdenSize=39321600, sun.gc.policy.youngCapacity=39321600, sun.gc.tlab.alloc=4934500}

rx.operators.OperatorSerializePerf.serializedSingleStream

      HS(GC) | difference: {sun.gc.collector.0.invocations=109, sun.gc.collector.0.lastEntryTime=5028125, sun.gc.collector.0.lastExitTime=5028068, sun.gc.collector.0.time=51921, sun.gc.generation.0.space.0.capacity=-12058624, sun.gc.generation.0.space.0.used=-46544136, sun.gc.generation.0.space.1.used=-32768, sun.gc.generation.0.space.2.used=65536, sun.gc.generation.1.space.0.used=49152, sun.gc.generation.2.space.0.used=360, sun.gc.policy.avgBaseFootprint=320, sun.gc.policy.avgPromotedDev=-3128, sun.gc.policy.avgPromotedPaddedAvg=-9385, sun.gc.policy.avgSurvivedAvg=-10919, sun.gc.policy.avgSurvivedDev=-417, sun.gc.policy.avgSurvivedPaddedAvg=-12168, sun.gc.policy.avgYoungLive=-10919, sun.gc.policy.decreaseForFootprint=6, sun.gc.policy.edenSize=-12058624, sun.gc.policy.freeSpace=-12058624, sun.gc.policy.increaseYoungGenForThroughput=-5, sun.gc.policy.liveSpace=-10592, sun.gc.policy.minorGcCost=-1, sun.gc.policy.minorPauseYoungSlope=1, sun.gc.policy.mutatorCost=1, sun.gc.policy.oldEdenSize=-6291456, sun.gc.policy.survived=32768, sun.gc.policy.youngCapacity=-6291456, sun.gc.tlab.alloc=-790550, sun.gc.tlab.maxSlowWaste=-32, sun.gc.tlab.slowWaste=-32}

r.o.OperatorSerializePerf.noSerializationSingleThreaded      1024   avgt         5       45.778        1.837    ns/op
r.o.OperatorSerializePerf.noSerializationSingleThreaded   1048576   avgt         5       57.498        1.259    ns/op
r.o.OperatorSerializePerf.serializedSingleStream             1024   avgt         5       68.725        1.119    ns/op
r.o.OperatorSerializePerf.serializedSingleStream          1048576   avgt         5       72.124        1.131    ns/op
r.o.OperatorSerializePerf.synchronizedSingleStream           1024   avgt         5       74.341        3.580    ns/op
r.o.OperatorSerializePerf.synchronizedSingleStream        1048576   avgt         5       78.460        0.347    ns/op


###### 0.17.2

rx.operators.OperatorSerializePerf.noSerializationSingleThreaded

      HS(GC) | difference: {sun.gc.collector.0.invocations=101, sun.gc.collector.0.lastEntryTime=5041703, sun.gc.collector.0.lastExitTime=5041691, sun.gc.collector.0.time=48663, sun.gc.generation.0.space.0.capacity=6815744, sun.gc.generation.0.space.0.used=-10509384, sun.gc.generation.0.space.2.used=-65536, sun.gc.generation.1.space.0.used=40960, sun.gc.generation.2.space.0.used=1728, sun.gc.policy.avgBaseFootprint=1728, sun.gc.policy.avgMinorIntervalTime=-7, sun.gc.policy.avgPromotedDev=-3565, sun.gc.policy.avgPromotedPaddedAvg=-10695, sun.gc.policy.avgSurvivedAvg=-8504, sun.gc.policy.avgSurvivedDev=3596, sun.gc.policy.avgSurvivedPaddedAvg=2285, sun.gc.policy.avgYoungLive=-8504, sun.gc.policy.edenSize=6815744, sun.gc.policy.freeSpace=6815744, sun.gc.policy.liveSpace=-6784, sun.gc.policy.oldEdenSize=7340032, sun.gc.policy.survived=-65536, sun.gc.policy.youngCapacity=7340032, sun.gc.tlab.alloc=920400, sun.gc.tlab.maxSlowWaste=-3, sun.gc.tlab.slowWaste=-3}


rx.operators.OperatorSerializePerf.serializedSingleStream

      HS(GC) | difference: {sun.gc.collector.0.invocations=113, sun.gc.collector.0.lastEntryTime=5005975, sun.gc.collector.0.lastExitTime=5006015, sun.gc.collector.0.time=53886, sun.gc.generation.0.space.1.used=-32768, sun.gc.generation.0.space.2.used=65536, sun.gc.generation.1.space.0.used=40960, sun.gc.generation.2.space.0.used=1056, sun.gc.policy.avgBaseFootprint=992, sun.gc.policy.avgPromotedDev=-3067, sun.gc.policy.avgPromotedPaddedAvg=-9201, sun.gc.policy.avgSurvivedAvg=3945, sun.gc.policy.avgSurvivedDev=-2143, sun.gc.policy.avgSurvivedPaddedAvg=-2485, sun.gc.policy.avgYoungLive=3945, sun.gc.policy.liveSpace=4928, sun.gc.policy.oldEdenSize=-6291456, sun.gc.policy.survived=32768, sun.gc.policy.youngCapacity=-6291456, sun.gc.tlab.alloc=-790200, sun.gc.tlab.maxSlowWaste=-26, sun.gc.tlab.slowWaste=-26}


Benchmark                                                  (size)   Mode   Samples         Mean   Mean error    Units
r.o.OperatorSerializePerf.noSerializationSingleThreaded      1024   avgt         5       46.274        1.587    ns/op
r.o.OperatorSerializePerf.noSerializationSingleThreaded   1048576   avgt         5       60.240       13.223    ns/op
r.o.OperatorSerializePerf.serializedSingleStream             1024   avgt         5       73.160        3.774    ns/op
r.o.OperatorSerializePerf.serializedSingleStream          1048576   avgt         5       76.042        2.867    ns/op
r.o.OperatorSerializePerf.synchronizedSingleStream           1024   avgt         5       72.692        0.996    ns/op
r.o.OperatorSerializePerf.synchronizedSingleStream        1048576   avgt         5       76.616        2.982    ns/op

benjchristensen · 2014-03-28T02:48:19Z

The shadow jar is now being correctly created as of pull request: #996

I have confirmed that builds from command-line, importing into Eclipse, execution benchmarks via Gradle and using the shadow generated jar are all working.

gvsmirnov added 2 commits March 13, 2014 23:44

Updated gradle build scripts so there is a separate benchmarks task

2052da3

Updated ObservableBenchmark to follow better benchmarking practices

2fa35e5

benjchristensen added a commit that referenced this pull request Mar 13, 2014

Merge pull request #963 from gvsmirnov/master

da289ec

A more robust JMH benchmarking set-up

benjchristensen merged commit da289ec into ReactiveX:master Mar 13, 2014

gvsmirnov mentioned this pull request Mar 13, 2014

Performance Testing #776

Closed

benjchristensen mentioned this pull request Mar 26, 2014

JMH Benchmark Build Config #991

Merged

benjchristensen added a commit to benjchristensen/RxJava that referenced this pull request Mar 27, 2014

JMH Benchmarks Gradle Config

ab7f408

- this allows running benchmarks - config is hardcoded right now - wasn't able to get uberjar/shadowjar functionality working (ReactiveX#963 (comment))

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

A more robust JMH benchmarking set-up #963

A more robust JMH benchmarking set-up #963

gvsmirnov commented Mar 13, 2014

benjchristensen commented Mar 13, 2014

headinthebox commented Mar 13, 2014

gvsmirnov commented Mar 13, 2014

benjchristensen commented Mar 13, 2014

benjchristensen commented Mar 15, 2014

gvsmirnov commented Mar 17, 2014

benjchristensen commented Mar 24, 2014

benjchristensen commented Mar 26, 2014

benjchristensen commented Mar 26, 2014

benjchristensen commented Mar 26, 2014

benjchristensen commented Mar 27, 2014

benjchristensen commented Mar 27, 2014

gvsmirnov commented Mar 27, 2014

gvsmirnov commented Mar 27, 2014

quidryan commented Mar 27, 2014

gvsmirnov commented Mar 27, 2014

quidryan commented Mar 27, 2014

benjchristensen commented Mar 27, 2014

benjchristensen commented Mar 27, 2014

benjchristensen commented Mar 27, 2014

benjchristensen commented Mar 27, 2014

benjchristensen commented Mar 28, 2014

A more robust JMH benchmarking set-up #963

A more robust JMH benchmarking set-up #963

Conversation

gvsmirnov commented Mar 13, 2014

benjchristensen commented Mar 13, 2014

headinthebox commented Mar 13, 2014

gvsmirnov commented Mar 13, 2014

benjchristensen commented Mar 13, 2014

benjchristensen commented Mar 15, 2014

gvsmirnov commented Mar 17, 2014

benjchristensen commented Mar 24, 2014

benjchristensen commented Mar 26, 2014

benjchristensen commented Mar 26, 2014

benjchristensen commented Mar 26, 2014

benjchristensen commented Mar 27, 2014

benjchristensen commented Mar 27, 2014

gvsmirnov commented Mar 27, 2014

gvsmirnov commented Mar 27, 2014

quidryan commented Mar 27, 2014

gvsmirnov commented Mar 27, 2014

quidryan commented Mar 27, 2014

benjchristensen commented Mar 27, 2014

benjchristensen commented Mar 27, 2014

benjchristensen commented Mar 27, 2014

benjchristensen commented Mar 27, 2014

benjchristensen commented Mar 28, 2014