Add Scala Doku Benchmark #173

davleopo · 2019-07-03T13:36:50Z

This PR adds the Soduko Benchmark - shortly called Doku:
This benchmarks solves 2 soduko puzzles one with a few holes and one with exaclty one hole.

Results from my machine

lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              8
On-line CPU(s) list: 0-7
Thread(s) per core:  2
Core(s) per socket:  4
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               94
Model name:          Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz
Stepping:            3
CPU MHz:             1400.096
CPU max MHz:         2700,0000
CPU min MHz:         800,0000
BogoMIPS:            5424.00
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            8192K
NUMA node0 CPU(s):   0-7

Output

$JAVA_HOME/bin/java -jar target/renaissance-gpl-0.9.0.jar scala-doku
====== scala-doku (scala-stdlib), iteration 0 started ======
====== scala-doku (scala-stdlib), iteration 0 completed (7481.557 ms) ======
====== scala-doku (scala-stdlib), iteration 1 started ======
====== scala-doku (scala-stdlib), iteration 1 completed (7484.406 ms) ======
====== scala-doku (scala-stdlib), iteration 2 started ======
====== scala-doku (scala-stdlib), iteration 2 completed (7744.104 ms) ======
====== scala-doku (scala-stdlib), iteration 3 started ======
====== scala-doku (scala-stdlib), iteration 3 completed (8192.163 ms) ======
====== scala-doku (scala-stdlib), iteration 4 started ======
====== scala-doku (scala-stdlib), iteration 4 completed (7987.966 ms) ======
====== scala-doku (scala-stdlib), iteration 5 started ======
====== scala-doku (scala-stdlib), iteration 5 completed (7934.384 ms) ======
====== scala-doku (scala-stdlib), iteration 6 started ======
====== scala-doku (scala-stdlib), iteration 6 completed (7948.975 ms) ======
====== scala-doku (scala-stdlib), iteration 7 started ======
====== scala-doku (scala-stdlib), iteration 7 completed (7946.754 ms) ======
====== scala-doku (scala-stdlib), iteration 8 started ======
====== scala-doku (scala-stdlib), iteration 8 completed (8001.953 ms) ======
====== scala-doku (scala-stdlib), final iteration started ======
====== scala-doku (scala-stdlib), final iteration completed (7977.516 ms) ======

vhotspur · 2019-07-03T14:03:57Z

Stupid question: why is CafeSat inside our repository as full source code? It seems that CafeSat is not available for Maven/Ivy as JAR but here they clone it in Sbt (but perhaps something prevents us to use the same approach?):

lazy val root = (project in file(".")).
     ...
  ).
  dependsOn(cafeSat)

lazy val cafeSat = {
  val commit = "e39e2fb11b66b5daf8002cb5bf457a2a2f889e05"
  val githubLink = s"git://github.com/regb/cafesat.git#$commit"
  RootProject(uri(githubLink))
}

davleopo · 2019-07-03T14:07:05Z

Stupid question: why is CafeSat inside our repository as full source code? It seems that CafeSat is not available for Maven/Ivy as JAR but here they clone it in Sbt (but perhaps something prevents us to use the same approach?):
lazy val root = (project in file(".")).
     ...
  ).
  dependsOn(cafeSat)

lazy val cafeSat = {
  val commit = "e39e2fb11b66b5daf8002cb5bf457a2a2f889e05"
  val githubLink = s"git://github.com/regb/cafesat.git#$commit"
  RootProject(uri(githubLink))
}

Sry accidentally closed the PR (buttons to close to each other ;-) )
@vhotspur I think those are basically the 2 options. Either having it as source directly in or get it from github (since there is no maven artifact). We already took the route for other benchmarks adding dependencies from source so I went for that route as well. We can change that if we want I dont have a strong opinion on this.

axel22 · 2019-07-03T14:07:10Z

I think that Maven has a decent guarantee that it will keep artifacts forever, while GitHub references are volatile. Using GitHub uris essentially opens an easier door to breaking the suite.
So, given that there is no Maven, a source inclusion seems safer.

And even though they are no longer actively maintaining CafeSAT, it still exercises a lot of functional-style code and collections transformations, and it seems useful to have it as a staging benchmark.

axel22 · 2019-07-03T14:11:49Z

benchmarks/scala-stdlib/cafesat/build.sbt

+    fork in IntegrationTest := true,
+    logBuffered in IntegrationTest := false,
+    parallelExecution in Test := true,
+    libraryDependencies += "org.scalatest" % "scalatest_2.11" % "2.2.1" % "test"


Dependency on scalatest 2.11 is weird, but since we're only running the main artifact, it shouldn't be important.

Might be a left over from some changes I did when pulling in the sources. I will check.

Might be that CafeSAT had been using Scala 2.11.

axel22 · 2019-07-03T14:14:55Z

benchmarks/scala-stdlib/build.sbt

@@ -1,5 +1,9 @@
 lazy val renaissanceCore = RootProject(uri("../../renaissance-core"))

+lazy val scalaSMTLib = RootProject(uri("scala-smtlib"))
+
+lazy val scalaCafeSAT = RootProject(uri("cafesat"))


IMO, we should open a new subproject for benchmark, such as cafesat, and not pollute scala-stdlib with this.
At some point, we will potentially transition to a newer Scala version in scala-stdlib, and at that point this will make migration harder.

You mean a new group altogether containing sub projects for smt and cafesat and the doku benchmark?

vhotspur · 2019-07-03T14:18:11Z

We already took the route for other benchmarks adding dependencies from source so I went for that route as well

And given that they are no longer actively maintaining CafeSAT (but it still exercises a lot of functional-style code and collections transformations) it seems useful to have it as a staging benchmark.

Okay, that makes sense. Not that I like it ;-)

farquet · 2019-07-03T14:28:09Z

benchmarks/scala-stdlib/src/main/scala/org/renaissance/scala/stdlib/doku/ScalaDoku.scala

+@Group("scala-stdlib")
+@Summary("Solves Sudoku Puzzles using Scala collections.")
+@Licenses(Array(License.MIT))
+@Repetitions(10)


Interesting how the performance degrades over time. Have tried a run with many more iterations to make sure that warmup is sufficient ?

Yes it degrades quite a bit.
On my machine it stays between 7.8sec and 8.3
See 100 iterations

$JAVA_HOME/bin/java -jar target/renaissance-gpl-0.9.0.jar scala-doku -r 100 ====== scala-doku (scala-stdlib), iteration 0 started ====== ====== scala-doku (scala-stdlib), iteration 0 completed (7295.563 ms) ====== ====== scala-doku (scala-stdlib), iteration 1 started ====== ====== scala-doku (scala-stdlib), iteration 1 completed (7740.79 ms) ====== ====== scala-doku (scala-stdlib), iteration 2 started ====== ====== scala-doku (scala-stdlib), iteration 2 completed (7651.956 ms) ====== ====== scala-doku (scala-stdlib), iteration 3 started ====== ====== scala-doku (scala-stdlib), iteration 3 completed (7443.26 ms) ====== ====== scala-doku (scala-stdlib), iteration 4 started ====== ====== scala-doku (scala-stdlib), iteration 4 completed (7442.337 ms) ====== ====== scala-doku (scala-stdlib), iteration 5 started ====== ====== scala-doku (scala-stdlib), iteration 5 completed (7880.45 ms) ====== ====== scala-doku (scala-stdlib), iteration 6 started ====== ====== scala-doku (scala-stdlib), iteration 6 completed (7876.23 ms) ====== ====== scala-doku (scala-stdlib), iteration 7 started ====== ====== scala-doku (scala-stdlib), iteration 7 completed (7911.213 ms) ====== ====== scala-doku (scala-stdlib), iteration 8 started ====== ====== scala-doku (scala-stdlib), iteration 8 completed (7897.584 ms) ====== ====== scala-doku (scala-stdlib), iteration 9 started ====== ====== scala-doku (scala-stdlib), iteration 9 completed (7860.457 ms) ====== ====== scala-doku (scala-stdlib), iteration 10 started ====== ====== scala-doku (scala-stdlib), iteration 10 completed (7941.158 ms) ====== ====== scala-doku (scala-stdlib), iteration 11 started ====== ====== scala-doku (scala-stdlib), iteration 11 completed (7954.829 ms) ====== ====== scala-doku (scala-stdlib), iteration 12 started ====== ====== scala-doku (scala-stdlib), iteration 12 completed (8069.049 ms) ====== ====== scala-doku (scala-stdlib), iteration 13 started ====== ====== scala-doku (scala-stdlib), iteration 13 completed (7930.362 ms) ====== ====== scala-doku (scala-stdlib), iteration 14 started ====== ====== scala-doku (scala-stdlib), iteration 14 completed (7935.664 ms) ====== ====== scala-doku (scala-stdlib), iteration 15 started ====== ====== scala-doku (scala-stdlib), iteration 15 completed (7930.514 ms) ====== ====== scala-doku (scala-stdlib), iteration 16 started ====== ====== scala-doku (scala-stdlib), iteration 16 completed (7965.733 ms) ====== ====== scala-doku (scala-stdlib), iteration 17 started ====== ====== scala-doku (scala-stdlib), iteration 17 completed (7952.007 ms) ====== ====== scala-doku (scala-stdlib), iteration 18 started ====== ====== scala-doku (scala-stdlib), iteration 18 completed (7934.641 ms) ====== ====== scala-doku (scala-stdlib), iteration 19 started ====== ====== scala-doku (scala-stdlib), iteration 19 completed (8126.194 ms) ====== ====== scala-doku (scala-stdlib), iteration 20 started ====== ====== scala-doku (scala-stdlib), iteration 20 completed (7937.435 ms) ====== ====== scala-doku (scala-stdlib), iteration 21 started ====== ====== scala-doku (scala-stdlib), iteration 21 completed (7903.58 ms) ====== ====== scala-doku (scala-stdlib), iteration 22 started ====== ====== scala-doku (scala-stdlib), iteration 22 completed (7918.003 ms) ====== ====== scala-doku (scala-stdlib), iteration 23 started ====== ====== scala-doku (scala-stdlib), iteration 23 completed (7873.369 ms) ====== ====== scala-doku (scala-stdlib), iteration 24 started ====== ====== scala-doku (scala-stdlib), iteration 24 completed (7853.768 ms) ====== ====== scala-doku (scala-stdlib), iteration 25 started ====== ====== scala-doku (scala-stdlib), iteration 25 completed (7892.393 ms) ====== ====== scala-doku (scala-stdlib), iteration 26 started ====== ====== scala-doku (scala-stdlib), iteration 26 completed (7970.432 ms) ====== ====== scala-doku (scala-stdlib), iteration 27 started ====== ====== scala-doku (scala-stdlib), iteration 27 completed (7912.716 ms) ====== ====== scala-doku (scala-stdlib), iteration 28 started ====== ====== scala-doku (scala-stdlib), iteration 28 completed (7910.841 ms) ====== ====== scala-doku (scala-stdlib), iteration 29 started ====== ====== scala-doku (scala-stdlib), iteration 29 completed (7868.315 ms) ====== ====== scala-doku (scala-stdlib), iteration 30 started ====== ====== scala-doku (scala-stdlib), iteration 30 completed (7904.546 ms) ====== ====== scala-doku (scala-stdlib), iteration 31 started ====== ====== scala-doku (scala-stdlib), iteration 31 completed (7923.313 ms) ====== ====== scala-doku (scala-stdlib), iteration 32 started ====== ====== scala-doku (scala-stdlib), iteration 32 completed (7895.506 ms) ====== ====== scala-doku (scala-stdlib), iteration 33 started ====== ====== scala-doku (scala-stdlib), iteration 33 completed (7917.952 ms) ====== ====== scala-doku (scala-stdlib), iteration 34 started ====== ====== scala-doku (scala-stdlib), iteration 34 completed (7954.212 ms) ====== ====== scala-doku (scala-stdlib), iteration 35 started ====== ====== scala-doku (scala-stdlib), iteration 35 completed (7938.572 ms) ====== ====== scala-doku (scala-stdlib), iteration 36 started ====== ====== scala-doku (scala-stdlib), iteration 36 completed (7965.098 ms) ====== ====== scala-doku (scala-stdlib), iteration 37 started ====== ====== scala-doku (scala-stdlib), iteration 37 completed (7942.275 ms) ====== ====== scala-doku (scala-stdlib), iteration 38 started ====== ====== scala-doku (scala-stdlib), iteration 38 completed (7986.694 ms) ====== ====== scala-doku (scala-stdlib), iteration 39 started ====== ====== scala-doku (scala-stdlib), iteration 39 completed (7922.645 ms) ====== ====== scala-doku (scala-stdlib), iteration 40 started ====== ====== scala-doku (scala-stdlib), iteration 40 completed (7918.326 ms) ====== ====== scala-doku (scala-stdlib), iteration 41 started ====== ====== scala-doku (scala-stdlib), iteration 41 completed (8010.682 ms) ====== ====== scala-doku (scala-stdlib), iteration 42 started ====== ====== scala-doku (scala-stdlib), iteration 42 completed (8050.451 ms) ====== ====== scala-doku (scala-stdlib), iteration 43 started ====== ====== scala-doku (scala-stdlib), iteration 43 completed (7895.714 ms) ====== ====== scala-doku (scala-stdlib), iteration 44 started ====== ====== scala-doku (scala-stdlib), iteration 44 completed (8159.713 ms) ====== ====== scala-doku (scala-stdlib), iteration 45 started ====== ====== scala-doku (scala-stdlib), iteration 45 completed (8165.301 ms) ====== ====== scala-doku (scala-stdlib), iteration 46 started ====== ====== scala-doku (scala-stdlib), iteration 46 completed (8181.688 ms) ====== ====== scala-doku (scala-stdlib), iteration 47 started ====== ====== scala-doku (scala-stdlib), iteration 47 completed (8228.609 ms) ====== ====== scala-doku (scala-stdlib), iteration 48 started ====== ====== scala-doku (scala-stdlib), iteration 48 completed (8175.911 ms) ====== ====== scala-doku (scala-stdlib), iteration 49 started ====== ====== scala-doku (scala-stdlib), iteration 49 completed (8146.763 ms) ====== ====== scala-doku (scala-stdlib), iteration 50 started ====== ====== scala-doku (scala-stdlib), iteration 50 completed (8163.341 ms) ====== ====== scala-doku (scala-stdlib), iteration 51 started ====== ====== scala-doku (scala-stdlib), iteration 51 completed (8159.749 ms) ====== ====== scala-doku (scala-stdlib), iteration 52 started ====== ====== scala-doku (scala-stdlib), iteration 52 completed (8239.117 ms) ====== ====== scala-doku (scala-stdlib), iteration 53 started ====== ====== scala-doku (scala-stdlib), iteration 53 completed (8157.191 ms) ====== ====== scala-doku (scala-stdlib), iteration 54 started ====== ====== scala-doku (scala-stdlib), iteration 54 completed (8136.763 ms) ====== ====== scala-doku (scala-stdlib), iteration 55 started ====== ====== scala-doku (scala-stdlib), iteration 55 completed (8165.647 ms) ====== ====== scala-doku (scala-stdlib), iteration 56 started ====== ====== scala-doku (scala-stdlib), iteration 56 completed (8214.203 ms) ====== ====== scala-doku (scala-stdlib), iteration 57 started ====== ====== scala-doku (scala-stdlib), iteration 57 completed (8272.253 ms) ====== ====== scala-doku (scala-stdlib), iteration 58 started ====== ====== scala-doku (scala-stdlib), iteration 58 completed (8198.186 ms) ====== ====== scala-doku (scala-stdlib), iteration 59 started ====== ====== scala-doku (scala-stdlib), iteration 59 completed (8219.298 ms) ====== ====== scala-doku (scala-stdlib), iteration 60 started ====== ====== scala-doku (scala-stdlib), iteration 60 completed (8115.016 ms) ====== ====== scala-doku (scala-stdlib), iteration 61 started ====== ====== scala-doku (scala-stdlib), iteration 61 completed (8276.586 ms) ====== ====== scala-doku (scala-stdlib), iteration 62 started ====== ====== scala-doku (scala-stdlib), iteration 62 completed (8423.493 ms) ====== ====== scala-doku (scala-stdlib), iteration 63 started ====== ====== scala-doku (scala-stdlib), iteration 63 completed (8177.066 ms) ====== ====== scala-doku (scala-stdlib), iteration 64 started ====== ====== scala-doku (scala-stdlib), iteration 64 completed (8182.497 ms) ====== ====== scala-doku (scala-stdlib), iteration 65 started ====== ====== scala-doku (scala-stdlib), iteration 65 completed (8222.866 ms) ====== ====== scala-doku (scala-stdlib), iteration 66 started ====== ====== scala-doku (scala-stdlib), iteration 66 completed (8239.579 ms) ====== ====== scala-doku (scala-stdlib), iteration 67 started ====== ====== scala-doku (scala-stdlib), iteration 67 completed (8170.603 ms) ====== ====== scala-doku (scala-stdlib), iteration 68 started ====== ====== scala-doku (scala-stdlib), iteration 68 completed (8198.62 ms) ====== ====== scala-doku (scala-stdlib), iteration 69 started ====== ====== scala-doku (scala-stdlib), iteration 69 completed (8150.633 ms) ====== ====== scala-doku (scala-stdlib), iteration 70 started ====== ====== scala-doku (scala-stdlib), iteration 70 completed (8127.022 ms) ====== ====== scala-doku (scala-stdlib), iteration 71 started ====== ====== scala-doku (scala-stdlib), iteration 71 completed (8118.76 ms) ====== ====== scala-doku (scala-stdlib), iteration 72 started ====== ====== scala-doku (scala-stdlib), iteration 72 completed (8137.789 ms) ====== ====== scala-doku (scala-stdlib), iteration 73 started ====== ====== scala-doku (scala-stdlib), iteration 73 completed (8298.117 ms) ====== ====== scala-doku (scala-stdlib), iteration 74 started ====== ====== scala-doku (scala-stdlib), iteration 74 completed (8153.761 ms) ====== ====== scala-doku (scala-stdlib), iteration 75 started ====== ====== scala-doku (scala-stdlib), iteration 75 completed (8178.487 ms) ====== ====== scala-doku (scala-stdlib), iteration 76 started ====== ====== scala-doku (scala-stdlib), iteration 76 completed (8170.637 ms) ====== ====== scala-doku (scala-stdlib), iteration 77 started ====== ====== scala-doku (scala-stdlib), iteration 77 completed (8193.188 ms) ====== ====== scala-doku (scala-stdlib), iteration 78 started ====== ====== scala-doku (scala-stdlib), iteration 78 completed (8237.173 ms) ====== ====== scala-doku (scala-stdlib), iteration 79 started ====== ====== scala-doku (scala-stdlib), iteration 79 completed (8150.02 ms) ====== ====== scala-doku (scala-stdlib), iteration 80 started ====== ====== scala-doku (scala-stdlib), iteration 80 completed (8143.898 ms) ====== ====== scala-doku (scala-stdlib), iteration 81 started ====== ====== scala-doku (scala-stdlib), iteration 81 completed (8192.958 ms) ====== ====== scala-doku (scala-stdlib), iteration 82 started ====== ====== scala-doku (scala-stdlib), iteration 82 completed (8156.796 ms) ====== ====== scala-doku (scala-stdlib), iteration 83 started ====== ====== scala-doku (scala-stdlib), iteration 83 completed (8159.5 ms) ====== ====== scala-doku (scala-stdlib), iteration 84 started ====== ====== scala-doku (scala-stdlib), iteration 84 completed (8174.045 ms) ====== ====== scala-doku (scala-stdlib), iteration 85 started ====== ====== scala-doku (scala-stdlib), iteration 85 completed (8157.696 ms) ====== ====== scala-doku (scala-stdlib), iteration 86 started ====== ====== scala-doku (scala-stdlib), iteration 86 completed (8191.271 ms) ====== ====== scala-doku (scala-stdlib), iteration 87 started ====== ====== scala-doku (scala-stdlib), iteration 87 completed (8200.355 ms) ====== ====== scala-doku (scala-stdlib), iteration 88 started ====== ====== scala-doku (scala-stdlib), iteration 88 completed (8309.617 ms) ====== ====== scala-doku (scala-stdlib), iteration 89 started ====== ====== scala-doku (scala-stdlib), iteration 89 completed (8221.984 ms) ====== ====== scala-doku (scala-stdlib), iteration 90 started ====== ====== scala-doku (scala-stdlib), iteration 90 completed (8277.728 ms) ====== ====== scala-doku (scala-stdlib), iteration 91 started ====== ====== scala-doku (scala-stdlib), iteration 91 completed (8203.674 ms) ====== ====== scala-doku (scala-stdlib), iteration 92 started ====== ====== scala-doku (scala-stdlib), iteration 92 completed (8176.549 ms) ====== ====== scala-doku (scala-stdlib), iteration 93 started ====== ====== scala-doku (scala-stdlib), iteration 93 completed (8193.98 ms) ====== ====== scala-doku (scala-stdlib), iteration 94 started ====== ====== scala-doku (scala-stdlib), iteration 94 completed (8163.47 ms) ====== ====== scala-doku (scala-stdlib), iteration 95 started ====== ====== scala-doku (scala-stdlib), iteration 95 completed (8133.637 ms) ====== ====== scala-doku (scala-stdlib), iteration 96 started ====== ====== scala-doku (scala-stdlib), iteration 96 completed (8229.286 ms) ====== ====== scala-doku (scala-stdlib), iteration 97 started ====== ====== scala-doku (scala-stdlib), iteration 97 completed (8148.785 ms) ====== ====== scala-doku (scala-stdlib), iteration 98 started ====== ====== scala-doku (scala-stdlib), iteration 98 completed (8163.858 ms) ====== ====== scala-doku (scala-stdlib), final iteration started ====== ====== scala-doku (scala-stdlib), final iteration completed (8179.694 ms) ======

davleopo · 2019-07-04T08:15:10Z

@axel22 & @farquet I created a new group and moved scalasat, smt and the docu benchmark there.
As a result I could downgrade to the originally (from the projects) scala versions.
So now stdlib can evolve separately.
This changes performance a bit thus I increased the default iterations to 20 see below

$JAVA_HOME/bin/java -jar target/renaissance-gpl-0.9.0.jar scala-doku
====== scala-doku (scala-sat), iteration 0 started ======
====== scala-doku (scala-sat), iteration 0 completed (6706.789 ms) ======
====== scala-doku (scala-sat), iteration 1 started ======
====== scala-doku (scala-sat), iteration 1 completed (9081.978 ms) ======
====== scala-doku (scala-sat), iteration 2 started ======
====== scala-doku (scala-sat), iteration 2 completed (8932.892 ms) ======
====== scala-doku (scala-sat), iteration 3 started ======
====== scala-doku (scala-sat), iteration 3 completed (9011.966 ms) ======
====== scala-doku (scala-sat), iteration 4 started ======
====== scala-doku (scala-sat), iteration 4 completed (8849.51 ms) ======
====== scala-doku (scala-sat), iteration 5 started ======
====== scala-doku (scala-sat), iteration 5 completed (8795.823 ms) ======
====== scala-doku (scala-sat), iteration 6 started ======
====== scala-doku (scala-sat), iteration 6 completed (8794.004 ms) ======
====== scala-doku (scala-sat), iteration 7 started ======
====== scala-doku (scala-sat), iteration 7 completed (8805.905 ms) ======
====== scala-doku (scala-sat), iteration 8 started ======
====== scala-doku (scala-sat), iteration 8 completed (8781.712 ms) ======
====== scala-doku (scala-sat), iteration 9 started ======
====== scala-doku (scala-sat), iteration 9 completed (8781.206 ms) ======
====== scala-doku (scala-sat), iteration 10 started ======
====== scala-doku (scala-sat), iteration 10 completed (8780.117 ms) ======
====== scala-doku (scala-sat), iteration 11 started ======
====== scala-doku (scala-sat), iteration 11 completed (8799.237 ms) ======
====== scala-doku (scala-sat), iteration 12 started ======
====== scala-doku (scala-sat), iteration 12 completed (8824.494 ms) ======
====== scala-doku (scala-sat), iteration 13 started ======
====== scala-doku (scala-sat), iteration 13 completed (8765.353 ms) ======
====== scala-doku (scala-sat), iteration 14 started ======
====== scala-doku (scala-sat), iteration 14 completed (8841.746 ms) ======
====== scala-doku (scala-sat), iteration 15 started ======
====== scala-doku (scala-sat), iteration 15 completed (8752.229 ms) ======
====== scala-doku (scala-sat), iteration 16 started ======
====== scala-doku (scala-sat), iteration 16 completed (8752.513 ms) ======
====== scala-doku (scala-sat), iteration 17 started ======
====== scala-doku (scala-sat), iteration 17 completed (8782.854 ms) ======
====== scala-doku (scala-sat), iteration 18 started ======
====== scala-doku (scala-sat), iteration 18 completed (8804.562 ms) ======
====== scala-doku (scala-sat), final iteration started ======
====== scala-doku (scala-sat), final iteration completed (8830.316 ms) ======

farquet

LGTM !

farquet · 2019-07-04T08:28:16Z

I'll wait for this PR to get in before doing the 0.10.0 release that will include scala-doku. Since it's not yet a 1.0 release and we have early adopters that report bugs, the more we ship in the release, the better.

axel22 · 2019-07-04T09:42:43Z

Looks good. The performance degradation is possibly to less frequently invoked methods (callers) getting compiled later than their callees, and overtaking their (previously separately compiled) callees.
But it seems to converge to 8.2s in the earlier example.

axel22

LGTM

axel22 · 2019-07-04T09:44:31Z

@farquet #103

I think it's ok to go ahead with 0.10.0, but for 0.11.0 we should make sure that the staging/release/deprecated statuses are added to all the benchmarks.

vhotspur

LGTM too.

davleopo added 2 commits July 3, 2019 15:26

topic: add scala sudoku benchmark

3bcaf96

update readme

3fa6fa8

davleopo requested review from axel22, farquet and lbulej July 3, 2019 13:36

davleopo requested a review from vhotspur July 3, 2019 14:04

davleopo closed this Jul 3, 2019

davleopo reopened this Jul 3, 2019

axel22 reviewed Jul 3, 2019

View reviewed changes

farquet reviewed Jul 3, 2019

View reviewed changes

davleopo added 3 commits July 4, 2019 10:00

move cafesat, smtlib and doku benchmark to new group

280a78b

update readme

6a986b6

scala doku: increase default repetitions to 20

67718bb

farquet approved these changes Jul 4, 2019

View reviewed changes

axel22 approved these changes Jul 4, 2019

View reviewed changes

vhotspur approved these changes Jul 4, 2019

View reviewed changes

davleopo merged commit 47fd9ce into master Jul 4, 2019

davleopo mentioned this pull request Jul 9, 2019

large variance in exec time for scala-doku #175

Open

lbulej deleted the topic/add-scala-doku branch July 10, 2019 11:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Scala Doku Benchmark #173

Add Scala Doku Benchmark #173

davleopo commented Jul 3, 2019

vhotspur commented Jul 3, 2019

davleopo commented Jul 3, 2019

axel22 commented Jul 3, 2019 •

edited

Loading

axel22 Jul 3, 2019

davleopo Jul 3, 2019

axel22 Jul 3, 2019

axel22 Jul 3, 2019

davleopo Jul 3, 2019

axel22 Jul 3, 2019

davleopo Jul 4, 2019

vhotspur commented Jul 3, 2019

farquet Jul 3, 2019

davleopo Jul 4, 2019

davleopo commented Jul 4, 2019

farquet left a comment

farquet commented Jul 4, 2019

axel22 commented Jul 4, 2019 •

edited

Loading

axel22 left a comment

axel22 commented Jul 4, 2019

vhotspur left a comment

Add Scala Doku Benchmark #173

Add Scala Doku Benchmark #173

Conversation

davleopo commented Jul 3, 2019

vhotspur commented Jul 3, 2019

davleopo commented Jul 3, 2019

axel22 commented Jul 3, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vhotspur commented Jul 3, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davleopo commented Jul 4, 2019

farquet left a comment

Choose a reason for hiding this comment

farquet commented Jul 4, 2019

axel22 commented Jul 4, 2019 • edited Loading

axel22 left a comment

Choose a reason for hiding this comment

axel22 commented Jul 4, 2019

vhotspur left a comment

Choose a reason for hiding this comment

axel22 commented Jul 3, 2019 •

edited

Loading

axel22 commented Jul 4, 2019 •

edited

Loading