Engine benchmark regression on 2024-03-04

[Akirathan]

We have some huge benchmark regressions on 2024-03-04.

Regressed Engine benchmarks:

• org_enso_interpreter_bench_benchmarks_semantic_IfVsCaseBenchmarks_caseBench3
  • +577%
• org_enso_interpreter_bench_benchmarks_semantic_IfVsCaseBenchmarks_caseBench6
  • +378%
• org_enso_interpreter_bench_benchmarks_semantic_IfVsCaseBenchmarks_ifBench3
  • +477%
• org_enso_interpreter_bench_benchmarks_semantic_IfVsCaseBenchmarks_ifBench6
  • +233%
• org_enso_interpreter_bench_benchmarks_semantic_StringBenchmarks_lengthOfStrings
  • +1400%
• org_enso_interpreter_bench_benchmarks_semantic_WarningBenchmarks_noWarningsVecSum
  • +2600%
• org_enso_interpreter_bench_benchmarks_semantic_WarningBenchmarks_randomElementsVecSum
  • +557%
• org_enso_interpreter_bench_benchmarks_semantic_EqualsBenchmarks_equalsTrees
  • -82%

stdlib benchmarks appear to be fine.

Commits that likely caused the regressions:

• good: 39af372
• bad: 5d00a61
• between good and bad: 39af372...5d00a61
• The one commit that is most likely responsible for the regression is Autoscoped constructors #9190

[Akirathan]

engine.BackgroundCompilation was newly set to false in https://github.com/enso-org/enso/pull/9190/files#diff-7c0bf5c634ebfb606d229c275ac2cba4f74f2b156a6a674a43a6bc22d81968aaR58-R71. This is very bad idea for benchmarks. Let's revert this.

GitHub

Autoscoped constructors by JaroslavTulach · Pull Request #9190 · enso-org/enso

Pull Request Description Fixes #8645 by recognizing ~ prefix to constructor names. Checklist Please ensure that the following checklist has been satisfied before submitting the PR:

The documentat...

[Akirathan]

@JaroslavTulach's theory is that engine.BackgroundCompilation=false should not affect performance. Let's ensure that this is the case (DISCLAIMER: It is not - it causes huge regressions).

Experimenting with:

sbt:runtime-benchmarks> run -wi 3 -w 10 -i 3 -r 10 org.enso.interpreter.bench.benchmarks.semantic.WarningBenchmarks.noWarningsVecSum

(noWarningsVecSum benchmark with 3x10 seconds warmup and 3x10 seconds measurement)

Local score with --engine.BackgroundCompilation=false: 0.183
Local score with --engine.BackgroundCompilation=true: 0.009

Setting --engine.MultiTier to either false or true does not seem to have any impact on the final score.

Run with CompilationStatisticDetails=true

Running with --engine.CompilationStatisticDetails reveals that:

BackgroundCompilation=false

Compilations: 25
Success: 24
Temporary bailouts: 0
Permanent bailouts: 1

BackgroundCompilation=true

Compilations: 83
Success: 76
Temporary bailouts: 6
Permanent bailouts: 1

This just tells us that there is a lot more compilation for engine.BackgroundCompilation=true, which is a bit obvious.

TL;DR;

--engine.BackgroundCompilation=false causes huge regressions. We should not use it. Moreover, it is not used in production, so let's not use it for benchmarking. The same is true for --engine.MultiTier=false. In production, we use --engine.MultiTier=true (the default). Why use something different in benchmarks?

[JaroslavTulach]

MultiTier

In production, we use --engine.MultiTier=true (the default). Why use something different in benchmarks?

The IGV graphs are simpler to analyze with MultiTier=false and as you write...

Setting --engine.MultiTier to either false or true does not seem to have any impact on the final score.

...it doesn't affect the result of final compilation. E.g. easier to disable multi tier compilation in benchmarks.