Ensure that bench downloading script downloads artifacts from correct job

[Akirathan]

A follow-up of #9322. The Benchmark Engine workflow now has two jobs. One that runs with Oracle GraalVM and another one that runs with GraalVM CE. Ensure that the website_regen.py fetches artifacts from the GraalVM CE job so far.

[JaroslavTulach]

Ensure that ... fetches artifacts from the GraalVM CE job so far.

It'd be even better if the official benchmark results displayed values from both GraalVM versions. Each in different color.

At the end the goal is to find out why using Oracle GraalVM is better than sticking with GraalVM CE. Performance is likely one of the reasons. Having such reason directly that visible in the benchmarks result might be the convincing point.

[Akirathan]

First successful engine-benchmark.yml workflow run is at https://github.com/enso-org/enso/actions/runs/8257237123:

• Engine Benchmarks (GraalVM CE)
• Engine Benchmarks (Oracle GraalVM)

The uploaded Runtime Benchmark Report artifact contains results from the Engine Benchmarks (GraalVM CE). There are no artifacts for the Oracle GraalVM job.

GitHub

Benchmark Engine · enso-org/enso@ef82637

Hybrid visual and textual functional programming. Contribute to enso-org/enso development by creating an account on GitHub.

[Akirathan]

Preliminary manual comparison of GraalVM CE VS Oracle GraalVM engine benchmarks:

Label	GraalVM CE	Oracle GraalVM
org.enso.compiler.benchmarks.module.ImportStandardLibrariesBenchmark.importStandardLibraries	159.612	94.404
org.enso.interpreter.bench.benchmarks.semantic.ArrayProxyBenchmarks.sumOverComputingProxy	0.120	0.090
org.enso.interpreter.bench.benchmarks.semantic.ArrayProxyBenchmarks.sumOverDelegatingProxy	0.120	0.089
org.enso.interpreter.bench.benchmarks.semantic.ArrayProxyBenchmarks.sumOverVector	0.081	0.033
org.enso.interpreter.bench.benchmarks.semantic.EqualsBenchmarks.equalsTrees	1.002	0.614
org.enso.interpreter.bench.benchmarks.semantic.IfVsCaseBenchmarks.caseBench3	2.800	2.201
org.enso.interpreter.bench.benchmarks.semantic.IfVsCaseBenchmarks.caseBench6	3.205	0.989
org.enso.interpreter.bench.benchmarks.semantic.IfVsCaseBenchmarks.ifBench3	2.566	0.446
org.enso.interpreter.bench.benchmarks.semantic.IfVsCaseBenchmarks.ifBench6	2.980	6.048
org.enso.interpreter.bench.benchmarks.semantic.IfVsCaseBenchmarks.ifBench6In	2.951	12.164

[JaroslavTulach]

Re. ifBench6 - the manual inlining written by us is used in this benchmark - possibly it doesn't work well with Oracle GraalVM. Workaround: disable it and let Oracle do its job!

Otherwise the improvements look very nice!

[Akirathan]

Re. ifBench6 - the manual inlining written by us is used in this benchmark - possibly it doesn't work well with Oracle GraalVM. Workaround: disable it and let Oracle do its job!

Related to the work done in #5709

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-06-10):

Progress: - Making sure that results from Oracle GraalVM benchmarks are persistent.

• bench result reporter in benchmarks-common refactored to report a single JSON file, according to a specific schema.
• Most of the bench_download.py tool will get deprecated soon, let's replace it with a typescript CLI project.
  • Much easier to maintain, test, etc.
  • There is official octokit library for the GH API, unlike to the Python one.
  • Struggling with even generating venv for Python
  • This will be a HUGE simplification - just upload data from a single Github Action, this should be just a few dozens of JS code. It should be finished by 2024-06-12.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-06-11):

Progress: - Closing stale PRs

• Adding more tests for BindingsMap.exportedSymbols.
• Implementing new tool for downloading benchmarking data and uploading them to engine-bench-results.
  • Is a breeze with ockokit and NodeJS. It should be finished by 2024-06-12.

[Akirathan]

Add #9355 (comment) : No artifacts for Oracle GraalVM benchmark job means that we have to gather the data from the job output. Note that job outputs also expire after 3 months, like artifacts.

[Akirathan]

Last 3 months of Oracle GraalVM bench results persisted in 70cb785

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-06-12):

Progress: - Results from Oracle GraalVM benchmark runs gathered in #10224 (comment)

• It was more work than expected, as I needed to parse the results from job outputs - there are no artifacts for EE benches. It should be finished by 2024-06-12.