Execute (and analyze) single Bench.measure

[JaroslavTulach]

The current Enso benchmarking infrastructure allows to take highlevel view of the benchmark runs, but doesn't allow simple transfer of such benchmarks for detailed analysis via lowlevel tools like IGV. As a result we are working with Enso benchmarks as a blackbox. We know something is slow, but we don't have a way to increase our cluelessness by easily running the same Bench.measure benchmark in isolation, with deep insight into the compilation of the single benchmark functionality, without being distracted by the rest.

Tasks

Preview Give feedback

1. Design API for declarative definition of Enso benchmarks - currently this API seems as an acceptable starting point - see Bench builder API #7324
2. Design sbt command to run all Enso benchmarks (written using Bench builder API #7324)
3. Design sbt command to run single benchmark given its name (or other ID)
4. Connect with IGV - play nicely with withDebug --dumpGraphs
5. Align with @Warmup(iterations) concept of JMH - Add benchmarks comparing performance of Table operations 'vectorized' in Java vs performed in Enso #7270 (comment)

The exploratory work has already been done in #7101. Time to finish it or to create new PR(s) to move us forward. As the discussion at #7270 (comment) shows, we need to start writing benchmarks in a way that is more lowlevel friendly (as the amount of work to modify original #5067 benchmark was too high to be manually repeated with every benchmark written).

Follow-up tasks

• Ability to invoke all std-benchmarks via JMH #7489
• Create a GitHub action for std-benchmarks #7490

[enso-bot]

Pavel Marek reports a new STANDUP for yesterday (2023-07-17):

Progress: - Working on the prototype of JMH benchmarks for Enso libs

• Moving the prototype into a separate SBT project It should be finished by 2023-07-28.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-07-18):

Progress: - Discussion about bench API specification - how to specify before and after functions? It should be finished by 2023-07-28.

[enso-bot]

Pavel Marek reports a new STANDUP for yesterday (2023-07-20):

Progress: - The CLI of the custom JMH runner conforms to the standard JMH CLI

• The only exception is the meaning of includes regex specification, where we have to specify group along with label
• Got working benchmark discovery, listing, and help message It should be finished by 2023-07-28.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-07-21):

Progress: - Troubleshooting the recent GraalVM version update:

• Trying to fix a broken build on MacOS
• Adding back GraalVM version check to sbt build
• Troubleshooting various problems with IDE build after the GraalVM update
• Trying to merge the fixes in Fix GraalVM distribution download on MacOS #7364 ASAP. It should be finished by 2023-07-28.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-07-24):

Progress: - Reverting some incompatible changes that I introduced after Graal Update

• I accidentally changd the naming of our artifacts, instead of only the name of downloaded graal artifacts
• Fix of a Dockerfile.
• Supporting Greg T
  • Consultation how to initialize local IDE build.
• JMHization:
  • Integrating annotation processor
  • I tried hard to not introduce yet another sbt project, but failed to do so.
  • There will probably be two projects: bench-libs and bench-processor. It should be finished by 2023-07-28.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-07-25):

Progress: - Figuring out how to cmdline options for frgaal compiler so that we have a custom class path in the annotation processor.

• At the end, I was able to do it with some custom option handling in the FrgaalCompiler sbt plugin.
• Finally got all the Truffle languages, along with Enso, in the benchmark annotation processor!. It should be finished by 2023-07-28.

[enso-bot]

Pavel Marek reports a new STANDUP for yesterday (2023-07-26):

Progress: - Struggling a bit with the sbt build config again.

• Integrating bench and benchOnly SBT commands. It should be finished by 2023-07-28.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-07-27):

Progress: - Finished the prototype, now we can generate all the JMH code for benchmarks, run a single benchmark, and run all the benchmarks

• Added some docs for benchmarks and for debugging annotation processors.
• Tomorrow, I will merge the PR. It should be finished by 2023-07-28.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-07-28):

Progress: - Integrating a lot of suggestions from the review.

• Refactoring other Enso benchmarks to the new builder API.
• Putting all the result collection logic to bench-processor.
• Adding some annotation parameters so that the processor can collect suites from more files. It should be finished by 2023-07-28.

[enso-bot]

Pavel Marek reports a new 🔴 DELAY for today (2023-07-31):

Summary: There is 9 days delay in implementation of the Execute (and analyze) single Bench.measure (#7323) task.
It will cause 9 days delay for the delivery of this weekly plan.

Delay Cause: We concluded that we want to finish the whole integration of JMH for stdlib benchmarks generation. In other words, the end goal of this task is to create a new CI job that runs all the std lib benchmarks and collects the data. This is much more involved than a simple ability to run just one benchmark locally.

[enso-bot]

Pavel Marek reports a new STANDUP for the provided date (2023-08-06):

Progress: - Bumped into yet another AssertionError in Truffle's source section - #5585

• Reproduing issue with blank screen after opening any project on all the AppImages from July
• Emphasizing discussion about mismatch of versions between IDE and Language Server/Engine.
• Review of BigInteger support.
• Writing some tests for SpecCollector.
  • Struggling with how to create and compile a new module via enso polyglot API. It should be finished by 2023-08-06.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-07-31):

Progress: - Bumped into yet another AssertionError in Truffle's source section - #5585

• Reproduing issue with blank screen after opening any project on all the AppImages from July
• Emphasizing discussion about mismatch of versions between IDE and Language Server/Engine.
• Review of BigInteger support.
• Writing some tests for SpecCollector.
  • Struggling with how to create and compile a new module via enso polyglot API. It should be finished by 2023-08-06.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-08-01):

Progress: - Added some reasonable annotation parameters specifying benchmark discovery.

• BenchProcessor is able to collect more than 1 suite now.
• Integrated some tests via a package in test resources.
• Some reviews
• Bookclub It should be finished by 2023-08-06.

[enso-bot]

Pavel Marek reports a new STANDUP for yesterday (2023-08-02):

Progress: - Integrating many review comments It should be finished by 2023-08-06.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-08-03):

Progress: - Integrating rest of the review comments

• The PR is ready for review
• Let's merge it ASAP
• Created follow-up tasks Ability to invoke all std-benchmarks via JMH #7489 and Create a GitHub action for std-benchmarks #7490 It should be finished by 2023-08-06.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-08-04):

Progress: - Some problems with CI - jobs are taking unusually long, cannot merge today. It should be finished by 2023-08-06.