Refactor Standard.Test to the builder API

[Akirathan]

We would like to first collect all the tests and then execute them, optionally with some filtering.

The builder API pattern for benchmarks was introduced in #7324

Ideas for setup and teardown methods are in these commits:

• 48a523f
• f4e287d

In the future, we might want to integrate this into vscode test runner.

Tasks

Preview Give feedback

1. Implement filter alá - Filter for test/Benchmarks #8391
2. No special support for setup - let users use lazy constructor arguments
3. after method registration alá specify registration

[JaroslavTulach]

Setup - let's do the setup the same way as as we do it in benchmarks API - e.g. Suspended constructor arguments which get initialized when needed once.

Teardown - let's add method after method next to specify - there can be multiple functions register as after in each group. They are guaranteed to be invoked after all specify functions are invoked.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-12-21):

Progress: - Follow-up fixes after #8374 - #8620, #8593

• Discussion with Hubert about correct test log provider
• Fixing Engine benchmark build, optimizing the build - don't require fat jar assembly
• Starting the design of the new std test builder API It should be finished by 2024-01-05.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-12-22):

Progress: - Finished analysing #8441 for now - didn't find a smaller reproducer yet.

• Designing the new builder API.
  • We may be able to not change the current API at all, if we use State, I will experiment with the design in upcomming days.
  • Created a new Test_New library, that will, for now, stay along the old Test library. It should be finished by 2024-01-05.

[enso-bot]

Pavel Marek reports a new STANDUP for yesterday (2023-12-27):

Progress: - Waiting for a discussion: two API propositions It should be finished by 2024-01-05.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-12-29):

Progress: - Continuing with the new builder design API It should be finished by 2024-01-05.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-01-03):

Progress: - Reviews

• Investigating slowdown of org.enso.interpreter.bench.benchmarks.semantic.VectorBenchmarks.averageOverVector engine benchmark reported by Hubert
  • Most likely, the data set is just too small for this benchmark.
  • My theory that the benchmark slowdown is caused by changing the logger from NOPLogger to org.enso...TestLogProvider was not prooved.
• Trying to migrate SQLite_Spec to the builder API
  • Demonstrate the lazy field initialization It should be finished by 2024-01-05.

[enso-bot]

Pavel Marek reports a new 🔴 DELAY for today (2024-01-08):

Summary: There is 14 days delay in implementation of the Refactor Standard.Test to the builder API (#7566) task.
It will cause 0 days delay for the delivery of this weekly plan.

There must be maybe multiple iterations of discussions with the lib team about the designed API. I don't even have a PoC yet.

Delay Cause: This delay is expected. I have provided very short estimated time for this issue.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-01-08):

Progress: - Noticed that our graphs for benchmarks are generated from wrong data.

• We need to ensure that benchmarks are run on clean machines.
• Benchmark downloader script should take the latest score, not the smallest #8707
• Figuring out how to provide before_each setup function to the Group_Builder.
  • The data initialization in SQLite_Spec cannot be initialized just by lazy fields, because they are initialized just once, we need before_each. Alternativelly, we can manually do setup in every spec, but that is very cumbersome. It should be finished by 2024-01-19.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-01-09):

Progress: - Helping out Hubert with the JPMS setup for the Arrow language.

• Providing some benchmark guidance.
• Continue with the migration of SQLite_Spec
  • Bumped into State is not propagated to lazy field evaluation #7571 - increasing its priority
  • Tried to add methods group_with_setup and specify_with_args to the API.
    • Will probably revert it, it might really be unnecessary and we should be able to do setup via lazy atom fields. It should be finished by 2024-01-19.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-01-10):

Progress: - Meetings.

• Finalized the builder API, until Friday, I should publish a working prototype for new SQLite_Spec and open a discussion. It should be finished by 2024-01-19.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-01-11):

Progress: - Investigating native image builds - #8724

• Bumped into a bug with Runtime.assert when there are suspended arguments and lazy fields.
  • Struggling to reproduce that case in a simpler scenario
  • Let's first fix Runtime.assert does not work when called from lazy atom field #8741 It should be finished by 2024-01-19.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-01-12):

Progress: - Review of arrow language PR

• Trying to improve the assertion error stack trace in Fix Runtime.assert #8742
  • Ended up removing the workaround, didn't get it working.
• Ensure that Runtime.assert is a nop in production.
  • Fix Runtime.assert #8742 (comment)
  • Merged Fix Runtime.assert #8742
• Provide a summary of my two prototypes and waiting for the discussion.
  • Refactor Table_Tests to the builder API #8622 (comment) It should be finished by 2024-01-19.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-01-15):

Progress: - Providing an assistance with problem in IsSameObjectNode.

• Figured out a minimal reproducer.
• Fix Truffle invariants for Nothing with a warning #8764
• Fixing an incorrect behavior of SBT's withDebug --dumpGraphs in runtime project.
  • Fix withDebug benchOnly command #8769
• Get rid of an annoying error/warning in build of runtime-test-instruments.
  • Add Truffle TCK libraryDependencies to runtime-test-instruments. #8766
• Scheduled review with Jaroslav for tomorrow on the TestBuilder API. It should be finished by 2024-01-19.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-01-16):

Progress: - Finished discussion about the design of the API, let's start refactoring in bulk It should be finished by 2024-01-19.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-01-17):

Progress: - More rewriting, fixing many compilation errors, trying to ensure lazy initialization. It should be finished by 2024-01-19.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-01-18):

Progress: - Still more rewritting.

• The migration of Table_Tests is almost done, I may be able to mark the PR as ready for review tomorrow.
• All SQLite tests work.
• JUnit output works. It should be finished by 2024-01-19.

[enso-bot]

Pavel Marek reports a new 🔴 DELAY for today (2024-01-19):

Summary: There is 6 days delay in implementation of the Refactor Standard.Test to the builder API (#7566) task.
It will cause 6 days delay for the delivery of this weekly plan.

Still need to migrate the Postgres tests. The PR is not yet ready to review. I expect that I will finish it in the begining of the next week.

Delay Cause: Didn't quite finish the migration of Table_Test to the new API.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-01-19):

Progress: - Assisting Hubert with some class loading problems in runtime tests and instruments.

• Still migrating a lot of sources to Test_New API.
• Didn't get to migration of Postgres tests, I will finish that in the begining of the next week. It should be finished by 2024-01-25.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-01-22):

Progress: - Book club

• Discussion about clean before any benchmark job
• Refactoring Postgres tests
  • Dealing with some "Too many clients connected" errors - need to add teardowns on many places that close the connection to the database.
• There are some connections that leak from teardowns. It should be finished by 2024-01-25.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-01-23):

Progress: - The refactor PR is ready for review

• Making some last bits, comparing the JUnit xml files.
• Created a collection of issues to track the progress of private keyword implementation:
  • Implement private methods #8834
  • Implement private types #8835
  • Implement private constructors & deconstructors/getters for types #8836 It should be finished by 2024-01-25.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-01-24):

Progress: - Blocked on timeouts in LibrariesTest

• Only Engine (Linux) is blocked - tried restarting it three times
• Engine (Windows) already succeeds.
• Will try merging and restarting again tomorrow. I might be able to merge it after that.
• Working on Avoid heavyweight (fansi, Truffle) runtime-compiler dependencies #8210 in the meantime. It should be finished by 2024-01-25.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-01-25):

Progress: - Comparing which tests are run on old API vs on the new API.

• A bit tedious work, but I've managed to ensure that all the tests from the old API are run.
• I had to refactor also Numbers_Spec from Base_Tests, as some test from Table_Test uses it.
  • They both transiently use Round_Spec from Base_Test which is already refactored to the new API.
• Resolving last issues, hoping to merge it tomorrow.
  • Still enountering a lot of transient issues. It should be finished by 2024-01-25.