Add tests namespace from VSCode API

[sgraband]

Feature Description:

Add function from the tests namespace:

• createTestController

Add required variables, functions and fields from root namespace:

• TestController
• TestItem
• TestItemCollection
• TestMessage
• TestRun
• TestRunProfile
• TestRunProfileKind
• TestRunRequest
• TestTag

VSCode API for reference: https://code.visualstudio.com/api/references/vscode-api#tests.

[fyudanov]

Would be nice to have, e.g. for integrating https://github.com/robocorp/robotframework-lsp with Test Explorer view in Theia.

[planger]

Note that the test API has been stubbed for now. So extensions using this API aren't blocked from running, but usages of the test API won't have any effect.

[KevinClapperton]

Hello, I speak on behalf of the student project team from Polytechnique Montreal (Kevin, Théo, Mathieu and Zakaria). This is our first time contributing and we would like to work on this API. We are still going through some tutorials and documentation to make sure we understand how to implement this feature (we might have a slow start).

Here is what we are doing right now in order to understand how to implement the Test API:

• Reading Plugin-API.md and how to add the test namespace
• Looking how the Test API was stubbed
• Looking how other VSCode API are implemented (path: theia/packages/plugin-ext/src/plugin/)
• Checking VSCode extension sample for the Test API and the guide plus the official documentation
• Looking at a partial implementation of the Test API in Haxe

Question:

• Is there something we should be aware that is part of the fact that Theia can be Desktop or Remote? We saw some functions using remote procedure call (RPC). Is that something we will have to use in order to implement the Test API?

Are we missing something? Is there some documentation or examples we should read?

[D-Zaq]

Hi, are the test coverage and observer part not to be considered in the implementation based on this issue as the functions like runTests, createTestObserver and onDidChangeTestResults that are part of the tests namespace and the different variables and fields that come along were not included in the description and the stubbed tests API

[D-Zaq]

Hi, are the test coverage and observer part not to be considered in the implementation based on this issue as the functions like runTests, createTestObserver and onDidChangeTestResults that are part of the tests namespace and the different variables and fields that come along were not included in the description and the stubbed tests API

@sgraband

[sgraband]

The functions you are listed are part of vscodes proposed api. If i rememeber correctly proposed api is not required to be compatible. @planger is this correct?

[KevinClapperton]

Hi, are the test coverage and observer part not to be considered in the implementation based on this issue as the functions like runTests, createTestObserver and onDidChangeTestResults that are part of the tests namespace and the different variables and fields that come along were not included in the description and the stubbed tests API

We talked about this in today's DevMeeting. From what I understood, if the Test API doesn't make heavy use of the proposed API, then there is no need to implement it.

[KevinClapperton]

I wanted to add some information of something that we found during the last month. There are some UI implementation to be done to make this API complete. There is something called testingDecoration that are the little play buttons that should appear in the editor's gutter when a test is detected. Here is more information on the UI implementations:

• testingDecoration in the common folder: https://github.com/microsoft/vscode/blob/1.72.2/src/vs/workbench/contrib/testing/common/testingDecorations.ts
• Some more files on the UI implementation in the browser folder : https://github.com/microsoft/vscode/tree/1.72.2/src/vs/workbench/contrib/testing/browser
• Video showing a example of what it should look like: https://youtu.be/AulgIP9P8Cw?t=60

When we first looked at this issue we didn't think there was any UI implementation to be migrated. Is this part of the issue?

[KevinClapperton]

I wanted to add some information of something that we found during the last month. There are some UI implementation to be done to make this API complete. There is something called testingDecoration that are the little play buttons that should appear in the editor's gutter when a test is detected. Here is more information on the UI implementations:

* testingDecoration in the `common` folder: https://github.com/microsoft/vscode/blob/1.72.2/src/vs/workbench/contrib/testing/common/testingDecorations.ts

* Some more files on the UI implementation in the `browser` folder : https://github.com/microsoft/vscode/tree/1.72.2/src/vs/workbench/contrib/testing/browser

* Video showing a example of what it should look like: https://youtu.be/AulgIP9P8Cw?t=60

When we first looked at this issue we didn't think there was any UI implementation to be migrated. Is this part of the issue?

We discussed this in a Dev Meeting and the answer is yes. The UI must be migrated as well.

[KevinClapperton]

@JonasHelming @D-Zaq @mathblin @paul-marechal @colin-grant-work @MatthewKhouzam @tsmaeder
Here is a condensed report of known issues and what is/isn't implemented in the current PR :

Known issues:

• No known issues in Plugin Host code
• Missing translation for the menu: There's no translation for menu 'view -> test' #12372
• The Main Browser code of the Test API doesn't use Inversify and needs a lot of changes to work without failing in runtime. See this comment

Implemented:

• Testing Workspace
  • packages/testing/src/common folder with the code necessary for Plugin Host (contains a lot of code necessary for Main Browser as well)
  • This workspace contains code from VS Code found in the following folders:
    • vscode/src/vs/workbench/contrib/testing/common
    • vscode/src/vs/base/common
    • vscode/src/vs/editor/core
  • Imported interfaces, classes, variables, functions and fields:
    • TestController (found in tests.ts implemented at run-time in function TestingExtImpl::createTestController)
    • TestItem (found in test-items.ts as TestItemImpl)
    • TestItemCollection (found in `test-item-collection.ts)
    • TestMessage (already found in types-impl.ts)
    • TestRunProfile (found in tests.ts as TestRunProfileImpl)
    • TestRunProfileKind (already found in types-impl.ts)
    • TestRun (found in test.ts)
    • TestRunRequest
    • TestTag
• Plugin Host logic in tests.ts (the equivalent of ExtHostTesting.ts in vscode)
  • Imported ExtHostTesting from vscode now called TestingExtImpl

Note: The following diagram shows the files that were added or modified to make Plugin Host work:

• In the diagram, each node represents a files path and each leaf (green or grey rectangle) represents a file. Ex: if we look at the top-left green leaf called constants.ts, we can look at its parent nodes common in light blue, testing/src in orange and theia/packages in dark blue and know the file constants.ts is found in the path theia/packages/testing/src/common/constants.ts
• Green leafs are added files by the current commit
• Grey leafs are modified files by the current commit

draw.io file: migrated-and-changed-files.zip

Not implemented:

• Main browser (rpc calls to the Main Browser do nothing)
• UI and UI logic

Note: The following diagram shows the files from VS Code that we think should be imported for the Main Browser and UI to work (other files might be missing) :

• See previous diagram for explanation on how this diagram works
• The colour of the leafs have no meaning in this diagram

draw.io file: files-to-migrate.zip

[KevinClapperton]

Feel free to ask any questions about the report or the code before the next meeting on May 2th. That way we will be able to answer them in more detail at the meeting.

[KevinClapperton]

Here is a simple extension that pokes the functions createTestController, createRunProfile, createTestItem and createTestRun of the Test API. It returns a notification for each of the functions with "SUCCESS" or "FAIL" with details if the functions throw an error.

vsix file: test-api-functional-test-0.0.2.zip
Source code: https://github.com/KevinClapperton/Test-API-functional-test

Once the extension is installed in Theia, execute the command Test API: Run All Test in the command palette to call the Test API functions.

[tsmaeder]

I'm thinking about what UI we would have to have to implement this API:

1. Test Explorer View
   Shows all tests in a tree structure and has toolbar/drop downs to run test, goto location, hide/show, etc. Shows failures, errors, etc.
2. Test Markers in the Editor
   You can run tests from the marker
3. Test output view
   VS Code uses a terminal widget. Not sure that would work for us, based on our implementation
4. Various commands ("configure test profile", "refresh", etc.)
5. Showing errors in the editor in an overlay

In a first step, we could leave out the marker support, since that duplicates functionality from the Test Explorer view. The "peek error" support in VS Code could maybe be replaced by a hover on failed tests. We could also kind of ignore "continuous run" capabilities by always requesting "continuous=false" when running tests. Any "proposed" functionality like coverage or TestObserver could be safely stubbed.

[tsmaeder]

So a preliminary plan might be:

1. Define a TestService package that allows for dynamic "TestController" contributions
2. Make a super-simple test contribution
3. Create a simple Test Explorer View (Tree View) It needs to be able to
   • show tests from contributions
   • refresh
   • show test status
   • show test failure messages
   • run tests
   • debug tests
4. Implement test output
5. Implement the "PluginTestController" contributions in the "plugin-ext" package (based on the existing PR).

[tsmaeder]

Some further notes...

1. The test API proper is pretty much a write-only API: there is no way for an extension to obtain references to any test controllers the extension did not create itself. The extension adds things (profiles, test runs, output messages) and writes them to the API, but it can mostly not read back the results of those writes (TestItemCollection being the exception).
2. The Implementation in VS Code seems to employ a tree-delta mechanism to propagate time-batched updates to the set of tree items in a controller. That makes sense, since discovering tests in a workspace (or refreshing a TestItem node) will likely generate many updates in a short time.
3. The proposed TestObserver API allows to observe tests and test results in the system. The API gives you access to TestItem instances, and you would think from the API that you can manipulate these, but in fact, they are effectively readonly. That makes sense, since an extension cannot be expected to run a test it did not create through its controller.
4. Since tests may by created in different plugin hosts (e.g. front-end and back-end), the set of observed tests is non-local: i.e. only the front end can know the set of tests. That means the tests in the TestObserver API is distinct from the set of test in the test API proper. We can therefore reasonably start without the TestObserver API.
5. Updates to the tree related data structures never happen by asking the extensions for anything: if we're refreshing a tree item, the extension actively CRUD's test items. In that respect, there is never any lazyness inside the API.

[tsmaeder]

Note that propagating test run results is also something that might happen at a fast pace an profit from batching. We have to keep in mind that we might have to match a particular result to the location in the test item tree and design data structures accordingly.