Run VSCode extension's tests on top of che-theia

[svor]

What does this PR do?

This PR contains an action VSCode Extension's Tests to run tests from VSCode extension on top of Che-Theia if it was changed in che-theia-plugins.yaml.

VSCode Extension's Tests workflow:
https://github.com/svor/che-plugin-registry/actions/workflows/vscode-extension-tests-check.yaml

1. The first step is Checkout. It clones the repository and makes checkout to the working branch.

2. Step 2 is Get changes. This step executes ./ci/get-changed-plugin.sh. In this script we check what exactly was changed in che-theia-plugins.yaml. If this file doesn't contain changes on lines with key revision - it means that none extension is going to be updated or added and in this way we set EXTENSION_REPO environment variable as empty and all next action's steps will be skipped (action should be finished). In another way, when some value of revision was changed, we find all information about the repository of changed extension (url and a revision).

3. Step 3 is Build image. It builds che-plugin-registry image with all updates. It is needed to run the workspace with updated/added plugin.

4. Step 4 is Start minikube. Needs to setup minikube

5. Step 5 is Load image. It loads che-plugin registry image into docker registry

6. Step 6 is Deploy Eclipse Che. This step uses che-incubator/che-deploy-action@next action to deploy Che with custom che-plugin-registry image into minikube.

7. Step 7 is Run VSCode extension's tests. This step executes ./ci/run-extension-tests.sh. The script has a few steps:

   • First of all we need to install all required dependencies into the runner (selenium, geckodriver)
   • By using url and revision from step add cico script and rhel dockerfile #2 we clone extension repository into the /tmp/projects/ folder and do checkout to the branch
   • Compile extension (yarn install and yarn compile --- I'm not sure if it will work for all extension)
   • Next step is to prepare a devfile to create tested workspace. A template of this devfile is in .ci/templates/extension-tests-devfile.yaml and it looks like

apiVersion: 1.0.0
metadata:
  name: extension-test-sample
components:
  - mountSources: true
    memoryLimit: 3Gi
    type: dockerimage
    image: 'quay.io/eclipse/che-theia-dev:next'
    alias: che-dev
  - type: chePlugin
    reference: 'https://raw.githubusercontent.com/svor/che-vscode-extension-tests/main/meta.yaml'
  - type: chePlugin
    id: @
  - id: eclipse/che-theia/next
    memoryLimit: 1024Mi
    type: cheEditor

• When the devile is ready we can start the workspace. By using chectl we start the workspace and check when its status will be Running. When the workspce is ready, we copy extension's sources into /projects in theia-ide container. After that by using python .ci/tests-runner.py we run selenium script to open the workspace and run extension's tests.
• The last step is to check /projects/test.log file which should contain the result of test execution.

So, this step builds id of tested extension by analysing che-theia-plugins.yaml. If the changed plugin contains id we just take it, if not we build it from extension's package.jon as publisher/name/latest. When we have the id we need to replace @ with it in the devfile.

How are tests running.

The workspace contains che-vscode-extension-tests plug-in (it locates in local repository https://github.com/svor/che-vscode-extension-tests) it is included in the devfile:

  - type: chePlugin
    reference: 'https://raw.githubusercontent.com/svor/che-vscode-extension-tests/main/meta.yaml'

When the workspace was started, this plugin automatically finds all *.test.js files in /projects folder inside the workspace and run them one by one via mocha framework.

For now tests are failed because each test needs to execute vscode.extensions.getExtension(id) to activate an extension and it gets Cannot read property 'getExtension' of undefined. There is an issue to investigate this problem: eclipse-che/che#19172

Screenshot/screencast of this PR

Some details of execution is here https://github.com/svor/che-plugin-registry/runs/1979560988?check_suite_focus=true

What issues does this PR fix or reference?

eclipse-che/che#18219

How to test this PR?

PR Checklist

As the author of this Pull Request I made sure that:

• The Eclipse Contributor Agreement is valid
• Code produced is complete
• Code builds without errors
• Tests are covering the bugfix
• The repository devfile is up to date and works
• Sections What issues does this PR fix or reference and How to test this PR completed
• Relevant user documentation updated
• Relevant contributing documentation updated
• CI/CD changes implemented, documented and communicated

Reviewers

Reviewers, please comment how you tested the PR when approving it.

[ericwill]

Nice work @svor 🎉

I left some comments, see them inline

[ericwill]

I think it would be good to extract out the arguments for git diff into arguments for the script. That way it's not hardcoded in the script, but takes input from the GitHub action job.

For example:
https://github.com/eclipse/che-plugin-registry/blob/master/.ci/sidecar-build-publish.sh#L16
https://github.com/eclipse/che-plugin-registry/blob/master/.github/workflows/sidecar-build-push.yaml#L48

[ericwill]

Let's try a better name than vscode-extension, maybe vscode-extension-test or something similar?

[ericwill]

Please create an variable for the geckodriver version, and use that variable in the URL. This makes it easier to read and update.

[ericwill]

It might be better to create a user level virtual environment.

[ericwill]

Looks like a print statement left over from development 😄

[ericwill]

Is there a central place we can put this, i.e. not your fork?

[svor]

@ericwill I think we can move this extension into che-incubator, but when eclipse-che/che#19172 will be resolved. I'm still working on this extension.

[benoitf]

let's move all the logic of this file to a github action

[svor]

@benoitf do you mean to create more steps in the action and each step will execute some part of this script?

[benoitf]

I was not clear enough

today you're using that script by doing:

name: Run VSCode extension's tests
        if: env.EXTENSION_REPO != ''
        run: |
          ./.ci/run-extension-tests.sh

but if you turn it to a github action, then we could just invoke it by calling

       if: env.EXTENSION_REPO != ''
        uses: che-incubator/vscode-extension-tests@next

(so plug-in registry is only a client of the action hosted elsewhere)

[benoitf]

and with github actions it's easy to get unit tests / code coverage / etc.

[openshift-ci]

@svor: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/v7-plugins-test-pr-check	80c52ff	link	/test v7-plugins-test-pr-check

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[benoitf]

outdated