Rethinking the Karma Test Adapter

[MortenHoustonLudvigsen]

When I initially implemented the Karma Test Adapter I made the assumption, that it would be possible to statically analyze test files to discover tests by parsing them.

Since then I have learned:

_There is no (easy) way to discover tests without running them!_

In this simple example it would be hard to determine test names just by parsing:

describe("Suite", function() {
    for (var i = 0; i < 10; i++) {
        it("sould not fail - " + i, function() {
        });
    }
});

A test adapter for the Visual Studio test explorer has three phases:

1. Discovery of test containers.
   In our case this would be karma.conf.js or karma-vs-reporter.json.
2. Discovery of tests for each test container.
   This is where the current version of the Karma Test Adapter parses the test files to discover tests. As I have learned, this is very fragile, and can easily fail.
3. Run the tests discovered
   The current version of the Karma Test Adapter "tells" Karma to run the tests.

Based on this I am working on a rewrite of the Karma Test Adapter. The new adapter will:

• Always run Karma in the background
• If karma.conf.js or karma-vs-reporter.json change karma in the background will be restarted using the changed settings.
• In karma-vs-reporter.json it will be possible to turn off the adapter.
• When Visual Studio wants to discover tests the background Karma process will be asked to run them. Test files will no longer be parsed to discover tests.
• When Visual Studio wants to run tests the background Karma process will be asked to return the result of the latest test run.

The fact that test files will no longer be parsed to find tests leads to the following:

• The file name and line number of each test will not be available from parsing of test files. My plan is to instrument the jasmine functions describe and it to include this information in the test results using a stack trace.

[MortenHoustonLudvigsen]

I have now made a prerelease of the next version of the karma test adapter.
There are some major changes:

• The node package karma-vs-reporter is no longer used, and is deprecated.
• The optional settings file is now called KarmaTestAdapter.json, not karma-vs-reporter.json.
• Temporary files are no longer used to communicate between different parts of the adapter. This means that it no longer makes sense to have an OutputDirectory option in KarmaTestAdapter.json.
• Karma is always run in "server" mode, and the port to use is generated automatically. This means that options ServerMode and ServerPort are no longer available in KarmaTestAdapter.json.
• It is possible to disable the adapter using the Disabled option in KarmaTestAdapter.json.
• The adapter no longer watches for changes in files (except for karma.conf.js and KarmaTestAdapter.json). This is handled by Karma. This means that option TestFiles is no longer available in KarmaTestAdapter.json.
• Tests are run immediately when any file, that Karma watches, is changed.

Before I publish this version, I would really appreciate it if some of the users of the adapter could test
it. The vsix file can be found here: v1.0.0-alpha.1.

[jitterbox]

Based on your comments, would another adapter extension of this sort have to be written in order to support running Protractor-based (Jasmine) tests from Visual Studio? I would imagine there would be a fair amount of code overlap...would it make more sense for something like that to be a fork of this project, or a separate project?

I'll test your alpha.

[MortenHoustonLudvigsen]

@jitterbox I must admit that I have not looked a Protractor before now. My initial feeling is that the Karma Test Adapter is currently quite closely coupled to Karma. So I would think that another adapter should be written for Protractor.

It might be possible to refactor the Karma Test Adapter so code overlap could be avoided. However, I think it would take a bit of work. 😄

Thank you very much for testing the alpha (now: v1.0.0-alpha.3), it is greatly appreciated!

[MortenHoustonLudvigsen]

As described in karma-runner/karma#1213 Karma sometimes fails when files it watches are changed. Therefore, as of v1.0.0-alpha.4, the adapter will restart Karma if it fails.

[MortenHoustonLudvigsen]

I have now published version 1.0.0 of the adapter.
Please create new issues for any problems with the new adapter.