TestCafé

This project uses TestCafé for automated testing.

It basically starts a virtualtabletop server, then starts a browser and automatically interacts with the virtualtabletop client. For the most part it loads public library games, clicks multiple game buttons and then checks if the resulting room state looks like it's supposed to.

This is done automatically for every commit to pull requests and the PRs can only be merged if these tests succeed.

They are defined by JavaScript files in tests/testcafe/ and a simple public library test looks like this:

publicLibraryButtons('Dice', 0, '46a339c348790b6df576c3d045602894', [ 'k18u', 'hy65', 'gghr', 'dsfa', 'f34a', 'fusq' ]);

This example loads the game Dice (a tutorial) from the public library, opens variant 0, then clicks the buttons k18u etc. and afterwards it checks if the MD5 hash of the room state is equal to 46a339c348790b6df576c3d045602894.

Running the tests

You can look at the output of the tests for nearly all commits to the repository by clicking at the green checkmark, yellow dot or red cross next a commit and selecting the TestCafe test.

You can run them on your local copy like this (note local server should be running in debug mode, use npm run debug in another terminal):

user@host:virtualtabletop$ npm run testcafe-firefox-headless-all

> [email protected] testcafe-firefox-headless-all
> testcafe firefox:headless tests/testcafe

 Running tests in:
 - Firefox 106.0 / Ubuntu 20.04

 virtualtabletop.io
 ✓ Create game using edit mode

 virtualtabletop.io
 ✓ Compute
 ✓ Dynamic expressions

 virtualtabletop.io
 ✓ Public library: Blue (variant 0)
 ✓ Public library: Bhukhar (variant 0)
 ✓ Public library: Dice (variant 0)
 ✓ Public library: Dots (variant 0)
 ✓ Public library: Solitaire (variant 0)
 ✓ Public library: Mancala (variant 0)
 ✓ Public library: Reversi (variant 0)
 ✓ Public library: Reward (variant 0)
 ✓ Public library: Rummy Tiles (variant 0)
 ✓ Public library: Undercover (variant 1)
 ✓ Public library: Functions - CALL (variant 0)
 ✓ Public library: Functions - CLICK (variant 0)
 ✓ Public library: Functions - ROTATE (variant 0)
 ✓ Public library: Functions - SELECT (variant 2)
 ✓ Public library: Functions - SORT (variant 1)
 ✓ Public library: Master Button (variant 0)


 19 passed (4m 13s)

There are 14 different commands that run the tests in different browsers as defined in package.json:

...
    "testcafe": "testcafe chrome",
    "testcafe-all": "testcafe chrome tests/testcafe",
    "testcafe-headless": "testcafe chrome:headless",
    "testcafe-headless-all": "testcafe chrome:headless tests/testcafe",
    "testcafe-chromium": "testcafe chromium",
    "testcafe-chromium-all": "testcafe chromium tests/testcafe",
    "testcafe-chromium-headless": "testcafe chromium:headless",
    "testcafe-chromium-headless-all": "testcafe chromium:headless tests/testcafe",
    "testcafe-firefox": "testcafe firefox",
    "testcafe-firefox-all": "testcafe firefox tests/testcafe",
    "testcafe-firefox-headless": "testcafe firefox:headless",
    "testcafe-firefox-headless-all": "testcafe firefox:headless tests/testcafe",
    "testcafe-gitpod": "bash tests/testcafe/gitpod.sh",
    "testcafe-gitpod-all": "bash tests/testcafe/gitpod.sh tests/testcafe"
...

The commands ending with -all run all the tests in test/testcafe; the other commands require the path to the tests that you want to run as the first command line argument. Currently the test sections are test/testcafe/editor.js, test/testcafe/functions.js and test/testcafe/publiclibrary.js. To run just the public library tests in firefox, use npm run testcafe-firefox test/testcafe/publiclibrary.js.

Tests can also be selected using pattern matching against the test name by using testcafe's --test-grep pattern option. To have the option passed to the testcafe process instead of being read as an option for the npm script, use -- before --test-grep. If specifying a test file, the -- can appear before or after the test file path. If running -all sections that have no matching tests will be skipped without a message. For instance, npm run testcafe-firefox-all -- --test-grep '.*Dice.*', npm run testcafe-firefox test/testcafe/publiclibrary.js -- --test-grep '.*Dice.*', and npm run testcafe-firefox -- test/testcafe/publiclibrary.js --test-grep '.*Dice.*' all end up with the same test running (there are no tests in editer.js or functions.js that match the pattern .*Dice.*).

For each browser there are headless versions that run the tests in the background and normal versions that actually show you a browser window with automatically running tests (looks really cool).

Running tests in Gitpod

Note: Gitpod deletes all software installed in the container outside of the /workspace directory whenever the container is stopped so any commands that install software must be repeated each time the container is restarted.

gitpod/workspace-full-vnc

If the branch you are working in is up to date with main, Gitpod will create new workspaces from gitpod/workspace-full-vnc. This image has Chrome installed. The normal method for running tests in Chrome should work by default. To watch the tests you can open a browser to the noVNC client being served on port 6080 in the container. Firefox can be installed with sudo apt-get update && sudo apt-get -y install firefox.

gitpod/workspace-full

Older branches of virtual tabletop may still be configured to create new Gitpod workspaces in Gitpod's default image (gitpod/workspace-full) which does not have any browser or graphical desktop installed. To run the tests headless in these containers, use npm run testcafe-gitpod-all. This first ensures that Firefox is installed then runs npm run testcafe-firefox-headless-all. You can install noVNC (and Firefox) using source ./tests/testcafe/gitpod-vnc/install.sh then open a browser to port 6080 to see the noVNC desktop. Then run npm run testcafe-firefox-all to start the tests.

When tests fail

If your PR runs into a failed test, you'll have to check why it fails. If the PR changes the public library game being tested, it's inevitable that its test fails. Apart from that basically only PRs needing a file version bump should expect tests to fail.

Let's look at PR #83 - changing SELECT mode default as an example that is expected to cause test failures.

The test for Dice mentioned above ran into this problem:

user@host:virtualtabletop$ npm run testcafe-firefox-all -- --test-grep '.*Dice.*'

> [email protected] testcafe-firefox
> testcafe firefox tests/testcafe/tests.js "--test-grep" ".*Dice.*"

 Running tests in:
 - Firefox 84.0 / Linux 0.0

 ✖ Public library: Dice (variant 0)

   1) AssertionError: expected '3cd34fc98732a94639d37e677c81b7f2' to deeply equal '46a339c348790b6df576c3d045602894'
      
      + expected - actual
      
      -3cd34fc98732a94639d37e677c81b7f2
      +46a339c348790b6df576c3d045602894
      

      Browser: Firefox 84.0 / Linux 0.0

         75 |    fs.writeFileSync(refFile, state);
         76 |
         77 |  if(!process.env.REFERENCE && hash != md5 && fs.existsSync(refFile))
         78 |    console.log(diffString(JSON.parse(fs.readFileSync(refFile)), JSON.parse(state)));
         79 |
       > 80 |  await t.expect(hash).eql(md5);
         81 |}
         82 |
         83 |async function getState() {
         84 |  // wait for 500ms
         85 |  await new Promise(resolve => setTimeout(resolve, 500));

         at <anonymous> (virtualtabletop/tests/testcafe/tests.js:80:24)
         at asyncGeneratorStep (virtualtabletop/tests/testcafe/tests.js:6:231)
         at _next (virtualtabletop/tests/testcafe/tests.js:6:569)



 1/1 failed (7s)

That is the check for the resulting room state and it says that it doesn't match. Since the fileupdater actually changed SELECT operations in buttons, this is to be expected. But we have to make sure that the result is correct before we update the MD5 hash in the test.

For that we have to run the test against a reference PR so it will save the actual room state belonging to the MD5 hash into a file. Just look at the list of currently open PRs and get the number of the latest one with a green checkmark (passing tests). At the time of writing that is #387.

Call the test like this and it should pass (note the REFERENCE=387):

user@host:virtualtabletop$ REFERENCE=387 npm run testcafe-firefox-all -- --test-grep '.*Dice.*'

> [email protected] testcafe-firefox
> testcafe firefox tests/testcafe/tests.js "--test-grep" ".*Dice.*"

 Running tests in:
 - Firefox 84.0 / Linux 0.0

 virtualtabletop.io
 ✓ Public library: Dice (variant 0)


 1 passed (7s)

Now run the test normally again and it should display a diff of the room state above a failing test:

user@host:virtualtabletop$ npm run testcafe-firefox-all -- --test-grep '.*Dice.*'

> [email protected] testcafe-firefox
> testcafe firefox tests/testcafe/tests.js "--test-grep" ".*Dice.*"

 Running tests in:
 - Firefox 84.0 / Linux 0.0

 virtualtabletop.io
 {
   dsfa: {
     clickRoutine: [
       {
-        mode: "set"
       }
       ...
       ...
       ...
       ...
       ...
       ...
       ...
       ...
     ]
   }
 }

 ✖ Public library: Dice (variant 0)

   1) AssertionError: expected '3cd34fc98732a94639d37e677c81b7f2' to deeply equal '46a339c348790b6df576c3d045602894'
      
      + expected - actual
      
      -3cd34fc98732a94639d37e677c81b7f2
      +46a339c348790b6df576c3d045602894
      

      Browser: Firefox 84.0 / Linux 0.0

         75 |    fs.writeFileSync(refFile, state);
         76 |
         77 |  if(!process.env.REFERENCE && hash != md5 && fs.existsSync(refFile))
         78 |    console.log(diffString(JSON.parse(fs.readFileSync(refFile)), JSON.parse(state)));
         79 |
       > 80 |  await t.expect(hash).eql(md5);
         81 |}
         82 |
         83 |async function getState() {
         84 |  // wait for 500ms
         85 |  await new Promise(resolve => setTimeout(resolve, 500));

         at <anonymous> (virtualtabletop/tests/testcafe/tests.js:80:24)
         at asyncGeneratorStep (virtualtabletop/tests/testcafe/tests.js:6:231)
         at _next (virtualtabletop/tests/testcafe/tests.js:6:569)



 1/1 failed (7s)

This tells you that the only change in the room state is in the clickRoutine of widget dsfa. One of the operations no longer has a mode: "set".

For this PR this is exactly what we expected because that's what the fileupdater does. Any other changes point to a problem in your PR because the game behavior should never change (unless the game itself gets updated).

So now you can change 46a339c348790b6df576c3d045602894 in tests/testcafe/tests.js to 3cd34fc98732a94639d37e677c81b7f2 and commit that change to your PR. Your PR should now pass this test and whenever your PR gets merged the code and your updated tests will be part of main.