Add visual regression tests with Playwright (#345)

Closes #208.

## How snapshots work

Playwright snapshot tests take screenshots of certain portions of our
site, such as the footer. They save that screenshot to Git, so that we
all have the exact copy. Then, for every PR, our tests will retake a new
screenshot and make sure it's the exact same as before. That way, we
always know when we make a change.

When there was a change, Playwright will output this:

![Screenshot 2023-05-26 at 12 09 19
PM](https://github.com/Qiskit/qiskit_sphinx_theme/assets/14852634/a5157d1d-f9de-4e0a-a93b-93c413dba848)

As stated there, Playwright will generate a before, after, and diff
photo. In CI, we upload those as GitHub artifacts.

If the change was intentional, then the author copies the updated
screenshot to overwrite the old file.

## Uses Docker for consistency

Websites render slightly differently between operating systems, e.g.
Linux and macOS. Naively, this would cause visual regression tests to
fail when running the tests locally on a mac.

So, we use Docker to make sure we always use the same environment. 

If users don't want to run Docker locally, they can rely on CI to get
the updated snapshots via GitHub Actions Artifacts.

More advanced users can also run the tests locally. They need to install
Docker and have the Docker daemon running. But otherwise, our scripts
automate everything. They don't need to know how to use Docker. They
only need to run `npm test`.

## How to run locally

(Reminder that contributors can rely on CI for visual regression
testing.)

The user has to first build the docs with `THEME=_qiskit_furo tox -e
docs`. Then, they only run `npm test` (after originally running `npm
install` & starting Docker).

Playwright will start up a server for us.

If the user changed the theme, they must rebuild the docs with Tox.

## Switches from Jest to Playwright

I originally tried doing this change via Jest, a common JavaScript test
runner. But I had major issues getting the browser automation library
Puppeteer #341 to work
properly on my M1.

So, this changes our test runner to
[Playwright](https://playwright.dev), a test runner from Microsoft
optimized for visual regression testing like we're doing. It can do
things like click buttons, e.g. our Translations menu.

## Only Furo

To keep things simple, this only adds infrastructure to snapshot test
Furo. Given that our goal is to migrate ASAP, this seemed okay to me.

.github/workflows/main.yml

@@ -10,6 +10,9 @@ jobs:
     name: Sample Docs Build
     runs-on: ubuntu-latest
     if: github.repository_owner == 'Qiskit'
+    container:
+      # Keep in sync with tests/js/Dockerfile's base image.
+      image: mcr.microsoft.com/playwright:v1.34.0-jammy
     steps:
       - uses: actions/checkout@v3
         with:
@@ -20,23 +23,25 @@ jobs:
           node-version: 18
       - name: Install Node.js dependencies
         run: npm ci
-      - name: Run Node.js tests
-        run: npm test
       - name: Set up Python
         uses: actions/setup-python@v4
         with:
           python-version: 3.9
       - name: Install Ubuntu deps
-        run: sudo apt-get install -y pandoc graphviz
+        run: apt-get update && apt-get install -y pandoc graphviz
       - name: Install tox
         run: python -m pip install -U tox
+
       - name: Run lint
         run: tox run -e lint
       - name: Run Pytest
         run: tox run -e py
       # For some reason, the Jupyter Extension fails to build properly if one theme has already
       # been built with a custom BUILD_DIR. So, we eagerly upload the artifacts for each theme
       # build, and then delete the BUILD_DIR before proceeding to the next theme.
+      #
+      # But, we build Furo with the default BUILD_DIR because our JavaScript Snapshot tests start
+      # the server at that standard location.
       - name: Create artifacts/ folder
         run: mkdir artifacts
       - name: Build Legacy theme
@@ -47,13 +52,21 @@ jobs:
           rm -rf example_docs/_build_legacy
       - name: Build Furo theme
         run: |
-          THEME=_qiskit_furo BUILD_DIR=example_docs/_build_furo/html tox run -e docs
-          tar -zcvf furo_html_docs.tar.gz example_docs/_build_furo/html
+          THEME=_qiskit_furo tox run -e docs
+          tar -zcvf furo_html_docs.tar.gz example_docs/docs/_build/html
           mv furo_html_docs.tar.gz artifacts/.
-          rm -rf example_docs/_build_furo
       - name: Upload Sphinx builds
         uses: actions/upload-artifact@v3
         if: always()
         with:
           name: html_docs
-          path: artifacts
+          path: artifacts
+
+      - name: Run JavaScript and snapshot tests
+        run: npm run _run-snapshot-tests
+      - name: Upload snapshot results
+        if: failure()
+        uses: actions/upload-artifact@v3
+        with:
+          name: snapshot_results
+          path: snapshot_results/

.gitignore

@@ -133,3 +133,4 @@ dmypy.json
 
 # JavaScript
 /node_modules
+/snapshot_results

CONTRIBUTING.md

@@ -47,16 +47,50 @@ Sometimes Sphinx's caching can get in a bad state. First, try running `tox -e do
 We are in the process of migrating our theme from Pytorch to Furo (see https://github.com/Qiskit/qiskit_sphinx_theme/issues/232). To build the local docs using the Furo theme, use `THEME=_qiskit_furo` in front of the command, e.g. `THEME=_qiskit_furo tox -e docs`.
 
 ------
-## Run JavaScript tests
+## Visual regression testing
 
-We write some tests in JavaScript (Node.js) to have automated checks of the theme, e.g. that certain components render properly.
+We use visual regression testing via [Playwright](https://playwright.dev/docs/test-snapshots) to take screenshots of the site and check that every change we make is intentional. If a screenshot has changed, the test will fail. 
 
-To run these tests, you first need to install Node.js on your machine. If you expect to use JavaScript in other projects, we recommend using [NVM](https://github.com/nvm-sh/nvm). Otherwise, consider using [Homebrew](https://formulae.brew.sh/formula/node) or installing [Node.js directly](https://nodejs.org/en).
+If the change was intentional, we need to update the screenshot. Otherwise, it means your change unintentionally impacted something, so you need to tweak your change.
 
-Then:
+The test runner creates a folder called `snapshot_results`, which is useful to determine what the difference is. For each failed test, there will be three files:
+
+* `<my-test-name>-actual.png`, what your change resulted in.
+* `<my-test-name>-expected.png`, what we expected.
+* `<my-test-name>-diff.png`, a heat map showing where the differences are.
+
+### How to check snapshot results in CI
+
+We upload `snapshot_results` in CI. So, you can get the changed snapshot from GitHub Actions:
+
+1. Navigate to the GitHub Actions page for the "Tests" action.
+2. Open the "Summary" page with the house icon.
+3. Under the "Artifacts" section, there should be a "snapshot_results" entry. Download it.
+
+### How to check snapshot results locally
+
+You can also run the tests locally for faster iteration, although it requires a little setup. If you don't want to install the below tools, it is okay to use CI for snapshot testing.
+
+First, you need to install:
+
+1. [Node.js](https://nodejs.org/en). If you expect to use JavaScript in other projects, consider using [NVM](https://github.com/nvm-sh/nvm). Otherwise, consider using [Homebrew](https://formulae.brew.sh/formula/node) or installing [Node.js directly](https://nodejs.org/en).
+2. [Docker](https://www.docker.com). You must also ensure that it is running.
+   * If you cannot install Docker Desktop (such as IBM contributors), you can use [Rancher Desktop](https://rancherdesktop.io). When installing, choose Moby/Dockerd as the engine, rather than nerdctl. To ensure it's running, open up the app "Rancher Desktop". 
+
+Then, to run the tests locally:
 
 1. `npm install`
-2. `npm test`
+2. Build the docs with Furo, `THEME=_qiskit_furo tox -e docs`
+3. `npm run test-snapshots`
+
+You must rebuild the docs with `THEME=_qiskit_furo tox -e docs` whenever you make changes to the theme or docs folder. The docs will not automatically rebuild.
+
+### How to update the expected snapshot for intentional changes
+
+First, get the `snapshot_results` folder, either by downloading it from CI or by running the tests locally. Then:
+
+1. Find the "actual" snapshot for the failing test, such as `footer-includes-page-analytics-1-actual.png`.
+2. Copy that snapshot into the folder `tests/js/snapshots.test.js-snapshots`. Rename the `-actual.png` file ending to be `-linux.png` and overwrite the prior file.
 
 ------
 ## Updating bundled web components

example_docs/docs/conf.py

@@ -42,7 +42,9 @@
     "qiskit_sphinx_theme",
 ]
 
-html_last_updated_fmt = "%Y/%m/%d"
+# Usually this would be something like "%Y/%m/%d", but we need a deterministic value for our
+# Playwright visual regression tests.
+html_last_updated_fmt = "2020/01/01"
 
 # This allows us to test both the Furo and Pytorch themes. In normal repositories, `html_theme`
 # would be set to one specific theme.