Skip to content

PennyLaneAI/plugin-test-matrix

Repository files navigation

Plugin Test Matrix

This repository contains regularly scheduled continuous-integration for the official PennyLane plugins.

The continuous integration is performed by GitHub actions. Tests are executed on every push to master, and are also run on a schedule of once per day.

Finally, tests can be triggered manually via the 'Actions' tab.

Testing matrix

All entries in the matrix are tested against PennyLane latest (GitHub master).

PyPI version Stable plugin/stable PennyLane Stable plugin/latest PennyLane Latest plugin/ stable PennyLane Latest plugin/latest PennyLane
Qiskit GitHub Workflow Status GitHub Workflow Status GitHub Workflow Status GitHub Workflow Status
Cirq GitHub Workflow Status GitHub Workflow Status GitHub Workflow Status GitHub Workflow Status
Qulacs GitHub Workflow Status GitHub Workflow Status GitHub Workflow Status GitHub Workflow Status
AQT GitHub Workflow Status GitHub Workflow Status GitHub Workflow Status GitHub Workflow Status
IonQ GitHub Workflow Status GitHub Workflow Status GitHub Workflow Status GitHub Workflow Status
Rigetti GitHub Workflow Status GitHub Workflow Status GitHub Workflow Status GitHub Workflow Status
Lightning GitHub Workflow Status GitHub Workflow Status GitHub Workflow Status GitHub Workflow Status
Braket GitHub Workflow Status GitHub Workflow Status GitHub Workflow Status GitHub Workflow Status

Notes:

  • The device integration tests currently do not support devices with a fixed number of wires; as a result, rigetti.qvm is not tested.

  • The Braket plugin device integration tests are run with -k “not Sample and not no_0_shots”, see #6

  • The Qiskit tests are run using local simulators. There are no tests that access the IBM Quantum backends.

QML repo

Status
build-branch-dev GitHub Workflow Status
build-branch-master GitHub Workflow Status
update-dev GitHub Workflow Status

Lightning Docker builds

Status
Docker build stable GitHub Workflow Status
Docker build latest GitHub Workflow Status

Catalyst compiler

The following table shows the current Catalyst compatibility with the Lightning and PennyLane packages, where each of the 3 packages is tested as "latest" and "stable". Latest versions are built directly from the repository's main or master branch, while stable is built from the repository at the most recent release tag. The reason this is needed rather than installing PyPI packages is that the Lightning dependency in Catalyst needs to be built from source and cannot be obtained from the PyPI package. PennyLane stable is installed from PyPI.

PennyLane stable PennyLane latest
Lightning stable Lightning latest Lightning stable Lightning latest
Catalyst stable Check CPL stable/stable/stable Check CPL stable/stable/latest Check CPL stable/latest/stable Check CPL stable/latest/latest
Catalyst latest Check CPL latest/stable/stable Check CPL latest/stable/latest Check CPL latest/latest/stable Check CPL latest/latest/latest

Latest reported warnings

PennyLane

UserWarning
DeprecationWarning
pytest.PytestRemovedIn9Warning
PendingDeprecationWarning
pytest.PytestUnraisableExceptionWarning
numpy.exceptions.ComplexWarning
RuntimeWarning
pytest.PytestCollectionWarning
FutureWarning
pennylane.PennyLaneDeprecationWarning

Interpreting the test matrix

  • Stable/stable: This reflects a scenario where PennyLane and the plugin are both installed from PyPI. This column should always pass. If this test fails, it indicates that either (a) a bug exists, or (b) an upstream dependency (such as Qiskit or Qulacs) were released with breaking changes. In both cases, a bugfix release of the plugin should be made.

  • Stable/latest passing: A new plugin release is not essential; the current PyPI release will continue to work once the new PennyLane version is released.

    • Latest/latest passing: Nothing to be done! However, it is still worth checking the plugin repository to see if new unreleased changes have been added that would be worth releasing.

    • Latest/latest failing: Indicates that recent changes have been made to the plugin repository post-release; further updates must be made to the plugin to ensure GitHub master passes.

  • Stable/latest failing: A new plugin release is required for compatibility with the upcoming PennyLane release.

    • Latest/latest failing: Updates must be made to the plugin to ensure GitHub master passes.

    • Latest/stable failing: The requirement of PennyLane in the plugin will have to be increased.

    • Latest/latest passing: Nothing to be done! The plugin is ready to be released with the upcoming PennyLane release.

Feature freeze testing matrix

The latest of all plugins are tested against the PennyLane release candidate branch (GitHub vX.XX.X-rc0 branch).

Status
Qiskit GitHub Workflow Status
Cirq GitHub Workflow Status
Qulacs GitHub Workflow Status
AQT GitHub Workflow Status
IonQ GitHub Workflow Status
Rigetti GitHub Workflow Status
Braket GitHub Workflow Status

The latest and release candidate of Lightning and Catalyst are tested with the release candidate of PennyLane:

Latest Release candidate
Lightning GitHub Workflow Status GitHub Workflow Status
Catalyst GitHub Workflow Status
Docker build GitHub Workflow Status

Adding a plugin

Two Jinja2 workflow templates are provided, that makes it easier to add a new plugin to the test matrix:

Simply add a new plugin to the workflows list in compile.py, with the following dictionary keys:

  • plugin (required): the name of the plugin not including the PennyLane prefix. The plugin repository and PyPI project are inferred from the plugin name. E.g., 'plugin': 'qiskit' will correspond to the GitHub repo PennyLaneAI/pennylane-qiskit and the PyPI project pennylane-qiskit.

  • gh_user (required): the GitHub user or organization which hosts the plugin repository.

  • which (required): a set specifying whether to generate a workflow to test the {"latest"} (GitHub master) version of the plugin, {"stable"} (PyPI) version of the plugin, or both {"latest", "stable"}.

  • requirements (optional): a list of Python packages that should be installed prior to plugin installation. You may use pip syntax, e.g., pyscf==1.7.2.

  • requirements_latest (optional): additional list of Python packages that should be installed prior to latest/GitHub master plugin installation. For example, if the latest version of the plugin depends on a development version of a particular framework.

  • device_tests (optional): a list of command line arguments to pass to the PennyLane device integration tests. Each list element corresponds to a single test run, e.g.,

    "device_tests":
        [
            "--device=cirq.simulator --tb=short --skip-ops --analytic=True",
            "--device=cirq.simulator --tb=short --skip-ops --analytic=False --shots=8000"
        ]
  • additional_setup (optional): multiline string containing additional GitHub actions for execution before the installment of plugins.

  • token (optional): sets an environment variable with the same name as the token. The value is loaded from the repository secrets. The token is required by some plugins for backend authentication.

  • additional_env_vars (optional): string containing additional environment variables you'd like to be set for the CI action. Double-check the whitespace when using this - variables should be separated with "\n "

  • runs_on (optional): string to override the workflow's default runs-on attribute of ubuntu-latest.

  • test_kwargs (optional): additional arguments to pass to pytest for the given plugin.

  • no_deprecation_error (optional): set to True to not raise PL deprecation warnings as errors when testing the latest version of the plugin. By default, PL deprecation warnings are raised as errors.

Once you have added your plugin, run

$ python compile.py

This will autogenerate up to five workflow files, depending on the variable which:

  • .github/workflows/plugin-stable-stable.yml
  • .github/workflows/plugin-stable-latest.yml
  • .github/workflows/plugin-latest-stable.yml
  • .github/workflows/plugin-latest-latest.yml
  • .github/workflows/plugin-latest-rc.yml

Finally, make sure to add a row to the testing matrix above!

About

Scheduled integration tests for the official plugins

Resources

Stars

Watchers

Forks

Packages

No packages published

Languages