To join the conversation, we're on the OpenActive Slack at #openactive-test-suite.
The general aim of this project is to allow end to end testing of the various flows and failure states of the Open Booking API.
Running npm start in the root will run the OpenActive Test Suite, which is actually comprised of these packages:
- Integration Tests: this performs automated tests against the API.
- Broker Microservice: this sits in between the test suite and the target Open Booking API implementation. This allows the integration tests to watch for changes to the various RPDE feeds.
- OpenID Client, OpenID Browser Automation and OpenID Test CLI: together these connect to the target Open Booking API's OpenID Provider. This allows the Broker and Integration tests to authorize with the implementation
- Test Interface Criteria: this allows test suite to tailor specific opportunities to specific tests by implementing the OpenActive Test Interface Criteria.
Running npm start will orchestrate running the Broker Microservice and the Integration Tests in order to test your Open Booking API implementation.
Note that the implementation under test will need to implement the OpenActive Test Interface to run in controlled mode, and for selected tests.
You can check that the test suite works in your local environment by running it against the hosted OpenActive Reference Implementation, simply by using the default configuration:
git clone [email protected]:openactive/openactive-test-suite.git
cd openactive-test-suite
nvm use
npm install
npm start -- coreNote that the above command only runs the "core" tests within the test suite, which should take around 60 seconds to complete.
The hosted OpenActive Reference Implementation is running on a basic developer tier Azure instance with a burst quota, so it will not handle the load of a test suite run for all tests (hence npm start -- core); if the hosted application shuts down, simply wait 5 minutes and try again.
In order to run the test suite against your own implementation, configure the test suite by creating a copy of config/default.json named config/{NODE_ENV}.json (where {NODE_ENV} is the value of your NODE_ENV environment variable), including the following properties:
The test suite uses the file config/{NODE_ENV}.json to override the settings in default.json. For development and deployment create a new file instead of making changes to the default.json file, so that any new required settings that are added in future versions can be automatically updated in default.json.
For more information about this use of NODE_ENV see this documentation.
By convention, much of the documentation assumes you to have created a config/dev.json file, which Test Suite will use when env var NODE_ENV=dev. But you can use any name you like, and have multiple configuration files for different environments.
The primary Seller is used for all tests, and random opportunities used when "useRandomOpportunities": true are selected from this Seller. The secondary Seller is used only for multiple-sellers tests.
An example, using OpenID Connect Authentication:
"sellers": {
"primary": {
"@type": "Organization",
"@id": "https://reference-implementation.openactive.io/api/identifiers/sellers/0",
"authentication": {
"loginCredentials": {
"username": "test1",
"password": "test1"
}
},
"taxMode": "https://openactive.io/TaxGross",
"paymentReconciliationDetails": {
"name": "AcmeBroker Points",
"accountId": "SN1593",
"paymentProviderId": "STRIPE"
}
},
"secondary": {
"@type": "Person",
"@id": "https://reference-implementation.openactive.io/api/identifiers/sellers/1",
"authentication": {
"loginCredentials": {
"username": "test2",
"password": "test2"
}
},
"taxMode": "https://openactive.io/TaxNet"
}
}Description of each field:
-
authentication: Check out the Configuration for Seller Authentication section. -
taxMode: Which Tax Mode is used for this Seller.Note: If testing both Tax Modes, make sure that there is at least one Seller with each. Alternatively, if not supporting multiple Sellers, you can run the Test Suite once with
"taxMode": "https://openactive.io/TaxNet"and once with"taxMode": "https://openactive.io/TaxGross". However, it is not currently possible to generate a certificate that covers both configurations unless multiple Sellers are supported. -
paymentReconciliationDetails: If testing Payment Reconciliation Detail Validation, include the required payment reconciliation details here.
In order to make bookings for a specific Seller's Opportunity data, some kind of authentication is required to ensure that the caller is authorized to make bookings for that Seller.
Test Suite allows for a few different options for Seller Authentication. This determines the data to put in the authentication field for each Seller:
You'll need the username/password that the Seller can use to log in to your OpenID Connect Provider.
Example:
"sellers": {
"primary": {
// ...
"authentication": {
"loginCredentials": {
"username": "test1",
"password": "test1"
}
}
},In order for Test Suite to be able to log in to your OpenID Connect Provider, you'll need to also configure broker.loginPagesSelectors (see Broker Microservice Configuration).
Just a set of request HTTP headers which will be used to make booking requests. There are no restrictions on the requestHeaders that can be specified.
Example:
"sellers": {
"primary": {
// ...
"authentication": {
"loginCredentials": null,
"requestHeaders": {
"X-OpenActive-Test-Client-Id": "booking-partner-1",
"X-OpenActive-Test-Seller-Id": "https://localhost:5001/api/identifiers/sellers/1"
}
}
},OAuth Client Credentials are used to make booking requests.
Example:
"sellers": {
"primary": {
// ...
"authentication": {
"loginCredentials": null,
"clientCredentials": {
"clientId": "clientid_XXX",
"clientSecret": "example"
}
}
},This is different from the behaviour in the Client Credentials sub-section mentioned within the OpenID Connect Booking Partner Authentication for Multiple Seller Systems section in the spec as, in this case, Client Credentials are used to make booking requests for this Seller, rather than just to view the Booking Partner's Orders Feed.
First, ensure that you have the correct Node.js version installed. We recommend managing this with Node Version Manager (NVM). If you use NVM, switch to the correct version of Node.js with:
nvm useOtherwise, get the correct version of Node.js from the .nvmrc file.
Then, install the dependencies needed for all packages in the test suite:
npm installFor developers that are customising the installation, for use in e.g. Docker, the directories ./packages/openactive-openid-browser-automation, ./packages/openactive-openid-client, ./packages/openactive-openid-test-cli, ./packages/openactive-openid-test-client and ./packages/test-interface-criteria are dependencies, and so must be present during npm install.
Where dev.json is the name of your {NODE_ENV}.json configuration file:
export NODE_ENV=dev
npm startThis will start the broker microservice (openactive-broker-microservice) and run all integration tests (openactive-integration-tests) according to the feature configuration. It will then kill the broker microservice upon test completion. The console output includes both openactive-broker-microservice and openactive-integration-tests. This is perfect for CI, or simple test runs.
Alternatively the Broker microservice and Integration tests may be run separately, for example in two different console windows. This is more useful for debugging.
Any extra command line arguments will be passed to jest in openactive-integration-tests. For example:
export NODE_ENV=dev
npm start -- --runInBand test/features/core/availability-check/It is also possible to use a category identifier or feature identifier as short-hand:
export NODE_ENV=dev
npm start -- coreexport NODE_ENV=dev
npm start -- availability-checkRead about Jest's command line arguments in their CLI docs.
Alternatively, Docker can be used to run Test Suite.
An example command to run Test Suite, with Docker, for the core tests, with the configuration in ./config/dev.json and output in ./output/ is as follows:
docker run -it \
-e INPUT_CONFIG=/config/dev.json \
-e NODE_ENV=dev \
-p 3000:3000 \
-v ${PWD}/config:/config \
-v ${PWD}/packages/openactive-integration-tests/output:/openactive-test-suite/packages/openactive-integration-tests/output \
ghcr.io/openactive/test-suite:latest \
-- coreThis is equivalent to running Test Suite with NODE_ENV=dev npm start -- core.
An explanation of each part of this command:
-
Use
-itto get the full interactive Test Suite experience.docker run -it \
-
Copy the
dev.jsonconfig file into the Docker container and instruct Test Suite to use it. Usually Test Suite is either configured usingconfig/dev.jsonor viaNODE_CONFIG.-e INPUT_CONFIG=/config/dev.json` \ -e NODE_ENV=dev \
-
Expose Broker Microservice's HTTP server to port 3000. This can be useful as the Broker Microservice API can help debug issues.
-p 3000:3000 \
-
Allow the Docker container to read the config file from
./configand to output results to./packages/openactive-integration-tests/output(assuming that you have the default value,./output/configured forintegrationTests.outputPathin yourdev.jsonconfig file)-v ${PWD}/config:/config \ -v ${PWD}/packages/openactive-integration-tests/output:/openactive-test-suite/packages/openactive-integration-tests/output \
-
Use the latest Test Suite docker image
ghcr.io/openactive/test-suite:latest \
-
Run all the tests from the
corecategory (see Running specific tests for more information)-- core
Further reading:
- Running with Docker for Continuous Integration
- See the documentation within
Dockerfileto see how it is built.
-
Build the image
docker build -t local-openactive-test-suite
-
Run the image
docker run -it <OPTIONS> local-openactive-test-suite <...ARGS>
See Running with Docker for more information on the options and arguments.
The configuration of the test suite can be overridden with the environment variable NODE_CONFIG, where any specified configuration will override values in both config/default.json. More detail can be found in the node-config docs. For example:
NODE_CONFIG='{ "waitForHarvestCompletion": true, "datasetSiteUrl": "https://localhost:5001/openactive", "sellers": { "primary": { "@type": "Organization", "@id": "https://localhost:5001/api/identifiers/sellers/0", "requestHeaders": { "X-OpenActive-Test-Client-Id": "test", "X-OpenActive-Test-Seller-Id": "https://localhost:5001/api/identifiers/sellers/0" } }, "secondary": { "@type": "Person", "@id": "https://localhost:5001/api/identifiers/sellers/1" } }, "useRandomOpportunities": true, "generateConformanceCertificate": true, "conformanceCertificateId": "https://openactive.io/openactive-test-suite/example-output/random/certification/" }' npm startDefaults to 3000.
Set PORT to override the default port that the openactive-broker-microservice will expose endpoints on for the openactive-integration-tests. This is useful in the case that you already have a service using port 3000.
E.g. FORCE_COLOR=1
Set this to force the OpenActive Test Suite to output in colour. The OpenActive Test Suite uses chalk, which attempts to auto-detect the color support of the terminal. For CI environments this detection is often inaccurate, and FORCE_COLOR=1 should be set manually.
Assuming configuration is set using the NODE_CONFIG environment variable as described above, the test suite can be run within a continuous integration environment, as shown below:
#!/bin/bash
set -e # exit with nonzero exit code if anything fails
# Get the latest OpenActive Test Suite
git clone [email protected]:openactive/openactive-test-suite.git
cd openactive-test-suite
# Install dependencies
npm install
# Start broker microservice and run tests
npm start"ci": true must be included in the supplied NODE_CONFIG to ensure correct console logging output within a CI environment.
Note that running npm start in the root openactive-test-suite directory will override waitForHarvestCompletion to true in default.json, so that the openactive-integration-tests will wait for the openactive-broker-microservice to be ready before it begins the test run.
You may find it useful to run Test Suite with Docker in your CI environment (general information on running with Docker here).
Similarly to the Continuous Integration section, if we assume that configuration is set using the NODE_CONFIG environment variable, CI can use a command like the following:
docker run -it \
-e NODE_CONFIG=< PUT YOUR NODE CONFIG HERE > \
-p 3000:3000 \
-v ${PWD}/config:/config \
# Change this if using a custom `integrationTests.outputPath`
-v ${PWD}/packages/openactive-integration-tests/output:/openactive-test-suite/packages/openactive-integration-tests/output \
ghcr.io/openactive/test-suite:latestAnd as in the parent section, NODE_CONFIG must include "ci": true.
Some things to note:
- When using a hostname other than
localhostfor your booking system, it must included at least one.to pass the validator. host.docker.internalmust be the host to access your booking system locally if it is not running in another Docker container. This hostname must also be used withinNODE_CONFIG.
This repository can also be referenced as a GitHub action, which conveniently wraps the Docker container.
The following steps may be used within a GitHub Actions script:
steps:
- name: Create output dirs
run: mkdir -p ./test-suite/output/json ./test-suite/conformance/
- name: Run OpenActive Test Suite
uses: openactive/openactive-test-suite@master
with:
config_file: ./test-suite-config.json
NODE_CONFIG: |
{"ci": true, "broker": {"outputPath": "/github/workspace/test-suite/output/", "datasetSiteUrl": "http://host.docker.internal/openactive"}, "integrationTests": { "outputPath": "/github/workspace/test-suite/output/", "conformanceCertificatePath": "/github/workspace/test-suite/conformance/", "conformanceCertificateId": "https://certificates.example.com/openactive/" }, "sellers": {"primary": { "@id": "http://host.docker.internal/api/identifiers/sellers/1","secondary": { "@id": "http://host.docker.internal/api/identifiers/sellers/2"}}}}
- name: Upload test output as artifact
uses: actions/upload-artifact@v4
if: ${{ success() || failure() }}
with:
name: openactive-test-suite
path: ./test-suite/output/
- name: Deploy conformance certificate to Azure Blob Storage (master branch only)
uses: bacongobbler/[email protected]
if: ${{ github.ref == 'refs/heads/master' }}
with:
source_dir: ./test-suite/conformance/
container_name: '$web'
connection_string: ${{ secrets.CONFORMANCE_CERTIFICATE_BLOB_STORAGE_CONNECTION_STRING }}
sync: falseAnd as in the parent section, NODE_CONFIG must include "ci": true.
Note that outputPath and conformanceCertificatePath must start with /github/workspace/ to ensure these outputs are accessible in subsequent steps.
Sub-directories must be created before the test suite runs (including a subdirectory of the output folder named json) e.g. mkdir -p ./test-suite/output/json ./test-suite/conformance/.
As in the previous section, host.docker.internal must be the host to access your booking system locally to the GitHub action if it is not running in another Docker container. This hostname must also be used within NODE_CONFIG for datasetSiteUrl and Seller @ids.
The Test Suite Certificate should be updated upon each successfull CI run.
In order to run the tests in random mode, the target Open Booking API implementation will need to have some Opportunity data pre-loaded. Use Test Data Generator to find out how much data is needed and in what configuration.
An OpenActive Conformance Certificate offers a mechanism by which implementing systems can prove their conformance to the OpenActive specifications. Test Suite can be configured to output a Conformance Certificate upon all tests passing.
An example conformance certificate can be found here: https://certificates.reference-implementation.openactive.io/examples/all-features/random/
For more information about Certification please see here.
The method by which a Booking Partner authenticates with the Open Booking API implementation. There are a number of supported strategies, including OpenID Connect, HTTP Header, etc.
Your impementation will need to support at least one Authentication Strategy for each of Orders Feed Authentication and Booking Authentication.
How a Booking Partner accesses the Orders Feed containing updates to Orders that they have created.
For Test Suite, the selected Orders Feed Authentication Strategy is configured with the broker.bookingPartners configuration property and documentation on the supported strategies can be found there.
How a Booking Partner accesses the booking endpoints (C1, C2, B, etc) for a specific Seller's data. This differs from Orders Feed Authentication as it can be specified at the per-Seller level for Multiple Seller Systems (relevant feature: multiple-sellers).
For Test Suite, the selected Booking Authentication Strategy is configured with the sellers configuration property and documentation on the supported strategies can be found there.