Webhook test coverage #15537

naz · 2022-10-06T04:30:32Z

Webhooks have been a part of Ghost since very first versions. This area has been constantly lagging behind with solid test coverage. Having tests would add confidence to webhooks stability. Snapshot tests would be a great source of example data webhooks deliver to external integrations.

To improve this situation, Ghost has introduced a special purpose Webhooks snapshot tests. These are snapshot-based tests (similar to the end-to-end API tests) with one big difference - they test the outgoing request payload instead of API responses.

To understand how end-to-end testing works at Ghost have a read through this guide. The concepts applied in end-to-end API tests are very similar to ones used in webhook tests.

Here is an example of a webhook test suite for for post.published and post.added events.

The above suite covers two event's which Webhook events listen to. There are 29 total events Webhooks can respond to, 27 left to cover. We need help writing tests for them!

Below are some pointers around what you could help us with. For the most part it's covering all of the webhook events with at least one "happy path" test.

Steps to cover a webhook event listener with a test

1. Write a test

First, find one webhook event listener to cover - search for an event that has no test coverage in the list of all webhook event listeners.

Copy of all events below for quick reference and ✅ for the ones with at least one test:

Before digging deep into work please double check if the list is up to date - check if there are any open or closed Pull Requests open with the name of the event you are about to pick up.

Let's pick site.changed event to cover with example a test.

Webhook tests are located in ghost/core/test/e2e-webhooks/{model_name}.test.js (e.g.: test/e2e-webhooks/posts.test.js). As there is no site.test.js suite in the e2e-webhooks folder - create a site.test.js file. Note, for post.rescheduled event, there is a posts.test.js test suite already. In this case, we would add test cases to existing posts.test.js test suite. The convention is to group test suites by common model name - {model_name}.test.js.

Write the test in the newly created file following concepts in end-to-end testing guide. For an easy start it's usually a good idea to copy a similar working test suite into your new file and tweak it.

Notes on Webhook test case structure

Webhook tests follow a setup->action->assert structure used commonly when writing other types of tests, like unit tests:

Setup

Create a webhook database entry

Action

Find under which conditions the event under test is triggered (usually a result of API call(s)). This might take a little bit of research and experimentation. If you are completely lost and did not find the pathway after searching feel free to leave a comment here for more guidance.

Assertion

Wait for webhook events propagation and write an assertion about the expected format of the snapshot.

2. Run the test and record a snapshot

Use the yarn test:single command inside ghost/core to run the test suite you've been working on:

yarn test:single test/e2e-webhooks/site.test.js

After the test run, there will be a snapshot file created with recorder webhook payload (example).

Check the Snapshot Testing section of the end-to-end testing guide for more information about the inner workings of snapshot-based tests.

3. Open a PR

To make sure you've not broken anything & followed our code standard, run yarn test:all. You can also use yarn lint to check formatting issues.

Commit and push the test suite code and the snapshot to your working branch. Open up a separate PR for each webhook even you cover.

🎉 All done! we will try to review your contribution ASAP.

Please follow our contribution guidelines as closely as you can, particularly being sure to reference this issue in your commit. Test coverage commits should not use emojis as they are not user-facing changes.

This issue is going to improve the ease of bug reproductions and will let all Ghost developers sleep better after modifying webhook-related code. This means a lot for the team and contributors!

Thank you 🙏 💚

The text was updated successfully, but these errors were encountered:

dshubhadeep · 2022-10-06T05:24:54Z

hi @naz
I would like to work on tag.* events

naz · 2022-10-06T06:06:32Z

Hey @dshubhadeep 👋 Feel free to pick one event at a time, will try my best to review ASAP. And feel free to ask questions if anything is unclear!

dshubhadeep · 2022-10-06T06:37:16Z

Thanks, I'll start with tag.added event.

illiteratewriter · 2022-10-06T15:26:01Z

I'd like to work on member.added event

JohnGrisham · 2022-10-06T18:53:42Z

I'll try my hand at page.added

refs #15537 refs 4110ffa - The test had minor formatting issues not worth an extra back-forth during the PR review

refs TryGhost#15537

jennyxchang · 2022-10-07T08:22:39Z

I can work on post.scheduled event.

refs #15537 - this adds an e2e test and test snapshot for the `tag.deleted` webhook so we can prevent regressions and bugs in the future

refs #15537 - this adds an e2e test and test snapshot for the `member.added` webhook so we can prevent regressions and bugs in the future

refs TryGhost#15537 - this adds an e2e test and test snapshot for the `tag.edited` webhook so we can prevent regressions and bugs in the future

TryGhost#15537

kritikash18 · 2022-10-09T05:21:41Z

working on post.deleted event

ref TryGhost#15537 - this adds an e2e test and test snapshot for the page.published webhook so we can prevent regressions and bugs in the future

refs: TryGhost#15537 This adds an e2e test and test snapshot for the page.tag.detached webhook so we can prevent regressions and bugs in the future