Adding plugins to QualWeb

[Tellus]

Hey guys!

We've talked about the possibility of extending QualWeb's interface in a way that would allow plugins (like CMP "cookie banner" handling, or login automation) to augment QualWeb in useful ways. The post is a bit long, so I hope you'll bear with me - I wanted to make sure I got the important details written down :)

Use cases

Two notable use cases were mentioned initially:

• CMP handling, being able to recognize and dismiss "cookie banners" on pages. In our work (Inqlude IT), we've seen several cases of these banners severely affecting the results of pages that we evaluate. In at least two cases, the only accessibility issues we found on the pages were the banners themselves, but since our automated tests don't dismiss/ignore them, the final scores for those sites are much lower than they should be. It would be a great benefit if we had a way of suppressing the banners on demand.
• Log in handling/automation. In cases where pages required authentication, there's currently not a good way to access that content without constant manual intervention. Being able to control puppeteer prior to QualWeb's evaluation would allow a developer to automate the login process. It might also be possible to bypass the authentication in other useful ways - like setting session cookies before the page loads, or setting authentication request headers.

Requirements

From those use cases, we've identified these initial requirements for QualWeb to support them:

• A plugin needs access to the puppeteer Page object in order to run its code.
• A plugin needs to be able to specify when to run in the QualWeb life cycle. Currently, at least:
  • After the page object is created, but before navigation to a target URL. ("beforePageLoad")
  • After the page has navigated to the target URL ("afterPageLoad"?). It's possible there may be a consideration as to whether this should further be divided into before/after the javascript context has been injected with QualWeb's javascript bundles. It has been noted, for example, that navigating to a different page after the bundle has been injected will cause the evaluation to fail, since the javascript context gets destroyed.
• Almost all useful methods on Page are asynchronous, which means that QualWeb must be able to delay its evaluation until a plugin has finished processing.

Possible interfaces

During our online discussion, we noted event emitters, callback parameters, and express-style middleware as inspirations for a design.

Event emitter and Express

Event emitters allow listeners to hook into specific events, typically with a generic 'on' method. You'd use it like qualweb.on('beforePageLoad', myCallback) or qualweb.on('afterPageLoad', anotherCallback). The benefit of event emitters is that the interface is quite simple, and the heavy lifting can be delegated to Node's EventEmitter class. When a specific event needs to be raised, you simply call EventEmitter#emit() with the event name and payload. However, I'm not 100% sure that EventEmitter waits for asynchronous functions by default, which is a hard requirement for most use cases that work with the puppeteer API.

Express-stye middleware looks a lot like event emitters, but without being able to specify the specific events you'd want to listen for, express#use(middlewarePlugin). This is fine for express, since the only "event" that passes through express is HTTP(S) requests, and middleware functions like transformations in a chain before a response is (possibly) sent back to the request source.

Passing a callback

Passing callback parameters to QualWeb#evaluate() would mean expanding QualwebOptions with new fields for specifying lifecycle callbacks:

const qw = new QualWeb();
qw.start();

qw.evaluate({
  beforePageLoad: async (page: Page): Promise<unknown> {
    // Do something after page is instantiate, but before URL navigation.
  }
});

Plugins for QualWeb

Initially, I had been thinking of suggesting an interface in the style of EventEmitter (QualWeb#on('beforePageLoad', ...)) because it would allow multiple different listeners and directly specify the event you wanted to hook onto.

After our talk (and the express-style "use" was mentioned), it struck me that a single plugin might want/need to hook into multiple events at different times. As an example, a CMP "banner" plugin might want to listen on "afterPageLoad" to detect and handle banners, but then store the resulting cookies and re-use them in subsequent URLs in a "beforePageLoad" call. While you could do that in a plugin by calling a "on" method twice (with different parameters), it might be more elegant if you pass a full plugin object/constructor instead. Example:

/// Interfaces
type PageCallback = (page: puppeteer.Page): Promise<void> | void;

interface QualwebPlugin {
  beforePageLoad?: PageCallback;
  afterPageLoad?: PageCallback;
}

/// Usage

// Injects known cookies into the page before navigating to a URL. This may
// suppress cookie banners or authorize puppeteer to a secured site.
class MyPlugin implements QualwebPlugin {
  cookieCache: Cookie[] = [];

  async beforePageLoad(page: puppeteer.Page): Promise<void> {
    await page.setCookies(... this.cookieCache);
  }
}

qw.start();

qw.use(new MyPlugin()); // Or possibly a constructor function?

qw.evaluate();

---

I think that's enough of a wall of text, to being with. I'll leave it at this as the first post :)

[Tellus]

I've forked the repo and written a basic approach. It should suffice for our next batch of evaluations starting in March.

Any thoughts?

[AntonioEstriga]

Hey!
I´m sorry for the delay. Your approach looks good. Can you please make a pull request with the new changes in their respective repositories?

[Tellus]

Awesome. I'm wrapping up a few high-priority issues this week and the next, I expect get to these changes after that. I'm hoping I'll get the updates to @qualweb/core out as well as our CMP (cookie banner) handling as a first plugin.

[Tellus]

I've created #22 and qualweb/types#6 to work on the changes.