Is there any interest in creating extension-specific test harness like the WPT one?

[bershanskiy]

WPT has a standard test harness capable of opening pages and running unit tests. Is there any interest in creating a similar tool which would generate test extensions, launch browsers, install extensions, and run the tests in the context of these extensions?

[yankovichv]

@bershanskiy , Anton, hello 👋.

I'm not sure I understood you correctly. But if you're asking if there are developers here interested in getting a test bench that will measure the impact of an extension on browser performance, I'm first in line.

I've been working with extensions for a long time, and almost always, "browser degradation" is one of the main reasons for deletions. Of course, this is often a subjective feeling of the user, which may have nothing to do with reality and your product. But as a rule, developers never have a tool to measure this objectively.

It seems to me that such a tool would be extremely useful for developers and store moderation teams.

I imagine it as a set of simple tests in two configurations (clean browser and browser with extension installed): browser start, new tab open, 10-30 tabs open with different hosts, new tab reopen.

@oliverdunk, have you ever thought about this? It seems to me that this could be as valuable as the server for testing the update process that you recently provided to the community.

@bershanskiy, did I understand you correctly?

If anyone has expertise in solving my problem, I will be happy to discuss cooperation.

[oliverdunk]

If I'm understanding correctly, there are two separate requests here - @bershanskiy was asking about testing for functionality (e.g do two browsers do the same thing when a certain API runs) whereas your request @yankovichv is more performance driven.

I think both are interesting, although up to this point I've thought mostly about the former (testing for functionality). That seems like it would align very closely with this group's goals of increasing interoperability between browsers. Both could likely benefit from similar groundwork.

I know @zombie has some thoughts here, and we've previously discussed in issues like #49 and #19.

[bershanskiy]

No, I meant something completely different, I'm sorry that I failed to get my point across. It is not about the performance, but rather just a way to express unit tests so that all browser vendors can add specific unit tests calling a single API. Like an assert library.

In the past, whenever I needed to test a specific aspect of API, I always just performed the action manually and observed result manually. This is inefficient (multiple browser and OS combos, etc.). Also, since API is not documented, the behavior can change from release to release.

For example, Duck Duck Go solved this by enumerating behaviors they rely on writing a bunch of "platform expectation" tests. These tests apparently run periodically (may be, manually, may be not) and in theory could warn DDG staff about API instability.

[xeenon]

Similar to issue #19.

[sammacbeth]

Related to this. In the past I wrote an extension that tests support for some MV3 APIs across Chrome, Safari and Firefox in order to verify some implementation inconsistencies I found: https://github.com/duckduckgo/mv3-compat-tests. This approach could potentially be extended to cover more of the WebExtension API surface.

[abitrolly]

WPT stands for Web Platform Tests - https://web-platform-tests.org/ (I was not aware of it)

[oliverdunk]

We discussed this briefly at our in-person meeting in San Diego. We're supportive of WPT like tests, and actively looking into it. Closing this issue though since we're trying to only keep things open with specific action items.