Quantify plugin and theme compatibility with Gutenberg

[danielbachhuber]

As we prepare to enter this brave new Gutenberg world, it would be helpful to know:

Which plugins/themes are compatible with Gutenberg, and which are not? For those that aren't, what sort of incompatibility exists?

It would be amazing if code scanning could determine this for us. Alas, I don't think it can.

Crowdsourcing could help us answer this question.

WordPress admin feedback call to action

The Gutenberg plugin could expose a lightweight mechanism for users to submit feedback as to whether or not a given plugin or theme is compatible with Gutenberg:

The feedback link would take the user to WordPress.org where they'd submit a form pre-filled with the plugin name. They could mark the plugin as "Compatible" or "Incompatible". If the latter, they'd be presented with a bug report form after they submit the initial form.

Cheap Docker containers

The WordPress.org plugin and theme directory could have a mechanism where by a user could easily spin up a Docker container with WordPress, Gutenberg and a specified plugin installed. They'd then complete the aforementioned feedback form to help track whether or not a given plugin is compatible with Gutenberg.

Other ideas on how to generate this data?

Related #3902

[danielbachhuber]

Couple of additional thoughts:

• The Guardian's MP Expenses project likely has some useful lessons for us re: crowdsourcing.
• Just so it's obvious, one key destination for this information is plugin authors. It'd be helpful if they had a dashboard combining these reports + links to relevant documentation on how to resolve.

[khromov]

@danielbachhuber In the old version of the plugin repository there was a way of marking a plugin as compatible with a specific version of WordPress, which would be shown as crowdsourced data. I never thought that feature was particularly useful because the data was poor quality (ie. 3 people would have marked the plugin as not working, but it was working fine), and I think that was probably the reason for that features removal.

If we try to learn a lesson from that, I think we'll see that this will suffer the same fate.

I've suggested a telemetry plugin based on existing open-source infrastructure that would catch javascript errors as they happen and aggregate those in a simple UI. (Which could even be made completely open to everyone). I feel that's vastly superior to crowdsourcing bug reports or spinning up Docker containers. And while resolution can't be automated, finding errors can be. See: #3902 (comment)

[danielbachhuber]

@khromov Great feedback — definitely worth considering!

[khromov]

@danielbachhuber We've recently started using Sentry and I'm really impressed with the ability to find issues using it. It groups errors and you can easily see which browsers are affected, get full stack traces and the output from JS console for that pageload. From my understanding both the frontend (Raven.js) and backend is fully open sourced and possible to set up: https://github.com/getsentry/onpremise

A screenshot to demonstrate:

[danielbachhuber]

I spent time today producing wireframes to better communicate what I think we should set out to build. Note: these are really bad wireframes. They're meant to communicate the general idea; layout, design, language, etc. is all open to improvement. Plus, I have a head cold so I'm not 100% lucid.

At a high-level, the implementation would be:

• plugin_compat custom post type, with one post generated for each plugin in the plugin directory (or limited to plugins with some minimum number of installations).
• Tester feedback is stored as a comment on the plugin compat post. When they respond, their "Yes" vs. "No" response assigns a corresponding custom taxonomy term to the post. The term lets us sort the database based on compatible vs. incompatible.
• Tester feedback is limited to logged-in WordPress.org users, to ensure accountability and possibility of follow-up.

---

Generally, there are two key ideas:

1. Specific URLs to point WordPress users to for one-off feedback or some highly motivated user can come through and test 100 plugins.
2. Quality feedback data over quantity. As long as it's accurate, we really only need one report per plugin.

Feedback I'm looking for:

1. Am I headed generally in the right direction?
2. If yes to 1, are there any specific details we need to be aware of prior to development?

[danielbachhuber]

I also did some data analysis of the WordPress.org plugin directory:

• 49,749 total WordPress.org plugins.
• Only 18,002 plugins have 200 or greater active installs.
• 168,623,000 (92.5%) of total active installs are represented by 3,440 plugins with >=5000 active installs.

[bph]

@danielbachhuber This would be very, very helpful! And I would be happy to help with user testing, I have to do it for many client projects anyway.

If there is a concern that plugins "step on each other's toes" when implementing for Gutenberg, and I don't know enough if that is a valid concern, but if it is there should be a category/flag/tag for 'plays well with others'

The numbers are a quite interesting on so many levels. Thanks for digging them up!

[karmatosed]

@danielbachhuber this is all super interesting and going to put the design of it to one side, focus on the concept. Similarly for copy, this is one area we're going to need to get right and we can.

Panning out I can see a few things this needs to solve right off:

• User question: what plugins that I am using are compatible with Gutenberg.
• Developer question: how can I test if I am compatible.
• Community contributor question: how can I help test plugins.

We also have the fact we can make this a positive achievement. Somehow showing on the profile and/or promoting those that are.

I think most of my above points are met with this, although we maybe need to work on the positive achievement and user question more. I also think we need probably more links to handbook pages on the 'why and what' of compatibility.

What do we define as compatible? Is the lowest point of this no errors? Are there levels?

[danielbachhuber]

Similarly for copy, this is one area we're going to need to get right and we can.

Agree 100%. My copy ain't great 😉

User question: what plugins that I am using are compatible with Gutenberg.

My thinking is that we'd eventually expose this data on the plugins screen in the WordPress admin. This would be Phase 2 or 3 even.

Developer question: how can I test if I am compatible.

Yep, exactly. I'm not sure how in-progress this is with the documentation efforts though.

What do we define as compatible? Is the lowest point of this no errors? Are there levels?

TBD based on what we decide. The two options in the wireframes are:

1. Compatible - Plugin has been updated to work with Gutenberg.
2. Compatible - Plugin doesn't touch the editor and isn't impacted by Gutenberg.

I think most of my above points are met with this, although we maybe need to work on the positive achievement and user question more.

What do you see as immediate next steps to get a v1 out the door? From my perspective, I think I could start on the functional components while we flesh out the non-functional details.

[danielbachhuber]

In a stroke of insight, I realized it would be helpful to do some "paper prototyping" of the database before I set about writing code. I considered using a spreadsheet, but I didn't want the cells to become garbled with long-form text (and screenshots, etc.). Instead, I created a GitHub repo to use as a database: danielbachhuber/gutenberg-plugin-compatibility.

Of the 50 top WordPress.org plugins I created issues for, I managed to test 40 in ~3 hours time. This works out to 4.5 minutes per plugin. With a more conservative estimate of 10 minutes per plugin, we'd need 5,733 person-hours to test the 3,440 plugins with >=5000 active installs.

Or, put another way, we'd need 5.733 hours from 1000 testers.

Of the 40 plugins I tested:

• 29 are compatible with Gutenberg.
  • These are compatible entirely because they don't expose any UI/functionality in the editor.
• 11 are incompatible with Gutenberg.
  • Most have features in the classic editor that don't appear in the Gutenberg editor.
  • Some have their features appear in the Gutenberg editor, but the feature is broken to some degree.

Lastly, a few observations from the experience:

• Most of the incompatibilities so far are caused by lack of media buttons (plugin adds a media button in the classic editor that isn't exposed in Gutenberg).
• Using a GitHub repo as a database isn't totally awful and may be worth continuing.
• We might be able to reduce our manual testing effort by machine-classifying plugins as "Likely to be compatible". For instance, most caching and backup plugins probably don't expose editor UI.

[danielbachhuber]

Another idea: given my testing approach was simply to activate a plugin and compare the classic editor to Gutenberg, we could likely automate this approach to at least weed out the plugins without any editor UI.

[danielbachhuber]

given my testing approach was simply to activate a plugin and compare the classic editor to Gutenberg, we could likely automate this approach to at least weed out the plugins without any editor UI.

I began on this approach today. Using danielbachhuber/puppet-photo-booth locally, I wrote a small script to:

1. Take a before screenshot for both classic editor and Gutenberg.
2. Install and activate the plugin.
3. Take an after screen shot for both editors.
4. Uninstall the plugin.

For the ManageWP Worker plugin, this these screenshots look like this:

Puppet Photo Booth also provides quantification of percent difference.

I think I'll run this against a reasonably wide sample and then see what's feasible to do with the data.

[danielbachhuber]

Yesterday, I ran my gutensnaps script against the top 248 WordPress.org plugins (250 minus a couple plugins that failed for reasons). For each plugin, the script performs these steps:

1. Installs the plugin.
2. Takes 'before' screenshots of the classic editor and Gutenberg editor.
3. Activates the plugin.
4. Takes 'after' screenshots of the classic editor and Gutenberg editor.
5. Deactivates and deletes the plugin.
6. Using ImageMagick, calculates the percentage difference between the two pairs of screenshots.

If you're interested in the raw data, here is:

• The full log output
• A gallery of all screenshots (warning: 1,488 total images and takes some time to load)

The more useful data is the log data parsed into CSV (using this parsing script). Each plugin row indicates percentage change and has a link to a gallery of its screenshots. As an example:

plugin	classic_percent_diff	gutenberg_percent_diff	gallery_link
contact-form-7	0%	0%	http://gutensnaps.handbuilt.co/index.php?plugin_name=contact-form-7
wordpress-seo	2.923%	0.82%	http://gutensnaps.handbuilt.co/index.php?plugin_name=wordpress-seo
akismet	0%	0%	http://gutensnaps.handbuilt.co/index.php?plugin_name=akismet

In total, 155 (62.5%) plugins recorded 0% change in screenshots. This means 93 (37.5%) plugins did have some UI in the classic editor.

This 0% change number could be an indicator of Gutenberg compatibility, but I'm not sure of my degree of confidence in it. Probably higher than 60% confidence but lower than 95% confidence. As examples:

• Advanced Custom Fields interacts with the editor but recorded 0% change.
• Akismet doesn't have any editor UI and correctly recorded 0% change.
• Jetpack recorded 0% change but requires modules to be enabled in order to display its editor UI.

In summary, the screenshot approach isn't a panacea. They're helpful for quickly skimming to see if anything changed. However, in the cases where something did change, further human intervention is needed to assess impact. Plus, some human judgement is needed to review the results of each test.

Other implementation notes:

• I hid the admin bar and admin menu to avoid false positive changes. Plugin activation often added a menu option that would get flagged by ImageMagick. We only care about the area within the editor.
• It took ~20 seconds to generate gutensnaps for each plugin. Running this on my local machine meant stopping all other development to avoid breaking the gutensnaps. Ideally, we'd spin up some isolated environments for each test. We'd be looking at ~19 hours runtime for 3,440 plugins.

I'm not entirely sure of my next steps at this point. Open to thoughts, opinions and feedback.

[omarreiss]

Just a few thoughts with regard to breakage:

Please also note #1316. I've documented all hooks there that are being removed with Gutenberg. Plugins using these hooks are probably your best candidates for finding breakage.

I also started documenting typical usecases:

Metaboxes and Shortcodes are well-known points of integration which have been considered and documented at https://wordpress.org/gutenberg/. They're supposed to mostly still be supported.

So far I could find the following common points of integration which will break:

• Rendering something after the title field ( edit_form_before_permalink and edit_form_after_title actions)
• Media buttons (media_buttons action)
• Editor toolbar buttons (mce_buttons filter)
• Using jQuery to watch or manipulate all the possible fields outputted by core.

[danielbachhuber]

This quantification project is now a work in progress.

[danielbachhuber]

I posted a summary of results to date.

[danielbachhuber]

@pento @getsource @jadonn Do we want to consider building the plugin compatibility database into WordPress.org at this point?

More specifically, make it more of a crowdsourcing project and less of a plugin testing workflow by enabling plugin authors and WordPress users to mark a plugin as compatible / incompatible.

One pro is that the plugin compatibility project would receive more visibility. We could also build in an audit log and other bespoke features we don't get out of Knack.

However, the con I see is that it could potentially become a point of contention and debate.

[danielbachhuber]

Do we want to consider building the plugin compatibility database into WordPress.org at this point?

Not going to pursue this project at this time. Quantification has had mixed results, but we have a good sense of the compatibility issues at this point. We'll see what comes up in a broader rollout.