Start using JavaScript Modules and Import Maps

[gziolo]

What problem does this address?

Related WordPress Trac ticket: https://core.trac.wordpress.org/ticket/48654.

JavaScript Modules

Resource: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules

JavaScript has had modules for a long time. However, they were implemented via libraries, not built into the language. ES2015 is the first time that JavaScript has built-in modules. Older browsers like Internet Explorer 11 didn't support ES Modules. Earlier this year, WordPress dropped support for IE11, which opened the door to using ES Modules in the Gutenberg plugin and later in WordPress core. The most significant benefit of moving the codebase to modules is the possibility of using native browser techniques for async loading scripts

Import Maps

Resource: https://wicg.github.io/import-maps/

Import maps allow web pages to control the behavior of JavaScript imports.

There's native support for import maps in the latest versions of Chrome and Edge browsers, and a shim is available for any other browser that WordPress supports. In practical terms, import maps let use shared libraries by importing dependencies through a predefined list of aliases.

What is your proposed solution?

A long-term goal would be to use JavaScript modules and import maps for all Gutenberg plugin scripts. @youknowriad did some very promising initial explorations:

• Explore ESM modules instead of regular WP scripts #34140
• Add an example esm-example package and use it in edit-post #34172
• Use ESM JSON imports instead of adhoc json imports #34176
• Tracking Issue: Script Modules API #55942

[gziolo]

We discussed the same topic during the bi-weekly WordPress Core JavaScript office hours (link requires registration at https://make.wordpress.org/chat/):

https://wordpress.slack.com/archives/C5UNMSU4R/p1637075934082700

A short summary:

1. We discussed if there is an overlap with a long-standing track ticket https://core.trac.wordpress.org/ticket/12009 about adding support for async and defer to the script tag. We came to the conclusion that it would be better to start exploration from a clean state (the reason of why this issue exists) and build an independent API for import maps first. We might want to exercise a future where you register only JavaScript entry points and the rest happens in the browser.
2. @sgomes raised a CORS concern for 3rd party scripts. There are other differences between modules and standard scripts to take into account. He will explore the topic further and report back his findings.
3. @luisherranz suggested to explore if there are any performance benefits by adopting ES modules. He was skeptical about the performance when the browser has to download 100s of small dependencies instead of just a few bigger ones.

[tomalec]

@luisherranz suggested to explore if there are any performance benefits by adopting ES modules. He was skeptical about the performance when the browser has to download 100s of small dependencies instead of just a few bigger ones.

Maybe we don't need to drop bundling, and just import the bundle as a module, and let it export individual packages which could be then consumed as modules. I believe import maps could make it a bit easier.

{
  "imports": {
    "bundle": "/bundle.js",
    "@wordpress/components": "/* inlined module that does something like `import { package } from "bundle"; export { ...package };*/"
  }
}

See https://stackblitz.com/edit/import-map-to-bundled-esm-package?file=index.html for a POC / more code-like explanation of the idea. (Chromium / import-maps env only)

[youknowriad]

For me the biggest performance improvement here is not something we can measure 1-1 by switching to ESM packages but it's more related to us being smarter when loading things, some examples:

• Only load the "syntax highlighter" module when needed (when switching to code editor or showing a code block)
• Lazy load "edit" implementation of heavy blocks
• Lazy load tinymce only if the classic editor is needed...

On the opposite side of the spectrum, making the "components" package for instance ESM won't have any impact on performance as it's probably a requirement on most JS UI pages so it's always loaded synchronously.

[sgomes]

2. @sgomes raised a CORS concern for 3rd party scripts. There are other differences between modules and standard scripts to take into account. He will explore the topic further and report back his findings.

I've looked into how modules are implemented in various browsers, and they follow the spec in this, with the use of the same origin policy.

Loading third party scripts will not be a problem, as long as they're served from the origin, but any module scripts served from a CDN or third-party provider would need the appropriate CORS headers in place, which is a new restriction when compared to classic scripts. This means that things like self-hosted module-based plugins and themes would not be an issue most of the time, unless they load scripts from external sources for some reason.

There is no workaround for this new restriction, as the behaviour of classic scripts is considered a flaw from the security point of view, and while the behaviour of classic scripts can't be changed because of legacy reasons, the flaw has been corrected in new features like modules.

There is an interesting discussion on this topic in the whatwg html repo, for anyone who'd like more details.

[sgomes]

3. @luisherranz suggested to explore if there are any performance benefits by adopting ES modules. He was skeptical about the performance when the browser has to download 100s of small dependencies instead of just a few bigger ones.

I don't think we should try to unbundle, or attempt extreme levels of modularisation. At this point, I believe the right tradeoff is to provide and encourage the same level of bundling as we currently do, but expose these bundles as ESM for native browser loading. If anyone's interested in some outdated explorations into unbundling JS through ESM, I ran benchmarks and wrote an article on the topic a few years ago. TL;DR: even with HTTP/2, there are persistent bottlenecks to this approach.

The main benefit to going all-in on ESM isn't increased modularity, as I see it, but rather better performance defaults.

Classic scripts are blocking by default, and that's how wp_enqueue_script makes use of them to guarantee script execution ordering. Blocking scripts halt execution when the parser reaches them, which can lead to a longer wait until the page reaches a useful state. And while blocking rendering (if they're in the head, for instance) may be desirable for some scripts, the vast majority of them only deal with user interaction, which can only happen when the page is done loading, and thus don't need to be set up that early in advance.

By contrast, module scripts use the defer behaviour, which means they only execute on DOMContentLoaded, when the document is done parsing. This means that they give us better performance characteristics by default, in that they don't have the potential to delay first paint (again, by default). They are also much easier to lazily load, simply by making use of dynamic import(), without having to deal with any of the tricky issues that attempting to do that in the status quo would entail.

Furthermore, a well-designed ESM API would still be able to maintain all the performance benefits of the current wp_enqueue_script approach:

• The browser handles deduplication, which means that each dependency will only load and execute once, regardless of how many dependants need it. This would match the deduplication that wp_enqueue_script's dependency handling currently does.
• There would still be a way of sharing dependencies between simple plugins and more complex pieces of code like the Editor, which use a build process.

With all of this in mind, a well-designed ESM API would get us much closer to good performance defaults for scripts, as well as an easy way to do the right thing where it comes to lazy loading. And anyone that's worked in designing or supporting APIs surely knows the importance of good defaults, since that's what most developers end up going for 🙂

[gziolo]

I've looked into how modules are implemented in various browsers, and they follow the spec in this, with the use of the same origin policy.

@sgomes, thank you for doing the research and sharing your findings.

With all of this in mind, a well-designed ESM API would get us much closer to good performance defaults for scripts, as well as an easy way to do the right thing where it comes to lazy loading. And anyone that's worked in designing or supporting APIs surely knows the importance of good defaults, since that's what most developers end up going for 🙂

100% agree with that. I filed the issue #2768 to discuss options for lazy loading blocks (or their parts) back in 2017! We haven't been able to come up with any solution that would work seamlessly for WordPress core, not mentioning 3rd party blocks.

There was some progress with the Block Directory that lets users install blocks on the fly, but it still doesn't use a proper lazy loading of JavaScript but instead injects all necessary scripts with a clever workaround:

gutenberg/packages/block-directory/src/store/load-assets.js

Lines 53 to 81 in 50b19e7

	export async function loadAssets() {
	/*
	* Fetch the current URL (post-new.php, or post.php?post=1&action=edit) and compare the
	* JavaScript and CSS assets loaded between the pages. This imports the required assets
	* for the block into the current page while not requiring that we know them up-front.
	* In the future this can be improved by reliance upon block.json and/or a script-loader
	* dependency API.
	*/
	const response = await apiFetch( {
	url: document.location.href,
	parse: false,
	} );
	const data = await response.text();
	const doc = new window.DOMParser().parseFromString( data, 'text/html' );
	const newAssets = Array.from(
	doc.querySelectorAll( 'link[rel="stylesheet"],script' )
	).filter( ( asset ) => asset.id && ! document.getElementById( asset.id ) );
	/*
	* Load each asset in order, as they may depend upon an earlier loaded script.
	* Stylesheets and Inline Scripts will resolve immediately upon insertion.
	*/
	for ( const newAsset of newAssets ) {
	await loadAsset( newAsset );
	}
	}

There was also a promising exploration #21244 started by @spacedmonkey to expose all the registered scripts with the new REST API endpoint. However, it didn't materialize because it uncovered the complexity of the existing system built on top of the WP_Scripts class that handles inline scripts and translations.

I think i18n support might still be a challenge when dealing with lazy loading, even in the ESM world. However, it feels like moving the management of all complex dependencies to the client would resolve many of the limitations the WP_Scripts has. Considering that many plugins and sites interact with the existing system, the best way to move forward would be to build an entirely new ESM API to avoid backward compatibility issues. Those two systems should still be able to co-exist without any problems. The transition to ESM will take years. We still have to wait until all underlying APIs mature. A good example is JSON import assertions (explored in #34176) that landed in Node.js 17.1 with nodejs/node@95e4d29eb4, but it's still a stage 3 proposal for JavaScript language.

• Only load the "syntax highlighter" module when needed (when switching to code editor or showing a code block)
• Lazy load "edit" implementation of heavy blocks
• Lazy load tinymce only if the classic editor is needed...

Those are great examples shared by @youknowriad that would immediately contribute to the better performance of the block editor. We could use also lazy load the metaboxes as proposed in #32665.

The same applies to the frontend. In #36176 @aristath explores adding handling for multiple scripts defined for the single block using the Navigation core block. With good support for lazy loading, we could move the complex logic from PHP to JavaScript and decide when a given chunk needs to be loaded.

[justlevine]

I'm betting this is beyond the scope, but I would hope any API solution includes some sort of version management. Any performance gains we get from ESM are barely make a dent when plugins are all multiple duplicates of the same libraries.

[sgomes]

I'm betting this is beyond the scope, but I would hope any API solution includes some sort of version management. Any performance gains we get from ESM are barely make a dent when plugins are all multiple duplicates of the same libraries.

Thank you, @justlevine, that is definitely a good point that hadn't yet been discussed as part of this thread!

Some consideration needs to be given to versioning, and how a situation like the one you describe could be avoided, I agree. There are several options for that ranging from full semver awareness and version handling in WordPress, to pushing the version management aspects to developers and external tools entirely, and it should definitely be discussed as part of an initial proposal 👍

[gziolo]

Leaving here a note with one possible way for updating existing WordPress packages to support ES Modules:
https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c

[tomalec]

Leaving here a note with one possible way for updating existing WordPress packages to support ES Modules:

Nicely aggregated migration guide. Thanks for sharing. ❤️ Sindre
Also,

I'm having problems with ESM and (Webpack|Jest)

The problem is either (Webpack|Jest) or your (Webpack|Jest) configuration.

---

I'm betting this is beyond the scope, but I would hope any API solution includes some sort of version management. Any performance gains we get from ESM are barely make a dent when plugins are all multiple duplicates of the same libraries.

I don't have any precise POC with all WordPress setup to present, but I think adopting ESM and ImportMaps gets us closer to be able to use Open Platform means to solve that problem, see https://github.com/WICG/import-maps#multiple-versions-of-the-same-module.

For example, individual plugins could contribute their own import maps, to either use versions provided by WP, or use their own.

Also, we (as WordPress instance) could generate/resolve those maps ourselves, given the version ranges from a plugin (link in package.json format), we could compare them to the one we have, and either map it to ours or to theirs.

I tried to draft the idea in #35630 (comment)

[gziolo]

There is an active proposal for Enhancing the Scripts API with a loading strategy. There is a comment thread that proposes that type="module" could also be covered so you could enqueue JavaScript in WordPress using ES Modules with a simple syntax. This probably wouldn't cover import maps initially but it would be a good starting point.

[tyxla]

It appears that import maps are supported in all major browsers now. Even Safari seems to support it in the latest version released 2 days ago: https://caniuse.com/import-maps

[tronicboy1]

We should add a hook for enqueuing the importmap script in the head before any other scripts. What do you all think about this?

Also, I think it would be very nice to be able to specify which tags we want to be rendered on the script tags in the wp_register_script function. I have to manually add type="module" in hooks to fix this and it is not clean.

[gziolo]

@luisherranz opened an issue to track progress on the exploration of the necessary low-level primitives to add native support in WordPress for registering and enqueueing JavaScript modules, including generating an import map:

• Tracking Issue: Script Modules API #55942

[gziolo]

It's now implemented in WordPress core and will be shipped with the upcoming 6.5 major release as noted in the Interactivity API merge announcement. Major kudos to everyone involved in making that happen 🎉