Blocks: Upgrade simple-html-tokenizer dependency

[gziolo]

Description

Originally raised by @swissspidy in #15138:

I am trying to use @wordpress/scripts for unit tests. Still running into #13364, but there's also another issue when running wp-scripts test-unit-js:

and further commented by @nerrad in #13364 after the previous discussion on WordPress.org Slack.

I'm guessing the issue here is that the import is coming from a specific es6 directory from the simple-html-tokenizer module:

gutenberg/packages/blocks/src/api/validation.js

Line 4 in 93c5ad3

import Tokenizer from 'simple-html-tokenizer/dist/es6/tokenizer';

We had this line added to Jest config:
https://github.com/WordPress/gutenberg/blob/master/test/unit/jest.config.js#L29
to ensure this unusual import statement works as expected with our test setup.

This PR tries to resolve this issue for projects which depend on @wordpress/blocks. It brings the latest version of the simple-html-tokenizer package and uses it as a usual npm module through the entry point rather than trying to use a file based on the filesystem structure.

Impact

There is only 1 kB difference in the size of uncompressed production build file which means we can ignore it for the benefit of fixing issues for 3rd party developers.

Before

After

Testing instructions

Based on #11771.

Verify that block invalidation is not triggered by encoding variations.

For example, inserting the following HTML as the contents of a post (in Text Mode, Classic Editor Text tab, or directly in the database) should not be presented as an invalid block when next viewing the Visual Mode of the editor:

<!-- wp:paragraph -->
<p>This works. &#128517;</p>
<!-- /wp:paragraph -->

[gziolo]

@aduth - as you worked on #11771 which introduced the changes which enforced the exclusion of simple-html-tokenizer from Jest config in the transform ignores section, I would appreciate your review.

[nerrad]

I think the 1kb size gain is reasonable considering the benefit to those that import this package. It was a gotcha that wasn't easily apparent on how to fix.

[aduth]

I'm guessing by your measure of the size difference, it has not regressed (possibly because of better ES module support than they had previously†), but the reason it was originally written this way was to avoid the side effect of importing the very very large character reference set.

https://github.com/tildeio/simple-html-tokenizer/blob/9a894bd75545e107638272f3e1e812933815953f/src/index.ts#L2-L4

https://github.com/tildeio/simple-html-tokenizer/blob/master/src/generated/html5-named-char-refs.ts

† I'd forgot I made this pull request: tildeio/simple-html-tokenizer#49 😅

[gziolo]

It would explain why it works so well now 😍

[swissspidy]

Thanks for merging this!

I assume for testing this in my project I need to manually apply the changes until new package versions are released?

[gziolo]

@swissspidy, yes. I think the only thing you need to include is:

transformIgnorePatterns: [	
	'node_modules/(?!(simple-html-tokenizer)/)',	
],

in Jest config.

[swissspidy]

Hmm I already had that in my config. But since I also had other issues with Babel and core-js, I probably have to wait for #15139 anyway.