Use ICU4X to run parts of util.unicode.org

[sffc]

Currently util.unicode.org runs on top of ICU4J. It works fine, but sometimes it is slow or hits rate limits that we've imposed to cap server costs, as it is doing as I write this message:

We should add ICU4X-backed tooling to parts of util.unicode.org via WebAssembly. This has the benefit of reducing latency (all calculations are client-side) and serving costs (the ICU4X wasm file can be cost-efficiently cached and served in a CDN).

The Unicode Tools are designed to run on the latest (even unreleased) version of the Unicode Standard, and so part of this project may involve improving some of the ICU4X tooling so that it can read raw UCD files. See unicode-org/icu4x#4602

CC @josh-hadley @eggrobin

[eggrobin]

There is much more to it than using draft data. We also expose properties that should never be part of APIs, things that are not properties, etc., and we use an extremely featureful version of UnicodeSet that should not be part of general-purpose libraries (in particular, this allows you to look at past versions of Unicode, or search property values using regular expressions).

In particular, the fact that the properties library is the same that we use to actually generate and test the standard matters when it comes to being confident that we know what we are publishing.

I suspect that the traffic we are seeing is some sort of crawling bot though:

All of these queries are « the characters that have some specific value of some property property » (typically with one result), but without much rhyme or reason to what values and properties are queried. Queries of this form are linked from the character.jsp page, so I suspect this is something following the links there.

[eggrobin]

Here’s the current traffic from one specific (slightly odd) user agent:

[macchiati]

Hmmm, maybe we could block that user, and throttle anyone with more than 1 query per 10 seconds?

…

On Sun, Jan 26, 2025, 06:30 Robin Leroy ***@***.***> wrote: Here’s the current traffic from one specific (slightly odd) user agent: image.png (view on web) <https://github.com/user-attachments/assets/b70a6309-d506-45f9-b321-959704609b7c> — Reply to this email directly, view it on GitHub <#1004 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACJLEMCIWJO35MX5HAIJ2ID2MTWPHAVCNFSM6AAAAABV4DY2O2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJUGQ2DQMJSHE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[srl295]

@sffc is this ticket a dup? or have we just talked about it without an issue?

if node is nodejs maybe someone is scraping.

[sffc]

This issue is to track migrating parts of until.unicode.org to ICU4X, which has been discussed in various forums, but I couldn't find a canonical issue.

Investigation on other server cost mitigation or rate limiting techniques could be discussed elsewhere. That doesn't however invalidate the motivation for making popular parts of the site run client-side.