Rewrite the i18n scripts (#5177)

* Rewrite the i18n scripts

- Convert CJS to MJS
- Use flat keys and simplify json handling
- Update ESLint rule to allow for dot in keys
* Update casing for Glotpress and simplify ESLint rule
* Deduplicate user-agent constant
* Add Vue to TermCasing

.codespell/ignore_lines.txt

@@ -8,7 +8,7 @@
 ;; frontend/i18n/data/en.json5
 ;; Prettier insists we escape a single quote rather than the double quotes and codespell
 ;; does not understand the escaped `\'t` as "couldn't". It instead just sees "couldn".
-    heading: 'We couldn\'t find anything for "{query}".',
+  "noResults.heading": 'We couldn\'t find anything for "{query}".',
 
 ;; catalog/tests/dags/providers/provider_api_scripts/test_wikimedia_commons.py
 ;; "Titel" matches "title", but the phrase is in Dutch, not English, so "titel"

.vale/styles/Openverse/TermCasing.yml

@@ -12,12 +12,13 @@ swap:
   # [^/\.] prevents matching things that look like URLs, file paths, or GitHub team mentions
   # For example: @WordPress/openverse-maintainers
   '[^/\.]openverse[^/\.]': Openverse
-  # OpenVerse should never be used, except as an example of something that is always wrong,
-  # in which case we'll tell Vale to ignore that line.
-  "OpenVerse": Openverse
   '[^/\.]wordpress[^/\.]': WordPress
-  # Wordpress is the same as OpenVerse
-  "Wordpress": WordPress
   '[^/\.]github[^/\.]': GitHub
-  # Github is the same as Wordpress and OpenVerse
+  '[^/\.]vue[^/\.]': Vue
+  # OpenVerse, Wordpress, Github and Glotpress should never be used, except as an example of
+  # something that is always wrong, in which case we'll tell Vale to ignore that line.
+  "OpenVerse": Openverse
+  "Wordpress": WordPress
   "Github": GitHub
+  "Glotpress": GlotPress
+  "vue": Vue

documentation/frontend/reference/i18n.md

@@ -9,7 +9,7 @@ WordPress uses GlotPress for managing translations, which is built on top of the
 Nuxt (and most JS-based i18n libraries) use
 [JSON](https://kazupon.github.io/vue-i18n/guide/formatting.html) for managing
 translations. This disconnect means that Openverse translations must convert
-from JSON to POT and back again. Hence there is quite a bit of scaffolding
+from JSON to POT and back again. Hence, there is quite a bit of scaffolding
 involved.
 
 ## Upload pipeline
@@ -25,7 +25,7 @@ frontend application and provided to GlotPress for translation.
 
 - Upload this
   [POT file](https://github.com/WordPress/openverse/blob/translations/openverse.pot)
-  to a fixed URL. Currently the file is hosted in the `translations` branch of
+  to a fixed URL. Currently, the file is hosted in the `translations` branch of
   the [WordPress/openverse](https://github.com/WordPress/openverse) repo.
 
 - GlotPress presents the strings, fuzzy translations and other helpful context
@@ -34,42 +34,39 @@ frontend application and provided to GlotPress for translation.
 ## Download pipeline
 
 This pipeline deals with how translations are retrieved from GlotPress,
-processed and loaded into Nuxt via the Nuxt i18n module.
+processed and loaded into Nuxt via the Nuxt i18n module. The main entry point,
+`i18n/scripts/setup.mjs`, orchestrates the entire process based on the command
+line arguments. There are three main modes of operation:
 
-### Steps
-
-- Parse and extract the list of all locales from GlotPress's PHP source code.
-  Then narrow down the list to locales available in the WP GlotPress instance
-  and populate their coverage percentage from the
-  [GlotPress stats](https://translate.wordpress.org/projects/meta/openverse/).
-
-  The output is written to `wp-locales.json`.
+- for production, it can download translations from GlotPress, parse the
+  locales, and update the Vue i18n plugin configuration.
+- for local development, it can convert `en.json5` to JSON format used by the
+  Nuxt app, and save the empty `valid-locales.json` file to be used by the Nuxt
+  app.
+- for testing, it can copy the test locale metadata, and test translations to
+  the main `i18n/locales` folder.
 
-  **Script:** `i18n:create-locales-list`
+### Production Steps
 
-- Download all translations from GlotPress as JED 1.x JSON files. The flattened
-  JED 1.x (derived from the flattened POT files) files are converted back into
-  the nested JSON as expected by Nuxt i18n.
+- Download all translations from GlotPress as NGX JSON files - flat json files.
 
   This script downloads all available translations in bulk as a ZIP file and
   then extracts JSON files from the ZIP file. This prevents excessive calls to
   GlotPress, which can be throttled and cause some locales to be missed.
 
-  **Script:** `i18n:get-translations`
+  **Script:** `i18n/scripts/translations.mjs`
 
-- Separate the locales into three groups based on the JSON files emitted by
-  `i18n:get-translations`.
+- Parse and extract the list of all locales from GlotPress's PHP source code.
+  Then narrow down the list to locales available in the WP GlotPress instance
+  and calculate their coverage percentage from the number of keys in the
+  translation and the number of keys in the main `en.json5` file. After that,
+  separate the locale metadata into two groups based on the JSON files emitted
+  by the previous step.
 
   - **translated:** JSON file is present with mappings, written to
     `valid-locales.json`.
-  - **untranslated:** JSON file is present but empty, written to both
-    `valid-locales.json` and `untranslated-locales.json`.
-  - **invalid:** JSON file is not present, written to `invalid-locales.json`.
-
-  **Script:** `i18n:update-locales`
-
-- Pass the list of valid locales (along with extra fields) into the Nuxt i18n
-  module. This is configured in the Nuxt configuration file, `nuxt.config.ts`.
+  - (only if --verbose flag is on) **untranslated:** JSON file is present but
+    empty, written to `untranslated-locales.json`.
 
 - Pass the fallback locale mappings to the underlying Vue i18n plugin. This is
   configured in the plugin configuration file,
@@ -78,7 +75,7 @@ processed and loaded into Nuxt via the Nuxt i18n module.
 ## Test locales
 
 Three locales are maintained in the code base (as opposed to downloaded from
-Glotpress) for use in testing, alongside the default English locale. These
+GlotPress) for use in testing, alongside the default English locale. These
 locales are **not** meant to be representative of actual final translations.
 They are merely used to approximate length and language features that help
 identify potential layout issues in the application.

frontend/i18n/README.md

@@ -0,0 +1,10 @@
+# Locales
+
+The primary internationalisation file is [`data/en.json5`](../data/en.json5).
+All `.json` files present in the [`locales`](./locales) directory are
+re-generated when updating translations, so they should not be modified.
+
+# Locale scripts
+
+Locale scripts should be run in the root of the repository using their
+respective pnpm commands, in the `i18n` namespace.