Update unsupported languages in Weblate

[eloquence]

Compare:

• German strings in Weblate - a supported language - 302 strings
• Danish strings in Weblate - an unsupported language - 254 strings

It looks like the unsupported languages haven't been updated in a while, which means that translators will see outdated information, and won't see new strings at all. That in turn seems to make it impossible for an unsupported language to become supported by achieving 100% translation.

[cfm]

@eloquence and I discussed this as a candidate for a future sprint, to determine whether this is a bug, a process glitch in the internationalization procedures, or merely an unintended behavior of the i18n_tool.py translate-messages command used to generate the .po[t]s pushed to Weblate. (Note that freedomofpress/securedrop-client#1317 will investigate deferring the .pot → .po update to Weblate itself, which might relieve i18n_tool.py of having to limit this operation to SUPPORTED_LANGUAGES only.)

[cfm]

@deeplow reports that this is blocking translation of PT_pt, now that we've disambiguated it from PT. PT_br is a supported_language; PT_pt is not. I'll nominate this for attention in the next sprint, to give it plenty of lead time before the next translation cycle for v2.4.0.

[cfm]

Tl;dr: We need to, and unless I'm missing something we do not, distinguish between:

1. languages open for translation in Weblate; and
2. languages supported for selection in securedrop-admin sdconfig.

Findings: We have three kinds of "supported" "languages" for four different stages of the localization process:

1. i18n_tool.I18NTool.supported_languages are those for which i18n_tool.py update-from-weblate will fetch translations from Weblate.
2. securedrop/translations/$LANG are those that i18n_tool.py translate-messages will:
   1. --extract-update from .py → .pot → .po; and
   2. --compile from .po to .mo.
3. securedrop/translations/$LANG are also those that securedrop-admin sdconfig will allow admins to configure as...
4. SUPPORTED_LOCALES are those that the Source and Journalist interfaces will allow users to select.

(2)(1) is what keeps a language in sync in Weblate. In our current workflow and tooling, we can't enable (2) without first doing (1). Doing (2) also enables (3). Therefore, adding a new $LANG via i18n_tool.py must be completed within a single release cycle, so that $LANG can be fully supported at least to (3), if not all the way to (4).

If we want to add or update $LANG outside of a release cycle, we can cheat by manually:

1. running i18n_tool.py update-from-weblate --supported-languages $LANG to get its current messages.po;
2. running i18n_tool.py translate --extract-update to update $LANG's messages.po against the current branch;
3. importing that messages.po into Weblate to allow up-to-date translation (select Replace existing translation file); and then
4. exporting that messages.po from Weblate, committing it to securedrop, and adding $LANG to i18n_tool.I18NTool.supported_languages as part of the string-freeze procedure for the next release cycle.

From then on it's available for translation, configuration, and selection like any other language.

Since there's been a specific request for pt_PT, tomorrow I'll use it as a test case for extra- (non-) release process. Weblate will commit its changes only to securedrop-i18n, so they will be safely reverted at the next string freeze if we don't affirmatively choose to keep them via this (or some other, better!) process. :-)

[cfm]

This procedure worked well enough to sync pt_PT in weblate-sandbox and then in weblate. I've updated #6156 (comment) to reflect that the Replace existing translation file option is the way to get Weblate to accept the new strings merged in (via i18n_too.py translate-messages) with the old (from i18n_tool.py update-from-weblate).

I'm adding this ticket to the v2.4.0 milestone just for visibility to that release's Localization Manager, who can evaluate translation coverage for pt_PT and decide whether to include it as a supported language. (Likewise for any other languages we might choose to sync up this way.)

Now: this is cumbersome. If we plan to move securedrop to continuous localization anytime soon, that will enable Weblate to pick up source-string changes for all translations automatically. Otherwise, we should consider adopting just a part of that workflow, the weblate.gettext.msgmerge add-on, to see if it can obsolete that portion of i18n_tool.py translate-messages.

I'll also file a companion ticket to freedomofpress/securedrop-client#1438, which is the complement to this problem. There, we have languages being translated that we don't yet want to support; here, we have languages that can't yet be translated because we don't support them.

[cfm]

This did the trick to include pt_PT in v2.4.0. Now that we've proven the hypothesis with a specific real-world case, I'm bumping this ticket to v2.5.0, where together with #6387 it will let us solve the general "language lifecycle" story.

[cfm]

Dropping from the v2.5.0 milestone, since this will be completed during that release and localization cycle with #6557.

[Matthaiks]

Could you add Polish as a supported language?

[cfm]

Thanks for the request, @Matthaiks! We're planning (#6592) to have a policy for adding new supported languages in our v2.6.0 release. Polish looks like a promising candidate for support, so I'll make sure we consider it specifically.