build: eliminate `i18n_tool.py` #6954

cfm · 2023-09-28T22:18:07Z

Status

Ready for review

Description of Changes

Closes #6917 by removing i18n_tool.py in favor of:

make extract-strings, which wraps pybabel extract;
make check-{desktop-files,strings}, which enforces make extract-strings in CI;
pybabel compile;
make supported-locales, which consumes i18n.json and can be further processed via jq;
ci(translation-tests): verify reproducibility of gettext machine objects #6953;
replace l10n.txt include with link to i18n.rst securedrop-docs#503; and
Document changes related to continuous localization securedrop-dev-docs#86.

Testing

Because this pull request offloads much of our existing localization workflow onto Weblate, very little self-contained, local testing is possible or worthwhile. Namely:

Inspect packages for differences between develop and here, with a fresh build image each time:
1. (docker|podman) image prune -a --force && make build-debs on develop and save them (e.g., cp -Rp build build@develop).
2. (docker|podman) image prune -a --force && make build-debs on this branch.
3. diffoscope build@develop/focal/securedrop-app-code*.deb build/focal/securedrop-app-code*.deb shows only:
  - diffs from this branch
  - metadata changes in .mo files

Instead...

Perform a mock release and deployment via `securedrop-i18n-sandbox`

A mock Localization Manager (@cfm) and Release Manager (@nathandyer) should pair their way through (approximately) the following:

Along the way, append fixup commits both here and in freedomofpress/securedrop-dev-docs#86 as needed for both a working implementation and correct documentation, respectively, of this workflow.

Rebase fixups (only) before the final approval and merge.

Deployment

Before merging:

Give Localization Lab a heads-up.
At weblate.securedrop.org, lock the securedrop/securedrop and securedrop/desktop components.
Coordinate with Infrastructure so that:
- weblate.securedrop.org uses...
- GitHub user @weblate-fpf to open pull requests in...
- freedomofpress/securedrop.

After merging:

At weblate.securedrop.org, configure securedrop/securedrop and securedrop-desktop (from freedomofpress/securedrop) like securedrop/securedrop-client (from freedomofpress/securedrop-client). Add-ons: as above.
At weblate.securedrop.org, unlock the securedrop/securedrop and securedrop/desktop components.
Smoke-test a translation change. It may take some cycling of the Weblate-side repository and add-ons before this yields as clean a diff as freedomofpress/securedrop-i18n-sandbox#3. It should also pass CI into securedrop.

Checklist

If you made changes to the server application code:

Linting (make lint) and tests (make test) pass in the development container

If you made non-trivial code changes:

I have written a test plan and validated it for this PR

Choose one of the following:

I have opened a PR in the docs repo for these changes, or will do so later
- Document changes related to continuous localization securedrop-dev-docs#86
I would appreciate help with the documentation
These changes do not require documentation

cfm · 2023-09-29T01:34:16Z

@nathandyer, I'd love your thoughts on this "test" plan, which is really a mock (compressed) QA/localization/release/deployment cycle. Feel free to leave comments here, and we can discuss this (and next week's timeline) in Monday's stand-up too.

nathandyer · 2023-09-29T13:41:38Z

@cfm I think this "test" plan makes a lot of sense, and is well thought out! I'm looking forward to pairing next week to work through this mock workflow together!

cfm · 2023-10-03T16:35:20Z

@nathandyer, now that we've postponed the testing session to Thursday, would you be able to run through the async portion of the test plan beforehand? (The very beginning of the "Test Plan" section, before the "Peform a Mock Release..." subsection.) It would be nice to go into the end-to-end operational testing having already done a preliminary review of the self-contained functionality.

legoktm

(very drive-by review)

securedrop/i18n.py

Makefile

babel.cfg

This list was *almost* in alphabetical order, so I've reordered it. If there's a logical order in which we want these targets to run, we should document that.

@legoktm

As @legoktm notes[1] on a previous version of this commit, it's a single line whether or not we wrap it in a Makefile target. [1]: #6954 (comment)

@legoktm

As @legoktm notes[1] on a previous version of this commit, it's a single line whether or not we wrap it in a Makefile target. [1]: #6954 (comment)

…top files

…op/desktop too

…s.po" structure

cfm · 2023-10-06T01:58:30Z

@nathandyer, all the securedrop/desktop bits are wired up as of 0bd3249 and demonstrated in freedomofpress/securedrop-i18n-sandbox@d6fcba4 into freedomofpress/securedrop-i18n-sandbox#3. I'll take a look at the lint failure tomorrow to unblock your Tails-side testing on Tuesday.

…-sort-output" I'm not sure what about the securedrop/desktop sources would otherwise introduce sorting drift between my local environment and CI, but there's no downside to applying this sorting wherever it turns out to be necessary.

cfm · 2023-10-06T19:51:48Z

Okay, we're out of the woods with 1735fa2. @nathandyer, this was truly a lint-level fix, the gettext equivalent of re-running black. So I won't bother pushing this to securedrop-i18n-sandbox, and you should be good to go on Tuesday morning per #6954 (comment).

nathandyer · 2023-10-10T16:12:56Z

Thank you @cfm!

I took this for a spin this morning, but it doesn't seem to be working as we might have hoped. The process I followed was:

I noticed where you had already modified a string in the extension template, changing Launch Source Interface to Launch Interface
In weblate-sandbox.securedrop.org, I found this change, and added a new translation for it under the French language
I committed the changes from the management view, then pushed them to the repository
In securedrop-i18n-sandbox, I reviewed the pull request, confirmed it contained the necessary translation changes, and then merged it in
From an Admin workstation, I downloaded and extracted a copy of the i18n branch from securedrop-i18n-sandbox, then ran the usual ./securedrop-admin setup and ./securedrop-admin --force tailsconfig
I rebooted and confirmed the extension loaded (in English), and that it contained your string change
I rebooted and selected French (fr) as the language

Result:
After doing so, all the items in the extension are in French, except our new change; however, instead of the translation I provided, it just says "Launch Interface"

cfm · 2023-10-10T22:34:26Z

@nathandyer and I discussed #6954 (comment) out of band and concluded that it's mostly passing:

https://github.com/weblate-sandbox-fpf/securedrop-i18n-sandbox/blob/73b5aea826e6e6ec7a8f4ebe0f4c9994b183cce0/install_files/ansible-base/roles/tails-config/templates/locale/fr/LC_MESSAGES/messages.po#L21-L22 is shown in this screenshot. The msgid Launch Interface (freedomofpress/securedrop-i18n-sandbox@d6fcba4) has an empty msgstr, so it's left untranslated, while all the other translations for fr are intact from Weblate.

https://github.com/weblate-sandbox-fpf/securedrop-i18n-sandbox/blob/73b5aea826e6e6ec7a8f4ebe0f4c9994b183cce0/install_files/ansible-base/roles/tails-config/templates/locale/fr/LC_MESSAGES/messages.po#L18-L19 is correct too, but make update-desktop-files still has to be run manually as part of the release process:

(.venv) user@sd-dev:~/securedrop-i18n-sandbox$ git whereami
aa6495c2e (HEAD -> i18n, origin/i18n, origin/HEAD) Merge pull request #3 from weblate-sandbox-fpf/weblate-securedrop-securedrop
(.venv) user@sd-dev:~/securedrop-i18n-sandbox$ make check-desktop-files
make[2]: Nothing to be done for 'install_files/ansible-base/roles/tails-config/templates/ssh_config.j2'.
Desktop files are out of date. Please run "make update-desktop-files" and commit the changes.
make: *** [Makefile:395: check-desktop-files] Error 1

This would have been enforced by make lint in CI, which isn't run for securedrop-i18n-sandbox.

(.venv) user@sd-dev:~/securedrop-i18n-sandbox$ make update-desktop-files
make[1]: Nothing to be done for 'install_files/ansible-base/roles/tails-config/templates/ssh_config.j2'.
(.venv) user@sd-dev:~/securedrop-i18n-sandbox$ git diff
diff --git a/install_files/ansible-base/roles/tails-config/templates/desktop-source-icon.j2 b/install_files/ansible-base/roles/tails-config/templates/desktop-source-icon.j2
index 50efd29c2..13369da04 100644
--- a/install_files/ansible-base/roles/tails-config/templates/desktop-source-icon.j2
+++ b/install_files/ansible-base/roles/tails-config/templates/desktop-source-icon.j2
@@ -6,27 +6,8 @@ Type=Application
 Terminal=false
 StartupNotify=true
 Categories=Network;
-Name[ar]=واجهة SecureDrop للمصدر
-Name[ca]=Interfície de font del SecureDrop
-Name[cs]=SecureDrop rozhraní zdroje
-Name[de_DE]=Quellenschnittstelle für SecureDrop
-Name[el]=Διεπαφή πηγής του SecureDrop
-Name[es_ES]=Interfaz de fuente de SecureDrop
-Name[fr]=SecureDrop - Interface des sources
-Name[hi]=SecureDrop स्रोत अंतराफलक
-Name[is]=Heimildarmannaviðmót SecureDrop
-Name[it]=Interfaccia Fonte di SecureDrop
-Name[nb_NO]=Kildegrensesnitt for SecureDrop
-Name[nl]=Securedrop - interface voor bronnen
-Name[pt_BR]=Interface de Fontes do SecureDrop
-Name[pt_PT]=Interface de Fontes do SecureDrop
-Name[ro]=SecureDrop - interfața pentru surse
-Name[ru]=SecureDrop — интерфейс источников
-Name[sk]=SecureDrop rozhranie zdroja
-Name[sv]=SecureDrop källgränssnitt
-Name[tr]=SecureDrop Kaynak Arayüzü
-Name[zh_Hans]=SecureDrop 线人界面
-Name[zh_Hant]=SecureDrop 線人使用介面
-Name=SecureDrop Source Interface
+Name[es_ES]=missing a word
+Name[fr]=Le SecureDrop Flufudo
+Name=SecureDrop Interface
 Icon={{ tails_config_securedrop_dotfiles }}/securedrop_icon.png
 Exec=/usr/local/bin/tor-browser {{ item.0.onion_url }}

All the subtractions are correct per freedomofpress/securedrop-i18n-sandbox@d6fcba4. The additions are correct per freedomofpress/securedrop-i18n-sandbox#3.

So, @nathandyer, as of freedomofpress/securedrop-i18n-sandbox@9b09f371a you should now see the Source Interface icon display the msgstr Le SecureDrop Flufudo.

I'll confirm this in a moment on my end. I'll also rebase my own local copy of securedrop-i18n-sandbox after #6976 so that I can check the staging environment without rebasing and force-pushing here.

cfm · 2023-10-10T23:02:05Z

#6954 (comment):

So, @nathandyer, as of freedomofpress/securedrop-i18n-sandbox@9b09f371a you should now see the Source Interface icon display the msgstr Le SecureDrop Flufudo.

Confirmed here:

Under en_US: Source Interface icon is labeled SecureDrop Interface
Under fr_FR: Source Interface icon is labeled Le SecureDrop Flufudo

cfm · 2023-10-10T23:26:10Z

A staging server installed from packages built from securedrop-i18n-sandbox contains the current translations from securedrop/securedrop.

Confirmed that freedomofpress/securedrop-i18n-sandbox@9b09f371a98522f5120b60996f63c837460fa264 + #6975 includes https://github.com/weblate-sandbox-fpf/securedrop-i18n-sandbox/blob/73b5aea826e6e6ec7a8f4ebe0f4c9994b183cce0/securedrop/translations/es_ES/LC_MESSAGES/messages.po#L866-L867.

root@sd-staging:~/securedrop-i18n-sandbox# git show 55a37b831206bfbfdffe3b57babec678166f1cfb
commit 55a37b831206bfbfdffe3b57babec678166f1cfb (HEAD -> i18n)
Merge: 9b09f371a d431061c7
Author: root <root@(none)>
Date:   Tue Oct 10 23:13:34 2023 +0000

    Merge remote-tracking branch 'main/develop' into i18n

nathandyer

Everything is looking great to me, and I'm satisfied that our test plan (as unusual as it was, given that the author of the PR was directly involved in the testing itself) is sufficient and covers all the necessary bases. From my perspective, we're OK to go ahead and merge, then move this forward.

cfm · 2023-10-11T23:32:56Z

Thanks, @nathandyer! I'm about to merge and begin the fiddly bits between here and Weblate.

Makefile

securedrop/requirements/python3/translation-requirements.txt

---for the greatest coverage as soon as possible.[1] [1]: #6954 (comment)

Since #6954 and freedomofpress/securedrop-dev-docs#86, we no longer derive translator credits from Git history at all, so this file is unused as well as out of date.

chore: remove "i18n_tool.py"

e9d1d03

cfm force-pushed the i18n-6917-c10s-l10n branch 5 times, most recently from 4241fd7 to 99d4eca Compare September 29, 2023 01:14

cfm requested a review from nathandyer September 29, 2023 01:19

cfm marked this pull request as ready for review September 29, 2023 01:19

cfm requested a review from a team as a code owner September 29, 2023 01:19

cfm assigned cfm and nathandyer Sep 29, 2023

cfm added this to the SecureDrop 2.7.0 milestone Sep 29, 2023

legoktm mentioned this pull request Sep 29, 2023

Replace bandit with ruff #6961

Merged

3 tasks

cfm mentioned this pull request Oct 3, 2023

replace l10n.txt include with link to i18n.rst freedomofpress/securedrop-docs#503

Merged

5 tasks

cfm mentioned this pull request Oct 3, 2023

ci(translation-tests): verify reproducibility of gettext machine objects #6953

Merged

4 tasks

legoktm reviewed Oct 3, 2023

View reviewed changes

securedrop/i18n.py Outdated Show resolved Hide resolved

Makefile Outdated Show resolved Hide resolved

babel.cfg Outdated Show resolved Hide resolved

cfm added 3 commits October 3, 2023 15:05

build({check,extract}-strings}): adapt from securedrop-client

2659e8d

build(extract-strings): skip "securedrop/tests/"

60002cd

build(lint): run "check-strings"

9016dbe

This list was *almost* in alphabetical order, so I've reordered it. If there's a logical order in which we want these targets to run, we should document that.

cfm added a commit that referenced this pull request Oct 3, 2023

build: inline "pybabel compile" invocation

c7cc91d

As @legoktm notes[1] on a previous version of this commit, it's a single line whether or not we wrap it in a Makefile target. [1]: #6954 (comment)

cfm force-pushed the i18n-6917-c10s-l10n branch from 39c2659 to 37a79f9 Compare October 3, 2023 22:06

cfm added 6 commits October 3, 2023 15:17

build: inline "pybabel compile" invocation

820b37f

As @legoktm notes[1] on a previous version of this commit, it's a single line whether or not we wrap it in a Makefile target. [1]: #6954 (comment)

build(supported-languages): read from "i18n.json"

25e8ab0

chore: "make extract-strings"

2144759

chore: remove "list[-all]-translators" targets

1dd85bb

chore: types

e517484

ci(translation-tests): split on output of "make supported-locales"

e0e8c9b

cfm added 5 commits October 5, 2023 16:24

build(update-supported-locales): also update desktop "LINGUAS" file

7dffc70

build({check,update}-desktop-files): enforce msgfmt rendering of desk…

086b9ad

…top files

build(extract-strings): use Babel for unified extraction for securedr…

986fa63

…op/desktop too

refactor(update-desktop-files): use "locale/$LANG/LC_MESSAGES/message…

6ffacc4

…s.po" structure

docs: subsection "Localization" section for clarity

0bd3249

cfm force-pushed the i18n-6917-c10s-l10n branch from 6afba25 to 1735fa2 Compare October 6, 2023 19:21

cfm mentioned this pull request Oct 10, 2023

Document changes related to continuous localization freedomofpress/securedrop-dev-docs#86

Merged

6 tasks

nathandyer approved these changes Oct 11, 2023

View reviewed changes

cfm merged commit 8d00ba5 into develop Oct 11, 2023

cfm deleted the i18n-6917-c10s-l10n branch October 11, 2023 23:41

This was referenced Oct 12, 2023

Translations update from Weblate #6984

Merged

ci(translation-tests): change filter prefix s/i18n/weblate/ #6985

Merged

legoktm reviewed Oct 12, 2023

View reviewed changes

Makefile Show resolved Hide resolved

legoktm reviewed Oct 12, 2023

View reviewed changes

securedrop/requirements/python3/translation-requirements.txt Show resolved Hide resolved

legoktm reviewed Oct 12, 2023

View reviewed changes

securedrop/requirements/python3/translation-requirements.txt Show resolved Hide resolved

cfm added a commit that referenced this pull request Oct 12, 2023

build(lint): run ruff first

ddea15f

---for the greatest coverage as soon as possible.[1] [1]: #6954 (comment)

This was referenced Oct 12, 2023

build(lint): run ruff first #6995

Merged

docs: translation requirements are production requirements #6996

Merged

cfm mentioned this pull request Sep 26, 2024

generation of screenshots is broken #7237

Closed

cfm mentioned this pull request Oct 16, 2024

i18n_tool.py update-from-weblate stages securedrop/desktop changes from unsupported languages #6879

Closed

2 tasks

cfm added a commit that referenced this pull request Dec 19, 2024

chore: remove Git ".mailmap"

0cc2a7c

Since #6954 and freedomofpress/securedrop-dev-docs#86, we no longer derive translator credits from Git history at all, so this file is unused as well as out of date.

cfm mentioned this pull request Dec 19, 2024

chore: remove Git .mailmap #7397

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build: eliminate `i18n_tool.py` #6954

build: eliminate `i18n_tool.py` #6954

cfm commented Sep 28, 2023 •

edited

Loading

cfm commented Sep 29, 2023

nathandyer commented Sep 29, 2023

cfm commented Oct 3, 2023

legoktm left a comment

cfm commented Oct 6, 2023

cfm commented Oct 6, 2023

nathandyer commented Oct 10, 2023

cfm commented Oct 10, 2023

cfm commented Oct 10, 2023

cfm commented Oct 10, 2023

nathandyer left a comment

cfm commented Oct 11, 2023

build: eliminate i18n_tool.py #6954

build: eliminate i18n_tool.py #6954

Conversation

cfm commented Sep 28, 2023 • edited Loading

Status

Description of Changes

Testing

Perform a mock release and deployment via securedrop-i18n-sandbox

Deployment

Checklist

If you made changes to the server application code:

If you made non-trivial code changes:

cfm commented Sep 29, 2023

nathandyer commented Sep 29, 2023

cfm commented Oct 3, 2023

legoktm left a comment

Choose a reason for hiding this comment

cfm commented Oct 6, 2023

cfm commented Oct 6, 2023

nathandyer commented Oct 10, 2023

cfm commented Oct 10, 2023

cfm commented Oct 10, 2023

cfm commented Oct 10, 2023

nathandyer left a comment

Choose a reason for hiding this comment

cfm commented Oct 11, 2023

build: eliminate `i18n_tool.py` #6954

build: eliminate `i18n_tool.py` #6954

cfm commented Sep 28, 2023 •

edited

Loading

Perform a mock release and deployment via `securedrop-i18n-sandbox`