feat: validate `SDConfig.SUPPORTED_LANGUAGES` for usable locales #6406

cfm · 2022-04-19T00:53:14Z

Status

Ready for review

Description of Changes

Closes #6366 by determining that a locale is usable if it is both (a) available in the filesystem and (b) configured by the administrator in SDConfig.SUPPORTED_LANGUAGES. Once we've determined which configured locales are actually usable, we:

warn if a configured locale is not available;
fall back to the hard-coded FALLBACK_LOCALE (en_US) if SDConfig.DEFAULT_LOCALE is not usable; and
error out if neither the default nor the fallback locale is usable.

Testing

Routine testing

In the development environment:

Confirm that the language menu works normally.
- zh_{Hans,Hant} are each (separately) listed, selectable, and loadable.

Configured language loses support

In the development environment:

First:

$ rm -rf securedrop/translations/de_DE  # or the language of your choice
$ make dev

Observe the error:

[2022-04-19 22:14:24,290] ERROR in i18n: Configured locales {Locale('de', territory='DE')} are not in the set of usable locales {Locale('es', territory='ES'), Locale('is'), Locale('pt', territory='BR'), Locale('zh', script='Hans'), Locale('ro'), Locale('it', territory='IT'), Locale('en', territory='US'), Locale('ca'), Locale('nl'), Locale('fr', territory='FR'), Locale('el'), Locale('sv'), Locale('ar'), Locale('cs'), Locale('ru'), Locale('hi'), Locale('nb', territory='NO'), Locale('tr'), Locale('zh', script='Hant'), Locale('sk')}

In the Source Interface, observe that Deutsch is no longer available in the language menu.

Default language loses support

In the development environment:

Apply the patch:

--- a/securedrop/config.py.example

+++ b/securedrop/config.py.example

@@ -82,7 +82,7 @@ DATABASE_ENGINE = 'sqlite'

 DATABASE_FILE = os.path.join(SECUREDROP_DATA_ROOT, 'db.sqlite')

 

 # Which of the available locales should be displayed by default ?

-DEFAULT_LOCALE = 'en_US'

+DEFAULT_LOCALE = 'de_DE'

 

 # How long a session is valid before it expires and logs a user out

 SESSION_EXPIRATION_MINUTES = 120

Then:

$ rm -rf securedrop/translations/de_DE # or the language of your choice

$ make dev

Observe the error:

[2022-04-19 22:14:24,290] ERROR in i18n: Configured locales {Locale('de', territory='DE')} are not in the set of usable locales {Locale('es', territory='ES'), Locale('is'), Locale('pt', territory='BR'), Locale('zh', script='Hans'), Locale('ro'), Locale('it', territory='IT'), Locale('en', territory='US'), Locale('ca'), Locale('nl'), Locale('fr', territory='FR'), Locale('el'), Locale('sv'), Locale('ar'), Locale('cs'), Locale('ru'), Locale('hi'), Locale('nb', territory='NO'), Locale('tr'), Locale('zh', script='Hant'), Locale('sk')}

In the Source Interface, observe that Deutsch is no longer available in the language menu.

No usable fallback locale

In the development environment:

Apply the patch:

--- a/Makefile

+++ b/Makefile

@@ -171,7 +171,6 @@ securedrop/config.py: ## Generate the test SecureDrop application config.

                 ctx.update(dict((k, {"stdout":v}) for k,v in os.environ.items())); \

                 ctx = open("config.py", "w").write(env.get_template("config.py.example").render(ctx))'

        @echo >> securedrop/config.py

-       @echo "SUPPORTED_LOCALES = $$(if test -f /opt/venvs/securedrop-app-code/bin/python3; then ./securedrop/i18n_tool.py list-locales --python; else DOCKER_BUILD_VERBOSE=false $(DEVSHELL) ./i18n_tool.py list-locales --python; fi)" | sed 's/\r//' >> securedrop/config.py

        @echo

 

 .PHONY: test-config

diff --git a/securedrop/config.py.example b/securedrop/config.py.example

index 92544087a..508e5db66 100644

--- a/securedrop/config.py.example

+++ b/securedrop/config.py.example

@@ -82,7 +82,8 @@ DATABASE_ENGINE = 'sqlite'

 DATABASE_FILE = os.path.join(SECUREDROP_DATA_ROOT, 'db.sqlite')

 

 # Which of the available locales should be displayed by default ?

-DEFAULT_LOCALE = 'en_US'

+DEFAULT_LOCALE = 'de_DE'

 

 # How long a session is valid before it expires and logs a user out

 SESSION_EXPIRATION_MINUTES = 120

+SUPPORTED_LOCALES = ['de_DE']

Then:

$ rm -rf securedrop/config.py
$ rm -rf securedrop/translations/de_DE # or the language of your choice

$ make dev

Observe the crash:


ValueError: None of the default locales {Locale('en', territory='US'), Locale('de', territory='DE')} is in the set of usable locales set()

OSSEC alerts

In the staging environment:

On the Monitor Server, tail -f /var/ossec/logs/alerts/alerts.log.

On the Application Server:

$ sudo rm -rf /var/www/securedrop/translations/de_DE  # or another locale configured in your staging environment

$ sudo systemctl restart apache2

On the Monitor Server, observe an alert like:

** Alert 1650408345.42633: mail  - Apache logs

2022 Apr 19 22:45:45 (sd-staging-app-base-focal) 10.137.0.50->/var/log/apache2/journalist-error.log

Rule: 400700 (level 7) -> 'Apache application error.'

[2022-04-19 22:14:24,290] ERROR in i18n: Configured locales {Locale('de', territory='DE')} are not in the set of usable locales {Locale('es', territory='ES'), Locale('is'), Locale('pt', territory='BR'), Locale('zh', script='Hans'), Locale('ro'), Locale('it', territory='IT'), Locale('en', territory='US'), Locale('ca'), Locale('nl'), Locale('fr', territory='FR'), Locale('el'), Locale('sv'), Locale('ar'), Locale('cs'), Locale('ru'), Locale('hi'), Locale('nb', territory='NO'), Locale('tr'), Locale('zh', script='Hant'), Locale('sk')}

Deployment

This is a disruptive change for an instance X if:

we remove a language L for lack of translation coverage; and
L is configured as X's DEFAULT_LANGUAGE.

This is a breaking change for X if, in addition:

the fallback locale en_US has been removed from X's SUPPORTED_LANGUAGES.

A report of instances' supported_languages from the Directory should tell us whether any instances will need proactive outreach about either of these scenarios. Otherwise, we should make sure to warn about them in the release notes.

Checklist

If you made changes to the server application code:

Linting (make lint) and tests (make test) pass in the development container

If you made non-trivial code changes:

I have written a test plan and validated it for this PR

Choose one of the following:

I have opened a PR in the docs repo for these changes, or will do so later
I would appreciate help with the documentation
These changes do not require documentation

codecov-commenter · 2022-04-19T01:10:15Z

Codecov Report

Merging #6406 (20c7212) into develop (f0dd9a8) will decrease coverage by 0.18%.
The diff coverage is 94.11%.

@@             Coverage Diff             @@
##           develop    #6406      +/-   ##
===========================================
- Coverage    84.02%   83.83%   -0.19%     
===========================================
  Files           61       62       +1     
  Lines         4312     4331      +19     
  Branches       524      525       +1     
===========================================
+ Hits          3623     3631       +8     
- Misses         565      575      +10     
- Partials       124      125       +1

Impacted Files	Coverage Δ
securedrop/i18n.py	`93.93% <93.93%> (+0.68%)`	⬆️
securedrop/sdconfig.py	`87.93% <100.00%> (+0.10%)`	⬆️
...versions/3d91d6948753_create_source_uuid_column.py	`42.85% <0.00%> (-2.60%)`	⬇️
securedrop/encryption.py	`89.43% <0.00%> (-0.80%)`	⬇️
securedrop/source_app/main.py	`92.75% <0.00%> (-0.49%)`	⬇️
...ns/b7f98cfd6a70_make_filesystem_id_non_nullable.py	`44.44% <0.00%> (ø)`
securedrop/source_app/utils.py	`93.22% <0.00%> (+0.11%)`	⬆️

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

legoktm · 2022-05-02T19:07:12Z

Still walking through the test plan (more feedback to come shortly...), but when I switched my locale to de_DE, deleted it and restarted the dev container, I got a stack trace of:

Traceback (most recent call last):
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/flask/app.py", line 2091, in __call__
    return self.wsgi_app(environ, start_response)
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/flask/app.py", line 2076, in wsgi_app
    response = self.handle_exception(e)
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/flask/app.py", line 2073, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/flask/app.py", line 1518, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/flask/app.py", line 1516, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/flask/app.py", line 1502, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/home/user/github/freedomofpress/securedrop/securedrop/source_app/main.py", line 38, in index
    return render_template('index.html')
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/flask/templating.py", line 147, in render_template
    return _render(
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/flask/templating.py", line 128, in _render
    rv = template.render(context)
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/jinja2/environment.py", line 1304, in render
    self.environment.handle_exception()
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/jinja2/environment.py", line 925, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "/home/user/github/freedomofpress/securedrop/securedrop/source_templates/index.html", line 66, in top-level template code
    {% include 'locales.html' %}
  File "/home/user/github/freedomofpress/securedrop/securedrop/source_templates/locales.html", line 4, in top-level template code
    {{ gettext('Selected language')}}: {{ g.locales[g.localeinfo.id].display_name }}
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/jinja2/environment.py", line 474, in getattr
    return getattr(obj, attribute)
jinja2.exceptions.UndefinedError: 'collections.OrderedDict object' has no attribute 'de_DE'

Resetting my Tor Browser identity/session fixed it so I don't think it's a problem in practice for production installs, but flagging it as it seems to reveal another place not using/checking USABLE_LOCALES first.

legoktm

dev test plan passed, minus the one stacktrace I posted. I didn't go onto the staging part yet, will do that in the next round of review.

securedrop/i18n.py

securedrop/tests/test_i18n.py

securedrop/i18n.py

cfm · 2022-05-02T20:24:06Z

Thanks, @legoktm! I'll try to turn this around this afternoon (Pacific) in the hope of getting it under the feature-freeze wire for v2.4.0.

A locale is considered usable if it is both (a) available in the filesystem and (b) configured by the administrator in SDConfig.SUPPORTED_LANGUAGES. Once we've determined which configured locales are actually usable, we: 1. warn if a configured locale is not available; 2. fall back to the hard-coded FALLBACK_LOCALE ("en_US") if SDConfig.DEFAULT_LOCALE is not usable; and 3. error out if neither the default nor the fallback locale is usable.

We can't iterate directly over USABLE_LOCALES: Set(Locale) because Locales aren't sortable. Instead, we parse config.SUPPORTED_LOCALES in order and skip those that aren't in USABLE_LOCALES.

… to disambiguate SUPPORTED_LOCALES thread: #6406 (comment)

This is a safe assumption except in the corner case where (e.g.): 1. A user begins a session and selects language L. 2. The Application Server upgrades to a version of securedrop-app-code that removes support for language L. 3. The user resumes their session, which still has language L selected. But, since this case has come up in testing[1], best to check for it. [1]: #6406 (comment)

As of 9c11950, get_locale() handled user/agent-provided locale preferences and deferred to resolve_fallback_locale() if none could be satisfied. Here we build a single list of locales, beginning with user/agent preferences and ending with the server-side {DEFAULT_FALLBACK}_LOCALEs, and call babel.core.negotiate_locale() once to negotiate one (if possible).

cfm · 2022-05-03T01:18:21Z

@legoktm in #6406 (comment):

Still walking through the test plan (more feedback to come shortly...), but when I switched my locale to de_DE, deleted it and restarted the dev container, I got a stack trace of:

[...]

Resetting my Tor Browser identity/session fixed it so I don't think it's a problem in practice for production installs, but flagging it as it seems to reveal another place not using/checking USABLE_LOCALES first.

Thanks for catching this failure-mode. I'm not able to reproduce it, but I've added what I think is the necessary defense-in-depth check in eb2a042. If you're still able to reproduce with this change, and think it worth addressing, could you specify a more-detailed reproduction in terms of the "delete de_DE" and "restart container" steps, to make sure I'm testing what you're seeing?

This change prompted the further refactoring proposed in eb2a042, to consolidate the locale-negotiation logic in get_locale(). Let me know what you think, or feel free to rebase out for the sake of an expedient review and merge for v2.4.0.

legoktm · 2022-05-03T01:22:15Z

Using your latest set of changes I can no longer reproduce it, so I assume it's fixed. Going through the rest of the test plan now...

legoktm

Tested on both dev and staging, LGTM!

After freedomofpress/securedrop#6557, in lieu of a Venn diagram. I regret that I did not manage to use identical terminology between freedomofpress/securedrop#6406 and freedomofpress/securedrop#6557.

cfm added feature i18n Anything related to translation or internationalization of SecureDrop labels Apr 19, 2022

cfm changed the title ~~feat: validate SDConfig.SUPPORTED_LANGUAGES for *usable* locales~~ feat: validate SDConfig.SUPPORTED_LANGUAGES for usable locales Apr 19, 2022

cfm force-pushed the 6366-check-usable-languages branch from 84ac53d to 925fefb Compare April 19, 2022 22:09

cfm marked this pull request as ready for review April 19, 2022 23:30

cfm requested a review from a team as a code owner April 19, 2022 23:30

legoktm self-assigned this Apr 26, 2022

legoktm requested changes May 2, 2022

View reviewed changes

cfm self-assigned this May 2, 2022

cfm mentioned this pull request May 2, 2022

runtime i18n configuration manipulates global state #6420

Closed

cfm added 2 commits May 2, 2022 16:45

feat: map_locale_display_names(): skip (hide) unusable locales

20c59fd

We can't iterate directly over USABLE_LOCALES: Set(Locale) because Locales aren't sortable. Instead, we parse config.SUPPORTED_LOCALES in order and skip those that aren't in USABLE_LOCALES.

cfm added a commit that referenced this pull request May 2, 2022

refactor(map_locale_display_names): use set rather than looping twice…

bc5e366

… to disambiguate SUPPORTED_LOCALES thread: #6406 (comment)

cfm force-pushed the 6366-check-usable-languages branch from 34eaea2 to bc5e366 Compare May 2, 2022 23:59

refactor(map_locale_display_names): use set rather than looping twice…

9621639

… to disambiguate SUPPORTED_LOCALES thread: #6406 (comment)

cfm force-pushed the 6366-check-usable-languages branch from bc5e366 to 9621639 Compare May 3, 2022 00:03

cfm added 2 commits May 2, 2022 17:45

cfm removed their assignment May 3, 2022

cfm requested a review from legoktm May 3, 2022 01:19

legoktm approved these changes May 3, 2022

View reviewed changes

legoktm merged commit 0d8138b into develop May 3, 2022

legoktm deleted the 6366-check-usable-languages branch May 3, 2022 03:45

zenmonkeykstop mentioned this pull request May 10, 2022

Release SecureDrop 2.4.0 #6440

Closed

35 tasks

cfm mentioned this pull request Jun 3, 2022

map_locale_display_names() does not set use_display_name for the first of multiple locales for a language #6475

Closed

cfm mentioned this pull request Jan 5, 2023

enable automatically-generated "pseudolocale" #6690

Closed

7 tasks

cfm mentioned this pull request Jun 22, 2023

eo locale is overloaded between tests and pseudolocale generation #6873

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: validate `SDConfig.SUPPORTED_LANGUAGES` for usable locales #6406

feat: validate `SDConfig.SUPPORTED_LANGUAGES` for usable locales #6406

cfm commented Apr 19, 2022 •

edited by legoktm

Loading

codecov-commenter commented Apr 19, 2022 •

edited

Loading

legoktm commented May 2, 2022

legoktm left a comment

cfm commented May 2, 2022

cfm commented May 3, 2022

legoktm commented May 3, 2022

legoktm left a comment

feat: validate SDConfig.SUPPORTED_LANGUAGES for usable locales #6406

feat: validate SDConfig.SUPPORTED_LANGUAGES for usable locales #6406

Conversation

cfm commented Apr 19, 2022 • edited by legoktm Loading

Status

Description of Changes

Testing

Routine testing

Configured language loses support

Default language loses support

No usable fallback locale

OSSEC alerts

Deployment

Checklist

If you made changes to the server application code:

If you made non-trivial code changes:

codecov-commenter commented Apr 19, 2022 • edited Loading

Codecov Report

legoktm commented May 2, 2022

legoktm left a comment

Choose a reason for hiding this comment

cfm commented May 2, 2022

cfm commented May 3, 2022

legoktm commented May 3, 2022

legoktm left a comment

Choose a reason for hiding this comment

feat: validate `SDConfig.SUPPORTED_LANGUAGES` for usable locales #6406

feat: validate `SDConfig.SUPPORTED_LANGUAGES` for usable locales #6406

cfm commented Apr 19, 2022 •

edited by legoktm

Loading

codecov-commenter commented Apr 19, 2022 •

edited

Loading