feat: Add full i18n support

[jpmckinney]

I don't understand how packaging works now, but babel.cfg was previously included in MANIFEST.in.

It's only needed in development, so maybe no action is needed.

Fix #257

[jpmckinney]

@choldgraf Ready for review

[choldgraf]

This looks good to me - in my opinion we should try to merge it in relatively quickly, since it is a nice iterative improvement (and note we have docs about this setup at the PR below)

• DOCS: Add internationalization instructions #1178

Three quick questions but I don't think any of them should block merge.

• Could we .gitignore the .mo files and then generate them at package build times? Since those are programmatic ones, we could not check them in to our project, and put a build for them in our "build the documentation" contributor docs. That way the only files in the repository are the ones that humans edit.
• If we do small things like change our HTML template structure, what do we do?. I noticed the docs mention that if you update the text you need to re-generate the .po files for each language. But those files also have hand-edited translations in them as well as specific line numbers. Are those updated automatically when you re-run pybabel?
• Could we add jobs to our noxfile for the generation steps? I feel like that would be a nice way to make this easier for folks: https://pydata-sphinx-theme--1178.org.readthedocs.build/en/1178/community/i18n.html#changing-natural-language-text
• Should we provide guidance for whether to include non-translateable text in translations? Given that it is common to have non-translateable words (like brand words such as GitHub) in elements, it might be helpful to explicitly note how to do this. For example you seem to use something like %(provider) around here but I don't know what this means or how to use it.

cc also @12rambau who I believe has also done a bit of translation work in Sphinx.

[12rambau]

Could we .gitignore the .mo files and then generate them at package build times?

I actually don't know pybabel very well so I don't know how it generate the .po file but we should not keep the .mo in the github for sure.

If we do small things like change our HTML template structure, what do we do?

The best way to proceed is to use an external tool to make the translation (I'm a big fan of crowdinbut Transifexalso has an open-source support). That way the translation platform keep tracks of what you have already translated and automatically update the other locales. here is an example of something I'm working on: https://crowdin.com/project/sepal-doc

Could we add jobs to our noxfile for the generation steps?

that would be neat. Even better would be a pre-commit hook.

Should we provide guidance for whether to include non-translateable text in translations

You can use parameters in the translated messages (i.e. "display (%number) image" ) and add rules for plurals. If you think it's needed then yes documentation would be welcome (I think a link to the .po formatting would be sufficient)

[jpmckinney]

The MO files must be in the package for translation to work. Right now they will be - if you later write a way to build them at the packaging step, even better. But, as is, there is no harm in having them in version control.

If you change non-translatable text, you can run pybabel and it will update the line numbers. However, this step is optional - the line numbers are to inform the human translator, not to perform the translation.

If you change translatable text, pybabel will usually try to fuzzy match the new text with an existing translation. If so it’ll add a comment like #, fuzzy. Otherwise, the new text will simply be untranslated in the file.

I’ve used Transifex, but for a small project with a few dozen strings, it’s fine to just hand-edit the file. Up to the maintainers.

When you do some Jinja like {% trans %}Hello {{ name }}{% endtrans %}, it’s extracted as the plain Python format string Hello %(name)s, such that it can be used in both Jinja templates and Python code. The translated text must contain the %(name)s verbatim.

Full sentences and clauses must always be a single translatable string. Otherwise, you get “next page” translated as “suivant page” instead of “page suivante” etc (a real bug before this PR)

[choldgraf]

That is super helpful knowledge to share, thanks @jpmckinney . Any chance you could work that into the i18n documentation? If it's too hard to find the right place, feel free to just create a "Miscellaneous tips" section at the end and just copy/paste your response above into it. I was worried that the line numbers would mean we'd have to hand-edit them every time something shifted by one line.

re: .mo files being checked into git, I'm fine just tracking that in a separate issue if @12rambau doesn't have strong opinions 👍

[jpmckinney]

The docs from the other PR can remove the warning at the top about things not being implemented - otherwise, all good!

[jpmckinney]

Any chance you could work that into the i18n documentation?

Can we create a new issue? I might not get around to it right away, and it’s better to merge the feature. The docs aren’t specific to this project - just helpful tips on how pybabel and gettext work.

[choldgraf]

OK I decided to give a quick pass at this, so just pushed a commit with a few updates:

• Our noxfile.py now has a command for translate, which supports an extra verb for each of the commands we describe here. I documented that too.
• I removed the .mo files from git history, and added a script to our webpack configuration to generate the .mo files when the theme is compiled. This ensures that they'll be included any time the theme is built (similar to how we treat other static assets). This also means adding pybabel as a dependency which I think is reasonable?
• I also added sudo apt-get install gettext to our publish job, because I found in the book theme that for some reason this wasn't installed normally and this bit us in the book theme.
• I added to the documentation from @jpmckinney 's information above.

I'd suggest that @12rambau takes a quick pass, and then we cut a release of the theme, then merge this quickly so that we have time to play around with it.

[choldgraf]

I'm not sure why the macos and the windows builds aren't working. I suspect that this is because of a dependency that isn't installed somewhere, or a path issue. But I can't figure out what it could be. @12rambau any ideas?

Given that we always build the package for this theme in linux, we could skip the translation test on windows + macosx so that this doesn't become a nagging problem...wdyt?

[jpmckinney]

Docs fixes.

[jpmckinney]

The windows and macos tests were passing in my last commit - not sure why they are failing now (other than the MO files not being committed).

[drammock]

current (6.1.x) sphinx has this:

[jinja2: **/themes/**.html]
encoding = utf-8
ignore_tags = script,style
include_attrs = alt title summary

IDK though which is correct for us?

[choldgraf]

I don't have strong opinions on this, I suggest we just pick one and see if it has any tangible impact, then change it later if we want. IMO unless we have a strong rationale for excluding placeholder, we should just keep it 🤷

[choldgraf]

I swapped the spanish / french sentences that @drammock mentioned. I suspect that was a one-off mistake because it was the same sentence in both languages.

I also accepted all of @jpmckinney's suggested docs edits.

Given that the translation compilation will always happen via Linux for the forseeable future, I suggest that we:

1. Skip the HTML translation test on MacOS and Windows jobs
2. Make a release of this theme
3. Merge this PR and iterate from there

What do others think?

[choldgraf]

I don't have strong opinions on this, I suggest we just pick one and see if it has any tangible impact, then change it later if we want. IMO unless we have a strong rationale for excluding placeholder, we should just keep it 🤷

[jpmckinney]

Skip the HTML translation test on MacOS and Windows jobs

Tests pass now.

[choldgraf]

Nice ! OK so looking back over the code, given that this PR is roughly strictly an enhancement / bugfix of previously broken translation infrastructure, and given that it doesn't change anything in the HTML other than providing translation strings, what do @drammock or @12rambau of getting this in the current release cycle instead of waiting for the next? I'm +1 on merge either way (before or after the next release)

[12rambau]

I'm +1 merging it whenever. that will be an excellent base to work on.

I have one last comment to make though. I realized that we are using nox in the test only for the translation build. My guess is that we avoid it because it make a virtual env within a very controlled env. What about simply adding the code run in nox instead ?

[choldgraf]

To answer part of your question, the only reason I avoid using nox in the GitHub actions tests themselves is because I haven't figured out how to cache the environment properly between builds, but i know that some projects do re-use nox for all their cicd. That way when you run it locally you are almost exactly replicating the environment that is used in the actions. IMO there's nothing inherently wrong with using nox inside of actions (but I'm fine on whatever implementation we want to go with here if others prefer to repeat the code by hand in the action)

[12rambau]

then let's go with that and open a new issues about moving toward nox, I think it would make our lives easier if everything is managed the same way from one file (I think we can take inspiration from here: https://github.com/pypa/packaging/blob/main/.github/workflows/lint.yml)

[drammock]

getting this in the current release cycle

Ok with me.

[choldgraf]

Ok cool - then I'll merge this and a couple minor fix PRs and cut another RC (because why not and people continue to find tiny bugfixes etc so I figure it's a reasonable thing to keep with the pre-releases as long as that's happening 😅)

[12rambau]

Instead of doing rc why don't we use patches (0.13.xx) ? so that everyone relying on "pydata-sphinx-theme" get the latest devs

[choldgraf]

@12rambau you mean just make a full release now? I would be fine with that as well, just felt like an RC is safer if we keep adding new things. I also want to double-check that the translation files are actually in the .whl file, because we've run into issues in the book theme where that didn't happen and had to be debugged. I'll make an RC now but if you'd like to go for a full release, I am +1 on it!

[12rambau]

@jpmckinney I'm asking here because I'm no expert on pybabel and I would need some insight from your side before doing a small refactoring of our actions (related to #1292).

• in the update command, you set up domain to "sphinx", any reason not to keep the default "message" ?
• in the extract function you have added multiple keywords (_, __, l_ and lazy_gettext) what are they used for ? why cannot we rely on the defaults ?

[jpmckinney]

I just copied Sphinx - we can probably set the domain to anything we want: https://github.com/sphinx-doc/sphinx/blob/2329fdef8c20c6c75194f5d842b8f62ebad5c79d/setup.cfg#L22

---

Unless --no-default-keywords is specified, then keywords specified with -k are appended: https://github.com/python-babel/babel/blob/56071c90116e6b9ebbb62ac072bcf032fc6987cb/babel/messages/frontend.py#L415-L417

The default keywords are here: https://github.com/python-babel/babel/blob/56071c90116e6b9ebbb62ac072bcf032fc6987cb/babel/messages/extract.py#L77-L88

pydata-sphinx-theme doesn't use __, l_ or lazy_gettext, and _ is already a default keyword, so we can remove this configuration.

I had simply copied Sphinx's configuration: https://github.com/sphinx-doc/sphinx/blob/2329fdef8c20c6c75194f5d842b8f62ebad5c79d/setup.cfg#L18

[12rambau]

perfect, thanks a lot.

I think I'll move it toward default behaviour to reduce the complexity of the command call