Meta: Translation Project for IPFS GUIs

[lidel]

Current State

It's alive! https://github.com/ipfs/i18n

• Thank you all for help and smoke-testing the setup! ❤️
• Crowdsourcing translations from IPFS Community
• Everyone can join using github account!
• Missing language? Join and request it via Transifex
• I moved docs and reading material to a dedicated project at https://github.com/ipfs/i18n ✨
  • Feel free to improve it with best practices and resources useful for others

Dear future reader:

If you have trouble with i18n framework, translation site, locale format or want to add project to Transifex please fill an issue here.

---

(Original issue below)

Background

Click to expand

In the past it was a mixed bag: different services, missing locales, no common corpus/dictionary to support crowdsourcing efforts.

• WebUI
  • Old WebUI and the ipfs.io website: https://www.transifex.com/ipfs/public/
    • created by a community member, but @lidel now has admin rights as well
  • New WebU (translation in progress: feat: i18n ipfs-webui#772 done)
• Companion (i18n): https://crowdin.com/project/ipfs-companion
  • right now under @lidel's private account, needs to be migrated
    done: Add Transifex for Crowdsourcing Translations ipfs-companion#566
• Desktop has no translation set up yet
• Other things with translations
  • http://js.ipfs.io
  • In future CLI interfaces of go-ipfs and other shell tools could add support for $LC_ALL

Milestones

Manual

• all GUI apps support translation and ship with English as plain text locale files
  this is very important:
  • establishes convention and people can PR with new locales via github
  • enables us to experiment with different crowdsourcing websites in async mode

Semi-Automated

• translations can be provided via a crowdsourcing website: https://www.transifex.com/ipfs/public/
• there is a single crowdsourcing website that user can join to start translating
  • we have multilingual dictionary for core concepts and most commonly used phrases ([NEW CONTENT] [BOUNTY] Glossary ipfs-inactive/docs#56)
  • when everything is set up we can reach out to https://discuss.ipfs.io for help in translating everything in bulk
• maintainer is responsible for synchronizing between github and crowdsource website
  • push source locale (Engligh) a week before release and/or on big changes
  • fetch translations on release day
  • (this does not scale well if we have multiple projects)
• licensing is solved (Copyright Hygiene  community#331 (comment))

Full Automation

• there is some kind of automation that removes friction
  • synchronizes new source strings from github (master) to crowdsourcing site
    • done: Meta: Translation Project for IPFS GUIs #50 (comment)
  • synchronizes new translations from crowdsourcing site to github
    • eg. making a PR once a day or by commiting to locales branch on daily basis to make sync as easy as merge to master

References

Transifex

• Installing the Transifex Client
• Understanding .tx/config file
• Manual sync via Transifex Client
  • Using Transifex with GitHub in Your Development Workflow
    • Syncing a local project to Transifex with the Transifex Client
    • Integrating the Client with CI (Travis CI, but same will work with Jenkins)
• Sync automation
  • Option A: Automatically updating source files 👌 👌 (Meta: Translation Project for IPFS GUIs #50 (comment))

    Transifex will check [Github] URL for updates twice per day and if there are any changes to the file

  • Option B: set up Jenkins: Localizing with the Transifex Client
  • Option C: Setting up an instance of Txgh (third-party sync server between Transifex and GitHub)
• Neat features:
  • Translation Memory:

    Without going into too much technical detail, Translation Memory is a database of previously completed translations. Each time a translation is submitted, it's saved to this database. In the future, if a phrase similar to the original source string appears, the translation is presented to the translator as a suggestion. Let's look at an example.

  • Sharing Translation Memory across all projects (paid feature)
  • Crowdsourcing Mode (paid)
• Bonus: eye savers for night owls 🦉
  • Dark Theme for OLED
  • Geeko Dark Theme

---

Click to expand notes for CrowdIn (alternative to Transifex)

CrowdIn

• Manual sync via CLI Tool
  • Console Client (CLI)
• Bonus perk: Localization That Does Not Delay the Releases
• Configuration File: crowdin.yaml
• Sync automation
  • Option A: Set up Jenkins with the CLI Client
  • Option B: Native GitHub Integration:

    Crowdin’s integration with GitHub makes source and translation files synchronized with GitHub repository and Crowdin translation project. All translated and approved files will be automatically pushed as a pull request to the l10n branch in GitHub repository.

    We do not commit directly to master, so that you have a chance to verify translations and keep your app safe and sound. By default, l10n_ is added to the created service branch name. In case of necessity it can be easily changed.

    • Good: looks easier to set up than Transifex
    • Bad: requires granting write access to repositories (we already use things like Waffle, but still...)

[lidel]

• Added References section with some notes about CLI for two most popular crowdsourcing sites and supported ways of integrating with Github.
• Thanks to @digital-dreamer we got Admin ownership over IPFS org at Transifex :)

[lidel]

• At CrowdIn every project has its own team and you need to manually assign people as managers/translators to each project.

• That is why I am leaning towards using Transifex: it enables us to have all apps under a single IPFS org (https://www.transifex.com/ipfs/public/) and one team of translators. We can share translation memory and glossary across all projects, and when a new one is added it gets the translation history for free.

[lidel]

Reposting some notes on i18next-react vs react-intl from ipfs/ipfs-webui#772 (review):

I was unable to find native support for i18next-react's translation format at Transifex (which I am leaning towards, here is why). People were successful with using Transifex+i18next-gettext-converter which has a support for plurals for most of languages, but there are edge cases with known issues which may produce frustrating bugs. I feel we could try to blackbox the sync by creating 1 shell script that takes care of conversion and runs at jenkins + is run manually before release, but this need for additional layer of indirection if a sort-of a warning flag.
Update: i18next supports ICU after all!

The alternative, react-intl is a part of FormatJS which is aligned with ECMAScript Internationalization API (ECMA-402), Unicode CLDR, and ICU Message syntax. ICU does not require additional conversion step and is supported out of the box by Transifex and other crowdsourcing websites. It kinda feels more future-proof, when looked at from that angle [..]

If anyone has any prior experience with either of them, feedback with pros/cons would be appreciated!

[olizilla]

Of note, i just saw https://github.com/lingui/js-lingui which may be the best of both react-intl and i18next in terms of dev experience and translation support... It uses ICU message format like react-intl under the hood, but provides a nicer high-level api for creating them. I'm wary of posting links to twitter popularity contests, but this https://twitter.com/kentcdodds/status/1036084711678861312 at surfaced that react-intl and i18next are popular now, and lingui is interesting.

[hacdias]

Also about react-intl: formatjs/formatjs#1160

[lidel]

Those are good data points. js-lingui is really lightweight and optimized + actively maintained. On the other hand, @hacdias noted there is a way to use i18next with ICU format, so if we happen to have a decision paralysis, we can always go with i18next-icu (uses yahoo/intl-messageformat).

So far I don't have any strong opinion on this, my only ask is to use framework-agnostic format, so we are not bound to a specific framework and can switch in the future.

So far I see two options:

• ICU Format
• gettext (.po files)

As long we store translations in either of them, we will be future-proof.

[lidel]

Update: I've managed to setup Transifex for IPFS Companion and new WebUI, for now it is a manual sync but web interface works and everyone can create Transifex account (or login with github one) and play with translating some strings.

See links to PRs below for more info.

• IPFS Companion
  https://www.transifex.com/ipfs/ipfs-companion/ | PR: Add Transifex for Crowdsourcing Translations ipfs-companion#566
• New WebUI (revamp)
  https://www.transifex.com/ipfs/ipfs-webui/ | PR: feat: i18n ipfs-webui#772 (comment)

Let me know if there are any issues with registration / joining the project there.

[olizilla]

thanks to @lidel and the transifex service Today I Learned about the trade-offs around using logical keys in 1i8n systems Vs. using en phrases as default keys...

logical keys are useful to pinpoint the location in the app where a string appears... this works well with translation services, as it means we don't lose the connection between translated values when we change an english value... the key is stable, and remains even if the value changes.

The translated values may now be out of date, but they are still connected to same logical location in the app. This has the downside of us having to generate pseudo-logical key names for things, and it means the same phrase could appear duplicated under many different keys, and need translating each time... which feels like it would slowly increase the workload over time

but

The translation service manages that by suggesting translations from other sections of your project where the english (or whatever the configured source language is) value matches this new one. So where you have different logical keys / namespaced keys, pointing to the same 'en' text, the translation service can automate the translation based on the values of existing keys.

From previous locaization experience I wanted english lang keys to

1. avoid having to make up "logical" keys
2. wrangle the phrase to a common set of strings that we re-use all over the place, and slowly reduce the translation burden over time, as all possible ways of saying "put the files in IPFS" get translated and reused

Point 1. is defeated by the value of having a stable key when existing translations change, and you need a way to keep the connection between a phrase, its usage site, and the previous other translations of the phrase that used to go there
2. would be nice, but we get that by auto-suggest magic in the translation service

so... hurrah for logical keys and translations services.

[lidel]

Ok, I took CI as far as I could:

• Transifex tx CLI Client in Docker: https://github.com/lidel/ci-transifex
  • Useful even on desktop, eg. if you don't want to install python/virtualenv/pip etc
• [WIP] PoC Transifex automation via Jenkins CI: [WIP] Transifex automation ipfs-inactive/jenkins-libs#28 nah, see Meta: Translation Project for IPFS GUIs #50 (comment) 🙃
  • Blocked by Jenkins supports having secret variables, for code signing and auth tokens ipfs-inactive/dev-team-enablement#18

Next:

• Close TODOs from Add Transifex for Crowdsourcing Translations ipfs-companion#566
• Make Transifex official, update community and project docs

[lidel]

Got some valuable feedback from Chinese Translators at Transifex :

• If we use zh everywhere, the operating system will handle the details. (follow the operation defaults)

• If we want to precise control of each language code, should use the standard writing as much as possible：zh_CN/HK/TW

• The thing is is that on the crowdsourcing side we don't want to put everything under zh:

  Basically, if you have both zh_CN and zh_TW, you should not move any one to 'zh', because both of them have a considerable amount of users and represent Chinese, any one of them representing the whole of the 'zh' will bring controversy.

  There are rarely Chinese native speakers who use the simple 'zh'.
  https://en.wikipedia.org/wiki/Traditional_Chinese_characters#Modern_usage_in_Chinese-speaking_areas

TL;DR we will use zh_CN (and will add zh_TW and other ones when requested) and there will be no zh

[lidel]

Not sure how I missed this before 🤦‍♂️
Transifex can be set to automatically check arbitrary URL for updates:

https://docs.transifex.com/projects/updating-content/#section-automatic-updates:

Automatically updating source files

Manually updating source files isn't fun or scalable if you've got frequent updates. To avoid this, you can have Transifex automatically check for updates to your source file. You simply need to provide Transifex with the public URL of the file. The file can be hosted on any service, such as GitHub or Dropbox.

Transifex will check that URL for updates twice per day and if there are any changes to the file, it'll fetch the file and update the resource in Transifex. Because the check only happens twice a day, there may be a delay between when you update a file and when Transifex detects the changes. The updated resource will automatically be available to your translators and reviewers.

This means we can point it at a raw resource at GitHub and don't need CI for pushing new strings ✨

I've just enabled this for IPFS Companion using this URL:

• https://github.com/ipfs-shipyard/ipfs-companion/raw/master/add-on/_locales/en/messages.json

[lidel]

A day later I am able to confirm it indeed updates automatically. I've enabled it for other projects as well.
This is a super cool thing to have 🚀

@olizilla @hacdias some additional notes regarding WebUI:

• If a new translation namespace is added to WebUI in future, we need to remember to set update URL for it.
• We need to switch update URL for WebUI after we rename revamp to master, right now it updates using values from revamp branch:

  

[lidel]

@mikeal asked if there a thread somewhere on starting the translation community and.. I had no doc to point at :)

I wrote a short overview in ProtoSchool/protoschool.github.io#39, but in the long term we should have a linkable resource with protocol for adding projects to Transifex and some cheat-sheet documenting our use of i18next and JSON-ICU.

Question: where should we keep it?

• Option A: https://github.com/ipfs/ipfs-gui/i18n/ or https://github.com/ipfs/community/i18n/ will probably be lost like tears in the rain and hard to discover
• Option B: https://github.com/ipfs/i18n/ or https://github.com/ipfs-shipyard/i18n/ – personally I'd go with one of these. The added value of a dedicated repo would be 1) discoverability 2) use of Github Issues to manage things like adding project to Transifex, asking questions etc.

Is it ok to create i18n repo? If so, in which org?

[mikeal]

+1 for Option B :)

[swedneck]

Here's a geeko dark theme for transifex.

[lidel]

I moved docs and reading material to a dedicated project.
Thank you all for help and smoke-testing the setup! ❤️

https://github.com/ipfs/i18n ✨

Dear future reader:

If you have trouble with i18n framework, translation site, locale format or want to add project to Transifex please fill an issue here.