Switch to using Weblate

[m-aciek]

Transifex:

• doesn't support translation memory download/upload between projects in the open-source plan since October 2022
• doesn't support machine translation as suggestions (Consider setting up machine translation #18),

One of options is to switch to Weblate (which supports both machine translations as suggestions and global translation memory propagation). Weblate has very good version control system integration – this switch would also solve #14 plus would remove the necessity of maintaining scripts to fetch the translations.

This issue is to track blockers for migrating the translation project to Weblate.

• updating translation source strings along with English original documentation changes
  • to be implemented by ourselves on repository side,
• Allow per-language Git repositories WeblateOrg/weblate#7271 – one target repository per language is not supported – validation for file mask in the target repository requires a placeholder with a language code,
• translation propagation between different Python versions
  as we would set up a project per Python stable version, we'd like to have working translation propagation between the projects
  • confirm Autotranslate add-on solves it
• Nested projects or project grouping WeblateOrg/weblate#263

[comradekingu]

Happy to help

[m-aciek]

I think the action point here for now is to create a feature request in Weblate for allowing the target repository to not have a language code in the filename pattern (second point from the list of blockers in the issue description). Point first and three should be possible to implement as GitHub Actions scripts.

[m-aciek]

I've added proof-of-concept repository and successfully pushed the first change: https://github.com/m-aciek/python-docs-pl-weblate-poc (https://hosted.weblate.org/projects/python-docs/). I've updated original issue with refined points that still require an effort. I think of scripts being fired with GitHub Actions for both updating source strings and translations propagation.

[m-aciek]

Translation propagation between components will be handled by Autotranslate add-on.

Discovery for adding or removing components will be handled by Discovery add-on.

[m-aciek]

According to this discussion setting up Autotranslate between projects (Python versions) is also possible.

[comradekingu]

@m-aciek
Population between components "spreading of translations" can be set up in https://hosted.weblate.org/settings/python-docs/bugs//#translation if that is the component the others are mirrored from. (Or individually per component where desirable.)

There is also the shared translation memory between other projects on Hosted Weblate, which can be turned on in
https://hosted.weblate.org/settings/python-docs/#workflow

---

Don't use CC0, it is not advisable. It is also not the license used for the documentation already.

This file is distributed under the same license as the Python package.

from https://github.com/python/python-docs-pl/blob/3.10/copyright.po

https://github.com/python/cpython/blob/main/LICENSE#L65

---

If you add "kingu" in https://hosted.weblate.org/access/remmina/#users I can see if I find other improvements to be made. :)

[m-aciek]

Population between components "spreading of translations" can be set up in https://hosted.weblate.org/settings/python-docs/bugs//#translation if that is the component the others are mirrored from. (Or individually per component where desirable.)

I already found and did that 👍 .

@comradekingu do you know if we add a separate project for Python 3.9 documentation translation, will we be able to set automatic propagation from Python 3.10 project? (Not only having a translation memory as a suggestions.) I wonder if this feature is visible when you have admin privileges to two projects (WeblateOrg/weblate#7200 seems to suggest that).

Don't use CC0, it is not advisable. It is also not the license used for the documentation already.

From what I understand the license of translations may be a bit different than the license of the original documentation. The PEP 545 suggests to use CC0. If needed we can/should discuss it further on Python documentation Discourse section.

I can see if I find other improvements to be made. :)

Sure, I've added you as an administrator.

[comradekingu]

The setup worked just fine, but the grace period timed out.
I meant to do this, but the date I had in my head was the expiry.
I suggest putting it up again and then requesting approval.

@m-aciek
Yes, it is possible to turn on sharing between components.
CC0 is a disaster. For collaborative works it does nothing to further move it towards the public domain in any usable sense, but it incurs every bit of legal trouble from doing it effectively part-way for different and incompatible jurisdictions.
Just use the license the software itself has.

[m-aciek]

I suggest putting it up again and then requesting approval.

According to this comment by nijel, our request would be put down, at least until WeblateOrg/weblate#7271 is done.

CC0 is a disaster. For collaborative works it does nothing to further move it towards the public domain in any usable sense, but it incurs every bit of legal trouble from doing it effectively part-way for different and incompatible jurisdictions.

Do you maybe have some external resource with the critique of CC0 in a context of similar works like this translation? I'd be keen to get to know more.

For what it's worth on main branch of this repository I started trying to put together a GitHub Action to update translations source strings.

[comradekingu]

@m-aciek

https://rd-alliance.org/sites/default/files/cc0-analysis-kreuzer.pdf
https://www.ivir.nl/publicaties/download/KnowRight08_Paper.pdf
https://halshs.archives-ouvertes.fr/halshs-00671622/document
https://wiki.osmfoundation.org/w/index.php?title=Licence/Licence_Compatibility&mobileaction=toggle_view_desktop#CC0

It always stops at the "what did you think you were doing" stage, and it is never popular, with nobody using it for collaborative works at scale, or with success at a smaller scale.

I couldn't find the one that hits the nail on the head, but basically moral rights can't be waived in some jurisdictions,
and the terms for actual public domain, which is both possible and not, differ.
You thereby get inalienable copyright, mixed in with a lot of non-endgame. The winning move is not to play, even though it somewhere down the road could make sense. It seeks no protections, and finds itself challenged by the protections it could have used. It is buying into the results of downfall of the commons as if it prevents it somehow.
My opinion is that copyright protection is better, even if it were possible.
TL;DR Collaborative works thereby get every bit of legal pain incurred in the differences, with no functional upside as a result.

https://www.gnu.org/licenses/license-recommendations.html is a particularly bad recommendation, and they also championing waiving copyright for translations.
In my opinion a better CLA system for maintainership is warranted, and it should be in the license.
Something along the line of it will be like this, but possibly have more protections/restrictions to that end.
That is another in-process discussion.

[m-aciek]

Another blocker (?) on Weblate side: WeblateOrg/weblate#263.

[rffontenelle]

How about presenting to all translation teams a migration to a single workflow using Weblate? If accepted by teams and by Python core devs, this would eliminate per-team customized scripts and would land translations directly in the main repository (or another suffixed with e.g. "-doc-translations" of their choice).

For PyPA (PyPI website/warehouse and the packaging guide) it is working fine.

One thing I know is that French teams gives great importance to FLOSS, so the Hosted Weblate linked with several machine translations might be a problem. So self-hosted weblate could be a solution. I wanted to present a proof-of-concept of a Weblate instance for python docs translations using Yunohost, but never went further.

[m-aciek]

How about presenting to all translation teams a migration to a single workflow using Weblate? If accepted by teams and by Python core devs, this would eliminate per-team customized scripts and would land translations directly in the main repository (or another suffixed with e.g. "-doc-translations" of their choice).

That would require an update in PEP 545, which now includes per-language repositories.

One thing I know is that French teams gives great importance to FLOSS, so the Hosted Weblate linked with several machine translations might be a problem. So self-hosted weblate could be a solution.

Would French team be keen to enable machine translations on self-hosted Weblate? In my opinion ML suggestions are a must-have in a modern translations workflow. On the other hand even shared repository could be used through Weblate only as opt-in (I mean PR flow without a service could also work there).

I started looking into Weblate core, to see how much of a hassle would be to allow multiple single-language Git repositories for a project. It's quite a big change (to the core models), but according to Weblate creator it would be accepted.

[rffontenelle]

That would require an update in PEP 545, which now includes per-language repositories.

Using a single workflow would probably mean eliminating the per-language repository, indeed. So if not all teams agree, that indeed would be a blocker. But I think it worth the shot to simplify and centralize the translation maintenance.

Would French team be keen to enable machine translations on self-hosted Weblate? In my opinion ML suggestions are a must-have in a modern translations workflow.

I don't know, but I assume they are not. I'm saying this because French moved from GitHub (now a mirror) to AFPy Gitea-based repo, and something else I don't recall right now. If I'm wrong, great, Hosted Weblate is a go.

Fedora migrated to Weblate and because they give great importance to FLOSS, they opted to self-host Weblate and to have machine translation disabled (i.e. not linked to Google, Microsoft, etc.).

I started looking into Weblate core, to see how much of a hassle would be to allow multiple single-language Git repositories for a project. It's quite a big change (to the core models), but according to Weblate creator it would be WeblateOrg/weblate#7271 (comment).

If at least we can make this work, I'd love to migrate to Weblate. Count me in.

[m-aciek]

I realized we could use Weblate "as is" by creating a repo like "python-docs-weblate" to use it as a "Weblate backend" and fetch the translations to the current repositories e.g. using weblate CLI or Python API. It would allow us to use translation memory (through global translation memory on Hosted Weblate). We would need to create separate Weblate projects for different Python versions with separate users management (until WeblateOrg/weblate#263 is done). We would preserve current authorship tracking style in the languages repositories (PO header with translators list).

I've created the repo and a project on Hosted Weblate for the POC (again, but with different approach this time). I will try to prepare the "client's" scripts (for the "mirrors") in the python-docs-pl-weblate-poc repository soon.

(By the way it looks like we have the addons and in particular autodiscovery addon blocked in the Weblate project right now, but I assume it's temporarily.)

According to Weblate's policy for open-source we'd need to eventually move python-docs-weblate into Python organization on GitHub to have the project approved on Weblate. (The trial is for 14 days.)

[rffontenelle]

According to Weblate's policy for open-source we'd need to eventually move python-docs-weblate into Python organization on GitHub to have the project approved on Weblate. (The trial is for 14 days.)

I think we might not get accepted in libre hosting because its limits are the same as Advanced Plan: 10k source strings and we have 24K+. See table and libre hosting right below the table in https://weblate.org/hosting/

[rffontenelle]

@m-aciek How did you create the components for your language? I'd rather not do it one by one. :D

[m-aciek]

I think we might not get accepted in libre hosting because its limits are the same as Advanced Plan: 10k source strings and we have 24K+. See table and libre hosting right below the table in https://weblate.org/hosting/

Ah, I see that there's that "libre has the same limits as advanced plan". Hm, we might need to reach out directly to Michal and ask for dedicated plan. 🤔

How did you create the components for your language? I'd rather not do it one by one. :D

I have put all PO files into the "backend" repository in the current state from the language repo. Weblate should just recognize a new language for component being added and display it inside the existing components. We also have autodiscovery plugin enabled, so in case of a new component it should be created in Weblate automatically.

[rffontenelle]

Ah, I see that there's that "libre has the same limits as advanced plan". Hm, we might need to reach out directly to Michal and ask for dedicated plan.

Agreed. Considering that this is not a high demanding project (lots os change, lots of translators working at the same time, I assume this will not result in a high resource consumption of Hosted Weblate resources. So worth the shot.

One thing I had in mind would be to ask PSF sysadmins to host a Weblate instance. Maybe this would allow as to improve the resources organization in different projects (not sure how and if it would help anyhow, but just saying).

[rffontenelle]

Nice, as I said about the string limit, Billing Overview says "Used 56,499 of 10,000"

[comradekingu]

Please avoid CC0, it generally does nothing, and when it does it is incompatible across jurisdictions.
It incurs all the problems of partial public domain, for none of the benefits of either full public domain, or keeping with only copyright.

Just use what the source is licensed as, and drop the CLA. Keep it snimple.

[rffontenelle]

@comradekingu Actually, CC0 and the CLA are requirements from PEP 545: https://peps.python.org/pep-0545/#setup-the-documentation-contribution-agreement

[comradekingu]

That wouldn't be any different from the core problem.
However, it says

A permissive software license (like Apache or MIT) would also get the job done

and then

although it is not quite fit for task.

which is a tiny problem that hasn't produced an actual problems for the other projects yet.
Meanwhile the CC0 isn't just not tested in court, it isn't tested in the myriad of implications between courts, at all.
That is an absolute joke, from a Creative Commons that hasn't ever done anything well.
It just proliferates the licensing ecosystem, for no upside and tremendous downside.

The link doesn't say "license" agreement.
When it is in a README, it is just information (for most jurisdictions).
The CLA posted on Weblate is redundant to 6.4 of https://hosted.weblate.org/legal/terms/

If it is OK enough to just put it in a README, making it so that everyone has to actively click through it is a hurdle.

[rffontenelle]

Please check the context of the quoted string. PEPs normally have some rationale and proposed changes before they conclude with specific commands. By "commands" I mean the contents in the section New Translation Procedure section in PEP545. It looks like Apache and MIT were considered, but they ended making CC0 mandatory with that CLA text.

https://devguide.python.org/documentation/translating/ confirms this in the list item "Each translation is under CC0 and marked as such in the README (as in the cookiecutter)."

[m-aciek]

One thing I had in mind would be to ask PSF sysadmins to host a Weblate instance. Maybe this would allow as to improve the resources organization in different projects (not sure how and if it would help anyhow, but just saying).

@rffontenelle Maybe Docs WG monthly meeting would be a good place to start this conversation?

Please avoid CC0, it generally does nothing, and when it does it is incompatible across jurisdictions.

@comradekingu Would you care to create a thread on Python forum about modifying the license part of PEP 545? I think that would be the way to go with it further.

The CLA posted on Weblate is redundant to 6.4 of https://hosted.weblate.org/legal/terms/
If it is OK enough to just put it in a README, making it so that everyone has to actively click through it is a hurdle.

I've cleared the "Contributor agreement" field in components. Edit: clearing manually base component didn't propagate to others, I need to do it programatically yet. I agree the flow of accepting the CLA before contributing to every component would be not optimal UX for translators.

FYI I've requested the hosting for the (first) project through the Weblate billing module, and still waiting for the response.

[m-aciek]

I've got the response from the Weblate representative.

Hello Maciej,

Thank you for your request; we are happy to see the growing interest of Python communities in Weblate. (…)

In this case, I am unable to approve the gratis Libre hosting plan for your project, at least for now.

• The project is too big for the free hosting tier.
• There are only two languages.

Would you mind connecting on call, so we could find a suitable model for Python docs translation hosting? You can reserve it at (…).

• I would like to know about the plans of growth for this project
• We offer a discount for open-source projects.

I'll try to move the discussion about the topic to Python forum for the broader audience.