Skip to content

Latest commit

 

History

History
421 lines (321 loc) · 18.6 KB

0019-oep-58-atlas-translations-design.rst

File metadata and controls

421 lines (321 loc) · 18.6 KB
Raw

Design for Refactoring Translations pull to use Atlas

Status

Accepted

Context

OEP-58 Translation Management overview

The Translation Management update OEP-58 proposal has been merged with the following changes to the way translations are managed in the Open edX platform:

If you're new to the OEP-58 proposal, please review the OEP-58 Specifications in addition to the Key Metrics and Expected Results section in the Approach Memo and Technical Discovery - Translations Infrastructure Implementation document before continuing.

Pre-OEP-58 Architecture/Implementation for XBlocks and Plugins

Before OEP-58, Open edX XBlocks and Open edX plugins had the following:

  • Translations live in the GitHub repository.
  • Translations are packaged with the rest of the code when published to pypi

Pros:

  • Translations are always available after installation.

Cons:

XBlockI18nService

The XBlockI18nService loads translations for installed XBlocks via its __init__ method. XBlock translations are only used during the during the execution of the XBlock.

The XBlockI18nService implementation pull request (2016) introduced support for XBlock translations in edx-platform and has the full context of the implementation.

JavaScript Translations for XBocks

As of September 2023, there is no centralized method to bundle JavaScript translations in XBlocks. Non-XBlock plugins lack JavaScript translation support altogether.

The de-facto standard method for bundling JavaScript translations in XBlocks is to use web_fragment and load the translations as part of the XBlock frontend static files on every XBlock load.

The LTI Consumer XBlock embeds the translations in its web_fragment via the LtiConsumerXBlock._get_statici18n_js_url and LtiConsumerXBlock.student_view methods.

In order to separate the XBlock translations from the platform, it's isolated in a separate gettext namespace. For example, the Drag and Drop XBlock namespace is DragAndDropI18N which is hardcoded in multiple places such as:

OEP-58 does not change this structure, it just makes the necessary changes to pull translations from the openedx-translations repo via atlas instead of having them live in the XBlock repository itself.

Decisions

Proposed Design for edX Platform conf/locale translations

We're going to use atlas in make pull_translations like we do in course-discovery atlas integration and frontend-app-learning atlas integration.

Proposed Design for XBlocks and Plugins

Instead of storing translation files for each XBlock and Plugin in their respective repositories, we will use openedx-atlas to pull them from the openedx-translations repo.

New pull_xblock_translations commands

Introduce new Django command to the edx-platform:

  • manage.py lms pull_xblock_translations: This command will pull translations for installed XBlocks and Plugins by module name:

    $ atlas pull --expand-glob \
    'translations/*/drag_and_drop_v2/conf/locale:conf/plugins-locale/drag_and_drop_v2' \
    'translations/*/done/conf/locale:conf/plugins-locale/done' \
    'translations/*/edx_proctoring/conf/locale:conf/plugins-locale/edx_proctoring'

    Resulting in the following file tree:

    $ tree conf/plugins-locale/
conf/plugins-locale/
├── done
│   ├── ar
│   │   └── LC_MESSAGES
│   │       └── django.po
│   ├── de
│   │   └── LC_MESSAGES
│   │       └── django.po
│   ├── en
│   │   └── LC_MESSAGES
│   │       └── django.po
│   └── fr_CA
│       └── LC_MESSAGES
│           └── django.po
├── drag_and_drop_v2
│   ├── ar
│   │   └── LC_MESSAGES
│   │       └── django.po
│   ├── en
│   │   └── LC_MESSAGES
│   │       └── django.po
│   └── fr_CA
│       └── LC_MESSAGES
│           └── django.po
└── edx_proctoring
    ├── ar
    │   └── LC_MESSAGES
    │       └── djangojs.po
    ├── de
    │   └── LC_MESSAGES
    │       └── djangojs.po
    ├── en
    │   └── LC_MESSAGES
    │       ├── djangojs.po
    │       └── django.po
    └── fr_CA
        └── LC_MESSAGES
            ├── djangojs.po
            └── django.po

Using XBlock python module names instead of repository names

There's more than one identifier for XBlocks and Plugins:

  1. The XBlock/plugin tag: Python plugins have an entry point name which is referred to as tag in Open edX. For example, the tag in the Drag and Drop XBlock setup.py file is drag-and-drop-v2:

    # xblock-drag-and-drop-v2/setup.py
entry_points={
    'xblock.v1': 'drag-and-drop-v2 = drag_and_drop_v2:DragAndDropBlock',
}

  2. The git repository name: Each XBlock has a unique git repository name. For example, the Drag and Drop XBlock has the xblock-drag-and-drop-v2 repository name in GitHub: https://github.com/openedx/xblock-drag-and-drop-v2/

  3. Python module name: The python module name appears in the path of XBlock translations in the openedx-translations repo. For example, the Drag and Drop XBlock will have drag_and_drop_v2 python module name in the translations directory structure:

    translations/xblock-drag-and-drop-v2/drag_and_drop_v2/conf/locale/...

The pull_xblock_translations command will use the Python module name instead of the repository name to pull translations from the openedx-translations repo via atlas.

Using the Python module name has the following pros and cons:

Pros:

  • The python module name is available without needing to install the XBlock, or parse the setup.py file.
  • It is available in Python runtime.
  • It is available in the openedx-translations repo file structure.
  • It is unique in the virtual environment which prevents collisions.
  • The python module name of XBlocks doesn't change often if at all.

Cons:

  • The python module name can be confused as the XBlock tag, which can be different in some XBlocks.
  • The unique and stable identifier of XBlocks is the tag, not the python module name. Therefore, this decision will implicitly make the python module name another unique identifier for XBlocks.

The trade-offs are acceptable and this decision is reversible in case the xblock.tag needs to be used. However, this will require parsing the setup.py file and/or installing the XBlock in order to get the tag in the extract-translation-source-files.yml workflow in the openedx-translations repo.

Using the django and djangojs gettext domains

This proposal standardizes the gettext domain for XBlocks and Plugins to django and djangojs. This helps to unify the file names and avoid the need to add more complexity to the openedx-translations repo tooling.

The DjangoTranslation class doesn't allow customizing the locale directory for django.mo files for caching reasons. Therefore, the GNUTranslations class will be used instead in the create_js_namespaced_catalog helper function for generating JavaScript catalogs from django.mo files.

BlockI18nService support for atlas Python translations

get_python_locale_directory will support two modes:

  1. If translations for the XBlock/plugin has been pulled by atlas from the openedx-translations repo, it will be used. For example, if the edx-platform/conf/plugins-locale/drag_and_drop_v2/ar/LC_MESSAGES/django.po path exists, it will be used for the Drag and Drop XBlock.
  2. Otherwise, the bundled translation files in the XBlock packages will be used. The fallback path for the Drag and Drop XBlock will be lib/python3.8/site-packages/drag_and_drop_v2/translations/ar/LC_MESSAGES/text.po.

This fallback is used to maintain backwards compatibility with existing XBlocks that may or may not be included in the openedx-translations repo. Third-party XBlocks that are not included in the xblocks Transifex project, such as the Lime Survey XBlock, will benefit from this backwards compatibility.

New compile_xblock_translations command

An XBlock.i18n_js_namespace property will be added for the compile_xblock_translations to generate JavaScript translations in a centrally managed manner for installed XBlocks.

A compile_xblock_translations command will loop over XBlock modules that has the i18n_js_namespace property set and compile the JavaScript translations via the compilejsi18n command.

For example if the Drag and Drop XBlock has i18n_js_namespace = 'DragAndDropI18N', the compile_xblock_translations command will execute the equivalent of the following commands:

i18n_tool generate -v  # Generate the .mo files
python manage.py compilejsi18n --namespace DragAndDropI18N --output conf/plugins-locale/drag_and_drop_v2/js/

XBlockI18nService support for atlas JavaScript translations

A get_javascript_locale_path method will be added to the XBlockI18nService to provide XBlocks the appropriate path to django.js translation files. This method will allow XBlocks to utilize legacy packaged translations or atlas.

A i18n_js_namespace property will be added to generate JavaScript translations in a centrally managed manner for all XBlocks as described in the :ref:`js-translations` section.

For example, the Drag and Drop XBlock get_static_i18n_js_url will need to be updated to support the new XBlockI18nService get_javascript_i18n_catalog_url method and the namespace.

  class DragAndDropBlock(XBlock):

+   i18n_js_namespace = 'DragAndDropI18N'

    @staticmethod
    def _get_statici18n_js_url():
        """
        Returns the Javascript translation file for the currently selected language, if any found by
        `pkg_resources`
        """
        lang_code = translation.get_language()
        if not lang_code:
            return None

+       # TODO: Make this the default once OEP-58 is implemented.
+       if hasattr(self.i18n_service, 'get_javascript_i18n_catalog_url'):
+           i18n_catalog_url = self.i18n_service.get_javascript_i18n_catalog_url()
+           if i18n_catalog_url:
+               return i18n_catalog_url

        text_js = 'public/js/translations/{lang_code}/text.js'
        country_code = lang_code.split('-')[0]
        for code in (translation.to_locale(lang_code), lang_code, country_code):
            if pkg_resources.resource_exists(loader.module_name, text_js.format(lang_code=code)):
                return text_js.format(lang_code=code)
        return None

Dismissed Proposals

XBlocks and plugins have their own "atlas pull" command

This dismissed proposal intends to have each XBlock and Plugin have their own make pull_translations and be responsible for managing pulling their own translations from the openedx-translations repo.

This proposal has been dismissed because it would require substantial work to get into the details for the lib/python3.8/site-packages/ directory and ensure that the make pull_translations command won't corrupt the virtual environment.

This is a non-trivial task and appears to add more complexity than necessary due to the fact that XBlocks and plugins won't be used outside the context of edx-platform.

Goals

  1. Use atlas pull for the edx-platform repo.
  2. Use atlas pull for the XBlocks and Plugins.
  3. Allow Tutor and other advanced uses to craft their own atlas pull commands by making the the plugins list available via Django commands.
  4. Allow atlas pull to use the Python module names instead of the repository name of XBlocks and Plugins which is supported via the atlas pull --expand-glob option.

Non-Goals

The following are non-goals for this proposal, although some are going to be tackled in the future as part of the Translation Management update OEP-58 proposal.

  1. Provide a fool-proof method for managing named-release translations. This will be a separate discussion.
  2. Discuss the merge/segment strategy of the edx-platform. This is being discussed in the decision no. 0018.
  3. Design a new XBlock frontend architecture. Instead this proposal works with the existing architecture.
  4. Provide a new translation method for theme translations. This will be tackled later on.
  5. Provide a new translation method for non-XBlock plugins such as edx-val. This will be tackled later on as part of the OEP-58 proposal.