Allow get_translated helper to fall back to base version of a language

[EricSoroos]

Proposed fixes:

The current get_translated helper gets data from the _translated.[lang] field in the data_dict. There are language variants (for example, en_GB) where the dialect is a minor change, and should fall back to the en value if en_GB is not available.

In my case, we've got interfaces that use the en_GB locale, but the data/schema is split between en and ga (Gaelic) languages.

Features:

• includes tests covering changes
• includes updated documentation
• includes user-visible changes
• includes API changes
• includes bugfix for possible backport

Please [X] all the boxes above that apply

[wardi]

@EricSoroos the test failure might be unrelated. want to try merging master?

[EricSoroos]

Yeah, I think it's good to go, with the 1 space removed that lint was complaining about. (and a rebase to remove the conflict)

[wardi]

@EricSoroos it looks like your test needs the request context set up to run properly

[amercader]

@EricSoroos You need this to use a request context in your test:

diff --git a/ckan/tests/lib/test_helpers.py b/ckan/tests/lib/test_helpers.py
index 43f247bb5..f45d0eaf3 100644
--- a/ckan/tests/lib/test_helpers.py
+++ b/ckan/tests/lib/test_helpers.py
@@ -887,6 +887,7 @@ def test_decode_view_request_filters(test_request_context):
     # Null case, falls all the way through.
     ({}, 'en', ''),
 ])
+@pytest.mark.usefixtures("with_request_context")
 def test_get_translated(data_dict, locale, result, monkeypatch):
     monkeypatch.setattr(flask_babel, "get_locale", lambda: locale)
     assert h.get_translated(data_dict, 'notes') == result

But after applying this, the tests don't seem to work as expected and there are two failures:

========================================================= FAILURES =========================================================                                                                                                                                                         
_____________________________________ test_get_translated[data_dict0-es-untranslated] ______________________________________                                                                                                                                                         
                                                                                                                                                                                                                                                                                     
data_dict = {'notes': 'untranslated', 'notes_translated': {'en': 'en', 'en_GB': 'en_GB'}}, locale = 'es'                                                                                                                                                                             
result = 'untranslated', monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f57e94d0d00>                                                                                                                                                                                    
                                                                                                                                                                                                                                                                                     
    @pytest.mark.parametrize("data_dict, locale, result", [                                                                                                                                                                                                                          
        # no matching local returns base                                                                                                                                                                                                                                             
        ({"notes": "untranslated",                                                                                                                                                                                                                                                   
          "notes_translated": {'en': 'en', 'en_GB': 'en_GB'}}, 'es', 'untranslated'),                                                                                                                                                                                                
        # specific returns specfic                                                                                                                                                                                                                                                   
        ({"notes": "untranslated",                                                                                                                                                                                                                                                   
          "notes_translated": {'en': 'en', 'en_GB': 'en_GB'}}, 'en_GB', 'en_GB'),                                                         
        # variant returns base                                       
        ({"notes": "untranslated",                                   
          "notes_translated": {'en': 'en'}}, 'en_GB', 'en'),         
        # Null case, falls all the way through.                      
        ({}, 'en', ''),                                              
    ])                                                               
    @pytest.mark.usefixtures("with_request_context")                 
    def test_get_translated(data_dict, locale, result, monkeypatch):                                                                      
        monkeypatch.setattr(flask_babel, "get_locale", lambda: locale)                                                                    
>       assert h.get_translated(data_dict, 'notes') == result                                                                             
E       AssertionError: assert 'en' == 'untranslated'                                                                                     
E         - untranslated                                             
E         + en                                                       
                 
ckan/tests/lib/test_helpers.py:893: AssertionError
---------------------------------------------------- Captured log setup ----------------------------------------------------
WARNING  ckan.config.middleware.flask_app:flask_app.py:265 Extensions are excluded from CSRF protection! We allow extensions to run without CSRF protection but it will be forced future releases. Read the documentation for more information on how to add CSRF protection to your extension.
_______________________________________ test_get_translated[data_dict1-en_GB-en_GB] ________________________________________

data_dict = {'notes': 'untranslated', 'notes_translated': {'en': 'en', 'en_GB': 'en_GB'}}, locale = 'en_GB'
result = 'en_GB', monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f57ea4362e0>

    @pytest.mark.parametrize("data_dict, locale, result", [
        # no matching local returns base
        ({"notes": "untranslated",
          "notes_translated": {'en': 'en', 'en_GB': 'en_GB'}}, 'es', 'untranslated'),
        # specific returns specfic
        ({"notes": "untranslated",
          "notes_translated": {'en': 'en', 'en_GB': 'en_GB'}}, 'en_GB', 'en_GB'),
        # variant returns base
        ({"notes": "untranslated",
          "notes_translated": {'en': 'en'}}, 'en_GB', 'en'),
        # Null case, falls all the way through.
        ({}, 'en', ''),
    ])
    @pytest.mark.usefixtures("with_request_context")
    def test_get_translated(data_dict, locale, result, monkeypatch):
        monkeypatch.setattr(flask_babel, "get_locale", lambda: locale)
>       assert h.get_translated(data_dict, 'notes') == result
E       AssertionError: assert 'en' == 'en_GB'
E         - en_GB
E         + en

ckan/tests/lib/test_helpers.py:893: AssertionError
---------------------------------------------------- Captured log setup ----------------------------------------------------
WARNING  ckan.config.middleware.flask_app:flask_app.py:265 Extensions are excluded from CSRF protection! We allow extensions to run without CSRF protection but it will be forced future releases. Read the documentation for more information on how to add CSRF protection to your extension.
================================================= short test summary info ==================================================
FAILED ckan/tests/lib/test_helpers.py::test_get_translated[data_dict0-es-untranslated] - AssertionError: assert 'en' == '...
FAILED ckan/tests/lib/test_helpers.py::test_get_translated[data_dict1-en_GB-en_GB] - AssertionError: assert 'en' == 'en_GB'
======================================= 2 failed, 2 passed, 155 deselected in 2.10s ========================================

[EricSoroos]

@amercader Additional logging isn't coming through in circleci -- is there a quick/docker-compose way that you run tests locally?

[amercader]

@EricSoroos If you don't have a local source install I'd try this:

# Clone this docker repo
git clone https://github.com/okfn/docker-ckan

# Replace the `build:` section in `docker-compose.yml` with `image: openknowledge/ckan-dev:2.10` or `image: openknowledge/ckan-dev:master`

# Clone ckan in the `src` folder, checkout your branch
cd src
git clone -b YourBranch YourFork

# boot everything up
docker-compose up 

# Run the tests in the container
docker-compose exec ckan bash -c "pytest -c /srv/app/ckan.ini /srv/app/src_extensions/ckan/ckan/tests"

I haven't tested it so you might have to tweak things!

[EricSoroos]

Yeah, I don't generally have a source install, we tend to do everything in docker-compose. Part of my frustration is coming from talking to someone who wants to get into doing their own CKAN development (on windows even), and the learning curve on following the directions is huge. And when I turn around and say "well, actually we ignore that part and do our own thing" it's not really a good view on anyone.

[EricSoroos]

Ok, with tests running locally (using -vv), I'm getting

------------------------------------------------------------------------------- Captured log call -------------------------------------------------------------------------------
ERROR    ckan.lib.helpers:helpers.py:2575 language: en
ERROR    ckan.lib.helpers:helpers.py:2576 data_dict: {'notes': 'untranslated', 'notes_translated': {'en': 'en', 'en_GB': 'en_GB'}}

for both cases, so the monkey patching for i18n isn't working in the test.

    def test_get_translated(data_dict, locale, result, monkeypatch):
        monkeypatch.setattr(flask_babel, "get_locale", lambda: locale)
        from ckan.lib import i18n
>       assert i18n.get_lang() == locale
E       AssertionError: assert 'en' == 'en_GB'
E         - en_GB
E         + en

[EricSoroos]

The circle-ci run failed here because it wasn't able to checkout the branch. Can someone with write privs on circleci kick it off again?