[QOL-7544] apply QGOV patches to 2.9.5 base

update psycopg2 to support PostgreSQL 12, ckan#5796 [QOL-9275] add config flag to control private dataset behaviour - If set to True, then return Forbidden or redirect to login page, instead of hiding the dataset's existence. Bump moment from 2.26.0 to 2.29.4 Bumps [moment](https://github.com/moment/moment) from 2.26.0 to 2.29.4. - [Release notes](https://github.com/moment/moment/releases) - [Changelog](https://github.com/moment/moment/blob/develop/CHANGELOG.md) - [Commits](moment/moment@2.26.0...2.29.4) --- updated-dependencies: - dependency-name: moment dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> fix logic import [DC-32816] update sqlparse to handle CSV contents more robustly, GitHub ckan#5822 - This is already fixed on master but not backported to 2.9 [DC-31965] handle missing resources in activity stream - Copy cleanup from upstream Revert "Add test and changelog" This reverts commit 3745031. Revert "Check if locale exists on i18n JS API" This reverts commit c906ee1. Revert "Fixing tests." This reverts commit 233be7e. Add test and changelog Fixing tests. Check if locale exists on i18n JS API fix errno2 check for resource remove extra comma (cherry picked from commit fb27f2a) [QOL-8785] only import Pylons response when relevant [QOL-8785] fix missing imports and make Flake8 happy [QOL-8785] reuse uploader mimetype if present [QOL-8785] store resource URL in uploader so we can retrieve it for Pylons downloads [QOL-8785] parse config flag as a boolean [QOL-8785] skip test for now [QOL-8785] drop test environment for post [QOL-8785] make mailer test syntax match other tests [QOL-8785] add blank password fields to pass validation [QOL-8785] restore 'save' flag to test form [QOL-8785] add debugging statement for test failure [QOL-8785] add username to test environment [QOL-8785] adjust test assertion for better failure message [QOL-8785] fix test assertion syntax [QOL-8785] adjust test username change parameters [QOL-8785] fix test assertion syntax [QOL-8785] fix test assertion syntax [QOL-8785] oops only add version to emails if flag is not set [QOL-8785] use string not bool for test config value [QOL-8785] add mail server parameter to test [QOL-8785] make post-reset landing page configurable [QOL-8785] use 'download' function to simplify controller [QOL-8785] drop 'REMOTE_USER' value in test as it's confusing the dashboard [QOL-8785] update assertion syntax for pytest [QOL-8515] get test app from fixtures instead of directly retrieving it [QOL-8515] make pep8 happy [QOL-8515] skip updating package modified timestamp if only updating resources - copied from 2.8.8 QGOV branch [QOL-8515] allow sysadmins to update usernames - copied from 2.8.8 QGOV branch [QOL-8515] optionally redirect to a different page after password resets - copied from 2.8.8 QGOV branch [QOL-8515] optionally hide CKAN version in site status and emails - copied from 2.8.8 QGOV branch [QOL-8515] make pep8 happy [QOL-8515] make pep8 happy [QOL-8515] fix import spelling [QOL-8515] make pep8 happy [QOL-8515] add functionality to delete uploaded files from storage on resource deletion [QOL-8515] adjust mocking of 'open' function to handle PY2 and PY3 [QOL-8515] oops provide necessary test import [QOL-8515] add testing of extra IUploader functions add metadata to uploader interface, next step, surface it via package * add metadata to IUploader interface's * Add new resource_file_metadata_show get api to surface metadata checks, usage case is archiver or similar verifying files have not change or disappeared * missed storage_path from file delete/download [QOL-8576] simplify imports [QOL-8576] fix isoformat syntax so lint is happy - raw string literals aren't allowed, but Python 2 'isoformat' can't handle unicode, so use the 'str' function, which is documented to have the same effect Fix isoformat argument for Python 2 [QOL-7544] handle storage directory already existing, ckan#6521 remove white space lint check if dir already exists [QOL-7544] apply QGOV patches to 2.9.5 base - render full name as 'Displayed Name' since that better describes its purpose - handle race condition in creation of site user - ignore authentication when loading user profile during password reset
qld-gov-au · Nov 24, 2022 · 4d601cc · 4d601cc
1 parent 65af260
commit 4d601cc
Show file tree

Hide file tree

Showing 34 changed files with 641 additions and 121 deletions.
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
@@ -9,6 +9,15 @@ Changelog
 
 .. towncrier release notes start
 
+v.2.9.x 2022-xx-xx
+==================
+
+Major features
+--------------
+
+- Update to Interface IUploader, on get_uploader and get_resource_uploader, new to include new method signature metadata() which can be utilised by archiver and other plugins instead of trying on local disk directly
+- Add get api `resource_file_metadata_show` which takes resource id and returns { 'content_type': content_type, 'size': length, 'hash': hash } if found
+
 v.2.9.5 2022-01-19
 ==================
 

diff --git a/ckan/cli/user.py b/ckan/cli/user.py
@@ -6,9 +6,7 @@
 from six import text_type
 from datetime import datetime
 
-import ckan.logic as logic
-import ckan.plugins as plugin
-import ckan.model as model
+from ckan import logic, model, plugins as plugin
 from ckan.cli import error_shout
 from ckan.common import json
 
@@ -56,8 +54,6 @@ def add_user(ctx, username, args):
     # pprint(u'Creating user: %r' % username)
 
     try:
-        import ckan.logic as logic
-        import ckan.model as model
         site_user = logic.get_action(u'get_site_user')({
             u'model': model,
             u'ignore_auth': True},
@@ -90,7 +86,6 @@ def get_user_str(user):
 
 @user.command(u'list', short_help=u'List all users')
 def list_users():
-    import ckan.model as model
     click.secho(u'Users:')
     users = model.Session.query(model.User).filter_by(state=u'active')
     click.secho(u'count = %i' % users.count())
@@ -102,7 +97,6 @@ def list_users():
 @click.argument(u'username')
 @click.pass_context
 def remove_user(ctx, username):
-    import ckan.model as model
     if not username:
         error_shout(u'Please specify the username to be removed')
         return
@@ -117,7 +111,6 @@ def remove_user(ctx, username):
 @user.command(u'show', short_help=u'Show user')
 @click.argument(u'username')
 def show_user(username):
-    import ckan.model as model
     if not username:
         error_shout(u'Please specify the username for the user')
         return
@@ -128,7 +121,6 @@ def show_user(username):
 @user.command(u'setpass', short_help=u'Set password for the user')
 @click.argument(u'username')
 def set_password(username):
-    import ckan.model as model
     if not username:
         error_shout(u'Need name of the user.')
         return
@@ -235,13 +227,13 @@ def list_tokens(username):
                 Strip out microseconds to force formatting as isoformat doesnt
                 have a timespec param on Python 2.
                 """
-                accessed = datetime(
+                accessed = str(datetime(
                     accessed.year,
                     accessed.month,
                     accessed.day,
                     accessed.hour,
                     accessed.minute,
-                    accessed.second).isoformat(u" ")
+                    accessed.second))
             else:
                 accessed = accessed.isoformat(u" ", u"seconds")
 

diff --git a/ckan/config/environment.py b/ckan/config/environment.py
@@ -328,6 +328,9 @@ def update_config():
     except sqlalchemy.exc.InternalError:
         # The database is not initialised.  Travis hits this
         pass
+    except sqlalchemy.exc.IntegrityError:
+        # Race condition, user already exists.
+        pass
 
     # Close current session and open database connections to ensure a clean
     # clean environment even if an error occurs later on

diff --git a/ckan/config/middleware/pylons_app.py b/ckan/config/middleware/pylons_app.py
@@ -154,12 +154,18 @@ def make_pylons_stack(conf, full_stack=True, static_files=True,
         storage_directory = uploader.get_storage_path()
         if storage_directory:
             path = os.path.join(storage_directory, 'storage')
-            try:
-                os.makedirs(path)
-            except OSError as e:
-                # errno 17 is file already exists
-                if e.errno != 17:
-                    raise
+
+            # check if the storage directory is already created by
+            # the user or third-party
+            if os.path.isdir(path):
+                pass
+            else:
+                try:
+                    os.makedirs(path)
+                except OSError as e:
+                    # errno 17 is file already exists
+                    if e.errno != 17:
+                        raise
 
             storage_app = StaticURLParser(path, cache_max_age=static_max_age)
             static_parsers.insert(0, storage_app)

diff --git a/ckan/controllers/package.py b/ckan/controllers/package.py
@@ -1166,19 +1166,11 @@ def resource_download(self, id, resource_id, filename=None):
 
         if rsc.get('url_type') == 'upload':
             upload = uploader.get_resource_uploader(rsc)
-            filepath = upload.get_path(rsc['id'])
-            fileapp = paste.fileapp.FileApp(filepath)
             try:
-                status, headers, app_iter = request.call_application(fileapp)
+                return upload.download(rsc['id'], filename)
             except OSError:
+                # includes FileNotFoundError
                 abort(404, _('Resource data not found'))
-            response.headers.update(dict(headers))
-            content_type, content_enc = mimetypes.guess_type(
-                rsc.get('url', ''))
-            if content_type:
-                response.headers['Content-Type'] = content_type
-            response.status = status
-            return app_iter
         elif 'url' not in rsc:
             abort(404, _('No download is available'))
         h.redirect_to(rsc['url'])

diff --git a/ckan/controllers/user.py b/ckan/controllers/user.py
@@ -493,13 +493,15 @@ def request_reset(self):
                           'or contact an administrator for help')
                     )
                     log.exception(e)
-                    return h.redirect_to(u'/')
+                    return h.redirect_to(config.get(
+                        u'ckan.user_reset_landing_page', u'home.index'))
             # always tell the user it succeeded, because otherwise we reveal
             # which accounts exist or not
             h.flash_success(
                 _(u'A reset link has been emailed to you '
                   '(unless the account specified does not exist)'))
-            return h.redirect_to(u'/')
+            return h.redirect_to(config.get(
+                u'ckan.user_reset_landing_page', u'home.index'))
         return render('user/request_reset.html')
 
     def perform_reset(self, id):
@@ -541,7 +543,8 @@ def perform_reset(self, id):
                 mailer.create_reset_key(user_obj)
 
                 h.flash_success(_("Your password has been reset."))
-                h.redirect_to(u'home.index')
+                return h.redirect_to(config.get(
+                    u'ckan.user_reset_landing_page', u'home.index'))
             except NotAuthorized:
                 h.flash_error(_('Unauthorized to edit user %s') % id)
             except NotFound as e:

diff --git a/ckan/i18n/en_AU/LC_MESSAGES/ckan.po b/ckan/i18n/en_AU/LC_MESSAGES/ckan.po
@@ -2,10 +2,10 @@
 # Copyright (C) 2020 ORGANIZATION
 # This file is distributed under the same license as the ckan project.
 # FIRST AUTHOR <EMAIL@ADDRESS>, 2020.
-# 
+#
 # Translators:
 # Adrià Mercader <[email protected]>, 2020
-# 
+#
 #, fuzzy
 msgid ""
 msgstr ""
@@ -4777,7 +4777,7 @@ msgstr "Change details"
 
 #: ckan/templates/user/edit_user_form.html:10
 msgid "Full name"
-msgstr "Full name"
+msgstr "Displayed name"
 
 #: ckan/templates/user/edit_user_form.html:10
 msgid "eg. Joe Bloggs"
@@ -4931,7 +4931,7 @@ msgstr "username"
 
 #: ckan/templates/user/new_user_form.html:6
 msgid "Full Name"
-msgstr "Full Name"
+msgstr "Displayed Name"
 
 #: ckan/templates/user/new_user_form.html:27
 msgid "Create Account"

diff --git a/ckan/lib/app_globals.py b/ckan/lib/app_globals.py
@@ -11,7 +11,7 @@
 
 import ckan
 import ckan.model as model
-from logic.schema import update_configuration_schema
+from ckan.logic.schema import update_configuration_schema
 
 
 log = logging.getLogger(__name__)

diff --git a/ckan/lib/changes.py b/ckan/lib/changes.py
@@ -1,5 +1,4 @@
 # encoding: utf-8
-
 '''
 Functions for generating a list of differences between two versions of a
 dataset
@@ -67,12 +66,12 @@ def check_resource_changes(change_list, old, new, old_activity_id):
     new_resource_set = set()
     new_resource_dict = {}
 
-    for resource in old.get(u'resources'):
+    for resource in old.get(u'resources', []):
         old_resource_set.add(resource['id'])
         old_resource_dict[resource['id']] = {
             key: value for (key, value) in resource.items() if key != u'id'}
 
-    for resource in new.get(u'resources'):
+    for resource in new.get(u'resources', []):
         new_resource_set.add(resource['id'])
         new_resource_dict[resource['id']] = {
             key: value for (key, value) in resource.items() if key != u'id'}
@@ -129,7 +128,7 @@ def check_resource_changes(change_list, old, new, old_activity_id):
                                 u'resource_id': resource_id,
                                 u'resource_name':
                                 new_resource_dict[resource_id].get(u'name'),
-                                u'org_id': new.get(u'organization')['id']
+                                u'org_id': new[u'organization']['id']
                                     if new.get(u'organization') else u'',
                                 u'format': new_metadata.get(u'format')})
 
@@ -142,7 +141,7 @@ def check_resource_changes(change_list, old, new, old_activity_id):
                                 u'resource_id': resource_id,
                                 u'resource_name':
                                 new_resource_dict[resource_id].get(u'name'),
-                                u'org_id': new.get(u'organization')['id']
+                                u'org_id': new[u'organization']['id']
                                     if new.get(u'organization') else u'',
                                 u'old_format': old_metadata.get(u'format'),
                                 u'new_format': new_metadata.get(u'format')})
@@ -392,30 +391,30 @@ def _org_change(change_list, old, new):
                             u'method': u'change',
                             u'pkg_id': new.get(u'id'),
                             u'title': new.get(u'title'),
-                            u'old_org_id': old.get(u'organization').get(u'id'),
+                            u'old_org_id': old[u'organization'].get(u'id'),
                             u'old_org_title':
-                            old.get(u'organization').get(u'title'),
-                            u'new_org_id': new.get(u'organization').get(u'id'),
+                            old[u'organization'].get(u'title'),
+                            u'new_org_id': new[u'organization'].get(u'id'),
                             u'new_org_title':
-                                new.get(u'organization').get(u'title')})
+                                new[u'organization'].get(u'title')})
     # if the dataset was not in an organization before and it is now
     elif not old.get(u'owner_org') and new.get(u'owner_org'):
         change_list.append({u'type': u'org',
                             u'method': u'add',
                             u'pkg_id': new.get(u'id'),
                             u'title': new.get(u'title'),
-                            u'new_org_id': new.get(u'organization').get(u'id'),
+                            u'new_org_id': new[u'organization'].get(u'id'),
                             u'new_org_title':
-                            new.get(u'organization').get(u'title')})
+                            new[u'organization'].get(u'title')})
     # if the user removed the organization
     else:
         change_list.append({u'type': u'org',
                             u'method': u'remove',
                             u'pkg_id': new.get(u'id'),
                             u'title': new.get(u'title'),
-                            u'old_org_id': old.get(u'organization').get(u'id'),
+                            u'old_org_id': old[u'organization'].get(u'id'),
                             u'old_org_title':
-                            old.get(u'organization').get(u'title')})
+                            old[u'organization'].get(u'title')})
 
 
 def _maintainer_change(change_list, old, new):
@@ -716,13 +715,13 @@ def _extra_fields(change_list, old, new):
     change_list.
     '''
     if u'extras' in new:
-        extra_fields_new = _extras_to_dict(new.get(u'extras'))
+        extra_fields_new = _extras_to_dict(new.get(u'extras', []))
         extra_new_set = set(extra_fields_new.keys())
 
         # if the old version has extra fields, we need
         # to compare the new version's extras to the old ones
         if u'extras' in old:
-            extra_fields_old = _extras_to_dict(old.get(u'extras'))
+            extra_fields_old = _extras_to_dict(old.get(u'extras', []))
             extra_old_set = set(extra_fields_old.keys())
 
             # if some fields were added
@@ -819,7 +818,7 @@ def _extra_fields(change_list, old, new):
                                     u'value_list': extra_fields_new})
 
     elif u'extras' in old:
-        deleted_fields = _extras_to_dict(old['extras']).keys()
+        deleted_fields = list(_extras_to_dict(old['extras']).keys())
         if len(deleted_fields) == 1:
             change_list.append({u'type': u'extra_fields',
                                 u'method': u'remove_one',

diff --git a/ckan/lib/i18n.py b/ckan/lib/i18n.py
@@ -76,7 +76,7 @@
 
 def get_ckan_i18n_dir():
     path = config.get(
-        u'ckan.i18n_directory', os.path.join(_CKAN_DIR, u'i18n'))
+        u'ckan.i18n_directory') or os.path.join(_CKAN_DIR, u'i18n')
     if os.path.isdir(os.path.join(path, u'i18n')):
         path = os.path.join(path, u'i18n')
 

diff --git a/ckan/lib/mailer.py b/ckan/lib/mailer.py
@@ -58,7 +58,8 @@ def _mail_recipient(recipient_name, recipient_email,
     msg['From'] = _("%s <%s>") % (sender_name, mail_from)
     msg['To'] = u"%s <%s>" % (recipient_name, recipient_email)
     msg['Date'] = utils.formatdate(time())
-    msg['X-Mailer'] = "CKAN %s" % ckan.__version__
+    if not ckan.common.asbool(config.get('ckan.hide_version')):
+        msg['X-Mailer'] = "CKAN %s" % ckan.__version__
     if reply_to and reply_to != '':
         msg['Reply-to'] = reply_to