[REF] module_auto_update: Deprecate buggy features, add new API

The previous implementation of this addon proved being extremely buggy:

- It supplied out of the box a enabled cron to update Odoo that didn't restart the server, which possibly meant that upgrades broke things.
- It overloaded standard Odoo upgrade methods that made i.e. installing an addon sometimes forced to upgrade all other addons in the database.
- The checksum system wasn't smart enough, and some files that didn't need a module upgrade triggered the upgrade.
- It was based on a dirhash library that was untested.
- Some updates were not detected properly.
- Storing a column into `ir.module.module` sometimes forbids uninstalling the addon.

For all of these reasons, the addon has been refactored:

- All buggy features are now deprecated and moved into `*_deprecated.*` files.
- A new update system has been added.
- More tests.
- New checksum system, tested too.
- Smarter diff detection system that triggers only on really needed upgrades.
- Cron is disabled by default.
- Old installations should keep most functionality intact.

Most of the job was done by Stéphane Bidoul (ACSONE), so credits are added where needed too.

module_auto_update/README.rst

@@ -6,36 +6,69 @@
 Module Auto Update
 ==================
 
-This module will automatically check for and apply module upgrades on a schedule.
-
-Upgrade checking is accomplished by comparing the SHA1 checksums of currently-installed modules to the checksums of corresponding modules in the addons directories.
-
-Installation
-============
-
-Prior to installing this module, you need to:
-
-#. Install checksumdir with `pip install checksumdir`
-#. Ensure all installed modules are up-to-date. When installed, this module will assume the versions found in the addons directories are currently installed.
+This addon provides mechanisms to compute sha1 hashes of installed addons,
+and save them in the database. It also provides a method that exploits these
+mechanisms to update a database by upgrading only the modules for which the
+hash has changed since the last successful upgrade.
 
 Configuration
 =============
 
-The default time for checking and applying upgrades is 3:00 AM (UTC). To change this schedule, modify the "Perform Module Upgrades" scheduled action.
+This module supports the following system parameters:
 
-This module will ignore .pyc and .pyo file extensions by default. To modify this, create a module_auto_update.checksum_excluded_extensions system parameter with the desired extensions listed as comma-separated values.
+* ``module_auto_update.exclude_patterns``: comma-separated list of file
+  name patterns to ignore when computing addon checksums. Defaults to
+  ``*.pyc,*.pyo,i18n/*.pot,i18n_extra/*.pot,static/*``.
+  Filename patterns must be compatible with the python ``fnmatch`` function.
+
+In addition to the above pattern, .po files corresponding to languages that
+are not installed in the Odoo database are ignored when computing checksums.
 
 Usage
 =====
 
-Modules scheduled for upgrade can be viewed by clicking the "Updates" menu item in the Apps sidebar.
+The main method provided by this module is ``upgrade_changed_checksum``
+on ``ir.module.module``. It runs a database upgrade for all installed
+modules for which the hash has changed since the last successful
+run of this method. On success it saves the hashes in the database.
+
+The first time this method is invoked after installing the module, it
+runs an upgrade of all modules, because it has not saved the hashes yet.
+This is by design, priviledging safety. Should this be an issue,
+the method ``_save_installed_checksums`` can be invoked in a situation
+where one is sure all modules on disk are installed and up-to-date in the
+database.
+
+An easy way to invoke this upgrade mechanism is by issuing the following
+in an Odoo shell session::
 
-To perform upgrades manually, click the "Apply Scheduled Upgrades" menu item in the Apps sidebar.
+  env['ir.module.module'].upgrade_changed_checksum()
 
 .. image:: https://odoo-community.org/website/image/ir.attachment/5784_f2813bd/datas
    :alt: Try me on Runbot
    :target: https://runbot.odoo-community.org/runbot/149/9.0
 
+Known issues / Roadmap
+======================
+
+* Since version ``2.0.0``, some features have been deprecated.
+  When you upgrade from previous versions, these features will be kept for
+  backwards compatibility, but beware! They are buggy!
+
+  If you install this addon from scratch, these features are disabled by
+  default.
+
+  To force enabling or disabling the deprecated features, set a configuration
+  parameter called ``module_auto_update.enable_deprecated`` to either ``1``
+  or ``0``. It is recommended that you disable them.
+
+  Keep in mind that from this version, all upgrades are assumed to run in a
+  separate odoo instance, dedicated exclusively to upgrade Odoo.
+
+* When migrating the addon to new versions, the deprecated features should be
+  removed. To make it simple all deprecated features are found in files
+  suffixed with ``_deprecated``.
+
 Bug Tracker
 ===========
 
@@ -58,6 +91,7 @@ Contributors
 * Brent Hughes <[email protected]>
 * Juan José Scarafía <[email protected]>
 * Jairo Llopis <[email protected]>
+* Stéphane Bidoul <[email protected]> (https://acsone.eu)
 
 Do not contact contributors directly about support or help with technical issues.
 

module_auto_update/__init__.py

@@ -4,4 +4,3 @@
 
 from . import models
 from . import wizards
-from .hooks import post_init_hook

module_auto_update/__openerp__.py

@@ -5,27 +5,21 @@
 {
     'name': 'Module Auto Update',
     'summary': 'Automatically update Odoo modules',
-    'version': '9.0.1.0.2',
+    'version': '9.0.2.0.0',
     'category': 'Extra Tools',
-    'website': 'https://odoo-community.org/',
+    'website': 'https://github.com/OCA/server-tools',
     'author': 'LasLabs, '
               'Juan José Scarafía, '
               'Tecnativa, '
+              'ACSONE SA/NV, '
               'Odoo Community Association (OCA)',
     'license': 'LGPL-3',
     'application': False,
     'installable': True,
-    'post_init_hook': 'post_init_hook',
-    'external_dependencies': {
-        'python': [
-            'checksumdir',
-        ],
-    },
     'depends': [
         'base',
     ],
     'data': [
-        'views/module_views.xml',
-        'data/cron_data.xml',
+        'data/cron_data_deprecated.xml',
     ],
 }

module_auto_update/addon_hash.py

@@ -0,0 +1,45 @@
+# -*- coding: utf-8 -*-
+# Copyright 2018 ACSONE SA/NV.
+# License LGPL-3.0 or later (http://www.gnu.org/licenses/lgpl).
+
+from fnmatch import fnmatch
+import hashlib
+import os
+
+
+def _fnmatch(filename, patterns):
+    for pattern in patterns:
+        if fnmatch(filename, pattern):
+            return True
+    return False
+
+
+def _walk(top, exclude_patterns, keep_langs):
+    keep_langs = {l.split('_')[0] for l in keep_langs}
+    for dirpath, dirnames, filenames in os.walk(top):
+        dirnames.sort()
+        reldir = os.path.relpath(dirpath, top)
+        if reldir == '.':
+            reldir = ''
+        for filename in sorted(filenames):
+            filepath = os.path.join(reldir, filename)
+            if _fnmatch(filepath, exclude_patterns):
+                continue
+            if keep_langs and reldir in {'i18n', 'i18n_extra'}:
+                basename, ext = os.path.splitext(filename)
+                if ext == '.po':
+                    if basename.split('_')[0] not in keep_langs:
+                        continue
+            yield filepath
+
+
+def addon_hash(top, exclude_patterns, keep_langs):
+    """Compute a sha1 digest of file contents."""
+    m = hashlib.sha1()
+    for filepath in _walk(top, exclude_patterns, keep_langs):
+        # hash filename so empty files influence the hash
+        m.update(filepath.encode('utf-8'))
+        # hash file content
+        with open(os.path.join(top, filepath), 'rb') as f:
+            m.update(f.read())
+    return m.hexdigest()

module_auto_update/data/cron_data.xml → module_auto_update/data/cron_data_deprecated.xml

@@ -2,7 +2,7 @@
 <odoo noupdate="1">
     <record model="ir.cron" id="module_check_upgrades_cron">
         <field name="name">Perform Module Upgrades</field>
-        <field name="active" eval="True"/>
+        <field name="active" eval="False"/>
         <field name="user_id" ref="base.user_root"/>
         <field name="interval_number">1</field>
         <field name="interval_type">days</field>

module_auto_update/migrations/9.0.2.0.0/pre-migrate.py

@@ -0,0 +1,24 @@
+# -*- coding: utf-8 -*-
+# Copyright 2018 Tecnativa - Jairo Llopis
+# License LGPL-3.0 or later (https://www.gnu.org/licenses/lgpl).
+import openupgradelib
+
+from openerp.addons.module_auto_update.models.module_deprecated import \
+    PARAM_DEPRECATED
+
+
+# TODO Remove when deprecated features are removed; see README's known issues
+@openupgradelib.migrate(no_version=True, use_env=True)
+def migrate(env):
+    """Move stored checksums to new JSON format."""
+    # Store module checksums in the new JSON format
+    env.cr.execute("""SELECT name, checksum_installed
+                      FROM ir_module_module
+                      WHERE checksum_installed""")
+    checksums = dict(env.cr.fetchall())
+    env["ir.module.module"]._save_checksums(checksums)
+    # Check if deprecation parameter existed
+    deprecated = env["ir.config_parameter"].get_param(PARAM_DEPRECATED)
+    if deprecated is False:
+        # If not, enable deprecated features now, to avoid breaking behavior
+        env["ir.config_parameter"].set_param(PARAM_DEPRECATED, "1")

module_auto_update/models/__init__.py

@@ -3,3 +3,4 @@
 # License LGPL-3.0 or later (http://www.gnu.org/licenses/lgpl).
 
 from . import module
+from . import module_deprecated