Implement a migration system

README.md

@@ -290,7 +290,16 @@ The environment also allows administrator overrides. Using the `:static` and `:t
 
 See `environment-change`, `environment`, `environment-directory`, `environment-module-directory`, `environment-module-pathname`, `check-environment`, `mconfig-pathname`, `mconfig-storage`, `mconfig`, `defaulted-mconfig`, `config`, `defaulted-config`, `template-file`, `@template`, `static-file`, `@static`
 
-### 1.12 Instance Management
+### 1.12 Migration System
+Sometimes systems evolve in backwards incompatible ways. In that case, for existing setups to continue functioning with the new version, runtime data migration is necessary. Radiance offers a system to automate this process and allow a smooth upgrade. 
+
+The migration between versions should occur automatically during Radiance's startup sequence. As an administrator or author you should not need to perform any additional steps for migrations to occur. However, as a module author, you will naturally have to provide the code to perform the necessary data migration steps for your module.
+
+In order for a module to be migratable, it needs to be loaded by an ASDF system that has a version specification. The version should follow the standard dotted number scheme, with an optional version hash that can be added at the end. You may then define migration steps between individual versions by using `define-version-migration`. Once defined, Radiance will automatically pick up on concrete versions and perform the necessary migrations in sequence to reach the current target version. For more information on the precise procedure and what you can do, see `migrate` and `migrate-versions`.
+
+See `last-known-system-version`, `migrate-versions`, `define-version-migration`, `ready-dependency-for-migration`, `ensure-dependencies-ready`, `versions`, `migrate`
+
+### 1.13 Instance Management
 Finally, Radiance provides a standard startup and shutdown sequence that should ensure things are properly setup and readied, and afterwards cleaned up nicely again. A large part of that sequence is just ensuring that certain hooks are called in the proper order and at the appropriate times.
 
 While you can start a server manually by using the appropriate interface function, you should not expect applications to run properly if you do it that way. Many of them will expect certain hooks to be called in order to work properly. This is why you should always, unless you exactly know what you're doing, use `startup` and `shutdown` to manage a Radiance instance. The documentation of the two functions should explain exactly which hooks are triggered and in which order. An implementation may provide additional, unspecified definitions on symbols in the interface package, as long as said symbols are not exported.

conditions.lisp

@@ -23,6 +23,17 @@
                                     This definition may lead to unintended overrides.~@[~%~a~]"
                                  (message c)))))
 
+(define-condition system-has-no-version (radiance-error)
+  ((system :initarg :system :initform (error "SYSTEM required.")))
+  (:report (lambda (c s) (format s "The system ~a has no version specified, so Radiance does not know how to migrate it to the latest point."
+                                 (asdf:component-name (slot-value c 'system))))))
+
+(define-condition backwards-migration-not-allowed (radiance-error)
+  ((from :initarg :from :initform (error "FROM required."))
+   (to :initarg :to :initform (error "TO required.")))
+  (:report (lambda (c s) (format s "Cannot migrate a system backwards from ~a to ~a."
+                                 (encode-version (slot-value c 'from)) (encode-version (slot-value c 'to))))))
+
 (define-condition environment-not-set (radiance-error) ()
   (:report "The application environment was not yet set but is required.
 This means you are either using Radiance for the first time or forgot to set it up properly.

defaults.lisp

@@ -228,3 +228,29 @@
               (T
                ;; Other refer, no change.
                ))))))
+
+;;; Default logger to make sure we can log even before the real impl is loaded.
+(defun l:log (level category log-string &rest format-args)
+  (format *error-output* "~&~a [~a] <~a> ~?~%"
+          (format-human-date (get-universal-time)) level category log-string format-args))
+
+(defun l:trace (category log-string &rest format-args)
+  (declare (ignore category log-string format-args)))
+
+(defun l:debug (category log-string &rest format-args)
+  (declare (ignore category log-string format-args)))
+
+(defun l:info (category log-string &rest format-args)
+  (apply #'l:log :info category log-string format-args))
+
+(defun l:warn (category log-string &rest format-args)
+  (apply #'l:log :warn category log-string format-args))
+
+(defun l:error (category log-string &rest format-args)
+  (apply #'l:log :error category log-string format-args))
+
+(defun l:severe (category log-string &rest format-args)
+  (apply #'l:log :severe category log-string format-args))
+
+(defun l:fatal (category log-string &rest format-args)
+  (apply #'l:log :fatal category log-string format-args))

documentation.lisp

@@ -283,6 +283,24 @@ systems.
 
 See RADIANCE-WARNING")
 
+  (type system-has-no-version
+    "Error signalled when an ASDF system does not store a version string.
+
+Without a version string, Radiance is incapable of tracking what the
+current version of a system is and is thus unable to automatically
+migrate it.
+
+This error should be continuable.
+
+See MIGRATE
+See RADIANCE-ERROR")
+
+  (type backwards-migration-not-allowed
+    "Error signalled when a migration from a later version to an earlier version is attempted.
+
+See MIGRATE
+See RADIANCE-ERROR")
+
   (type environment-not-set
     "Error signalled when an action was performed that requires an initialised environment, but no environment has been configured yet.
 
@@ -1257,6 +1275,289 @@ information until later.
 
 See DEFINE-INTERFACE"))
 
+;; migration.lisp
+(docs:define-docs
+  (function encode-version
+    "Encodes the given version into a version name as a keyword.
+
+Each part of the version spec is concatenated as follows:
+  VERSION-NAME    ::= (string|integer) (VERSION-STRING | VERSION-INTEGER)*
+  VERSION-STRING  ::= '-' string
+  VERSION-INTEGER ::= '.' integer
+
+Essentially meaning that integer sub-versions are preceded by a
+dot and string sub-versions by a dash.")
+
+  (function parse-version
+    "Parses the given string into a version spec.
+
+The string should be of the following format:
+  VERSION-NAME ::= VERSION-PART (('.' | '-') VERSION-PART)+
+  VERSION-PART ::= integer | string
+
+See ENSURE-PARSED-VERSION")
+
+  (function ensure-parsed-version
+    "Ensures that the given version is in the version spec format.
+
+Returns a version spec:
+  VERSION-SPEC ::= (VERSION-PART+)
+  VERSION-PART ::= integer | string
+
+See PARSE-VERSION")
+
+  (function ensure-versions-comparable
+    "Ensures the two versions are of the same length.
+
+If one of the versions is shorter, it is extended by 0 elements
+in its spec. Both of the versions are returned, with the specs
+having equal length.
+
+See ENSURE-PARSED-VERSION")
+
+  (function version-part=
+    "Returns T if the two version parts are considered equal.
+
+The following table should cover all cases:
+ ↓A B→ INTEGER  STRING
+INTEGER    =      NIL
+STRING    NIL   STRING=")
+
+  (function version-part<
+    "Returns T if the first version part is considered lower.
+
+The following table should cover all the cases:
+ ↓A B→ INTEGER  STRING
+INTEGER    <       T
+STRING    NIL   STRING<")
+
+  (function version=
+    "Returns T if the two version specs are considered equal.
+
+First ensures both versions are of equal length, then calls
+VERSION-PART= on each part of the versions. If they are all
+VERSION-PART=, then T is returned and NIL otherwise.
+
+A special exemption is made for NIL version specs. If both
+versions are NIL, T is returned. Otherwise, if only one spec is
+NIL, NIL is returned.
+
+See ENSURE-VERSIONS-COMPARABLE
+See VERSION-PART=")
+
+  (function version<
+    "Returns T if the first version is considered lower.
+
+First ensures both versions are of equal length, then calls
+VERSION-PART= on each part of the versions. If it returns NIL,
+then VERSION-PART< is called. If this returns T, T is returned,
+otherwise NIL is returned. If all parts are VERSION-PART=, NIL is
+returned.
+
+A special exemption is made for NIL version specs. If both
+versions are NIL, NIL is returned. Otherwise, if the first spec
+is NIL, T is returned. If the second spec is NIL, NIL is returned.
+
+See ENSURE-VERSIONS-COMPARABLE
+See VERSION-PART=
+See VERSION-PART<")
+
+  (function version<=
+    "Returns T if the first version is considered lower.
+
+First ensures both versions are of equal length, then calls
+VERSION-PART= on each part of the versions. If it returns NIL,
+then VERSION-PART< is called. If this returns T, T is returned,
+otherwise NIL is returned. If all parts are VERSION-PART=, T is
+returned.
+
+A special exemption is made for NIL version specs. If both
+versions are NIL, T is returned. Otherwise, if the first spec
+is NIL, T is returned. If the second spec is NIL, NIL is returned.
+
+See ENSURE-VERSIONS-COMPARABLE
+See VERSION-PART=
+See VERSION-PART<")
+
+  (function version-region
+    "Returns the versions in VERSIONS that are between START and END, inclusive.
+
+For instance, the version list '(1 2 4 5 6) for the bounds 3 and 6
+would return '(4 5 6).
+
+The list of versions is assumed to be sorted by VERSION<.
+
+See VERSION<=")
+
+  (function version-bounds
+    "Returns the versions in VERSIONS that are between START and END, inclusive, ensuring the sequence starts with START and ends with END, if provided.
+
+For instance, the version list '(1 2 4 5 6) for the bounds 3 and 6
+would return '(3 4 5 6).
+
+The list of versions is assumed to be sorted by VERSION<.
+
+See VERSION-REGION")
+
+  (function last-known-system-version
+    "Return the last known version of this system that had been migrated to.
+
+Returns the version as an encoded keyword or NIL if the system
+has not seen a migration previously. This version is automatically
+adapted after MIGRATE-VERSIONS on the system completes
+successfully.
+
+See MIGRATE-VERSIONS")
+
+  (function migrate-versions
+    "Perform a migration of the system from the given source version to the given target version.
+
+If a system or module requires manual intervention to upgrade
+data or other parts in order to move between versions, the author
+of the system should specialise a method on this function that
+performs the requested upgrade step.
+
+FROM and TO should be encoded versions in keyword form. FROM can
+also be the NIL symbol, which is useful to migrate from previously
+unknown versions to another.
+
+Note that the version steps between migrate-version methods on the
+same system should be contiguous. This means that if a system has
+the concrete versions 1, 2, 3, and 4, then there should be methods
+(if necessary) to upgrade from 1 to 2, from 2 to 3, from 3 to 4.
+Migration steps with gaps, such as from 2 to 4, will not be
+triggered by the system.
+
+Also note that by default the list of concrete versions a system
+offers are inferred from the methods defined on this function.
+There is no need to further inform the system on available
+concrete versions of a system.
+
+A default method that performs no action is provided on this
+function, as in the majority of cases no migration step is
+required. As such, methods need only be added to this function if
+an action is required.
+
+Before the primary method is executed, ENSURE-DEPENDENCIES-READY
+is called on the system and the source version. This should
+ensure that dependant systems are on a required version for this
+system to perform its own actions.
+
+After the primary method has completed, the target version is
+recorded as the last known concrete system version.
+
+The user should NOT call this function. It is called by MIGRATE
+as appropriate.
+
+See DEFINE-VERSION-MIGRATION
+See VERSIONS
+See MIGRATE
+See ENSURE-DEPENDENCIES-READY
+See LAST-KNOWN-SYSTEM-VERSION")
+
+  (function define-version-migration
+    "A shorthand to define a version migration method for a system.
+
+SYSTEM may be a string or symbol naming the ASDF system to define
+a migration action for. FROM may be NIL, a symbol, or a string
+naming the source version. TO may be a symbol or a string naming
+the target version.
+
+BODY should be a number of forms to be executed when the system
+is moved from the source version to the target version.
+
+See MIGRATE-VERSIONS")
+
+  (function ready-dependency-for-migration
+    "This function should ensure that the dependency conforms to the system's required state to migrate away from the given version.
+
+By default this will invoke MIGRATE on the dependency with both
+source and from versions being T.
+
+The user should supply methods on this function to customise the
+precise versions or features required to perform the migration of
+the system properly.
+
+See MIGRATE")
+
+  (function ensure-dependencies-ready
+    "This function should ensure that all dependencies of the given system are ready for the system to perform a migration away from the given version.
+
+By default this will call READY-DEPENDENCY-FOR-MIGRATION on all
+systems that are recorded in the system's defsystem-dependencies
+and regular dependencies, AND are virtual-module systems.
+
+The user should supply methods on this function in case it is
+necessary to perform actions on other systems for the migration
+to perform smoothly.
+
+See READY-DEPENDENCY-FOR-MIGRATION
+See ASDF:SYSTEM-DEFSYSTEM-DEPENDS-ON
+See ASDF:SYSTEM-DEPENDS-ON")
+
+  (function versions
+    "Returns a list of concrete version designators that are known for the given system.
+
+This list is deduplicated and sorted in such a way that lower
+versions come earlier in the list.
+
+By default this list is computed by inspecting the list of primary
+methods on MIGRATE-VERSION, extracting the version specifiers, and
+subsequently deduplicating and sorting the resulting list.
+
+The user may supply methods on this function in case the
+automatically computed list of versions is inadequate somehow.
+
+See VERSION<
+See MIGRATE
+See MIGRATE-VERSIONS")
+
+  (function migrate
+    "Migrates the given system between the given versions.
+
+Sometimes when versions change, a migration of runtime data is
+required to ensure that the system still operates correctly on
+existing data.
+
+This function should ensure that this compatibility is achieved.
+
+The system should be a designator for an ASDF system.
+
+FROM may be one of the following:
+  T       --- The system is migrated from its last known version.
+  NIL     --- The system is migrated from a time before migrations.
+  version --- The system is migrated from this specific version.
+
+TO may be one of the following:
+  T       --- The system is migrated to its current system version.
+  version --- The system is migrated to this specific version.
+
+When this function is called it determines the list of concrete
+versions in the range of the specified FROM and TO versions.
+It then calls MIGRATE-VERSION on the system and each pair of
+versions in the computed range. For instance, if the list of
+versions is (1 2 3 4), then MIGRATE-VERSION is called with the
+pairs (1 2) (2 3) (3 4).
+
+If the target version is T and the system has no recorded version,
+an error of type SYSTEM-HAS-NO-VERSION is signalled. If the target
+version is considered less than the source version, an error of
+type BACKWARDS-MIGRATION-NOT-ALLOWED is signalled.
+
+Two restarts are established during migration:
+  ABORT         --- Aborts migration of the current system,
+                    leaving the last known system version the
+                    same.
+  FORCE-VERSION --- Aborts the migration of the current system,
+                    but forces the last known system version to
+                    the requested target version.
+
+See ENSURE-PARSED-VERSION
+See VERSIONS
+See MIGRATE-VERSION
+See SYSTEM-HAS-NO-VERSION
+See BACKWARDS-MIGRATION-NOT-ALLOWED"))
+
 ;; modules.lisp
 (docs:define-docs
   (function module-domain