diff --git a/README.md b/README.md index 9717180..9cc76c9 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ ## Need Applications often need to load seed data into their databases before they can go live. -This data could be inserts, updates (whole record replacement) or updateAttributes (replacement of one or more field data). +This data could be inserts, updates (whole record replacement) or updateAttributes (replacement of one or more field data). Data may also need to be processed and loaded into the DB by application-specific JS files. They may also need to load incremental amounts of data from time to time after go-live, incorporating any structural changes to the tables. @@ -32,25 +32,30 @@ This migration module has the following features: 1. App list module - This module is built as an [app-list module](#bookmark0) which can be added to the application on a need basis. -2. Allows loading of data from [JSON files present in project sub-folder](#bookmark2) to database +2. Allows loading of data from [JSON files present in a project sub-folder](#bookmark2) or any other folder, to database -3. Allows [versioned data migration](#bookmark2), with data-folder name as the DB version. (de-linked from app package version). +3. Allows loading of data from [JSON files present in the project's app-list modules](#bookmark2b), to database -4. Re-running of migration script without adding any new data causes no harm +4. Allows [execution of JS files](#Running Custom JS files) for custom processing and migration to database -5. Migration can be [triggered via a standalone command](#bookmark1) at the command prompt (node server/migratejs) +5. Allows [versioned data migration](#bookmark2), with data-folder name as the DB version. (de-linked from app package version). -6. Allows sequential DDL/DML changes: +6. Re-running of migration script without adding any new data causes no harm + +7. Migration can be [triggered via a standalone command](#bookmark1) at the command prompt (node server/migratejs) + +8. Allows sequential DDL/DML changes: - [Allows writing version-specific meta-data](#bookmark2) (model definitions) to specify structural changes to tables - Auto [populate default data](#bookmark2a) in existing records for a new column -7. Has an [option to download the existing data](#bookmark3) in the DB (arbitrary list of tables, or all tables) as a zip file -8. Allows migration of data over http as a [zip file upload](#bookmark4) -9. Allows [download of migration logs](#bookmark5) over http (logs persisted in MigrationLogs table) -10. Has option to [skip oeCloud Validations](#bookmark6) during migration -11. Has option to [use updateAttributes instead of upsert](#bookmark6) during migration -12. Has option to [clear the data in one or more tables](#bookmark6) before commencing data migration. -13. Has option to [rollback](#bookmark6a) to a previous db version +9. Has an [option to download the existing data](#bookmark3) in the DB (arbitrary list of tables, or all tables) as a zip file +10. Allows migration of data over http as a [zip file upload](#bookmark4) +11. Allows [download of migration logs](#bookmark5) over http (logs persisted in MigrationLogs table) +12. Has option to [skip oeCloud Validations](#bookmark6) during migration +13. Has option to [use updateAttributes instead of upsert](#bookmark6) during migration +14. Has option to [clear the data in one or more tables](#bookmark6) before commencing data migration. +15. Has option to [rollback](#bookmark6a) to a previous db version +16. Allows [execution of custom JS files](#Running Custom JS files) pre- and/or post- migration. @@ -113,7 +118,8 @@ The code snippets below show how steps 1 and 2 can be done: ## Usage This module can be used for the following purposes: -- Migration from Command-Line +- Migration of application seed data from Command-Line +- Migration of app-list module's seed data from Command-Line - Downloading zip file of DB data - Uploading zip file for migration @@ -136,10 +142,12 @@ Once the above changes are done to the application, the migration can be done as | |-default- | | |-Customer.json | | |-Account.json + | | |-custom-script-1.js | | | |-tenant1- | | |-Customer.json | | |-Account.json + | | |-custom-script-2.js | | | |-meta.json | | @@ -154,6 +162,7 @@ Once the above changes are done to the application, the migration can be done as | | | |-tenant1- | | |-Account.json + | | |-custom-script-3.js | | | |-meta.json | | @@ -167,6 +176,7 @@ Once the above changes are done to the application, the migration can be done as | |-tenant1- | |-Account.json + | |-custom-script-4.js | |-meta.json @@ -197,7 +207,7 @@ then the records that existed in the table prior to the current db version migra ``` This file creation is a one-time activity, and the file itself can be part of your application. - **Note:** This file does not pass the `options` parameter to the `migrate()` function. However, it is possible to configure some aspects of migration if you pass the appropriate `options` object. See [**migrate() function**](#migrate function) under the [**Configuration**](#Configuration) section below, for details. + **Note:** This sample file does not pass the `options` parameter to the `migrate()` function. However, it is possible to configure some aspects of migration if you pass the appropriate `options` object. See [**migrate() function**](#migrate function) under the [**Configuration**](#Configuration) section below, for details. 3. From a command prompt at the root of your application, run the following: @@ -247,32 +257,110 @@ An example of MigrationLogs is shown below: The `server/migrate.js` can be executed repeatedly with/without additional data in the `/db` folder. - -### Running Custom JS files -In cases where migration needs additional complex logic to be executed, you can wrap the `migrate` call in custom javascript module callback. -```javascript + +### Migration of app-list module seed data + +*oe-migration* supports the migration of seed-data from *app-list* modules by leveraging the `options.basePath` parameter of the `migrate()` function. + +For this to work, + +1. Each *app-list* module should have its own seed-data in a folder named `db` at the module's root. +2. Create and run a new js file, say, *migrate-all.js* in your `/server/` folder with the following contents: + + ```javascript + var app = require('oe-cloud'); - var preMigrate = require('some/path/pre-migrate.js'); - var postMigrate = require('some/path/post-migrate.js'); + var async = require('async'); + var m = require('oe-migration'); + var path = require('path'); + var fs = require('fs'); + app.boot(__dirname, function (err) { - if (err) { console.log(err); process.exit(1); } - - var m = require('oe-migration'); - preMigrate('arguments', function(err, data) { - if(err) process.exit(1); - m.migrate(function(err, oldDbVersion, migratedVersions) { - if(err) process.exit(1); - postMigrate('arguments', function(err, data){ - if(err) process.exit(1); else process.exit(0); - } - }); + if (err) { console.log(err); process.exit(1); } + + var appList = getAppList(); + + async.eachOfSeries(appList, function (mdl, key, cb) { + m.migrate({ moduleName: mdl.moduleName, basePath: mdl.basePath }, function (err, oldDbVersion, migratedVersions) { + cb(err); + }); + }, function (err) { + if (err) { + console.log(err); + process.exit(1); + } else { + console.log('migration done'); + process.exit(0); } + }); }); + + + function getAppList() { + var appList = []; + var mdls = require('../server/app-list.json'); + mdls.forEach(function (o) { + var bPath; + if (o.path === './') { + bPath = path.resolve(process.cwd(), 'db'); + } else { + bPath = path.resolve(process.cwd(), 'node_modules', o.path, 'db'); + } + var isDir = false; + try { + isDir = fs.statSync(bPath).isDirectory(); + if (isDir === true) appList.push({ moduleName: o.path, basePath: bPath}); + } catch (e) { + if (e) console.log('Ignoring module', o.path, ' : No db folder'); + } + }); + return appList; + } + ``` + This file creation is a one-time activity, and the file itself can be part of your application. + +**Notes:** + +1. The above script, when run, does the migration of data in the `db` folders from each *app-list* module, in the order in which the module is specified in the application's `app-list.json` file. +2. The above script also migrates the main application's `db` folder, again, as per the order of the `./` module specified in `app-list.json` +3. The application developer is free to modify the sample migrate-all.js file to meet the application's needs. For example, the developer could choose not to migrate data from one or more app-list modules, change the location from where data is migrated, etc., +4. While modifying the *migrate-all.js* script, keep in mind the `moduleName` option of `migrate()` function. This is used by *oe-migration* as a key to maintain the latest DB migration version in the *SystemConfig* table. This should be set as the name of the module whose data is being migrated. If the module is the main application itself, `moduleName` can be omitted or set to the default value, `./` + + +### Running Custom JS files + +*oe-migration* supports running arbitrary JS files in addition to loading data from json files. These files are to be placed and configured similar to how this is done for json files, in `meta.json`. See [Configuration](#Configuration) for details. +JS files are run by *oe-migration* in the order it appears in `meta.json`. + +The JS files that should be run as part of migration need to use the following standard: + +1. The JS file/script needs to export a single function +2. The exported function needs to have the following 2 arguments - + + a) opts - This Object would contain the context as defined in `meta.json` + b) cb - This is a callback function that needs to be called from within the script to signal the end of processing in the script. +3. The callback function may be called with an error object. This will halt the migration. +4. Failure to call `cb()` would cause the migration process to wait indefinitely. + +An example JS file is shown below: + +```javascript + +module.exports = function(opts, cb) { + console.log(opts); // contains the ctx as specified in meta.json + // do processing + // more processing + + cb(err); // err should be undefined or null if all is well. + // Otherwise migration is halted. + +} + ``` -You must ensure the rerunnability of your custom javascript code for a rerunnable migration. + @@ -353,7 +441,7 @@ For e.g., one project could have the following files for each of the db versions /db/2.0.0/meta.json ``` -Each of these `meta.json` files define the contexts and data file details for its corresponding DB version migration. In addition, the `meta.json` can also optionally configure - +Each of these `meta.json` files define the contexts and data file/JS script details for its corresponding DB version migration. In addition, the `meta.json` can also optionally configure - - whether all tables or specified tables are cleared or not before migration - whether oeCloud validations are skipped for migration or not - whether the json data is to be used to do an updateAttributes instead of an upsert or not @@ -386,23 +474,23 @@ The structure of the `meta.json` file along with these configuration parameters, "files": [ { - "skipValidation": true, // Optional property, value is boolean. Default: false + "skipValidation": true, // Optional property, value is boolean. Default: false. Ignored if file is JS script // If true, skips oeCloud validation during migration of this file - "updateAttributes": true, // Optional property, value is boolean. Default: false + "updateAttributes": true, // Optional property, value is boolean. Default: false. Ignored if file is JS script // If true, does an updateAttributes instead of an upsert for this file. // If this is set to true, the json data needs to have an 'id' field for each // record. Alternatively, the 'key' property (see below) needs to be specified. - "key": "field2", // Optional property. Value is a string fieldname. Used in conjunction with + "key": "field2", // Optional property. Ignored if file is JS script. Value is a string fieldname. Used in conjunction with // "updateAttributes" (see above) Specifies a unique field other than 'id' to be // used as PK for performing updateAttributes using data in this file. - "model": "Customer", // Mandatory property. The model name to use for this file's migration + "model": "Customer", // Mandatory property if file is json. The model name to use for this json file's migration. Ignored if file is JS script "enabled": true, // Optional property. If false, skips migration from this file. Default: true - "file": "default/customer.json", // Mandatory property. Relative path under "db" folder to the json file's location + "file": "default/customer.js(on)", // Mandatory property. Relative path under "db" folder to the json/JS file's location "ctxId": "/default" // Mandatory property. Should match one of the keys under "contexts" }, @@ -432,13 +520,20 @@ The `options` object can have the following properties, illustrated with an exam ```js { + basePath: '/data/db', // Optional String. Specifies the absolute path to a folder to be used as the "db" folder. + // Default value is the "db" folder at the root of the oeCloud application. + + moduleName: 'my-app', // Optional String. Specifies the name of the module/app for which this migration is done. + // Default value is "./", the same string that is specified for "current application" in app-list.json. + // This is used by oe-migration as a key to maintain the latest DB migration version in the SystemConfig table. + force: true, // Optional. If true, repeats migration from all db versions present in "db" folder, // ignoring the last version that was migrated (last version being in SystemConfig table) fromVersion: '2.0.0', // Optional. When used with force: true, sets the starting db version for // forced migration. Default: '0.0.0' - toVersion: '4.0.0' // Optional. When used with force: true, sets the ending db version for + toVersion: '4.0.0', // Optional. When used with force: true, sets the ending db version for // forced migration. Default: last available version in "db" folder verbose: true // Optional. Prints additional info to the console during migration, for diff --git a/lib/migration.js b/lib/migration.js index 3eafcef..1f55afe 100644 --- a/lib/migration.js +++ b/lib/migration.js @@ -10,6 +10,7 @@ var opts = { var options; var MigrationLog; var basePath = path.resolve(process.cwd(), typeof global.it !== 'function' ? '' : 'test', 'db'); +var moduleName = './'; var validationFunctions = {}; var startTime; var endTime; @@ -30,6 +31,8 @@ function migrate(mOptions, mCb) { mCb = mOptions; mOptions = {}; } + basePath = (mOptions && mOptions.basePath) || basePath; + moduleName = (mOptions && mOptions.moduleName) || './'; var migrationCb = function (err, oldDbVersion, migratedVersions) { endTime = new Date().getTime(); // eslint-disable-next-line no-console @@ -93,8 +96,9 @@ function migrate(mOptions, mCb) { }); } else { var SystemConfig = loopback.findModel('SystemConfig'); + var versionKey = 'dbVersion.' + moduleName; SystemConfig.create({ - 'key': 'dbVersion', + 'key': versionKey, 'value': lastVersion }, opts, function (err, data) { /* istanbul ignore if */ @@ -299,30 +303,32 @@ function migrateFromPath(migrationPath, cb) { } function proceedWithMigration() { - var model = loopback.findModel(value.model); - if (!model) { - msg = value.model + ' model not found in application'; - mLog = {'logType': 'FATAL', 'model': value.model, 'dbVersion': dbVersion, 'filePath': filePath, - 'migrationDate': new Date(), 'tenant': value.ctxId, 'log': {'message': msg}}; - err = new Error(msg); - err.mLog = mLog; - return asyncCb(err); - } - if (value.skipValidation === true) { - // eslint-disable-next-line no-console - console.log('Skipping validations for model ' + value.model + ' (tenant ' + value.ctxId + ')'); - /* istanbul ignore else */ - if (!validationFunctions[value.model]) { - validationFunctions[value.model] = model.prototype.isValid; - model.prototype.isValid = function (done) { - done(true); - }; + if (filePath.endsWith('json')) { + var model = loopback.findModel(value.model); + if (!model) { + msg = value.model + ' model not found in application'; + mLog = {'logType': 'FATAL', 'model': value.model, 'dbVersion': dbVersion, 'filePath': filePath, + 'migrationDate': new Date(), 'tenant': value.ctxId, 'log': {'message': msg}}; + err = new Error(msg); + err.mLog = mLog; + return asyncCb(err); + } + if (value.skipValidation === true) { + // eslint-disable-next-line no-console + console.log('Skipping validations for model ' + value.model + ' (tenant ' + value.ctxId + ')'); + /* istanbul ignore else */ + if (!validationFunctions[value.model]) { + validationFunctions[value.model] = model.prototype.isValid; + model.prototype.isValid = function (done) { + done(true); + }; + } + } else if (validationFunctions[value.model]) { + model.prototype.isValid = validationFunctions[value.model]; + delete validationFunctions[value.model]; + // eslint-disable-next-line no-console + console.log('Restoring validation for model ' + value.model + ' (tenant ' + value.ctxId + ')'); } - } else if (validationFunctions[value.model]) { - model.prototype.isValid = validationFunctions[value.model]; - delete validationFunctions[value.model]; - // eslint-disable-next-line no-console - console.log('Restoring validation for model ' + value.model + ' (tenant ' + value.ctxId + ')'); } var dataList; try { @@ -339,6 +345,24 @@ function migrateFromPath(migrationPath, cb) { if (Object.prototype.toString.call(dataList) === '[object Object]') { dataList = [dataList]; } + if (Object.prototype.toString.call(dataList) === '[object Function]') { + dataList(opts, function (err) { + if (err) { + // eslint-disable-next-line no-console + console.error(err.message || err); + logMigration({'logType': 'ERROR', 'model': null, 'dbVersion': dbVersion, 'filePath': filePath, + 'migrationDate': new Date(), 'tenant': value.ctxId, 'log': {'message': err.message}}); + return asyncCb(err); + } + // eslint-disable-next-line no-console + console.log('Executed script ' + filePath + ' for tenant ' + value.ctxId); + + logMigration({'logType': 'INFO', 'model': null, 'dbVersion': dbVersion, 'filePath': filePath, + 'migrationDate': new Date(), 'tenant': value.ctxId, 'log': {'message': {filePath: filePath}}}); + return asyncCb(); + }); + return; + } var success = 0; var failed = 0; async.eachSeries(dataList, function (data, asyncCb2) { @@ -473,6 +497,9 @@ function clearTable(table, cb) { function getListOfMigrationPaths(options, cb) { + basePath = (options && options.basePath) || basePath; + moduleName = (options && options.moduleName) || './'; + var versionKey = 'dbVersion.' + moduleName; var pathErrors = []; if (!(fs.existsSync(basePath) && fs.lstatSync(basePath).isDirectory())) { return cb(new Error('Path ' + basePath + ' does not exist. Not migrating.'), null); @@ -480,7 +507,7 @@ function getListOfMigrationPaths(options, cb) { var SystemConfig = loopback.findModel('SystemConfig'); SystemConfig.findOne({ where: { - key: 'dbVersion' + key: versionKey } }, opts, function (err, dbVersionInstance) { var fromVersion; @@ -531,12 +558,6 @@ function getListOfMigrationPaths(options, cb) { migrationDirs: retList, dbVersionInstance: dbVersionInstance }); - } else if (retList.length === 0) { - return cb(new Error('No (new) directories matched for migration. Nothing migrated.'), { - migratedVersions: migDirs, - migrationDirs: retList, - dbVersionInstance: dbVersionInstance - }); } cb(null, { migratedVersions: migDirs, diff --git a/package.json b/package.json index 28a0e0f..de8d432 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "oe-migration", - "version": "2.2.0", + "version": "2.3.0", "description": "This module is an app-list module for oe-Cloud based applications. It provides the ability to perform data migration and/or data seeding into the app db from json files", "engines": { "node": ">=6" diff --git a/test/db1/25.0.1/default/migration-test1.js b/test/db1/25.0.1/default/migration-test1.js new file mode 100644 index 0000000..a491dd3 --- /dev/null +++ b/test/db1/25.0.1/default/migration-test1.js @@ -0,0 +1,4 @@ +module.exports = function(opts, cb) { + console.log(opts); + cb(); +} diff --git a/test/db1/25.0.1/default/migration-test2.js b/test/db1/25.0.1/default/migration-test2.js new file mode 100644 index 0000000..a491dd3 --- /dev/null +++ b/test/db1/25.0.1/default/migration-test2.js @@ -0,0 +1,4 @@ +module.exports = function(opts, cb) { + console.log(opts); + cb(); +} diff --git a/test/db1/25.0.1/meta.json b/test/db1/25.0.1/meta.json new file mode 100644 index 0000000..8839792 --- /dev/null +++ b/test/db1/25.0.1/meta.json @@ -0,0 +1,20 @@ +{ + "contexts": { + "/default": { + "id": 0, + "tenantId": "/default", + "remoteUser": "defaultuser" + } + }, + "files": [{ + "enabled": true, + "file": "default/migration-test1.js", + "ctxId": "/default" + }, + { + "enabled": true, + "file": "default/migration-test2.js", + "ctxId": "/default" + } + ] +} \ No newline at end of file diff --git a/test/test.js b/test/test.js index 451a639..a89bc5e 100644 --- a/test/test.js +++ b/test/test.js @@ -118,9 +118,7 @@ describe(chalk.blue('oe-migration tests'), function (done) { // This function deletes all records in the MasterJobExecutorTestData table function clearTestData(cb) { var TAG = 'clearTestData:'; - SystemConfig.remove({ - key: 'dbVersion' - }, opts, function findCb(err, res) { + SystemConfig.remove({ }, opts, function findCb(err, res) { if (err) { console.error(TAG, 'Could not remove dbVersion record from SystemConfig ' + JSON.stringify(err)); cb(err); @@ -240,10 +238,9 @@ describe(chalk.blue('oe-migration tests'), function (done) { console.log(chalk.yellow('[' + new Date().toISOString() + '] : ', 'Starting ' + TAG)); var options = {}; migration.migrate(options, function (err, oldDbVersion, data) { - expect(err).not.to.be.null; + expect(err).to.be.null; expect(oldDbVersion).to.equal('2.5.0'); expect(data.migratedVersions).to.be.null; - expect(err.message).to.equal('No (new) directories matched for migration. Nothing migrated.'); done(); }); }); @@ -314,10 +311,9 @@ describe(chalk.blue('oe-migration tests'), function (done) { console.log(chalk.yellow('[' + new Date().toISOString() + '] : ', 'Starting ' + TAG)); var options = {}; migration.migrate(options, function (err, oldDbVersion, data) { - expect(err).not.to.be.null; + expect(err).to.be.null; expect(data.migratedVersions).to.be.null; expect(oldDbVersion).to.equal('2.5.0'); - expect(err.message).to.equal('No (new) directories matched for migration. Nothing migrated.'); done(); }); }); @@ -620,8 +616,7 @@ describe(chalk.blue('oe-migration tests'), function (done) { console.log(chalk.yellow('[' + new Date().toISOString() + '] : ', 'Starting ' + TAG)); migration.setBasePath(null); // Setting it back to default migration.migrate(function (err, oldDbVersion, data) { - expect(err).not.to.be.null; - expect(err.message).to.equal('No (new) directories matched for migration. Nothing migrated.'); + expect(err).to.be.null; expect(oldDbVersion).to.equal('13.0.0'); expect(data.migratedVersions).to.be.null; done(); @@ -1112,6 +1107,24 @@ describe(chalk.blue('oe-migration tests'), function (done) { }); }); + + it('should successfuly perform migration with js files', function (done) { + var TAG = '[should successfuly perform migration with js files]'; + this.timeout(10000); + console.log(chalk.yellow('[' + new Date().toISOString() + '] : ', 'Starting ' + TAG)); + var options = {verbose: true}; + + migration.setBasePath(path.resolve(process.cwd(), 'test', 'db')); + copyRecursiveSync(path.join(process.cwd(), 'test', 'db1', '25.0.1'), path.join(basePath, '25.0.1')); + + migration.migrate(options, function (err, oldDbVersion, data) { + expect(oldDbVersion).to.equal('25.0.0'); + expect(data.migratedVersions).to.eql(['25.0.1']); + done(); + }); + }); + + it('The new data should have the default values populated', function (done) { var TAG = '[The new data should have the default values populated]'; this.timeout(1000000); @@ -1150,7 +1163,7 @@ describe(chalk.blue('oe-migration tests'), function (done) { expect(err).to.be.defined; expect(err.message).to.contain('Model MigrationTest4 specified in'); expect(err.message).to.contain('does not exist'); - expect(oldDbVersion).to.equal('25.0.0'); + expect(oldDbVersion).to.equal('25.0.1'); expect(data.migratedVersions).to.be.null; done(); });