Skip to content

Commit

Permalink
Merge branch '2.3.0' of http://10.73.97.24/oecloud.io/oe-migration
Browse files Browse the repository at this point in the history
  • Loading branch information
vamsee committed Oct 6, 2020
2 parents 33b02ae + f0ee3f6 commit abf25b8
Show file tree
Hide file tree
Showing 7 changed files with 239 additions and 82 deletions.
175 changes: 135 additions & 40 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
<a name="Need"></a>
## Need
Applications often need to load seed data into their databases before they can go live.
This data could be inserts, updates (whole record replacement) or updateAttributes (replacement of one or more field data).
This data could be inserts, updates (whole record replacement) or updateAttributes (replacement of one or more field data). Data may also need to be processed and loaded into the DB by application-specific JS files.

They may also need to load incremental amounts of data from time to time after go-live, incorporating any structural changes to the tables.

Expand All @@ -32,25 +32,30 @@ This migration module has the following features:

1. App list module - This module is built as an [app-list module](#bookmark0) which can be added to the application on a need basis.

2. Allows loading of data from [JSON files present in project sub-folder](#bookmark2) to database
2. Allows loading of data from [JSON files present in a project sub-folder](#bookmark2) or any other folder, to database

3. Allows [versioned data migration](#bookmark2), with data-folder name as the DB version. (de-linked from app package version).
3. Allows loading of data from [JSON files present in the project's app-list modules](#bookmark2b), to database

4. Re-running of migration script without adding any new data causes no harm
4. Allows [execution of JS files](#Running Custom JS files) for custom processing and migration to database

5. Migration can be [triggered via a standalone command](#bookmark1) at the command prompt (node server/migratejs)
5. Allows [versioned data migration](#bookmark2), with data-folder name as the DB version. (de-linked from app package version).

6. Allows sequential DDL/DML changes:
6. Re-running of migration script without adding any new data causes no harm

7. Migration can be [triggered via a standalone command](#bookmark1) at the command prompt (node server/migratejs)

8. Allows sequential DDL/DML changes:
- [Allows writing version-specific meta-data](#bookmark2) (model definitions) to specify structural changes to tables
- Auto [populate default data](#bookmark2a) in existing records for a new column

7. Has an [option to download the existing data](#bookmark3) in the DB (arbitrary list of tables, or all tables) as a zip file
8. Allows migration of data over http as a [zip file upload](#bookmark4)
9. Allows [download of migration logs](#bookmark5) over http (logs persisted in MigrationLogs table)
10. Has option to [skip oeCloud Validations](#bookmark6) during migration
11. Has option to [use updateAttributes instead of upsert](#bookmark6) during migration
12. Has option to [clear the data in one or more tables](#bookmark6) before commencing data migration.
13. Has option to [rollback](#bookmark6a) to a previous db version
9. Has an [option to download the existing data](#bookmark3) in the DB (arbitrary list of tables, or all tables) as a zip file
10. Allows migration of data over http as a [zip file upload](#bookmark4)
11. Allows [download of migration logs](#bookmark5) over http (logs persisted in MigrationLogs table)
12. Has option to [skip oeCloud Validations](#bookmark6) during migration
13. Has option to [use updateAttributes instead of upsert](#bookmark6) during migration
14. Has option to [clear the data in one or more tables](#bookmark6) before commencing data migration.
15. Has option to [rollback](#bookmark6a) to a previous db version
16. Allows [execution of custom JS files](#Running Custom JS files) pre- and/or post- migration.

<a name="bookmark0"></a>
<a name="Implementation"></a>
Expand Down Expand Up @@ -113,7 +118,8 @@ The code snippets below show how steps 1 and 2 can be done:
## Usage
This module can be used for the following purposes:

- Migration from Command-Line
- Migration of application seed data from Command-Line
- Migration of app-list module's seed data from Command-Line
- Downloading zip file of DB data
- Uploading zip file for migration

Expand All @@ -136,10 +142,12 @@ Once the above changes are done to the application, the migration can be done as
| |-default-
| | |-Customer.json
| | |-Account.json
| | |-custom-script-1.js
| |
| |-tenant1-
| | |-Customer.json
| | |-Account.json
| | |-custom-script-2.js
| |
| |-meta.json
| |
Expand All @@ -154,6 +162,7 @@ Once the above changes are done to the application, the migration can be done as
| |
| |-tenant1-
| | |-Account.json
| | |-custom-script-3.js
| |
| |-meta.json
| |
Expand All @@ -167,6 +176,7 @@ Once the above changes are done to the application, the migration can be done as
|
|-tenant1-
| |-Account.json
| |-custom-script-4.js
|
|-meta.json
Expand Down Expand Up @@ -197,7 +207,7 @@ then the records that existed in the table prior to the current db version migra
```
This file creation is a one-time activity, and the file itself can be part of your application.

**Note:** This file does not pass the `options` parameter to the `migrate()` function. However, it is possible to configure some aspects of migration if you pass the appropriate `options` object. See [**migrate() function**](#migrate function) under the [**Configuration**](#Configuration) section below, for details.
**Note:** This sample file does not pass the `options` parameter to the `migrate()` function. However, it is possible to configure some aspects of migration if you pass the appropriate `options` object. See [**migrate() function**](#migrate function) under the [**Configuration**](#Configuration) section below, for details.
<a name="bookmark1"></a>
3. From a command prompt at the root of your application, run the following:

Expand Down Expand Up @@ -247,32 +257,110 @@ An example of MigrationLogs is shown below:
The `server/migrate.js` can be executed repeatedly with/without additional data in the `<server>/db` folder.
<a name="Running Custom JS files"></a>
### Running Custom JS files
In cases where migration needs additional complex logic to be executed, you can wrap the `migrate` call in custom javascript module callback.
```javascript
<a name="bookmark2b"></a>
### Migration of app-list module seed data
*oe-migration* supports the migration of seed-data from *app-list* modules by leveraging the `options.basePath` parameter of the `migrate()` function.
For this to work,
1. Each *app-list* module should have its own seed-data in a folder named `db` at the module's root.
2. Create and run a new js file, say, *migrate-all.js* in your `<PROJECT_ROOT>/server/` folder with the following contents:
```javascript
var app = require('oe-cloud');
var preMigrate = require('some/path/pre-migrate.js');
var postMigrate = require('some/path/post-migrate.js');
var async = require('async');
var m = require('oe-migration');
var path = require('path');
var fs = require('fs');
app.boot(__dirname, function (err) {
if (err) { console.log(err); process.exit(1); }

var m = require('oe-migration');
preMigrate('arguments', function(err, data) {
if(err) process.exit(1);
m.migrate(function(err, oldDbVersion, migratedVersions) {
if(err) process.exit(1);
postMigrate('arguments', function(err, data){
if(err) process.exit(1); else process.exit(0);
}
});
if (err) { console.log(err); process.exit(1); }
var appList = getAppList();
async.eachOfSeries(appList, function (mdl, key, cb) {
m.migrate({ moduleName: mdl.moduleName, basePath: mdl.basePath }, function (err, oldDbVersion, migratedVersions) {
cb(err);
});
}, function (err) {
if (err) {
console.log(err);
process.exit(1);
} else {
console.log('migration done');
process.exit(0);
}
});
});
function getAppList() {
var appList = [];
var mdls = require('../server/app-list.json');
mdls.forEach(function (o) {
var bPath;
if (o.path === './') {
bPath = path.resolve(process.cwd(), 'db');
} else {
bPath = path.resolve(process.cwd(), 'node_modules', o.path, 'db');
}
var isDir = false;
try {
isDir = fs.statSync(bPath).isDirectory();
if (isDir === true) appList.push({ moduleName: o.path, basePath: bPath});
} catch (e) {
if (e) console.log('Ignoring module', o.path, ' : No db folder');
}
});
return appList;
}
```
This file creation is a one-time activity, and the file itself can be part of your application.
**Notes:**
1. The above script, when run, does the migration of data in the `db` folders from each *app-list* module, in the order in which the module is specified in the application's `app-list.json` file.
2. The above script also migrates the main application's `db` folder, again, as per the order of the `./` module specified in `app-list.json`
3. The application developer is free to modify the sample migrate-all.js file to meet the application's needs. For example, the developer could choose not to migrate data from one or more app-list modules, change the location from where data is migrated, etc.,
4. While modifying the *migrate-all.js* script, keep in mind the `moduleName` option of `migrate()` function. This is used by *oe-migration* as a key to maintain the latest DB migration version in the *SystemConfig* table. This should be set as the name of the module whose data is being migrated. If the module is the main application itself, `moduleName` can be omitted or set to the default value, `./`
<a name="Running Custom JS files"></a>
### Running Custom JS files
*oe-migration* supports running arbitrary JS files in addition to loading data from json files. These files are to be placed and configured similar to how this is done for json files, in `meta.json`. See [Configuration](#Configuration) for details.
JS files are run by *oe-migration* in the order it appears in `meta.json`.
The JS files that should be run as part of migration need to use the following standard:
1. The JS file/script needs to export a single function
2. The exported function needs to have the following 2 arguments -
a) opts - This Object would contain the context as defined in `meta.json`
b) cb - This is a callback function that needs to be called from within the script to signal the end of processing in the script.
3. The callback function may be called with an error object. This will halt the migration.
4. Failure to call `cb()` would cause the migration process to wait indefinitely.
An example JS file is shown below:
```javascript
module.exports = function(opts, cb) {
console.log(opts); // contains the ctx as specified in meta.json
// do processing
// more processing
cb(err); // err should be undefined or null if all is well.
// Otherwise migration is halted.
}
```
You must ensure the rerunnability of your custom javascript code for a rerunnable migration.
<a name="bookmark3"></a>
<a name="Downloading zip file of DB data"></a>
Expand Down Expand Up @@ -353,7 +441,7 @@ For e.g., one project could have the following files for each of the db versions
<PROJECT_DIR>/db/2.0.0/meta.json
```
<a name="bookmark6"></a>
Each of these `meta.json` files define the contexts and data file details for its corresponding DB version migration. In addition, the `meta.json` can also optionally configure -
Each of these `meta.json` files define the contexts and data file/JS script details for its corresponding DB version migration. In addition, the `meta.json` can also optionally configure -
- whether all tables or specified tables are cleared or not before migration
- whether oeCloud validations are skipped for migration or not
- whether the json data is to be used to do an updateAttributes instead of an upsert or not
Expand Down Expand Up @@ -386,23 +474,23 @@ The structure of the `meta.json` file along with these configuration parameters,
"files": [
{
"skipValidation": true, // Optional property, value is boolean. Default: false
"skipValidation": true, // Optional property, value is boolean. Default: false. Ignored if file is JS script
// If true, skips oeCloud validation during migration of this file
"updateAttributes": true, // Optional property, value is boolean. Default: false
"updateAttributes": true, // Optional property, value is boolean. Default: false. Ignored if file is JS script
// If true, does an updateAttributes instead of an upsert for this file.
// If this is set to true, the json data needs to have an 'id' field for each
// record. Alternatively, the 'key' property (see below) needs to be specified.
"key": "field2", // Optional property. Value is a string fieldname. Used in conjunction with
"key": "field2", // Optional property. Ignored if file is JS script. Value is a string fieldname. Used in conjunction with
// "updateAttributes" (see above) Specifies a unique field other than 'id' to be
// used as PK for performing updateAttributes using data in this file.
"model": "Customer", // Mandatory property. The model name to use for this file's migration
"model": "Customer", // Mandatory property if file is json. The model name to use for this json file's migration. Ignored if file is JS script
"enabled": true, // Optional property. If false, skips migration from this file. Default: true
"file": "default/customer.json", // Mandatory property. Relative path under "db" folder to the json file's location
"file": "default/customer.js(on)", // Mandatory property. Relative path under "db" folder to the json/JS file's location
"ctxId": "/default" // Mandatory property. Should match one of the keys under "contexts"
},
Expand Down Expand Up @@ -432,13 +520,20 @@ The `options` object can have the following properties, illustrated with an exam
```js
{
basePath: '/data/db', // Optional String. Specifies the absolute path to a folder to be used as the "db" folder.
// Default value is the "db" folder at the root of the oeCloud application.
moduleName: 'my-app', // Optional String. Specifies the name of the module/app for which this migration is done.
// Default value is "./", the same string that is specified for "current application" in app-list.json.
// This is used by oe-migration as a key to maintain the latest DB migration version in the SystemConfig table.
force: true, // Optional. If true, repeats migration from all db versions present in "db" folder,
// ignoring the last version that was migrated (last version being in SystemConfig table)
fromVersion: '2.0.0', // Optional. When used with force: true, sets the starting db version for
// forced migration. Default: '0.0.0'
toVersion: '4.0.0' // Optional. When used with force: true, sets the ending db version for
toVersion: '4.0.0', // Optional. When used with force: true, sets the ending db version for
// forced migration. Default: last available version in "db" folder
verbose: true // Optional. Prints additional info to the console during migration, for
Expand Down
Loading

0 comments on commit abf25b8

Please sign in to comment.