Synchronize client/server data using your existing RESTful APIs.
- Under MIT license.
- Database agnostic and schema-less.
- Works against different [database types][database-definition-link] (SQL, NoSQL, Graph, etc.) and service ([MySQL][mysql-home-link], [SQL Server][sql-server-home-link], [Oracle][oracle-home-link], [Postgress][postgress-home-link], [MongoDB][mongodb-home-link], [Redis][redis-home-link], etc.).
- Supports multiple databases sources.
- No need to propagate schema changes to a client.
- Reconciliation
- Automatic reconciliation of offline changes.
- In
clientOnly
mode, RESToff automatically provides a complete RESTful offline service for your app. - Reconciliation works without soft deletes or last modified date columns.
- Note: reconciliation is much faster with soft delete and last modified date columns available.
- Conflicts are resolved by default by saving both versions. Setting concurrency to 'overwrite' changes this behavior to optimistic, last-one-wins strategy where merge conflicts are overwritten
- Note: reconciliation is much faster with soft delete and last modified date columns available.
- Supports synchronous RESTful calls.
- Simplifies source code and improves client responsiveness.
- Supports resources in the following formats:
- Json
- Works in following frameworks
- Standalone - RESToff can be used without a framework.
- [Angular][angular-home-link] - RESToff wrapped in an Angular 1 provider.
- Limitations and Expectations:
- Every resource must have one unique
primaryKey
field. - Best to follow [RESTful best known practices][rest-best-practices]
- Every resource must have one unique
- TODO
- Tests to verify order of pending changes
- edit record A, edit A again: 2nd edit also contains 1st edit
- Tests to verify order of pending changes
- Next features
- Support rolling back a pending change
- Replace concurrency option with a plugin model
- This also will allow support for custom merge conflict resolution
- support non-standard get/put/post.
- Example: a request GET actually does a delete.
- support put/post updates where resource is changed on the server.
- requires better mockable REST api backend for testing.
- support nested resources (example: /users/45/addresses).
- support non-standard restful api: ability to map a user.
Examples:
// Create a restoff service instance.
var roff = restoff({
rootUri : "http://api.example.com/" // Required Option
});
roff.get("users").then(function(user) {
console.log(roff.getRepo("users")); // Access repository directly
});
roff.deleteOne("users",{id:301378d5}).then(function(deletedUserId) {
console.log("User " + deletedUserId + " deleted");
});
var aUser = {
"id" : "ffa454",
"first_name": "Happy",
"last_name": "User"
}
roff.post("users", aUser).then(function(user){
console.log("Posted user %O.", user);
});
aUser.last_name = "put";
roff.put("users/ffa454", aUser).then(function(user){
console.log("Put user %O.", user);
});
Let's synchronize subsets of resources on the backend service using the user id:
var roff = restoff({
"rootUri" : "http://api.example.com/"
});
function synchronize(roff, userId) {
roff.get("users/" + userId);
roff.get("books?ownedby="+userId);
roff.get("addresses?userid="+userId);
...
roff.forcedOffline = true; // Let's go offline or unplug from the internet
roff.get("users/" + userId).then(function(user) {
// Yep! We can work offline.
});
}
Row | Action | 1) Server Only Change | 2) Client Only Change | 3) Changes in Both |
---|---|---|---|---|
A | Insert (Post) | Get and Overwrite on Client | Post and Overwrite on Server | Same Primary Key. Apply B3. |
B | Update (Post/Put) | Get and Overwrite on Client | Post and Overwrite on Server | Brent Reconciliation |
C | Delete | Delete from Client | Delete on Server | Nothing to Do: Both deleted |
D | Delete with Update on client | Honor Delete: ignore client update |
- A1, A2, B1, B2, C1, C2, C3: No potential merge conflicts
- A3: This is the case where the same primary key was generated on each client. In this case we will simply view it as a reconciliation of changes because:
- You should be using UUIDs for keys and the chance of collisions is very low. If you are using incrementing IDs, then you will need an algorithm to generate these IDs uniquely on the client. See the
generateId
function.
- You should be using UUIDs for keys and the chance of collisions is very low. If you are using incrementing IDs, then you will need an algorithm to generate these IDs uniquely on the client. See the
- D3: The client/server deleted a record that was updated on the server/client. In this case, we honor the delete.
- B3: Using Brent reconciliation which means we assign a new resource id (primaryKey) to the resource and post it.
- This does not work with hierarchical data. The intent is to allow a user to take advantage of the existing User Interface of an Application to deal with the merge conflict.
- TODO: We will be providing a call back that lets the developer provide a custom merge conflict screen.
rootUri
is appended to RESTful calls when an incomplete uri
is provided. rootUri
is also used, under the hood, to determine the name of the repository.
Examples:
var roff = restoff({
"rootUri" : "http://api.example.com/",
});
// uri becomes http://api.example.com/users
// repository name is users
roff.get("users").then(function(result){
// use the result here
}
// uri is http://another.example.com/users
// repository name is users
roff.get("http://another.example.com/users").then(function(result){
// use the result here
}
When true, storage of RESTful results on the client are disabled.
- Effectivly bypass the core feature of RESToff causing it to act like a standard http library.
- Useful for debugging.
Example configuration:
var roff = restoff({
rootUri : "http://api.example.com/",
persistenceDisabled : true
});
Creates a backend RESTful service on your client.
- Allows your RESTful API to run 100% on the client.
- Note: By design, the
pending
repository runs inclientOnly
mode meaning the pending RESTful endpoint never hits the backend service.
var rest = restoff({
"rootUri" : "http://api.example.com/",
"clientOnly" : true
});
rest.put("users", { Name : "Happy User"}).then(function(result) {
rest.get("users").then(function(result)) {
// never hit a backend RESTful service
});
});
onCallPending
is the function called by RESToff before a pending call is executed. Provided is all information about the pending action and the uri record.
// TODO: Provide the json format for pendingAction and the uri record.
var rest = restoff({
rootUri: "http://api.example.com/",
onCallPending: function(pendingAction, uri) {
console.log("Pending will execute %O %O", pendingAction, uri);
}
});
primaryKeyName
is the name of the primary key field for a given repository.
- RESToff requires every repository to have a single primary key
- the primary key can't be based on a composite key
- The default value is 'id'.
You can globally set the primaryKeyName
or on a per RESTful call basis:
var roff = restlib.restoff({
rootUri: "api.example.com",
dbService: {
primaryKeyName : "ID",
},
});
roff.get("users").then(function(result) {
console.log("Primary key name used was ID.");
});
roff.get("users", {primaryKeyName:"USER_ID"}).then(function(result) {
console.log("Primary key name used was USER_ID.");
});
primaryKeyNotOnPost
is a boolean indication that we should not send a primary key on post. Some servers
create a primary key on post and send it back via response.
Forces the application to run in offline mode.
- Useful to see how an application behaves when it is offline.
- Example: Reload all information, go offline, and see how the application behaves.
isOnline
will return false whenforcedOffline
is true.
var roff = restoff({
rootUri: "http://api.example.com/"
});
synchronize(roff, "user456");
roff.forceOffline();
if (!roff.isOnline) {
console.log ("We are offline!");
}
NOTE: There is a slight difference between clientOnly
mode and forcedoffline
mode.
- In
forcedOffline
mode, RESToff will store any put/post/delete actions in thepending
repository. - In
clientOnly
mode, Retsoff will not store any put/post/delete actions in thepending
repository.
generateId
is the function called by RESToff to generate a primary key. Bey default, restOff uses RestOff.prototype._guidGenerate()
but you can define your own.
var rest = restoff({
rootUri: "http://api.example.com/",
generateId: function() { return Math.floor((1 + Math.random()) * 0x10000); } // not a great unique key generator
});
onReconciliation
is the function called by RESToff after reconciliation of a given resource is complete.
var rest = restoff({
rootUri: "http://api.example.com/",
onReconciliation: function(completedAction) {
console.log("The following completed action was reconciled and saved to the server %O.", completedAction);
}
});
Note: Placed features will provide a more robust reconciliation process allowing the developer to provide their own custom reconciliation process. Currently, RESToff always applies Brent Reconciliation.
The concurrency
option defaults to null and conflicts are resolved through Brent reconciliation (keep both versions)
Setting concurrency = 'overwrite'
will result in conflicts resolved by being overwritten by the local version
When offline, RESToff places any put/post/delete RESTful operations in a pending
repository. RESToff uses it's own persistence engine meaning it calls itself using get/post/delete.
Use pendingUri
and pendingRepoName
to configure the URI and repository name of the pending
repository.
- Note: In
clientOnly
mode, no changes are recorded in thepending
repository.
The resourceFilter
option defaults to null, when set to an function we can manipulate the received resourcedata before writing it to the lowDB. Systems like Backendless.comm return paged REST results to reduce long waits.
//Example for BackendLess.com ,plain functions removes all internal objects from the code
var rest = restoff({
rootUri: "http://api.example.com/",
resourceFilter: function(resources,uri){
if (resources.hasOwnProperty('totalObjects') && resources.hasOwnProperty('data')) {
resources=resources.data;
for (var i=0;i<resources.length;i++) resources[i]=self.plain(resources[i]);
} else {
resources = self.plain(resources);
}
return resources;
}
});
The errorHandler
option defaults to null, when set to a function it will we called allowing you to intercept the error, before
it is passed to the reject part of promise.
var rest = restoff({
rootUri: "http://api.example.com/",
errorHandler: function(error) {
/* error layout
error = {
message: message,
messageDetail: messageDetail,
status: request.status,
response:request.response || request.responseText,
uri: uri.uriFinal,
restMethod: uri.restMethod
};
*/
}
});
The pageHandler
option defaults to null, when set to a function it will we called allowing you to handle paged results form your backend system. The url returned by this functions will be loaded and added to the result set.
//Example for BackendLess.com
var rest = restoff({
rootUri: "http://api.example.com/",
pageHandler : function(resources,uri) {
if (resources.hasOwnProperty('totalObjects') && resources.hasOwnProperty('data') && resources.nextPage ) {
var p = uri.uriFinal.indexOf('?');
var ret = (p>0 ? uri.uriFinal.substr(0,p) : uri.uriFinal) + "?" + resources.nextPage.substr( resources.nextPage.indexOf('?')+1) ;
return ret;
}
return false;
}
});
The defaultParams
option defaults to null, but when set an object, the key values will be used as parameters in all
rest calls. See also autoQueryParamSet
The defaultHeader
option defaults to null, but when set an object, the key values will be used as header parameters in all
rest calls. See also autoHeaderParamSet
A parameter of name
with value
will be added/appended to EVERY RESTful api call. Useful for adding parameters such as an access token.
Example usage:
var roff = restoff({
rootUri : "http://api.example.com/"
});
roff.autoQueryParamSet("access_token", "rj5aabcea");
// uri becomes http://api.example.com/users?access_token=rj5aabcea
roff.get("users").then(function(user)) {
...
}
Returns the value of the query parameter with the provided name
.
Example usage:
var roff = restoff({
rootUri : "http://api.example.com/"
});
roff.autoQueryParamSet("access_token", "rj5aabcea");
var paramValue = roff.autoQueryParamGet("access_token");
TODO: Add autoQueryParameters as an option.
A header of name
with value
will be added to the header of every RESTful api call. Useful for adding parameters such as an access token.
Example usage:
var roff = restoff({
rootUri : "http://api.example.com/"
});
roff.autoHeaderParamSet("access_token", "rj5aabcea");
Returns the value of the header parameter with the provided name
.
Example usage:
var roff = restoff({
rootUri : "http://api.example.com/"
});
roff.autoHeaderParamSet("access_token", "rj5aabcea");
var headerValue = roff.autoHeaderParamGet("access_token");
TODO: Add autoHeaderParams as an option.
Clears the cache of the given repository name if the repository exists.
- Does not clear the repository if there are pending changes.
- Optional parameter
forced
- Pass a value oftrue
to force clearing a repository even if it has pending changes.
- Optional parameter
- Does not create a repository named
repoName
to the repository if it doesn't exist.
Example:
var roff = restoff({
rootUri : "http://api.example.com/"
});
roff.clear("users", true);
console.log("User repository, even if it had pending, was cleared ")
Clears the cache of all repositories.
- Does not clear any repositories if even one of them have pending changes.
- Optional parameter
forced
- Pass a value oftrue
to force clearing all repositories even if there are any pending changes.
- Optional parameter
Example:
var roff = restoff({
rootUri : "http://api.example.com/"
});
roff.clearAll(true);
console.log("User repository, even if it had pending, was cleared ");
delete(uri, [options])
asynchronously deletes a resource from a remote server and in the local repository.deleteRepo(uri, [options])
synchronously deletes a resource from in the local repository.deleteOne(uri, resource, [options])
asynchronously deletes a resource from a remote server using primarykey for uri extention and in the local repository.
Note:
- A 404 (not found) is "ignored" and the resource is still considered removed from the local repository.
- If
delete
is called on a non-existent repository, an empty repository is created.
Example usage:
var roff = restoff();
return roff.delete("http://test.development.com:4050/users/553fdf")
.then(function(result){
// user was deleted
});
roff.forcedOffline = true;
var result = roff.deleteRepo("http://test.development.com:4050/users/553fdf");
get(uri, [options])
asynchronously retrieves a resource from a remote server. Uses the local repository when offline.getRepo(uri, [options])
synchronously retrieves a resource from the local repository.getOne(uri, resource, [options])
asynchronously retrieves one resource using primarykey for uri extention from a remote server. Uses the local repository when offline.
Example usage:
var roff = restoff();
return roff.get("http://test.development.com:4050/testsweb/testdata/users")
.then(function(result){
// use the result here
});
roff.forcedOffline = true;
var result = roff.getRepo("http://test.development.com:4050/testsweb/testdata/users");
var user = {
"id" : "ffa454",
"first_name": "Happy",
"last_name": "User"
}
return roff.getOne("http://test.development.com:4050/testsweb/testdata/users",user)
.then(function(result){
// use the result of single get here
});
post(uri, resource, [options]), postRepo(uri, resource, [options]), postOne(uri, resource, [options])
-
post(uri, resource, [options])
asynchronously posts a resource to a remote server and in the local repository adding the resource if it doesn't exist or overwriting the existing resource. -
postRepo(uri, resource, [options])
synchronously posts a resource in the local repository adding the resource if it doesn't exist or overwriting the existing resource. -
postOne(uri, resource, [options])
asynchronously posts a resource to a remote server using the primarykey for uri extention and in the local repository adding the resource if it doesn't exist or overwriting the existing resource. -
When online,
post(...)
calls will happen immediately. -
With
postRepo(...)
, updates will happen when aget(...)
is executed on that resource.
Example usage:
var roff = restoff();
var newUser = {
"id" : "ffa454",
"first_name": "Happy",
"last_name": "User"
}
return roff.post("http://test.development.com:4050/users", newUser)
.then(function(result){
// use the result here
});
roff.forcedOffline = true;
var result = roff.postRepo("http://test.development.com:4050/users", newUser)
return roff.postOne("http://test.development.com:4050/testsweb/testdata/users",newUser)
.then(function(result){
// use the result of single post here
});
put(uri, resource, [options])
asynchronously puts a known resource on a remote server and in the local repository updating the resource id provided in the uri.putRepo(uri, resource, [options])
synchronously puts a known resource in the local repository updating the resource id provided in the uri.put(uri, resource, [options])
asynchronously puts a known resource on a remote server using the primarykey for uri extention and in the local repository updating the resource id provided in the uri.
Example usage:
var roff = restoff({
"rootUri" : "http://test.development.com:4050/"
});
var existingUser = {
"id" : "ffa454",
"first_name": "Happy",
"last_name": "User"
}
return roff.put("users/ffa454", existingUser)
.then(function(result){
// use the result here
});
roff.forcedOffline = true;
var result = roff.putRepo("users/ffa454", existingUser)
return roff.putOne("http://test.development.com:4050/testsweb/testdata/users",existingUser)
.then(function(result){
// use the result here
});
restOff may add additional query parameters when restOff.get(uri) is called. Use this method to see the final uri sent to the backend server.
Example usage:
var roff = restoff().autoQueryParamSet("access_token", "rj5aabcea");
var actualUri = roff.uriFromClient("http://test.development.com:4050/emailaddresses");
expect(actualUri)).to.equal("http://test.development.com:4050/emailaddresses?access_token=rj5aabcea");
you may need a clone of the restOff with add other config. config parts of orginal are copied to clone.
Example usage:
var roff = restoff({rootUri:'http://test.development.com:4050'});
var cloneOff = roff.clone({rootUri:'http://test.production.com:4050'});
- What is the difference between
offlineOnly
andforcedOffline
?forcedOffline
will line up any pending changes that are then synchronized when the client comes back online.offlineOnly
does not log any "pending changes" because the repository will never be synchronzied with a backend service.
Want to help out? Here is what you need to do to get started.
// TODO: Finish this part of documentation
Change directory to your node project.
$ npm install --save restoff
$ npm install
$ npm install -g json-server // for testing
$ npm outdated
- [jsonplaceholder][jsonplaceholder-link]
See ./dist for files we build.
$ gulp
$ gulp webtests
Read documentation in gulpfile.js to see how to setup automated web testing.
$ gulp webserver
First time?
$ npm adduser # Need to do one time
-
Verifty package.json version is newer than one in npm (visit https://www.npmjs.com/package/restoff)
-
Verify no pending changes
$ git status
$ npm publish ./
$ git tag -a 0.1.4 -m "v0.1.4" // 0.1.4 is an example
$ git push origin --tags
- Increment version in package.json and update check-build test. Re-run tests.
- Install LiveReload Chrome extension.
Add to your /etc/hosts file:
127.0.0.1 test.development.com
$ npm install
(NPM install stuff ommitted)
$ mocha tests
3 passing (or something like this)
$ gulp
(gulp stuff happens, look for this line...)
Server started http://test.development.com:4050
... then open the test suite in your browser http://test.development.com:4050
RESToff is wraped in an angular provider.
Example Usage:
angular.module("fakeRoot", ["restoff"])
.config(["restoffProvider", function (restoffProvider) {
restoffProvider.setConfig({
rootUri:"http://localhost/"
});
}]);
When using angular you should add a resourceFilter to remove $$hashKey added by angular if you want to keep your database clean. Example config for Angular, using the backendless.com backend servivce
var self = restoff.clone({
rootUri : $common.getENV('backendURL')
,pageHandler : function(resources,uri) {
if (resources.hasOwnProperty('totalObjects') && resources.hasOwnProperty('data') && resources.nextPage ) {
var p = uri.uriFinal.indexOf('?');
var ret = (p>0 ? uri.uriFinal.substr(0,p) : uri.uriFinal) + "?" + resources.nextPage.substr( resources.nextPage.indexOf('?')+1) ;
return ret;
}
return false;
}
,errorHandler : function(error){
// error = { message,messageDetail,status,response,uri,restMethod};
try {
var resp = JSON.parse(response);
if (resp.code==3064) {
console.log('Got an invalid user response');
self.logout();
$rootScope.$broadcast('baas.invaliduser',resp);
}
} catch (ex){}
}
,resourceFilter: function(resources,uri){
if (resources.hasOwnProperty('totalObjects') && resources.hasOwnProperty('data')) {
resources=resources.data;
for (var i=0;i<resources.length;i++) resources[i]=self.plain(resources[i]);
} else {
resources = self.plain(resources);
}
return resources;
}
,dbService: {
primaryKeyName : "objectId"
,primaryKeyNotOnPost: true
,dbName: $common.getENV('appName','restoff') +".json"
,repoOptions: []
,reconSettings: {
lastUpdatedFieldName: "updated"
, softDeleteFieldName: ""
, softDeleteValue: ""
}
,format: {
deserialize: function(str){
try {
return JSON.parse(CryptoJS.AES.decrypt(str, $rootScope.secretKey || 'undefined').toString(CryptoJS.enc.Utf8));
} catch (ex) {
try {return JSON.parse(str);} catch (exx){return {}}
}
},
serialize: function (obj) {
return CryptoJS.AES.encrypt(JSON.stringify(obj),$rootScope.secretKey || 'undefined');
}
}
}
,onReconciliation: function(completedAction) {
console.log("The following completed action was reconciled and saved to the server "+ completedAction);
}
// ,concurrency : 'overwrite'
// ,persistenceDisabled : true
});
self.plain = function (data){
if (!data) return data;
var obj;
try {
if (data instanceof Array) {
obj =[];
angular.forEach(data,function(value){
this.push(self.plain(value));
},obj);
} else {
if (typeof data == "string") {
data=JSON.parse(data);
}
obj = {};
angular.forEach(data,function(value,key){
if (key.indexOf('__')!=0 && key.indexOf('$$hashKey')!=0) {
if (typeof value === 'object') this[key]=self.plain(value);
else this[key]=value;
}
},obj);
}
return obj;
} catch (ex) {
//console.log("Failed to convert to plain ",JSON.stringify(data),ex.toString());
return data;
}
}
```javascript
Note that we "hard code" the configuration, but you could also get the configuration from another service.
## Development Issues
### Live Reload Isn't Reloading
* In chrome, navigate to ```chrome://extensions/``` then find the LiveReload extension and check Allow access to file URLs.
* Click on livereload icon in chrome browser: small circle in center should become solid.
* Is more than one instance of gulp running?
### Hide HTTP Network Messages During Testing
* In chrome, right mouse click and inspect.
* From the console tab, click on filter (icon next to Top)
* Check "Hide network messages"
[rest-best-practices]: https://codeplanet.io/principles-good-restful-api-design/
[jsonplaceholder-link]: http://jsonplaceholder.typicode.com/
[angular-home-link]: https://angularjs.org/
[database-definition-link]:https://en.wikipedia.org/wiki/Database
[mysql-home-link]: https://www.mysql.com/
[sql-server-home-link]: https://www.microsoft.com/en-us/server-cloud/products/sql-server/features.aspx
[oracle-home-link]: https://www.oracle.com/index.html
[postgress-home-link]: https://www.postgresql.org
[mongodb-home-link]: https://www.mongodb.org/
[redis-home-link]: http://redis.io