Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added support for local dynamodb #3

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions .git-commit-template.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# <type>: (If applied, this commit will...) <subject> (Max 50 char)
# |<---- Using a Maximum Of 50 Characters ---->|


# Explain why this change is being made
# |<---- Try To Limit Each Line to a Maximum Of 72 Characters ---->|

# Provide links or keys to any relevant tickets, articles or other resources
# Example: Github issue #23

# --- COMMIT END ---
# Type can be
# feat (new feature)
# fix (bug fix)
# refactor (refactoring production code)
# style (formatting, missing semi colons, etc; no code change)
# docs (changes to documentation)
# test (adding or refactoring tests; no production code change)
# chore (updating grunt tasks etc; no production code change)
# --------------------
# Remember to
# Capitalize the subject line
# Use the imperative mood in the subject line
# Do not end the subject line with a period
# Separate subject from body with a blank line
# Use the body to explain what and why vs. how
# Can use multiple lines with "-" for bullet points in body
# --------------------
# For more information about this template, check out
# https://gist.github.com/adeekshith/cd4c95a064977cdc6c50
31 changes: 18 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,15 @@ There's no blueprint in AWS Lambda that allows to dump DynamoDB data into Elasti

---

### Installation
## Installation

```sh
$ npm install dynamodb-to-elasticsearch
```

## [Guide to configure AWS to use this blueprint](https://aws.amazon.com/blogs/compute/indexing-amazon-dynamodb-content-with-amazon-elasticsearch-service-using-aws-lambda)

### Documentation
## Documentation

`module.exec (table, region, es_endpoint, es_data = { id: 'sortKey', type: 'datatype', indiceName: ''})`

Expand All @@ -31,24 +31,29 @@ $ npm install dynamodb-to-elasticsearch
| es_data.type | object | Name of class of objects, which document represents. By default it is set to `datatype`.
| es_data.indiceName | object | Index name of elastic-search on which you can perform query.

### Example
### Working locally

When working on a Dynamodb running on localhost, make sure the `IS_OFFLINE` environment varible is set to `true`.
The default port will be `8000` but it can be overriden by `LOCAL_DYNAMODB_PORT` environment variable.

## Example

```javascript
const d2es = require('dynamodb-to-elasticsearch');

const table = 'table',
region = 'region',
es_endpoint = 'es_endpoint_value',
es_data = { id: 'sortKey', type: 'data', indiceName: 'candidates' }};
region = 'region',
es_endpoint = 'es_endpoint_value',
es_data = { id: 'sortKey', type: 'data', indiceName: 'candidates' }};

exports.handler = function(event, context, callback) {
d2es.exec(table, region, es_endpoint, es_data, (err, success) => {
if (err) {
callback(err, null);
} else {
callback(null, success);
}
});
d2es.exec(table, region, es_endpoint, es_data, (err, success) => {
if (err) {
callback(err, null);
} else {
callback(null, success);
}
});
}

```
Expand Down
8 changes: 7 additions & 1 deletion index.js
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,13 @@ const AWS = require('aws-sdk'),

exports.exec = function (table, region, es_endpoint, es_data = { id: 'sortKey', type: 'datatype' }, callback) {
if (!es_data.indiceName) throw new Error('You should provide es_data.indiceName')
const docClient = new AWS.DynamoDB.DocumentClient({region: region})

const docClient = (process.env.IS_OFFLINE) ? new AWS.DynamoDB.DocumentClient({
region: 'localhost',
endpoint: `http://localhost:${process.env.LOCAL_DYNAMODB_PORT || 8000}`,
}) :
new AWS.DynamoDB.DocumentClient({region: region});


// Promise - Describe the ES Domain in order to get the endpoint url
when.promise(function (resolve) {
Expand Down
196 changes: 196 additions & 0 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

24 changes: 14 additions & 10 deletions package.json
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
{
"name": "dynamodb-to-elasticsearch",
"version": "0.2.1",
"description": "Module to dump DynamoDB data to ElasticSearch indices.",
"name": "@cheftonic/dynamodb-to-elasticsearch-offline",
"version": "0.3.1",
"description": "Module to dump DynamoDB data to ElasticSearch indexes.",
"repository": {
"type": "git",
"url": "https://github.com/sohamdodia/dynamodb-to-elasticsearch"
"url": "git+https://github.com/Cheftonic/dynamodb-to-elasticsearch.git"
},
"main": "index.js",
"scripts": {
Expand All @@ -18,13 +18,17 @@
"elasticsearch",
"elastic-search"
],
"author": "Soham Dodia",
"author": "Carlos Delgado",
"license": "MIT",
"dependencies": {
"aws-sdk": "^2.122.0",
"elasticsearch": "^13.3.1",
"http-aws-es": "^3.1.0",
"lodash": "^4.17.4",
"aws-sdk": "^2.403.0",
"elasticsearch": "^15.4.1",
"http-aws-es": "^6.0.0",
"lodash": "^4.17.11",
"when": "^3.7.8"
}
},
"bugs": {
"url": "https://github.com/Cheftonic/dynamodb-to-elasticsearch/issues"
},
"homepage": "https://github.com/Cheftonic/dynamodb-to-elasticsearch#readme"
}