AWS DynamoDB output plugin for Embulk

Embulk output plugin to load/insert data into AWS DynamoDB

Overview

• Plugin type: output
• Load all or nothing: no
• Resume supported: no
• Cleanup supported: no

Configuration

General Parameters

• mode (string optional, default:upsert)

  • upsert

    Creates a new item, or replaces an old item with a new item. If an item that has the same primary key as the new item already exists in the specified table, the new item completely replaces the existing item. This mode uses BatchWriteItem and you can set max_put_items parameter.

  • upsert_with_expression

    Edits an existing item's attributes, or adds a new item to the table if it does not already exist. If you have data with following schema, you can set update_expression as follows.

    - col1(Number primaryKey)
    - col2(Number)
    - col3(String)
    - col4(Number)
    - col5(String)
    

    mode: upsert_with_expression
    # update where col1 has #col1
    update_expression: "add #col2 :col2 set #col3 = :col3 set #col4 = #col4 - :col4 remove #col5"

• region (string required) Region of Amazon DynamoDB

• auto_create_table (boolean optional, default: false) See below

• table (string required) Table name of Amazon DynamoDB

  table and option accept Time#strftime format to construct table name. Table name is formatted at runtime using the local time of the embulk server.

  For example, with the configuration below, data is inserted into tables table_2015_04, table_2015_05 and so on.

  out:
    type: dynamodb
    table: table_%Y_%m
  

• write_capacity_units (int optional) Provisioned write capacity units

  • normal (int optional) value that will be set after execution
  • raise (int optional) value that will be set before execution

• read_capacity_units (int optional) Provisioned read capacity units

  • normal (int optional) value that will be set after execution

  • raise (int optional) value that will be set before execution

    • Up to 30 before execution and down to 5 after execution.

    write_capacity_units:
    normal: 5
    raise: 30

    • Up to 30 before execution

    write_capacity_units:
    raise: 30

    • Down to 5 after execution

    write_capacity_units:
    normal: 5

    NOTICE: You can decrease the read_capacity_units or write_capacity_units settings for a table, but no more than 4 times per table in a single UTC calendar day.

• max_put_items (int optional, default:25) Max put items at every batch writing requests when upsert mode. A single call to BatchWriteItem can write up to 16 MB of data, which can comprise as many as 25 put requests. If you have exceeded data, decrease this value.

Authentication Parameters

6 authentication methods are supported.

• basic

  Use access_key_id and secret_access_key

  auth_method: basic
  access_key_id: YOUR_ACCESS_KEY_ID
  secret_access_key: YOUR_SECRET_ACCESS_KEY

  • access_key_id: AWS access key ID (string, required)

  • secret_access_key: AWS secret access key (string, required)

• env

  Read AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY) from environment variables.

  auth_method: env

• instance

  Use EC2 instance profile

  auth_method: instance

• profile

  Read credentials from file. Format of the file is as following, where [...] is a name of profile.

  auth_method: profile
  profile_file: /path/to/profilefile
  profile_name: PROFILE_NAME

  • profile_file: path to a profiles file. (string, default: given by AWS_CREDENTIAL_PROFILES_FILE environment varialbe, or ~/.aws/credentials).

  • profile_name: name of a profile. (string, default: "default")

  [default]
  aws_access_key_id=YOUR_ACCESS_KEY_ID
  aws_secret_access_key=YOUR_SECRET_ACCESS_KEY
  
  [profile2]
  ...
  

• properties

  Read aws.accessKeyId and aws.secretKey from Java system properties

  auth_method: properties

• session

  Use temporary-generated access_key_id, secret_access_key and session_token.

  auth_method: session
  access_key_id: YOUR_ACCESS_KEY_ID
  secret_access_key: YOUR_SECRET_ACCESS_KEY
  session_token: SESSION_TOKEN

  • access_key_id: AWS access key ID (string, required)

  • secret_access_key: AWS secret access key (string, required)

  • session_token: session token (string, required)

Example

out:
  type: dynamodb
  mode: upsert
  region: us-west-2
  auth_method: basic
  access_key_id: ABCXYZ123ABCXYZ123
  secret_access_key: ABCXYZ123ABCXYZ123
  auto_create_table: true
  table: dynamotable
  write_capacity_units:
    normal: 5
    raise: 20
  read_capacity_units:
    normal: 6
    raise: 30

Build

$ ./gradlew gem  # -t to watch change of files and rebuild continuously

Test

$ ./gradlew test  # -t to watch change of files and rebuild continuously

To run unit tests, you need to configure the following environment variables.

When environment variables are not set, skip some test cases.

DYNAMO_REGION
DYNAMO_TABLE
DYNAMO_ACCESS_KEY_ID
DYNAMO_SECRET_ACCESS_KEY

If you're using Mac OS X El Capitan and GUI Applications(IDE), like as follows.

$ vi ~/Library/LaunchAgents/environment.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>my.startup</string>
  <key>ProgramArguments</key>
  <array>
    <string>sh</string>
    <string>-c</string>
    <string>
      launchctl setenv DYNAMO_REGION us-west-1
      launchctl setenv DYNAMO_TABLE dynamotest
      launchctl setenv DYNAMO_ACCESS_KEY_ID ABCXYZ123ABCXYZ123
      launchctl setenv DYNAMO_SECRET_ACCESS_KEY ABCXYZ123ABCXYZ123
    </string>
  </array>
  <key>RunAtLoad</key>
  <true/>
</dict>
</plist>

$ launchctl load ~/Library/LaunchAgents/environment.plist
$ launchctl getenv DYNAMO_REGION //try to get value.

Then start your applications.