local-data-api - Local Data API for AWS Aurora Serverless Data API

If you want to run tests on your local machine and CI then, local-data-api can run in your local machine with MySQL and PostgreSQL Servers.

What's AWS Aurora Serverless's Data API?

https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/data-api.html

How does local-data-api work?

local-data-api is "proxy server" to real databases.

The API converts RESTful request to SQL statements.

Support Database Types

• MySQL
• PostgreSQL

How to insert array data into PostgreSQL

DataAPI have not support inserting array data with SqlParameter yet. But, @ormu5 give us this workaround to insert array data.

insert into cfg.attributes(id, type, attributes)
values(:id, :type, cast(:attributes as text[]));

where the value for attributes parameter is a string properly formatted as Postgres array, e.g., '{"Volume","Material"}'.

Thanks to @ormu5.

Version 0.6.0

local-data-api has been re-written in Kotlin

Motivation

This project was written in Python at the start.

But, We need a JDBC driver that needs Java to reproduce real Data API behavior.

I have re-written local-data-api in Kotlin. The design is the same as the old version.

Also, I wrote unittest and made coverage 100%. We welcome your feedback and error reports.

Related issue: #70

How to use this image

You set your MYSQL Server configs as environments.

docker run --rm -it --name my-data-api -p 8080:80  -e MYSQL_HOST=<YOUR_MYSQL_HOST> -e MYSQL_PORT=<YOUR_MYSQL_PORT> -e MYSQL_USER=<YOUR_MYSQL_USER> -e MYSQL_PASSWORD=<YOUR_MYSQL_PASS>  -e RESOURCE_ARN=arn:aws:rds:us-east-1:123456789012:cluster:dummy -e SECRET_ARN=arn:aws:secretsmanager:us-east-1:123456789012:secret:dummy  koxudaxi/local-data-api

In this case, you give local-data-api URL to aws client (like aws-cli).

$ aws --endpoint-url http://127.0.0.1:8080 rds-data execute-statement --resource-arn "arn:aws:rds:us-east-1:123456789012:cluster:dummy" --sql "show databases"  --secret-arn "arn:aws:secretsmanager:us-east-1:123456789012:secret:dummy" --database 'test'

docker-compose

MySQL

docker-compose-mysql.yml

version: '3.1'

services:
  local-data-api:
    image: koxudaxi/local-data-api
    restart: always
    environment:
      MYSQL_HOST: db
      MYSQL_PORT: 3306
      MYSQL_USER: root
      MYSQL_PASSWORD: example
      RESOURCE_ARN: 'arn:aws:rds:us-east-1:123456789012:cluster:dummy'
      SECRET_ARN: 'arn:aws:secretsmanager:us-east-1:123456789012:secret:dummy'
    ports:
      - "8080:80"
  db:
    image: mysql:5.6
    command: --default-authentication-plugin=mysql_native_password
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: example
      MYSQL_DATABASE: test
    ports:
        - "3306:3306"

docker-compose with Python's aws-sdk client(boto3)

1. start local-data-api containers

$ docker-compose up -d

2. change a endpoint to local-data-api in your code.

$ ipython

In [1]: import boto3; client = boto3.client('rds-data', endpoint_url='http://127.0.0.1:8080', aws_access_key_id='aaa',  aws_secret_access_key='bbb') 

3. execute a sql statement

In [2]: client.execute_statement(resourceArn='arn:aws:rds:us-east-1:123456789012:cluster:dummy', secretArn='arn:aws:secretsmanager:us-east-1:123456789012:secret:dummy', sql='show databases', database='test')

If you use PostgreSQL, then you should run this line to check databases.

In [2]: client.execute_statement(resourceArn='arn:aws:rds:us-east-1:123456789012:cluster:dummy', secretArn='arn:aws:secretsmanager:us-east-1:123456789012:secret:dummy', sql='SELECT datname FROM pg_database', database='test')

4. local-data-api return the result from a MySQL Server.

Out[2]: {'ResponseMetadata': {'HTTPStatusCode': 200,
 'HTTPHeaders': {'date': 'Sun, 09 Jun 2019 18:35:22 GMT',
 'server': 'uvicorn',
 'content-length': '492',
 'content-type': 'application/json'},
 'RetryAttempts': 0},
 'numberOfRecordsUpdated': 0,
 'records': [[{'stringValue': 'information_schema'}],
  [{'stringValue': 'mysql'}],
  [{'stringValue': 'performance_schema'}],
  [{'stringValue': 'sys'}],
  [{'stringValue': 'test'}]]}

If a table has some records, then the local-data-api can run select

In [3]: client.execute_statement(resourceArn='arn:aws:rds:us-east-1:123456789012:cluster:dummy', secretArn='arn:aws:secretsmanager:us-east-1:123456789012:secret:dummy', sql='select * from users', database='test')

Out[3]: {'ResponseMetadata': {'HTTPStatusCode': 200,
 'HTTPHeaders': {'date': 'Sun, 09 Jun 2019 18:35:22 GMT',
 'server': 'uvicorn',
 'content-length': '492',
 'content-type': 'application/json'},
 'RetryAttempts': 0},
 'numberOfRecordsUpdated': 0,
 'records': [[{'longValue': 1}, {'stringValue': 'ichiro'}, {'longValue': 17}],
  [{'longValue': 2}, {'stringValue': 'ken'}, {'longValue': 20}],
  [{'longValue': 3}, {'stringValue': 'lisa'}, {'isNull': True}],}

PostgreSQL

Now, local-data-api supports PostgreSQL

docker-compose-postgres.yml

version: '3.1'

services:
  local-data-api:
    image: koxudaxi/local-data-api
    restart: always
    environment:
      ENGINE: PostgreSQLJDBC
      POSTGRES_HOST: db
      POSTGRES_PORT: 5432
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: example
      RESOURCE_ARN: 'arn:aws:rds:us-east-1:123456789012:cluster:dummy'
      SECRET_ARN: 'arn:aws:secretsmanager:us-east-1:123456789012:secret:dummy'
    ports:
      - "8080:80"
  db:
    image: postgres:10.7-alpine
    restart: always
    environment:
      POSTGRES_PASSWORD: example
      POSTGRES_DB: test
    ports:
        - "5432:5432"

Contribute

We are waiting for your contributions to local-data-api.

How to contribute

## 1. Clone your fork repository
$ git clone [email protected]:<your username>/local-data-api.git
$ cd local-data-api

## 2. Open the project with IDE/Editor
The project path is `./kotlin/local-data-api`

## 3. Run on your local machine
### Shell on Linux or MacOS 
$ cd ./kotlin/local-data-api
$ ./gradlew run

### Command Prompt on Windows
$ cd ./kotlin/local-data-api
$ gradlew.bat run

## 4. Create new branch and rewrite code.
$ git checkout -b new-branch

## 5. Run unittest
$ cd ./kotlin/local-data-api
$ ./gradlew test jacocoTestReport

## 6. Run integration-test
$ ./scripts/integration-test.sh

## 7. Commit and Push...

Related projects

py-data-api

DataAPI client for Python

https://github.com/koxudaxi/py-data-api

Docker Image

https://hub.docker.com/r/koxudaxi/local-data-api

Source Code

https://github.com/koxudaxi/local-data-api

Documentation

https://koxudaxi.github.io/local-data-api

License

local-data-api is released under the MIT License. http://www.opensource.org/licenses/mit-license