diff --git a/CHANGES.md b/CHANGES.md index 7105262..10de89b 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -1,7 +1,7 @@ # Changelog ## in progress -- Started unlocking AWS Kinesis stream sources +- Carabas: A subsystem to divert workloads to other people’s computers ## 2024-07-10 v0.0.2 - Initial working version, supporting MQTT, AMQP, and SQLAlchemy/CrateDB diff --git a/doc/carabas/dms/index.md b/doc/carabas/dms/index.md index f253a48..7420bbe 100644 --- a/doc/carabas/dms/index.md +++ b/doc/carabas/dms/index.md @@ -26,7 +26,7 @@ help. ### Install Install LorryStream. ```shell -pip install lorrystream +pip install --upgrade 'lorrystream[carabas]' ``` Acquire IaC driver program. ```shell diff --git a/doc/carabas/kcl/dynamodb.md b/doc/carabas/kcl/dynamodb.md index 6575b4e..589bc16 100644 --- a/doc/carabas/kcl/dynamodb.md +++ b/doc/carabas/kcl/dynamodb.md @@ -121,15 +121,16 @@ aws kinesis delete-stream --stream-name dynamodb-cdc --enforce-consumer-deletion Acquire sources and initialize sandbox. ```shell -git clone https://github.com/daq-tools/lorrystream --branch=kinesis +git clone https://github.com/daq-tools/lorrystream cd lorrystream python3 -m venv .venv source .venv/bin/activate +pip install --editable='.[carabas]' ``` Install dependencies, mainly the [amazon-kclpy] package. ```shell -cd lorrystream/dynamodb_cloud +cd lorrystream/spike/kcl_dynamodb pip install wheel pip install --verbose -r requirements.txt ``` @@ -141,7 +142,7 @@ virtualenv on the top-level directory. Then, navigate to the playground directory, and seed AWS credentials. ```shell source .venv/bin/activate -cd lorrystream/dynamodb_cloud +cd lorrystream/spike/kcl_dynamodb export AWS_ACCESS_KEY=... export AWS_SECRET_ACCESS_KEY=... ``` diff --git a/doc/carabas/kcl/kinesis.md b/doc/carabas/kcl/kinesis.md index 567f595..1a03a0f 100644 --- a/doc/carabas/kcl/kinesis.md +++ b/doc/carabas/kcl/kinesis.md @@ -34,15 +34,16 @@ permissions to do so. Acquire sources and initialize sandbox. ```shell -git clone https://github.com/daq-tools/lorrystream --branch=kinesis +git clone https://github.com/daq-tools/lorrystream cd lorrystream python3 -m venv .venv source .venv/bin/activate +pip install --editable='.[carabas]' ``` Install dependencies, mainly the [amazon-kclpy] package. ```shell -cd lorrystream/kinesis +cd lorrystream/spike/kcl_kinesis pip install wheel pip install --verbose -r requirements.txt ``` @@ -65,7 +66,7 @@ virtualenv on the top-level directory. Then, navigate to the playground directory, and seed AWS credentials. ```shell source .venv/bin/activate -cd lorrystream/kinesis +cd lorrystream/spike/kcl_kinesis export AWS_ACCESS_KEY=... export AWS_SECRET_ACCESS_KEY=... ``` diff --git a/doc/source/kinesis.md b/doc/source/kinesis.md index 6933778..75d997b 100644 --- a/doc/source/kinesis.md +++ b/doc/source/kinesis.md @@ -1,9 +1,19 @@ # Kinesis Source -## LocalStack Testbed -The recipe uses the LocalStack AWS environment to run an Amazon Kinesis surrogate. +This recipe uses the LocalStack AWS environment to run an Amazon Kinesis surrogate. The walkthrough follows the [Get started with Kinesis on LocalStack] tutorial. +If you intend to invoke the commands on a real AWS environment, just use `aws` +instead of `awslocal`. + +:::{tip} +LocalStack is a cloud service emulator that runs in a single container on your +laptop or in your CI environment. With LocalStack, you can run your AWS +applications or Lambdas entirely on your local machine without connecting to +a remote cloud provider. +::: + +## Setup Start the LocalStack service using Docker. ```shell docker run \ @@ -13,17 +23,13 @@ docker run \ -v /var/run/docker.sock:/var/run/docker.sock \ localstack/localstack:3.6 ``` -:::{tip} -LocalStack is a cloud service emulator that runs in a single container on your -laptop or in your CI environment. With LocalStack, you can run your AWS -applications or Lambdas entirely on your local machine without connecting to -a remote cloud provider. -::: Install LorryStream including LocalStack CLI programs. ```shell -pip install lorrystream +pip install --upgrade 'lorrystream[carabas]' ``` + +## Configure Create a Kinesis Data Stream called `testdrive`. ```shell awslocal kinesis create-stream \ @@ -43,6 +49,7 @@ Display Stream ARN. awslocal kinesis describe-stream --stream-name testdrive | jq -r .StreamDescription.StreamARN ``` +## Usage Submit an item to the data stream, using `awslocal`. ```shell awslocal kinesis put-record \ @@ -64,4 +71,10 @@ export AWS_ENDPOINT_URL="http://localhost:4566" python examples/aws/kinesis_subscribe.py testdrive ``` +:::{todo} +Demonstrate how to add a processor pipeline element using both either +AWS Lambda, or a dedicated processor instance. +::: + + [Get started with Kinesis on LocalStack]: https://docs.localstack.cloud/user-guide/aws/kinesis/