Add LocalStack integration with MaC (#306)

* Add inital docker compose file * WIP * Add inital docker compose file * Initial documentation about LocalStack. * Add some basic syntax highlighting. * Make pre-commit hook executable. * Update diagram. * Update README.md * Update README.md * Update README.md * Update diagram. * Add static config to YACE. * Add pre-requisite step for network creation. Co-authored-by: Ariful Haque <[email protected]>
HO-CTO · Oct 17, 2022 · ef32705 · ef32705
1 parent c53ed13
commit ef32705
Show file tree

Hide file tree

Showing 8 changed files with 316 additions and 1 deletion.
diff --git a/.githooks/pre-commit b/.githooks/pre-commit
diff --git a/example-apps/localstack/.gitignore b/example-apps/localstack/.gitignore
@@ -0,0 +1 @@
+volume/
diff --git a/example-apps/localstack/README.md b/example-apps/localstack/README.md
@@ -0,0 +1,118 @@
+# LocalStack Integration with MaC Framework
+
+[LocalStack](https://localstack.cloud/) can be used to simulate an AWS environment and Cloudwatch metrics telemetry to aid the development of the monitoring as code framework, without necessarily needing access to an actual AWS account.
+
+This can be useful for offline development and testing of the MaC framework, especially for new team members who may not have access to an AWS account for testing purposes.
+
+## Overview
+
+![Architecture Overview](./arch-overview.png)
+
+## Prerequisites
+
+You will need to have installed the following:
+
+- [docker](https://docs.docker.com/engine/install/)
+- [docker-compose](https://docs.docker.com/compose/install/)
+- [awscli](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)
+
+## Getting Started
+
+1. Run the MaC local development environment. This will implicitly create the local_default network, which the LocalStack docker-compose configuration will use.
+
+    ```sh
+    cd ../local/
+    docker-compose up
+    ```
+
+1. Run LocalStack using the provided `docker-compose.yml` file.
+
+    ```sh
+    docker-compose up
+    ```
+
+1. The LocalStack service shares its network configuration with the MaC local development environment.
+
+    This allows the YACE CloudWatch exporter, running within the MaC local development environment, to connect to LocalStack at http://localstack:4566.
+
+1. The YACE configuration file has been set up to allow auto-discovery of metrics from an AWS account. When running against an actual AWS account, this is desireable and a useful default configuration.
+
+    However, when working with LocalStack it does not appear to be possible to rely on this automatic discovery mechanism since metrics are produced manually using the AWS CLI, or related SDK, e.g. using [put-metric-data](https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/put-metric-data.html).
+
+    Fortunately, YACE also supports [static metric configuration](https://github.com/nerdswords/yet-another-cloudwatch-exporter#static-configuration) which allows additional metrics to be specified, without relying on discovery using resource tags.
+
+    The AWS metrics that are currently supported by the MaC framework have been configured in the static configuration section to allow them to be used with LocalStack.
+
+1. In order to point YACE at the LocalStack AWS emulator, you must add the following environment variables in the `yace` service section of the MaC local development envionrment `docker-compose.yml` file:
+
+    ```yml
+            environment:
+                - AWS_ENDPOINT_URL=http://localstack:4566
+                - AWS_REGION=eu-west-2
+    ```
+
+    Since YACE mounts the `~\.aws\credentials` file at runtime, you can authenticate against the LocalStack environment simply by running `aws configure` and specifying "test" for the ACCESS_KEY and SECRET_ACCESS_KEY values. Your resulting configuration file will look something like this:
+
+    ```ini
+    [default]
+    aws_access_key_id = test
+    aws_secret_access_key = test
+    ```
+
+1. Run the MaC local development environment ensuring that you have selected the `monitoring-fullstack` profile.
+
+    ```sh
+    docker-compose --profile monitoring-fullstack up
+    ```
+
+1. Supply metrics to the LocalStack CloudWatch API.
+
+    You can interact directly with the LocalStack environment using the standard AWS CLI tools. Simply ensure that you have supplied the `--endpoint-url=http://localhost:4566` when running your commands.
+
+    e.g.
+
+    ```sh
+    aws --endpoint-url="http://localhost:4566" sqs create-queue --queue-name sample-queue
+    ```
+
+    For convenience it might be desirable to alias the command
+
+    ```sh
+    alias awslocal="aws --endpoint-url=http://localhost:4566"
+    ```
+
+    Alternatively LocalStack provide an alernative `awslocal` command line tool which doesn't require supplying the endpoint URL on each command. See [the LocalStack Integration documentation](https://docs.localstack.cloud/integrations/aws-cli/#localstack-aws-cli-awslocal) for further details.
+
+    ### Examples
+
+    Here are some example commands that can be used to push simulated SQS metrics into LocalStack.
+
+    ```sh
+    awslocal cloudwatch put-metric-data --namespace "AWS/SQS" --metric-data '[{"MetricName": "ApproximateNumberOfMessagesVisible", "Value": 0}]'
+
+    awslocal cloudwatch put-metric-data --namespace "AWS/SQS" --metric-data '[{"MetricName": "ApproximateAgeOfOldestMessage", "Value": 5}]'
+
+    awslocal cloudwatch put-metric-data --namespace "AWS/SQS" --metric-data '[{"MetricName": "NumberOfMessagesSent", "Value": 10}]'
+
+    awslocal cloudwatch put-metric-data --namespace "AWS/SQS" --metric-data '[{"MetricName": "NumberOfMessagesDeleted", "Value": 0}]'
+    ```
+
+    Metrics for other services can be simulated by changing the `--namespace` value, the `MetricName` and adjusting the `Value`.
+
+1. Once the metrics have been pushed into LocalStack, they should appear on the YACE `/metrics` endpoint at `http://localhost:5000/metrics`.
+
+1. The MaC local development environment has already configured Prometheus to use YACE as a target, and so the metrics should also become visible from within the Prometheus console and from within Grafana dashboards.
+
+1. The `generic` mixin contains an SLI configuration for AWS SQS which can be useful for quickly testing the end-to-end flow of metrics from LocalStack to a Grafana Dashboard.
+
+    This can be deployed and run using the command:
+
+    ```sh
+    docker run --mount type=bind,source="$PWD"/output,target=/output --mount type=bind,source="$PWD"/mixin-defs,target=/input -it sre-monitoring-as-code:latest -m generic -rd -i input -o output
+    ```
+
+    It may be necessary to restart the local development environment after adding any new SLI configuration or rules.
+
+## Limitations
+
+At present, it does not appear to be possible to use YACE automatic discovery of LocalStack metrics using resource tags. Pushing metrics into LocalStack is very much a manual process for now, but may be made easier in the future via scripting or additional tooling.
diff --git a/example-apps/localstack/arch-overview.png b/example-apps/localstack/arch-overview.png
diff --git a/example-apps/localstack/docker-compose.yml b/example-apps/localstack/docker-compose.yml
@@ -0,0 +1,22 @@
+version: "3.8"
+
+services:
+  localstack:
+    container_name: "${LOCALSTACK_DOCKER_NAME-localstack_main}"
+    image: localstack/localstack
+    ports:
+      - "127.0.0.1:4566:4566"            # LocalStack Gateway
+      - "127.0.0.1:4510-4559:4510-4559"  # external services port range
+    environment:
+      - DEBUG=${DEBUG-}
+      - PERSISTENCE=${PERSISTENCE-}
+      - LAMBDA_EXECUTOR=${LAMBDA_EXECUTOR-}
+      - DOCKER_HOST=unix:///var/run/docker.sock
+    volumes:
+      - "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack"
+      - "/var/run/docker.sock:/var/run/docker.sock"
+    networks:
+      - local_default
+networks:
+  local_default:
+    external: true
diff --git a/example-apps/localstack/overview.drawio b/example-apps/localstack/overview.drawio
@@ -0,0 +1,55 @@
+<mxfile host="65bd71144e">
+    <diagram id="CZQeM4Upb8wU9XLdOa43" name="Page-1">
+        <mxGraphModel dx="1911" dy="487" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="850" pageHeight="1100" math="0" shadow="0">
+            <root>
+                <mxCell id="0"/>
+                <mxCell id="1" parent="0"/>
+                <mxCell id="19" value="" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#f5f5f5;strokeColor=#666666;fontColor=#333333;" vertex="1" parent="1">
+                    <mxGeometry x="-20" y="50" width="890" height="400" as="geometry"/>
+                </mxCell>
+                <mxCell id="6" value="MaC Local Environment" style="rounded=0;whiteSpace=wrap;html=1;labelPosition=center;verticalLabelPosition=top;align=center;verticalAlign=bottom;fillColor=#f5f5f5;strokeColor=#666666;fontColor=#333333;" vertex="1" parent="1">
+                    <mxGeometry x="480" y="110" width="340" height="220" as="geometry"/>
+                </mxCell>
+                <mxCell id="9" value="LocalStack" style="rounded=0;whiteSpace=wrap;html=1;labelPosition=center;verticalLabelPosition=top;align=center;verticalAlign=bottom;fillColor=#f5f5f5;strokeColor=#666666;fontColor=#333333;" vertex="1" parent="1">
+                    <mxGeometry x="30" y="110" width="340" height="220" as="geometry"/>
+                </mxCell>
+                <mxCell id="18" style="edgeStyle=none;html=1;fontColor=#000000;fillColor=#f5f5f5;strokeColor=#666666;" edge="1" parent="1" source="10" target="15">
+                    <mxGeometry relative="1" as="geometry"/>
+                </mxCell>
+                <mxCell id="10" value="Emulated CloudWatch&lt;br&gt;API" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#f5f5f5;strokeColor=#666666;fontColor=#333333;" vertex="1" parent="1">
+                    <mxGeometry x="140" y="190" width="120" height="60" as="geometry"/>
+                </mxCell>
+                <mxCell id="12" style="edgeStyle=none;html=1;entryX=1;entryY=0.5;entryDx=0;entryDy=0;fontColor=#000000;fillColor=#f5f5f5;strokeColor=#666666;" edge="1" parent="1" source="3" target="10">
+                    <mxGeometry relative="1" as="geometry"/>
+                </mxCell>
+                <mxCell id="13" value="Docker Compose&lt;br&gt;Network" style="rounded=1;whiteSpace=wrap;html=1;labelPosition=center;verticalLabelPosition=top;align=center;verticalAlign=bottom;fillColor=#f5f5f5;strokeColor=#666666;fontColor=#333333;" vertex="1" parent="1">
+                    <mxGeometry x="340" y="150" width="170" height="10" as="geometry"/>
+                </mxCell>
+                <mxCell id="15" value="http://localhost:4566" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;fillColor=#f5f5f5;strokeColor=none;fontColor=#333333;" vertex="1" parent="1">
+                    <mxGeometry x="170" y="370" width="60" height="30" as="geometry"/>
+                </mxCell>
+                <mxCell id="16" value="http://localhost:5000/metrics" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;fillColor=#f5f5f5;strokeColor=none;fontColor=#333333;" vertex="1" parent="1">
+                    <mxGeometry x="520" y="370" width="60" height="30" as="geometry"/>
+                </mxCell>
+                <mxCell id="17" style="edgeStyle=none;html=1;fontColor=#000000;fillColor=#f5f5f5;strokeColor=#666666;" edge="1" parent="1" source="3" target="16">
+                    <mxGeometry relative="1" as="geometry"/>
+                </mxCell>
+                <mxCell id="3" value="YACE" style="whiteSpace=wrap;html=1;aspect=fixed;fillColor=#f5f5f5;strokeColor=#666666;fontColor=#333333;" vertex="1" parent="1">
+                    <mxGeometry x="510" y="180" width="80" height="80" as="geometry"/>
+                </mxCell>
+                <mxCell id="4" value="Prometheus" style="whiteSpace=wrap;html=1;aspect=fixed;fillColor=#f5f5f5;strokeColor=#666666;fontColor=#333333;" vertex="1" parent="1">
+                    <mxGeometry x="610" y="180" width="80" height="80" as="geometry"/>
+                </mxCell>
+                <mxCell id="5" value="Grafana" style="whiteSpace=wrap;html=1;aspect=fixed;fillColor=#f5f5f5;strokeColor=#666666;fontColor=#333333;" vertex="1" parent="1">
+                    <mxGeometry x="710" y="180" width="80" height="80" as="geometry"/>
+                </mxCell>
+                <mxCell id="7" style="edgeStyle=none;html=1;entryX=1;entryY=0.5;entryDx=0;entryDy=0;fontColor=#000000;fillColor=#f5f5f5;strokeColor=#666666;" edge="1" parent="1" source="4" target="3">
+                    <mxGeometry relative="1" as="geometry"/>
+                </mxCell>
+                <mxCell id="8" style="edgeStyle=none;html=1;entryX=1;entryY=0.5;entryDx=0;entryDy=0;fontColor=#000000;fillColor=#f5f5f5;strokeColor=#666666;" edge="1" parent="1" source="5" target="4">
+                    <mxGeometry relative="1" as="geometry"/>
+                </mxCell>
+            </root>
+        </mxGraphModel>
+    </diagram>
+</mxfile>
diff --git a/local/docker-compose.yaml b/local/docker-compose.yaml
@@ -228,6 +228,7 @@ services:
         environment:
             - AWS_PROFILE
             - AWS_SHARED_CREDENTIALS_FILE=/exporter/.aws/credentials
+            - AWS_ENDPOINT_URL=http://host.docker.internal:4566
         env_file:
             - global.env
         restart: on-failure:5

diff --git a/local/yace/config/config.yml b/local/yace/config/config.yml
@@ -163,4 +163,122 @@ discovery:
             - Average
           period: 60
           length: 120
-          nilToZero: true
+          nilToZero: true
+static:
+  - namespace: AWS/SQS
+    name: my-id
+    regions:
+      - eu-west-2
+    metrics:
+      - name: ApproximateNumberOfMessagesVisible
+        statistics:
+          - Sum
+        period: 60
+        length: 120
+        nilToZero: true
+      - name: ApproximateAgeOfOldestMessage
+        statistics:
+          - Sum
+        period: 60
+        length: 120
+        nilToZero: true
+      - name: NumberOfMessagesSent
+        statistics:
+          - Sum
+        period: 60
+        length: 120
+        nilToZero: true
+      - name: NumberOfMessagesDeleted
+        statistics:
+          - Sum
+        period: 60
+        length: 120
+        nilToZero: true
+    customTags:
+      - key: Environment
+        value: localstack
+  - namespace: AWS/RDS
+    name: my-id
+    regions:
+      - eu-west-2
+    metrics:
+      - name: ReadLatency
+        statistics:
+          - Average
+        period: 60
+        length: 120
+        nilToZero: true
+      - name: ReadThroughput
+        statistics:
+          - Average
+        period: 60
+        length: 120
+        nilToZero: true
+      - name: ReadIOPS
+        statistics:
+          - Average
+        period: 60
+        length: 120
+        nilToZero: true
+      - name: WriteLatency
+        statistics:
+          - Average
+        period: 60
+        length: 120
+        nilToZero: true
+      - name: WriteThroughput
+        statistics:
+          - Average
+        period: 60
+        length: 120
+        nilToZero: true
+      - name: WriteIOPS
+        statistics:
+          - Average
+        period: 60
+        length: 120
+        nilToZero: true
+    customTags:
+      - key: Environment
+        value: localstack
+  - namespace: AWS/ES
+    name: my-id
+    regions:
+      - eu-west-2
+    metrics:
+      - name: SearchLatency
+        statistics:
+          - Sum
+        period: 60
+        length: 120
+        nilToZero: true
+      - name: SearchRate
+        statistics:
+          - Sum
+        period: 60
+        length: 120
+        nilToZero: true
+      - name: 2xx
+        statistics:
+          - Sum
+        period: 60
+        length: 120
+        nilToZero: true
+      - name: 3xx
+        statistics:
+          - Sum
+        period: 60
+        length: 120
+        nilToZero: true
+      - name: 4xx
+        statistics:
+          - Sum
+        period: 60
+        length: 120
+        nilToZero: true
+      - name: 5xx
+        statistics:
+          - Sum
+        period: 60
+        length: 120
+        nilToZero: true