Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: cloudwatch metrics example #970

Merged
merged 33 commits into from
Nov 6, 2023
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
9fbc621
chore: add cloudwatch experimental metrics lambda example
anitarua Sep 18, 2023
1bef4d1
add CDK code for metric filters and cloudwatch dashboard
anitarua Oct 4, 2023
2c66ce6
add ECS example
anitarua Oct 6, 2023
a4068a5
update readme and fix ecs example
anitarua Oct 9, 2023
8822967
add dashboard only option
anitarua Oct 9, 2023
6ea5341
revised after testing with dotnet lambda
anitarua Oct 17, 2023
9fb138c
set logger level to info instead of debug
anitarua Oct 18, 2023
3648487
use node latest version like ecs instead of node 16
anitarua Oct 18, 2023
13713ae
refactored and removed unnecessary dashboard widget args
anitarua Oct 18, 2023
e822603
specify all configs via env file and update readme
anitarua Oct 19, 2023
6a8349e
remove unnecessary region hardcoding in ecs dockerfile
anitarua Oct 19, 2023
6a87541
remove unnecessary es-lint hints
anitarua Oct 19, 2023
66b7f89
revise ecs example iam policy to restrict access to specific resources
anitarua Oct 19, 2023
b522b9f
Merge branch 'main' into cloudwatch-metrics-example
anitarua Oct 23, 2023
2302ed0
address pr feedback
anitarua Oct 23, 2023
a282991
Merge branch 'main' into cloudwatch-metrics-example
anitarua Oct 31, 2023
970d6b1
addressed some feedback
anitarua Oct 31, 2023
79500b7
added new npm targets
anitarua Oct 31, 2023
08c11c3
updated package lock
anitarua Oct 31, 2023
74971a2
Merge branch 'main' into cloudwatch-metrics-example
anitarua Oct 31, 2023
5112cd4
add p50, p90, p99 series to graphs
anitarua Oct 31, 2023
958f459
figured out dimensions and revised graphs
anitarua Nov 1, 2023
3aa686c
Merge branch 'main' into cloudwatch-metrics-example
anitarua Nov 1, 2023
ec27f26
remove repeated export
anitarua Nov 1, 2023
616b1ea
update metrics import
anitarua Nov 1, 2023
62c8b7b
update comments in cdk stack
anitarua Nov 1, 2023
9885bb7
incorporate feedback: grpc codes to human readable, relabel requests …
anitarua Nov 1, 2023
97bfd8a
update metric filters
anitarua Nov 1, 2023
4dc4161
change to momento prefix instead
anitarua Nov 1, 2023
4b0d47e
Merge branch 'main' into cloudwatch-metrics-example
anitarua Nov 2, 2023
8d37c08
Merge branch 'main' into cloudwatch-metrics-example
anitarua Nov 2, 2023
588dcf7
updates accounting for momento field in json logs
anitarua Nov 6, 2023
14a8842
Merge branch 'main' into cloudwatch-metrics-example
anitarua Nov 6, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
119 changes: 119 additions & 0 deletions examples/nodejs/lambda-examples/cloudwatch-metrics/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
<head>
anitarua marked this conversation as resolved.
Show resolved Hide resolved
<meta name="Momento Node.js Client Library Documentation" content="Node.js client software development kit for Momento Cache">
</head>
<img src="https://docs.momentohq.com/img/logo.svg" alt="logo" width="400"/>

[![project status](https://momentohq.github.io/standards-and-practices/badges/project-status-official.svg)](https://github.com/momentohq/standards-and-practices/blob/main/docs/momento-on-github.md)
[![project stability](https://momentohq.github.io/standards-and-practices/badges/project-stability-stable.svg)](https://github.com/momentohq/standards-and-practices/blob/main/docs/momento-on-github.md)

<br>

## CloudWatch Metrics using Momento's Experimental Metrics Middleware

This directory contains an AWS CDK stack that will create and populate a custom CloudWatch dashboard for Momento client-side metrics. You can deploy an example Lambda function or ECS cluster that uses the Momento Node.js SDK's Experimental Metrics Middleware to generate example dashboard data for 5 minutes at a time. Or you can deploy just the CloudWatch dashboard and use your own application to emit Momento metrics.

## Prerequisites

- Node version 18 or higher is required
- To get started with Momento, you will need a Super User Momento API key. You can get one from the [Momento Console](https://console.gomomento.com). Check out the [getting started](https://docs.momentohq.com/getting-started) guide for more information on obtaining an API key.
- To deploy the CDK app, you will need to have [configured your AWS credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-authentication.html#cli-chap-authentication-precedence).

## Deploying the CloudWatch Dashboard (along with an example Momento Lambda Function)

Follow the instructions in this section to deploy a CloudWatch dashboard along with a Node.js Lambda function that uses Momento's Experimental Metrics Middleware.

First make sure to start Docker and install the dependencies in the `lambda` directory, which is where the Lambda code lives.

```bash
cd lambda
npm install
```

The source code for the CDK application lives in the `infrastructure` directory.
To build and deploy it you will first need to install the dependencies:

```bash
cd infrastructure
npm install
```

Then create a `.env` file in the `infrastructure` directory, which is where you will define all required configuration variables.
You will need the Super User API key you generated from the [Momento Console](https://console.gomomento.com). You will also set the `EXAMPLE_MOMENTO_APPLICATION` variable to `lambda` like so:

```bash
MOMENTO_API_KEY=<YOUR_MOMENTO_API_KEY>
EXAMPLE_MOMENTO_APPLICATION="lambda"
```

Then deploy the CDK stack for the example Lambda function and CloudWatch dashboard:

```
npm run cdk -- deploy
anitarua marked this conversation as resolved.
Show resolved Hide resolved
anitarua marked this conversation as resolved.
Show resolved Hide resolved
```

To run the lambda, go to the `MomentoMetricsMiddlewareCDKExample` function in the AWS Lambda console and click the `test` button to invoke the example.

After a few minutes, you should be able to see the metrics populating several charts by navigating to CloudWatch > Dashboards > Custom Dashboards > MomentoMetricsCDKExampleDashboard in the AWS console.

## Deploying the CloudWatch Dashboard (along with an example Momento ECS Cluster)

Follow the instructions in this section to deploy a CloudWatch dashboard along with a small ECS cluster for a Node.js Docker image that uses Momento's Experimental Metrics Middleware.

First make sure to start Docker and install the dependencies in the `docker` directory, which is where the ECS container code lives.

```bash
cd docker
npm install
anitarua marked this conversation as resolved.
Show resolved Hide resolved
```

The source code for the CDK application lives in the `infrastructure` directory.
To build and deploy it you will first need to install the dependencies:

```bash
cd infrastructure
npm install
```

Then create a `.env` file in the `infrastructure` directory, which is where you will define all required configuration variables.
You will need the Super User API key you generated from the [Momento Console](https://console.gomomento.com). You will also set the `EXAMPLE_MOMENTO_APPLICATION` variable to `ecs` like so:

```bash
MOMENTO_API_KEY=<YOUR_MOMENTO_API_KEY>
EXAMPLE_MOMENTO_APPLICATION="ecs"
```

Then deploy the CDK stack for the example Lambda function and CloudWatch dashboard:

```
npm run cdk -- deploy
```

The ECS cluster should automatically start running the task and generating logs. After a few minutes, you should be able to see the metrics populating several charts by navigating to CloudWatch > Dashboards > Custom Dashboards > MomentoMetricsCDKExampleDashboard in the AWS console.

## Deploying only the CloudWatch Dashboard

Follow the instructions in this section to deploy only the CloudWatch metric filters and dashboard. This configuration option assumes you have an existing application that uses the Node.js or .NET Momento SDK Experimental Metrics Middleware that will emit metrics to the log group created by this CDK stack.

First make sure to start Docker and install the dependencies in the `infrastructure` directory.

```bash
cd infrastructure
npm install
```

Then create a `.env` file in the `infrastructure` directory, which is where you will define all required configuration variables.
You will need the Super User API key you generated from the [Momento Console](https://console.gomomento.com). You will also set the `EXAMPLE_MOMENTO_APPLICATION` and `LOG_GROUP_NAME` variables like so:

```bash
MOMENTO_API_KEY=<YOUR_MOMENTO_API_KEY>
EXAMPLE_MOMENTO_APPLICATION="dashboard-only"
LOG_GROUP_NAME="<YOUR_LOG_GROUP_NAME>"
```

Deploy the CDK stack for the metric filters and CloudWatch dashboard:

```
npm run cdk -- deploy
```

And then finally deploy your existing application in the usual way. Please note that this setup assumes your application will emit logs to the log group created for the dashboard, so be sure to double check that the log group names are aligned.
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
FROM --platform=linux/amd64 public.ecr.aws/docker/library/node:latest
anitarua marked this conversation as resolved.
Show resolved Hide resolved

WORKDIR /usr/src/app

COPY ecs-code/index.ts ./
COPY ecs-code/package.json ./
RUN npm install

ENV MOMENTO_API_KEY_SECRET_NAME="MomentoMetricsApiKey"
CMD ["npm", "run", "start"]

Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
node_modules
dist
**/*.d.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
{
"root": true,
"env": {
"es2021": true
},
"extends": [
"eslint:recommended",
"plugin:@typescript-eslint/recommended",
"plugin:@typescript-eslint/recommended-requiring-type-checking",
"plugin:import/recommended",
"plugin:prettier/recommended",
"plugin:node/recommended"
],
"parser": "@typescript-eslint/parser",
"parserOptions": {
"ecmaVersion": 12,
"project": "./tsconfig.json"
},
"plugins": ["@typescript-eslint"],
"rules": {
"semi": ["error", "always"],
"import/no-extraneous-dependencies": ["error", {}],
"node/no-unsupported-features/es-syntax": "off",
"node/no-missing-import": [
"error",
{
"tryExtensions": [".js", ".ts", ".json", ".node"]
}
],
"prettier/prettier": "error",
"block-scoped-var": "error",
"eqeqeq": "error",
"no-var": "error",
"prefer-const": "error",
"eol-last": "error",
"prefer-arrow-callback": "error",
"no-trailing-spaces": "error",
"quotes": ["warn", "single", {"avoidEscape": true}],
"no-restricted-properties": [
"error",
{
"object": "describe",
"property": "only"
},
{
"object": "it",
"property": "only"
}
],
// async without await is often an error and in other uses it obfuscates
// the intent of the developer. Functions are async when they want to await.
"require-await": "error",
"import/no-duplicates": "error"
},
"settings": {
"import/resolver": {
"node": {
"extensions": [".js", ".jsx", ".ts", ".tsx"]
}
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
advanced-middlewares-example-metrics.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
*.ts
!*.d.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"bracketSpacing": false,
"singleQuote": true,
"trailingComma": "es5",
"arrowParens": "avoid",
"printWidth": 120
}

Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
import {GetSecretValueCommand, SecretsManagerClient} from '@aws-sdk/client-secrets-manager';
import {
CacheClient,
Configurations,
DefaultMomentoLoggerFactory,
DefaultMomentoLoggerLevel,
CredentialProvider,
} from '@gomomento/sdk';
import {ExperimentalMetricsLoggingMiddleware} from '@gomomento/sdk/dist/src/config/middleware/experimental-metrics-logging-middleware';

const _secretsClient = new SecretsManagerClient({});
const _cachedSecrets = new Map<string, string>();
let _cacheClient: CacheClient | undefined = undefined;

async function main() {
try {
const loggerFactory = new DefaultMomentoLoggerFactory(DefaultMomentoLoggerLevel.INFO);
const logger = loggerFactory.getLogger('CloudWatchMetricsMiddlewaresExample');

const cacheClient = await getCacheClient(loggerFactory);
logger.info('Created a CacheClient configured with metrics middleware');

logger.info('Issuing 5 minutes of set and get requests to generate data for the dashboard example');
for (let i = 0; i < 60 /* seconds */ * 5 /* minutes */; i++) {
anitarua marked this conversation as resolved.
Show resolved Hide resolved
await cacheClient.set('cache', 'metrics-example-{i}', 'VALUE');
await cacheClient.get('cache', 'metrics-example-{i}');
await delay(1000);
anitarua marked this conversation as resolved.
Show resolved Hide resolved
}

logger.info('Momento metrics middleware example complete!');
} catch (err) {
console.log('Error occurred in the ECS example code:', err);
}
}

async function getCacheClient(loggerFactory: DefaultMomentoLoggerFactory): Promise<CacheClient> {
const apiKeySecretName = process.env.MOMENTO_API_KEY_SECRET_NAME;
if (apiKeySecretName === undefined) {
throw new Error("Missing required env var 'MOMENTO_API_KEY_SECRET_NAME");
}
if (_cacheClient === undefined) {
const momentoApiKey = await getSecret(apiKeySecretName);

_cacheClient = await CacheClient.create({
configuration: Configurations.Lambda.latest(loggerFactory).withMiddlewares([
new ExperimentalMetricsLoggingMiddleware(loggerFactory),
]),
credentialProvider: CredentialProvider.fromString({apiKey: momentoApiKey}),
defaultTtlSeconds: 60,
});
}
return _cacheClient;
}

async function getSecret(secretName: string): Promise<string> {
if (!_cachedSecrets.has(secretName)) {
const secretResponse = await _secretsClient.send(new GetSecretValueCommand({SecretId: secretName}));
if (secretResponse) {
_cachedSecrets.set(secretName, secretResponse.SecretString!);
} else {
throw new Error(`Unable to retrieve secret: ${secretName}`);
}
}
return _cachedSecrets.get(secretName)!;
}

function delay(ms: number): Promise<unknown> {
return new Promise(resolve => setTimeout(resolve, ms));
}

main().catch(err => {
console.log('Error in example run:', err);
});
Loading
Loading