Merge branch 'main' into postgresql/DBM-Screen-Metrics

middleware-labs · Jun 11, 2024 · 49981b7 · 49981b7
2 parents d6fbb0d + 148cf00
commit 49981b7
Show file tree

Hide file tree

Showing 34 changed files with 5,614 additions and 889 deletions.
diff --git a/receiver/awscloudwatchmetricsreceiver/README.md b/receiver/awscloudwatchmetricsreceiver/README.md
@@ -1,4 +1,4 @@
-# CloudWatch Metrics Receiver
+# AWS CloudWatch Metrics Receiver
 
 <!-- status autogenerated section -->
 | Status        |           |
@@ -12,12 +12,14 @@
 [contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
 <!-- end autogenerated section -->
 
-Receives Cloudwatch metrics from [AWS Cloudwatch](https://aws.amazon.com/cloudwatch/) via the [AWS SDK for Cloudwatch Logs](https://docs.aws.amazon.com/sdk-for-go/api/service/cloudwatchlogs/)
+Receives CloudWatch metrics from [AWS CloudWatch](https://aws.amazon.com/cloudwatch/) via the [AWS SDK for CloudWatch Metrics](https://docs.aws.amazon.com/sdk-for-go/api/service/cloudwatch/)
 
 ## Getting Started
 
 This receiver uses the [AWS SDK](https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/) as mode of authentication, which includes [Profile](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) and [IMDS](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html) authentication for EC2 instances.
 
+By default the receiver collects *no* metrics.
+
 ## Configuration
 
 ### Top Level Parameters
@@ -27,26 +29,37 @@ This receiver uses the [AWS SDK](https://aws.github.io/aws-sdk-go-v2/docs/config
 | `region`        | *required* | string | The AWS recognized region string  |
 | `profile`       | *optional* | string | The AWS profile used to authenticate, if none is specified the default is chosen from the list of profiles  |
 | `IMDSEndpoint`  | *optional* | string | The IMDS endpoint to authenticate to AWS  |                                                                                                                                 
-| `poll_interval`   | `default=1m` | duration   | The duration waiting in between requests | 
-| `metrics`          | *optional* | `Metrics` | Configuration for metrics ingestion of this receiver    |
+| `poll_interval`   | `default=5m` | duration   | The duration waiting in between requests | 
+| `metrics`          | *required* | `Metrics` | Configuration for metrics ingestion of this receiver    |
 
 ### Metrics Parameters
 
 
 | Parameter                | Notes        | type                   | Description                                                                                |
 | ------------------------ | ------------ | ---------------------- | ------------------------------------------------------------------------------------------ |
-| `named`                 | *required*   | `See Named Parameters` | Configuration for Named Metrics, by default no metrics are collected |
+| `group`                 | *one key required*   | `See group parameters` | Configuration for named metrics |
+| `autodiscover`        | *one key required* | `See autodiscover parameters` | Configuration for autodiscovery of metrics
 
 
-### Named Parameters
+### Group Parameters
 
 | Parameter                | Notes        | type                   | Description                                                                                |
 | ------------------------ | ------------ | ---------------------- | ------------------------------------------------------------------------------------------ |
 | `namespace`                 | *required*   | `string` | AWS Metric namespace, all AWS namespaces are prefixed with `AWS`, eg: `AWS/EC2` for EC2 metrics |
-| `metric_name` | *required* | `string` | AWS metric name |
-| `period` | `default=5m` | duration | Aggregation period |
+| `period` | `default=5m` | duration | Aggregation period | 
+| `name` | *required* | `See named metric config` | Configuration for metric name
+
+
+
+### Named Metric Parameters
+| Parameter                | Notes        | type                   | Description                                                                                |
+| ------------------------ | ------------ | ---------------------- | ------------------------------------------------------------------------------------------ |
+| `metric_name` | required | string | name of metric |
 | `aws_aggregation` | `default=sum` | string | type of AWS aggregation, eg: sum, min, max, average |
-| `dimensions` | *optional* | `see Dimensions Parameters` | Configuration for metric dimensions |
+| `dimensions` | *optional* | `see Dimensions Parameters` | Configuration for metric dimensions. You should note AWS CloudWatch cannot unroll metrics, and so some dimensions are required for metrics data to be returned such a `InstanceId` for EC2 metrics |
+
+
+
 
 ### Dimension Parameters
 
@@ -59,90 +72,101 @@ This receiver uses the [AWS SDK](https://aws.github.io/aws-sdk-go-v2/docs/config
 #### Named Example
 
 ```yaml
-awscloudwatchmetrics:
-  region: us-east-1
-  poll_interval: 1m
-  metrics:
-    named:
-      - namespace: "AWS/EC2"
-        metric_name: "CPUUtilization"
-        period: "5m"
-        aws_aggregation: "Sum"
-        dimensions:
-          - Name: "InstanceId"
-            Value: "i-1234567890abcdef0"
-      - namespace: "AWS/S3"
-        metric_name: "BucketSizeBytes"
-        period: "5m"
-        aws_aggregation: "p99"
-        dimensions:
-          - Name: "BucketName"
-            Value: "OpenTelemetry"
-          - Name: "StorageType"
-            Value: "StandardStorage"
+region: "eu-west-2"
+profile: "my_profile"
+imds_endpoint: ""
+poll_interval: "1m"
+metrics:
+  group:
+    - namespace: "AWS/EC2"
+      period: "5m"
+      name:
+        - metric_name: "CPUUtilization"
+          aws_aggregation: "Average"
+          dimensions:
+            - Name: "InstanceId"
+              Value: "i-1234567890abcdef0"
+    - namespace: "AWS/EC2"
+      period: "1h"
+      name:
+        - metric_name: "DiskReadBytes"
+          aws_aggregation: "Sum"
+          dimensions:
+            - Name: "InstanceId"
+              Value: "i-9876543210abcdef0"
+
 ```
 
-## Sample Configs
+## Autodiscovery Example
 
 ```yaml
-receivers:
-  awscloudwatchmetrics:
-    region: eu-west-1
-    poll_interval: 10m
-    metrics:
-      named:
-        - namespace: "AWS/EC2"
-          metric_name: "CPUUtilization"
-          period: "5m"
-          aws_aggregation: "Sum"
-          dimensions:
-            - Name: "InstanceId"
-              Value: "i-035e091c31292427a"
+
+region: "us-west-2"
+profile: "my_profile"
+imds_endpoint: ""
+poll_interval: "1m"
+metrics:
+  autodiscover:
+    namespace: "AWS/EC2"
+    limit: 100
+    aws_aggregation: "Average"
+    period: "5m"
+    dimensions:
+      - Name: "InstanceId"
+        Value: "i-1234567890abcdef0"
+      - Name: "InstanceId"
+        Value: "i-9876543210abcdef0"
+
 
 processors:
 
 exporters:
-  logging:
+  debug:
     verbosity: detailed
 
 service:
   pipelines:
     metrics:
       receivers: [awscloudwatchmetrics]
       processors: []
-      exporters: [logging]
+      exporters: [debug]
 ```
 
 ## AWS Costs
 
 This receiver uses the [GetMetricData](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html) API call, this call is *not* in the AWS free tier. Please refer to [Amazon's pricing](https://aws.amazon.com/cloudwatch/pricing/) for further information about expected costs.
 
+## Features not supported
 
-[alpha]:https://github.com/open-telemetry/opentelemetry-collector#alpha
-[contrib]:https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
-[Issue]:https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/15667
+- This receiver currently does not support [metric math](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html) to return new time series based on a mathematical expressions.
 
-## Troubleshooting / Debugging
+- For better performance AWS recommends modifying the `StartTime` and `EndTime` to align with the metric's period and start/end of the hour. For example, a metric with a period of 5 minutes and a start-time of 18:55 can get better [performance than specifying a start-time of 18:51](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudwatch/client/get_metric_data.html). This receiver does not currently implement this logic.
 
-## My metrics are intermittent / not receing any metrics
+## Troubleshooting
 
-Try a bigger `poll_interval`. CloudWatch returns no data if the period of the metric, by default for AWS supplied metrics, it's 300 seconds (5 minutes). Try out a period of 600 seconds and a poll interval of 600 seconds.
+### My metrics are not being discovered
 
-## Help, I'm getting IAM permission denied
+AWS CloudWatch [ListMetrics API call](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/cloudwatch/list-metrics.html) does not return information about metrics if those metrics haven't had any updates in the past two weeks
+
+### My metrics are intermittent / not receiving any metrics
+
+Try a bigger `poll_interval`. CloudWatch returns no data if the period of the metric, by default for AWS supplied metrics this is 300 seconds (5 minutes), is less than the `poll_interval`. Try out a period of 600 seconds and a poll interval of 600 seconds.
+
+### IAM Permission errors
 
 Make sure your IAM role/user has the required permissions:
 
 ```yaml
 "cloudwatch:GetMetricData",
-"cloudwatch:GetMetricStatistics",
 "cloudwatch:ListMetrics"
 ```
 
 The following IAM permissions are required for transit gateways to work:
 
-```
+```yaml
 "ec2:DescribeTags",
 "ec2:DescribeInstances",
 "ec2:DescribeRegions",
 "ec2:DescribeTransitGateway*"
 ```
+