[MetricBeat] add a field to represent the amount of MetricStat in CloudWatch GetMetricData API

[kwinstonix]

Describe the enhancement:

add a field metricstatscount to represent the amount of MetricStat in CloudWatch GetMetricData api request . Then we can figure out how many metrics were retrieved from AWS and the monthly bill amount.

with the example, there are six MetricNames of CloudWatch and each MetricName has only one statistic. so the total amount of MetricStat is 6 * 1 = 6

this only shows the MetricDataResults with valid datapoint, please refer below comment

# demo metricbeat event, only aws field
#  metricbeat version:  7.6.2 

    "aws": {
       # demo field
      "metricstatscount": 6

      "tags": {
        "Key1": "V1"
      },
      "applicationelb": {
        "metrics": {
          "ProcessedBytes": {
            "sum": 263484
          },
          "NewConnectionCount": {
            "sum": 255
          },
          "ActiveConnectionCount": {
            "sum": 135
          },
          "TargetResponseTime": {
            "avg": 0.0007011730769230769
          },
          "ConsumedLCUs": {
            "avg": 0.0005777777777777778
          },
          "RequestCount": {
            "sum": 208
          }
        }
      },
      "cloudwatch": {
        "namespace": "AWS/ApplicationELB"
      },
      "dimensions": {
        "LoadBalancer": "app/some-name1/111111aaaaa"
      }
    }




# cloudwatch  API

client.get_metric_data(
    MetricDataQueries=[
        {
            'Id': 'string',
            'MetricStat': {
                'Metric': {
                    'Namespace': 'string',
                    'MetricName': 'string',
                    'Dimensions': [
                        {
                            'Name': 'string',
                            'Value': 'string'
                        },
                    ]
                },
                'Period': 123,
                'Stat': 'string',
                'Unit': 'xxx,
            },
            'Expression': 'string',
            'Label': 'string',
            'ReturnData': True|False,
            'Period': 123
        },
    ],
    StartTime=datetime(2015, 1, 1),
    EndTime=datetime(2015, 1, 1),
    NextToken='string',
    ScanBy='TimestampDescending'|'TimestampAscending',
    MaxDatapoints=123
)

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[kwinstonix]

beats/x-pack/metricbeat/module/aws/cloudwatch/cloudwatch.go

Lines 443 to 461 in 1dde9f4

	func (m *MetricSet) createEvents(svcCloudwatch cloudwatchiface.ClientAPI, svcResourceAPI resourcegroupstaggingapiiface.ClientAPI, listMetricWithStatsTotal []metricsWithStatistics, resourceTypeTagFilters map[string][]aws.Tag, regionName string, startTime time.Time, endTime time.Time) (map[string]mb.Event, error) {
	// Initialize events for each identifier.
	events := map[string]mb.Event{}
	// Construct metricDataQueries
	metricDataQueries := createMetricDataQueries(listMetricWithStatsTotal, m.Period)
	m.logger.Debugf("Number of MetricDataQueries = %d", len(metricDataQueries))
	if len(metricDataQueries) == 0 {
	return events, nil
	}
	// Use metricDataQueries to make GetMetricData API calls
	metricDataResults, err := aws.GetMetricDataResults(metricDataQueries, svcCloudwatch, startTime, endTime)
	m.logger.Debugf("Number of metricDataResults = %d", len(metricDataResults))
	if err != nil {
	return events, errors.Wrap(err, "GetMetricDataResults failed")
	}

I want to create a panel of Metric visualization in kibana to show how many cloudwatch metrics are queried and estimated cost.
I have read aws cloudwatch module, I think if len(metricDataQueries) can be added into event , it could be convenient for showing the usage of GetMetricData API. Is it ok ? @kaiyan-sheng

[kwinstonix]

I run a demo test. Except of the amount of MetricDataQueries, we should also pay attention to the amoun of emtpy values in MetricDataResults. This would be helpful to observe the cost of metricbeat.

• add a new event

index 5674a45ec..d4d55335a 100644
--- a/x-pack/metricbeat/module/aws/cloudwatch/cloudwatch.go
+++ b/x-pack/metricbeat/module/aws/cloudwatch/cloudwatch.go

@@ -458,6 +461,17 @@ func (m *MetricSet) createEvents(svcCloudwatch cloudwatchiface.ClientAPI, svcRes
        if err != nil {
                return events, errors.Wrap(err, "GetMetricDataResults failed")
        }
+       emptyValueMetricCount := 0
+       for _, item := range metricDataResults {
+               if item.Values == nil {
+                       emptyValueMetricCount += 1
+               }
+       }
+       e := aws.InitEvent(regionName, m.AccountName, m.AccountID)
+       e.RootFields.Put("aws.cloudwatch.getmetricdata.query.amount", len(metricDataQueries))
+       e.RootFields.Put("aws.cloudwatch.getmetricdata.result.amount", len(metricDataResults))
+       e.RootFields.Put("aws.cloudwatch.getmetricdata.result.emptyvalues", emptyValueMetricCount)
+       events["some_random_strings"] = e

• elasticsearch doc

• kibana TSVB

this indicates that 48.4% of the MetricDataQueries got emtpy result, we should optimize metricbeat config to reduce the wast of cost.

refer:
https://aws.amazon.com/cloudwatch/pricing/

GetMetricData, GetInsightRuleReport $0.01 per 1,000 metrics requested

[kaiyan-sheng]

@kwinstonix Thanks for all the effort! Good point it would be nice to show users the query number to help understanding the API cost. What do you think about adding these information in debug logs for example instead of in each event?

[kaiyan-sheng]

When you mention optimize metricbeat config when there is a high empty value rate, do you mean adding filters in aws.yml for example dimensions/regions to lower the number of queries?

[kwinstonix]

@kwinstonix Thanks for all the effort! Good point it would be nice to show users the query number to help understanding the API cost. What do you think about adding these information in debug logs for example instead of in each event?

Yes， debug log is also helpful .

When I run metricbeat deployment in production env, I should report the estimated AWS API cost, so I think it is better to have a visualization panel in Kibana. Besides, metricbeat uses AWS API that is not free, aws.cloudwatch.getmetricdata.query.amount may be a key metric of metricbeat itself.

These metrics in the example are not included in each event. Each aws.GetMetricDataResults call will generate one event that contains only fields aws.cloudwatch.getmetricdata.* , so the events amount may be acceptable.

Above all is just the situation I'm facing, and I hope metricbeat could export some metrics of AWS API call cost. Maybe others people have the same situation ?

Thanks for your reply

[kwinstonix]

When you mention optimize metricbeat config when there is a high empty value rate, do you mean adding filters in aws.yml for example dimensions/regions to lower the number of queries?

Region or dimensions filters are helpful, and I reported another issue #18050 😁

When I run tests, there's a odd metric BackendConnectionErrors in ELB namespace, It is a nonzeor value according to AWS doc, but ListMetrics result contains this metric and metricdata result is empty. So I remove this metricname in MetricBeat config. empty value rate just remind me to check metrics.

There is another situaction that aws. GetMetricDataResults has empty value。When cloud resources are terminated, their metrics will keep for 14 days, so aws.GetMetricDataResults call will get empty metricdata. empty value rate is just another observable indicator.