Add AWS Cost and Usage reports to Metricbeat

[stevewritescode]

Describe the enhancement:
Create a Metricbeat module to import AWS cost and usage data, e.g. detailed billing information.

Describe a specific use case for the enhancement or feature:
The current AWS Billing Metricbeat module pulls in billing data from AWS. From what I can tell, the only data that's available through this module is EstimatedCharges since that's the only metric sent through AWS CloudWatch under AWS/Billing - I'd like to have rich billing information so that I can monitor AWS account usage in Elastic without needed to go log in to the AWS console cost explorer. Of course, once it's in Elastic then I can leverage all the visualization, ML, and alerting capabilities :)

I seen a few open-source projects that are built around this concept but none that use metricbeat. The AWS Go SDK supports costexplorer functions, so it would be natural to integrate with metricbeat the same way that the AWS Cloudwatch module works.

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[kaiyan-sheng]

Thank you @stevewritescode for creating this issue. Yes, billing metricset currently only collect EstimatedCharges from CloudWatch. I will look into costexplorer and let you know!

[kaiyan-sheng]

Hi @stevewritescode , I started a draft PR on adding cost explorer billing metrics for AWS. Please feel free to comment on the PR. Event output examples are here: https://github.com/elastic/beats/pull/20527/files?file-filters%5B%5D=.json

[stevewritescode]

Wanted to follow-up on this and let you know that I built 7.x branch and setup aws billing with the example steps in the PR, and I was able to see daily unblended costs by service, which is exactly what I wanted! I double checked with the AWS cost explorer console to make sure the numbers matched. Works great!

[kaiyan-sheng]

😬 @stevewritescode That's awesome. Thank you for testing this!!!