[awsfirehose] Add routing rules for metrics ingested from Firehose

[kaiyan-sheng]

Proposed commit message

This PR is to add routing_rules.yml for metrics ingested from Firehose. Without this integration or user specific set the parameter in Firehose, metrics coming into Elasticsearch by default goes to metrics-aws.cloudwatch-default. Idea to reroute is by checking the value of aws.cloudwatch.namespace field in each document to decide which data stream to route the document to. For example if aws.cloudwatch.namespace equals to AWS/EC2, then this document should go to metrics-aws.ec2_metrics-default.

For s3 daily storage and s3 request, we check for the actual metric name. If metric name is either BucketSizeBytes or NumberOfObjects, this document goes to aws.s3_daily_storage. If it's other metric name but still with AWS/S3 namespace, documents go to aws.s3_request.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Related issues

• Relates https://github.com/elastic/obs-infraobs-team/issues/834

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[agithomas]

Do you want to consider adding the details of permissions needed, mentioned here?

[agithomas]

In the documentation, i could find the below text

(Optional) To change the output format from the default format for your scenario, choose Change output format. The supported formats are JSON, OpenTelemetry 1.0.0, and OpenTelemetry 0.7.0.

Among these supported formats, which one the user must select?

[felixbarny]

LGTM. Non-blocking suggestion to add subobjects: false to dimensions.

[kaiyan-sheng]

@agithomas Thanks for the review!

Do you want to consider adding the details of permissions needed, mentioned here?

Good point! I will add this in https://github.com/elastic/observability-docs/blob/main/docs/en/observability/cloud-monitoring/aws/monitor-aws-firehose.asciidoc!

In the documentation, i could find the below text
(Optional) To change the output format from the default format for your scenario, choose Change output format. The supported formats are JSON, OpenTelemetry 1.0.0, and OpenTelemetry 0.7.0.
Among these supported formats, which one the user must select?

User must choose JSON or otel 1.0.0! Good point, I need to add it in https://github.com/elastic/observability-docs/blob/main/docs/en/observability/cloud-monitoring/aws/monitor-aws-firehose.asciidoc as well! Thanks!!

Hmm after reviewing the integration, I think these info and more doc needs to be added for this package too. I will update both doc!

[zmoog]

LGTM!

I would only double-check if we can completely leverage ecs@mappings for all remaining fields and drop the ecs.yml files.

[agithomas]

LGTM!

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 7f3616d

History

• 💔 Build #13510 failed 64cf10a
• 💔 Build #13507 failed b0cab48
• 💚 Build #13453 succeeded dc69ca6

cc @kaiyan-sheng

[elastic-sonarqube]

Quality Gate failed

Failed conditions
2.1% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

[elasticmachine]

Package awsfirehose - 1.1.0 containing this change is available at https://epr.elastic.co/search?package=awsfirehose