[AWS][EC2] Add processors to rename and drop fields

[constanca-m]

What does this PR do?

This PR syncs the EC2 fields from integrations with the ones in metricbeat.

In metricbeat, there are processors that rename and drop fields. Example: aws.ec2.metrics.CPUCreditUsage.avg is renamed to
aws.ec2.cpu.credit_usage (code here). However, there were no such processors in the integration, which caused the values from the ec2 group to be empty.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

1. Go to AWS integration and run elastic-package build.
2. Update the elastic-package registry with elastic-package stack up -v -d --services package-registry.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-06-30T09:19:24.771+0000

• Duration: 52 min 6 sec

Test stats 🧪

Test	Results
Failed	0
Passed	195
Skipped	4
Total	199

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (16/16)	💚
Files	94.118% (16/17)	👎 -5.882
Classes	94.118% (16/17)	👎 -5.882
Methods	85.816% (242/282)	👍 25.056
Lines	86.044% (7460/8670)	👎 -13.492
Conditionals	100.0% (0/0)	💚

[tommyers-elastic]

do you think this could be considered a breaking change for customers who have potentially built dashboards on the existing (un-renamed) metrics fields?

[constanca-m]

Yes, regarding the metrics.* maybe it is a breaking change, but it is not in sync with beats. And the documentation is not correct because the fields are mixed up. But maybe the best option is to support both for now. What do you think?

[tommyers-elastic]

yeh it would avoid worrying about breaking things, but for all other customers means a bunch of duplication which sucks. @vinaychandrasekhar @SubhrataK what do you think? so long as the dashboards still work can we remove fields? the public documentation does mention the fields in the exported fields table, as shown below.

@constanca-m i also added a pipeline test to validate whatever decision we make here.

[constanca-m]

@tommyers-elastic @vinayamohandoss @SubhrataK Any updates on this?

[girodav]

Just my two cents.

do you think this could be considered a breaking change for customers who have potentially built dashboards on the existing (un-renamed) metrics fields?

It might break the dashboards, but I do not think it should be considered a breaking change. Users have to explicitly upgrade the integration, no functionality breaks automatically. We had similar cases in the past (example).

In any case, we can consider bump the version to 1.5.0 or even 2.0.0 to signal the field naming change, if anything.

But maybe the best option is to support both for now. What do you think?

This is a possible option, but it would only delay the inevitable (e.g removing aws.ec2.metrics fields). I would opt for this only we are really on a tight schedule, to unblock this PR.

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[girodav]

@constanca-m I see no change made on the OOTB dashboards. Have you verified that they work properly or they need to be updated?

[constanca-m]

It works for me as well @girodav

[constanca-m]

/test

[kaiyan-sheng]

After discussion with @constanca-m and @tommyers-elastic , we decide to not rename these fields to match Metricbeat at this point in AWS integration. Instead we want to keep the name in the format from cloudwatch input and will change the fields.yml to make sure the field names are defined.