[AWS] Move S3, SQS, SNS and lambda lightweight module config into integration

[kaiyan-sheng]

What does this PR do?

This PR is to move lightweight module configuration from metricbeat into integrations for s3_daily_storage, s3_request, SQS, SNS, lambda.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Related issues

• Closes Remove lightweight modules from aws s3_daily_storage data stream #3601
• Closes Remove lightweight modules from aws s3_request data stream #3602
• Closes Remove lightweight modules from aws sqs data stream #3604
• Closes Remove lightweight modules from aws sns data stream #3603
• Closes Remove lightweight modules from aws lambda data stream #3599

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-07-27T16:13:54.379+0000

• Duration: 36 min 8 sec

Test stats 🧪

Test	Results
Failed	0
Passed	334
Skipped	0
Total	334

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (11/11)	💚
Files	91.667% (11/12)	👎 -5.454
Classes	91.667% (11/12)	👎 -5.454
Methods	82.464% (174/211)	👎 -6.768
Lines	92.894% (2157/2322)	👍 2.302
Conditionals	100.0% (0/0)	💚

[endorama]

This is a change I do not understand! From what I understand this means that the events from this package do not contain this data structure anymore. But is this expected? I still see lambda.metrics.<fields> (es https://github.com/elastic/integrations/pull/3838/files#diff-ab8d12efcf395a213dbef900621dfe0f32e523b7f5a9692896869971264a1a4eR40) in the events so I'm not sure I'm looking in the right place.

[endorama]

Maybe this is because there is the renaming in the ingest pipeline, so in the end these fields will not in the stored event?

[kaiyan-sheng]

Ha we are removing this field (tech debt, not related to lightweight module config) because the metrics we are collecting for lambda are already all listed and documented in fields.yml: https://github.com/elastic/integrations/blob/main/packages/aws/data_stream/lambda/fields/fields.yml#L16

[endorama]

Thanks! So is this expected to be removed everywhere? Just to check for it in other integrations too ;)

[kaiyan-sheng]

Yeah at least in aws package we added all the field names individually into fields.yml so we should remove this '*.metrics.*.*' from all data streams under aws.

[endorama]

Looks good to me! Just a nitpick, I would bump the minor and not the patch version.

[kaiyan-sheng]

Looks good to me! Just a nitpick, I would bump the minor and not the patch version.

😂 Too late haha I think all the PRs that moves lightweight module all only bumped the patch version 😂 Well...... I'm trying to be consistent haha