Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherry-pick #20875 to 7.x: [Metricbeat] Add latency config option into aws module #21537

Merged
merged 2 commits into from
Oct 5, 2020
Merged

Cherry-pick #20875 to 7.x: [Metricbeat] Add latency config option into aws module #21537

merged 2 commits into from
Oct 5, 2020

Conversation

kaiyan-sheng
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng commented Oct 5, 2020

Cherry-pick of PR #20875 to 7.x branch. Original message:

What does this PR do?

This PR adds the latency config parameter for aws module. With this config parameter, users can collect CloudWatch metrics with a latency larger than Metricbeat collection period. For example, for S3 request metrics, the latency can be 3 minutes. The recommended collection period is 1min. Without the latency config parameter, no data points will be collected ever.

In this screenshot: current timestamp is 14:21 but we only see the latest data at 14:18 with a 3-minute delay. In this case, without this fix, our Metricbeat module will keep trying to collect data from the last 2-minute time frame, which will always be empty:
Screenshot 2020-09-03 at 16 22 19

(@gbanasiak Thank you for your screenshot!)

Why is it important?

latency config sets CloudWatch API start time and end time with a delay. For example, if the current time is 2020-08-31T18:30:00.000Z with collection period 1m.

Without latency:
startTime = 2020-08-31T18:29:00.000Z
endTime = 2020-08-31T18:30:00.000Z

With latency set to 5min:
startTime = 2020-08-31T18:24:00.000Z
endTime = 2020-08-31T18:25:00.000Z

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

  1. Enable aws module: ./metricbeat modules enable aws
  2. Change modules.d/aws.yml to:
- module: aws
  period: 5m
  credential_profile_name: elastic-beats
  latency: 5m
  metricsets:
    - ec2
  1. Start Metricbeat with debug level logging: ./metricbeat -e -d "*"
  2. You should see the startTime and endTime in debug log with a latency of 5 minutes applied.

* Add latency config option into aws module

(cherry picked from commit 7400f43)
@kaiyan-sheng kaiyan-sheng added [zube]: In Review backport Team:Platforms Label for the Integrations - Platforms team labels Oct 5, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-platforms (Team:Platforms)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Oct 5, 2020
@kaiyan-sheng kaiyan-sheng self-assigned this Oct 5, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented Oct 5, 2020

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #21537 updated]

  • Start Time: 2020-10-05T16:15:13.241+0000

  • Duration: 67 min 31 sec

Test stats 🧪

Test Results
Failed 0
Passed 2618
Skipped 596
Total 3214

@kaiyan-sheng kaiyan-sheng merged commit e75e3a4 into elastic:7.x Oct 5, 2020
@kaiyan-sheng kaiyan-sheng deleted the backport_20875_7.x branch October 5, 2020 17:31
@zube zube bot removed the [zube]: Done label Jan 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport Team:Platforms Label for the Integrations - Platforms team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants