Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow set scrape interval to avoid throttling errors #103

Open
nhlushak opened this issue Jan 4, 2024 · 1 comment
Open

Allow set scrape interval to avoid throttling errors #103

nhlushak opened this issue Jan 4, 2024 · 1 comment
Assignees
Labels
question Further information is requested

Comments

@nhlushak
Copy link

nhlushak commented Jan 4, 2024

Is your feature request related to a problem? Please describe.

After running RDS exporter in single region, containing 65 databases we encountered periodic errors in logs about being throttled:

04 January 2024 at 17:40 (UTC+2:00)	{"time":"2024-01-04T15:40:30.006779978Z","level":"INFO","msg":"get RDS metrics"}	rds-exporter
04 January 2024 at 17:40 (UTC+2:00)	{"time":"2024-01-04T15:40:21.450411631Z","level":"ERROR","msg":"can't scrape metrics: can't fetch RDS metrics: can't compute instance metrics for xxxxxxx: can't get log files size for 274885408640: can't describe db logs files for xxxxxxx: operation error RDS: DescribeDBLogFiles, exceeded maximum number of attempts, 3, https response error StatusCode: 400, RequestID: f1e80964-6a77-xxxxxx, api error Throttling: Rate exceeded"}	rds-exporter
04 January 2024 at 17:40 (UTC+2:00)	{"time":"2024-01-04T15:40:20.003623961Z","level":"INFO","msg":"get RDS metrics"}	rds-exporter
04 January 2024 at 17:40 (UTC+2:00)	{"time":"2024-01-04T15:40:10.001978348Z","level":"INFO","msg":"get RDS metrics"}

Describe the solution you'd like

We would like to see some option to set scrape interval for exporter, so we may find some sweetspot between metrics freshness and not being throttled by AWS API.

Describe alternatives you've considered

Run exporter as scheduled operation to scrape data once, give it back to scraper and die until next run.

@nhlushak nhlushak added the enhancement New feature or request label Jan 4, 2024
@vmercierfr
Copy link
Collaborator

Hello,

Sorry for this late reply, but you can actually set the scrape interval in serviceMonitor.interval if you are using our Helm chart or scrape_interval parameter on any Prometheus server.

In you context, I'm suspecting that the AWS API throttling is due to RDS instances log files size compute.
You may try to disable logs size collection.

@vmercierfr vmercierfr self-assigned this Mar 25, 2024
@vmercierfr vmercierfr added invalid This doesn't seem right question Further information is requested and removed enhancement New feature or request invalid This doesn't seem right labels Mar 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants