Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Throttling exception using ssm maintenance window #191

Closed
suttph01 opened this issue Aug 10, 2020 · 4 comments
Closed

Throttling exception using ssm maintenance window #191

suttph01 opened this issue Aug 10, 2020 · 4 comments
Labels

Comments

@suttph01
Copy link

I am supporting an account with a large number of EC2 instances across multiple regions. The Instance scheduler is located in a single region, supporting itself & all other regions (4 in total). The schedules are configured to use the SSM maintenance window ( weekly event on saturday mornings).
The scheduler logs are reporting frequent ThrottlingException errors, e.g.
ERROR : Caught Exception while getting the maintenance window: An error occurred (ThrottlingException) when calling the DescribeMaintenanceWindows operation (reached max retries: 4): Rate exceeded
This appears to be causing the instances started for the maintenance window to stop frequently during the window.
Can the scheduler be updated to include a 'back-off' routine to prevent this?

@maykays maykays added the bug label Aug 21, 2020
@maykays
Copy link

maykays commented Aug 21, 2020

@suttph01 - potential bug; attempting to reproduce the issue.

@maykays
Copy link

maykays commented Oct 8, 2020

We recommend you update to v1.3.3 (or v1.3.2 at a minimum.) v1.3.2 contains a fix related to the SSM feature (change log notes below). This fix is carried over into the CDK v1.3.3.

1.3.2 - 2020-06-22
*Fix the issue to start instances before the SSM maintenance window beings #101
*Updated the SSM feature to reduce lambda cost
*Added HIBERNATE to the list of valid schedule properties

@zbornstein
Copy link

Does the new version of the scheduler 1.3.3 support retry backoff for EC2 and RDS scheduling? We see issues not related to SSM, but general API limits in our accounts.

@maykays
Copy link

maykays commented Apr 27, 2021

Fix shipped in v1.4

@maykays maykays closed this as completed Apr 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants