Support for AWS MSK services has been added #143
Conversation
There was a problem hiding this comment.
Thanks @oukooveu. Permissions look good, just one missing when I tested with your new module PR and one of your classes has a boto call that needs to be updated.
We have a default age limit on CI resources of 20 minutes which it looks like won't be enough for create_cluster. You'll need to add an age_limit to both of your terminator classes to override this, you can see an example here: https://github.com/mattclay/aws-terminator/blob/master/aws/terminator/networking.py#L291
From some limited local testing it looks like we'll need at least 30 (maybe 40) minutes, is that consistent with your experiences using MSK? (I tested in both us-east-1 and us-west-2, so I don't think it was an anomaly.)
Which one is missing? Should I add it?
I've added it for
Practically it's great than 40 minutes, here is a real execution sample: ansible-collections/community.aws#547 (comment) |
There was a problem hiding this comment.
Thanks @oukooveu. Permissions look good, just one missing when I tested with your new module PR and one of your classes has a boto call that needs to be updated.
Which one is missing? Should I add it?
I meant the self.client.delete_cluster(ClusterArn=self.id) line in the terminator - you got it!
We have a default age limit on CI resources of 20 minutes which it looks like won't be enough for create_cluster. You'll need to add an
age_limitto both of your terminator classes to override this, you can see an example here: https://github.com/mattclay/aws-terminator/blob/master/aws/terminator/networking.py#L291I've added it for
KafkaClusterbut not forKafkaConfigurationbecause it's pretty fast (less than one minute usually). Please let me know if I have to add it forKafkaConfigurationalso.
Configurations that have a deployed cluster can't be deleted, so they need to have the same timeout. Otherwise the application would try to delete the Configuration while the Cluster is still active, fail, and spam errors every few minutes into our CI cloudwatch logs. We monitor these for issues, so it's better to avoid false positives.
age_limit has been added for KafkaConfiguration |
feat(kafka): aws_msk_config and aws_msk_cluster modules have been added SUMMARY Modules for Amazon Managed Streaming for Apache Kafka (Amazon MSK) have been added. There are two modules: aws_msk_config for managing MSK configuration and aws_msk_cluster -- for clusters. ISSUE TYPE New Module Pull Request COMPONENT NAME aws_msk_config aws_msk_cluster ADDITIONAL INFORMATION All MSK cluster related operations are time consuming, for example create takes 20-30 minutes, update Kafka version -- more than one hour, update configuration -- 10-15 minutes. Integration tests work in my own account, please let me know if I can help with an evidences. REFERENCES Related aws-terminator PR: mattclay/aws-terminator#143 Boto3: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/kafka.html Reviewed-by: Mark Chappell <None> Reviewed-by: Daniil Kupchenko <None> Reviewed-by: Alina Buzachis <None> Reviewed-by: Gonéri Le Bouder <[email protected]>
feat(kafka): aws_msk_config and aws_msk_cluster modules have been added SUMMARY Modules for Amazon Managed Streaming for Apache Kafka (Amazon MSK) have been added. There are two modules: aws_msk_config for managing MSK configuration and aws_msk_cluster -- for clusters. ISSUE TYPE New Module Pull Request COMPONENT NAME aws_msk_config aws_msk_cluster ADDITIONAL INFORMATION All MSK cluster related operations are time consuming, for example create takes 20-30 minutes, update Kafka version -- more than one hour, update configuration -- 10-15 minutes. Integration tests work in my own account, please let me know if I can help with an evidences. REFERENCES Related aws-terminator PR: mattclay/aws-terminator#143 Boto3: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/kafka.html Reviewed-by: Mark Chappell <None> Reviewed-by: Daniil Kupchenko <None> Reviewed-by: Alina Buzachis <None> Reviewed-by: Gonéri Le Bouder <[email protected]> This commit was initially merged in https://github.com/ansible-collections/community.aws See: ansible-collections@3bfcfe3
feat(kafka): aws_msk_config and aws_msk_cluster modules have been added SUMMARY Modules for Amazon Managed Streaming for Apache Kafka (Amazon MSK) have been added. There are two modules: aws_msk_config for managing MSK configuration and aws_msk_cluster -- for clusters. ISSUE TYPE New Module Pull Request COMPONENT NAME aws_msk_config aws_msk_cluster ADDITIONAL INFORMATION All MSK cluster related operations are time consuming, for example create takes 20-30 minutes, update Kafka version -- more than one hour, update configuration -- 10-15 minutes. Integration tests work in my own account, please let me know if I can help with an evidences. REFERENCES Related aws-terminator PR: mattclay/aws-terminator#143 Boto3: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/kafka.html Reviewed-by: Mark Chappell <None> Reviewed-by: Daniil Kupchenko <None> Reviewed-by: Alina Buzachis <None> Reviewed-by: Gonéri Le Bouder <[email protected]> This commit was initially merged in https://github.com/ansible-collections/community.aws See: ansible-collections@3bfcfe3
feat(kafka): aws_msk_config and aws_msk_cluster modules have been added SUMMARY Modules for Amazon Managed Streaming for Apache Kafka (Amazon MSK) have been added. There are two modules: aws_msk_config for managing MSK configuration and aws_msk_cluster -- for clusters. ISSUE TYPE New Module Pull Request COMPONENT NAME aws_msk_config aws_msk_cluster ADDITIONAL INFORMATION All MSK cluster related operations are time consuming, for example create takes 20-30 minutes, update Kafka version -- more than one hour, update configuration -- 10-15 minutes. Integration tests work in my own account, please let me know if I can help with an evidences. REFERENCES Related aws-terminator PR: mattclay/aws-terminator#143 Boto3: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/kafka.html Reviewed-by: Mark Chappell <None> Reviewed-by: Daniil Kupchenko <None> Reviewed-by: Alina Buzachis <None> Reviewed-by: Gonéri Le Bouder <[email protected]> This commit was initially merged in https://github.com/ansible-collections/community.aws See: ansible-collections@7d39179
feat(kafka): aws_msk_config and aws_msk_cluster modules have been added SUMMARY Modules for Amazon Managed Streaming for Apache Kafka (Amazon MSK) have been added. There are two modules: aws_msk_config for managing MSK configuration and aws_msk_cluster -- for clusters. ISSUE TYPE New Module Pull Request COMPONENT NAME aws_msk_config aws_msk_cluster ADDITIONAL INFORMATION All MSK cluster related operations are time consuming, for example create takes 20-30 minutes, update Kafka version -- more than one hour, update configuration -- 10-15 minutes. Integration tests work in my own account, please let me know if I can help with an evidences. REFERENCES Related aws-terminator PR: mattclay/aws-terminator#143 Boto3: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/kafka.html Reviewed-by: Mark Chappell <None> Reviewed-by: Daniil Kupchenko <None> Reviewed-by: Alina Buzachis <None> Reviewed-by: Gonéri Le Bouder <[email protected]>
feat(kafka): aws_msk_config and aws_msk_cluster modules have been added SUMMARY Modules for Amazon Managed Streaming for Apache Kafka (Amazon MSK) have been added. There are two modules: aws_msk_config for managing MSK configuration and aws_msk_cluster -- for clusters. ISSUE TYPE New Module Pull Request COMPONENT NAME aws_msk_config aws_msk_cluster ADDITIONAL INFORMATION All MSK cluster related operations are time consuming, for example create takes 20-30 minutes, update Kafka version -- more than one hour, update configuration -- 10-15 minutes. Integration tests work in my own account, please let me know if I can help with an evidences. REFERENCES Related aws-terminator PR: mattclay/aws-terminator#143 Boto3: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/kafka.html Reviewed-by: Mark Chappell <None> Reviewed-by: Daniil Kupchenko <None> Reviewed-by: Alina Buzachis <None> Reviewed-by: Gonéri Le Bouder <[email protected]>
feat(kafka): aws_msk_config and aws_msk_cluster modules have been added SUMMARY Modules for Amazon Managed Streaming for Apache Kafka (Amazon MSK) have been added. There are two modules: aws_msk_config for managing MSK configuration and aws_msk_cluster -- for clusters. ISSUE TYPE New Module Pull Request COMPONENT NAME aws_msk_config aws_msk_cluster ADDITIONAL INFORMATION All MSK cluster related operations are time consuming, for example create takes 20-30 minutes, update Kafka version -- more than one hour, update configuration -- 10-15 minutes. Integration tests work in my own account, please let me know if I can help with an evidences. REFERENCES Related aws-terminator PR: mattclay/aws-terminator#143 Boto3: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/kafka.html Reviewed-by: Mark Chappell <None> Reviewed-by: Daniil Kupchenko <None> Reviewed-by: Alina Buzachis <None> Reviewed-by: Gonéri Le Bouder <[email protected]>
feat(kafka): aws_msk_config and aws_msk_cluster modules have been added SUMMARY Modules for Amazon Managed Streaming for Apache Kafka (Amazon MSK) have been added. There are two modules: aws_msk_config for managing MSK configuration and aws_msk_cluster -- for clusters. ISSUE TYPE New Module Pull Request COMPONENT NAME aws_msk_config aws_msk_cluster ADDITIONAL INFORMATION All MSK cluster related operations are time consuming, for example create takes 20-30 minutes, update Kafka version -- more than one hour, update configuration -- 10-15 minutes. Integration tests work in my own account, please let me know if I can help with an evidences. REFERENCES Related aws-terminator PR: mattclay/aws-terminator#143 Boto3: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/kafka.html Reviewed-by: Mark Chappell <None> Reviewed-by: Daniil Kupchenko <None> Reviewed-by: Alina Buzachis <None> Reviewed-by: Gonéri Le Bouder <[email protected]>
Modules implementation PR: ansible-collections/community.aws#547