-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
kafka get-bootstrap-brokers doesn't return all brokers #500
Comments
Hi @karankand thanks for reaching out. Could you share your debug logs (with any sensitive info redacted) by adding |
`aws kafka get-bootstrap-brokers --cluster-arn arn:aws:kafka:us-east-1:510**********:cluster/m1-plm-stg-npe-kafka/d5c1c host:kafka.us-east-1.amazonaws.com host;x-amz-date |
Thanks for following up and sharing your logs. I didn't see any issues in the logs, but upon further research I found that this issue has come up before in other places. I think this comment helps provide some context: hashicorp/terraform-provider-aws#19659 (comment). As noted in the
I'm not sure why the API was designed this way but can try to get more information. Regardless this behavior should be clarified and noted in the CLI command documentation. I will transfer this to our cross-SDK repo as the issue applies to other SDKs, and I'll reach out to the Kafka team for further escalation. Please let us know if you have any follow up questions. |
P84110006 |
@tim-finnigan Our team is also running into this issue. Are you aware of any updates being made to the AWS SDK to allow you to return all broker connection strings, instead of just 3 from random AZs? Similar to the Hashicorp issue you linked to, if this string updates (which it does every time you call the CLI or SDK to get brokers), Terraform will force the MSK connector to rebuild when nothing actually changed, and this can cause a ton of issues. |
Hi @eduardoasantana I haven't heard back from the Kafka team regarding the documentation issue yet. I'm not aware of any updates made to the API — you can check the CHANGELOG of whichever SDK you're using to see changes in functionality by version. But I think the API returning 3 brokers is expected behavior based on this documentation: https://docs.aws.amazon.com/msk/latest/developerguide/msk-get-bootstrap-brokers.html.
I can't speak to the design of the API as I don't have context on that, but hoping the Kafka documentation writes can provide more insight. |
Hello, I think there has been some confusion. The The Kafka producer and consumer clients will auto discover other brokers in the cluster by making a getMetadata request to one of the bootstrap brokers. This metadata is periodically updated by the client and also updated in failure cases if a broker goes offline. The only reason for having 3 is having redundancy when performing initial discovery. |
Thanks for clarifying here, and I'll also note that API/command documentation has been updated: https://docs.aws.amazon.com/msk/1.0/apireference/clusters-clusterarn-bootstrap-brokers.html
|
This issue is now closed. Comments on closed issues are hard for our team to see. |
Describe the bug
kafka get-bootstrap-brokers doesn't return all brokers. Itried executing couple of time , every time only any 3 brokers will be returned.
Expected Behavior
Should return all brokers in cluster.
Current Behavior
only any 3 brokers will be returned.
Reproduction Steps
Cluster with more than 3 brokers.
aws kafka get-bootstrap-brokers --cluster-arn arn:aws:kafka:::cluster/*****
Brokers per zone - 2
Total number of brokers - 4
Possible Solution
No response
Additional Information/Context
No response
CLI version used
aws-cli/2.11.2
Environment details (OS name and version, etc.)
macOs Monterey 12.6.3
The text was updated successfully, but these errors were encountered: