kafka get-bootstrap-brokers doesn't return all brokers

[karankand]

Describe the bug

kafka get-bootstrap-brokers doesn't return all brokers. Itried executing couple of time , every time only any 3 brokers will be returned.

Expected Behavior

Should return all brokers in cluster.

Current Behavior

only any 3 brokers will be returned.

Reproduction Steps

Cluster with more than 3 brokers.
aws kafka get-bootstrap-brokers --cluster-arn arn:aws:kafka:::cluster/*****
Brokers per zone - 2
Total number of brokers - 4

Possible Solution

No response

Additional Information/Context

No response

CLI version used

aws-cli/2.11.2

Environment details (OS name and version, etc.)

macOs Monterey 12.6.3

[tim-finnigan]

Hi @karankand thanks for reaching out. Could you share your debug logs (with any sensitive info redacted) by adding --debug to your command? I'm not very familiar with the get-bootstrap-brokers or sure what this issue might be but the logs may help provide more insight. Thanks!

[karankand]

`aws kafka get-bootstrap-brokers --cluster-arn arn:aws:kafka:us-east-1:510**********:cluster/m1-plm-stg-npe-kafka/d5c1c
50d-c3df-4b58-8883-d5fd969dd4f7-1 --debug
2023-03-20 15:35:05,710 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.11.2 Python/3.11.2 Darwin/21.6.0 exe/x86_64
2023-03-20 15:35:05,710 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['kafka', 'get-bootstrap-brokers', '--cluster-arn', 'arn:aws:kafka:us-east-1:510**********:cluster/m1-plm-stg-npe-kafka/d5c1c50d-c3df-4b58-8883-d5fd969dd4f7-1', '--debug']
2023-03-20 15:35:05,820 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x10e2759e0>
2023-03-20 15:35:05,820 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x10e0aaf20>
2023-03-20 15:35:05,820 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
2023-03-20 15:35:05,820 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x10dd36fc0>
2023-03-20 15:35:05,820 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x10dd4e480>
2023-03-20 15:35:05,820 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x10e277ec0>
2023-03-20 15:35:05,820 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x10e0f5940>
2023-03-20 15:35:05,820 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
2023-03-20 15:35:05,820 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x10e277d80>
2023-03-20 15:35:05,820 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x10e32d210>>
2023-03-20 15:35:05,823 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/data/cli.json
2023-03-20 15:35:05,824 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_types at 0x10e1af100>
2023-03-20 15:35:05,824 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function no_sign_request at 0x10e1af420>
2023-03-20 15:35:05,825 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x10e1af380>
2023-03-20 15:35:05,825 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x10e1af560>
2023-03-20 15:35:05,825 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x10e1af4c0>
2023-03-20 15:35:05,825 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x10e32a4c0>
2023-03-20 15:35:05,825 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.11.2 Python/3.11.2 Darwin/21.6.0 exe/x86_64 prompt/off
2023-03-20 15:35:05,825 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['kafka', 'get-bootstrap-brokers', '--cluster-arn', 'arn:aws:kafka:us-east-1:510**********:cluster/m1-plm-stg-npe-kafka/d5c1c50d-c3df-4b58-8883-d5fd969dd4f7-1', '--debug']
2023-03-20 15:35:05,825 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x10e276340>
2023-03-20 15:35:05,825 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x10dafc9a0>
2023-03-20 15:35:05,825 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x10e2f5300>
2023-03-20 15:35:05,825 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function no_pager_handler at 0x10dad4fe0>
2023-03-20 15:35:05,826 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x10db2a160>
2023-03-20 15:35:05,829 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2023-03-20 15:35:05,830 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x10e0de200>
2023-03-20 15:35:05,830 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x10e0971a0>
2023-03-20 15:35:05,923 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/kafka/2018-11-14/service-2.json
2023-03-20 15:35:05,927 - MainThread - botocore.hooks - DEBUG - Event building-command-table.kafka: calling handler <function add_waiters at 0x10e277d80>
2023-03-20 15:35:06,026 - MainThread - botocore.hooks - DEBUG - Event building-command-table.kafka: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x10e32d210>>
2023-03-20 15:35:06,026 - MainThread - awscli.clidriver - DEBUG - OrderedDict([('cluster-arn', <awscli.arguments.CLIArgument object at 0x10e32e5d0>)])
2023-03-20 15:35:06,026 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.kafka.get-bootstrap-brokers: calling handler <function add_streaming_output_arg at 0x10e276840>
2023-03-20 15:35:06,027 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.kafka.get-bootstrap-brokers: calling handler <function add_cli_input_json at 0x10db2aac0>
2023-03-20 15:35:06,027 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.kafka.get-bootstrap-brokers: calling handler <function add_cli_input_yaml at 0x10db2ab60>
2023-03-20 15:35:06,027 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.kafka.get-bootstrap-brokers: calling handler <function unify_paging_params at 0x10e0ab560>
2023-03-20 15:35:06,125 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/kafka/2018-11-14/paginators-1.json
2023-03-20 15:35:06,125 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.kafka.get-bootstrap-brokers: calling handler <function add_generate_skeleton at 0x10e1ad800>
2023-03-20 15:35:06,125 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.kafka.get-bootstrap-brokers: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x10e45e950>>
2023-03-20 15:35:06,125 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.kafka.get-bootstrap-brokers: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x10e445490>>
2023-03-20 15:35:06,125 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.kafka.get-bootstrap-brokers: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x10e446d10>>
2023-03-20 15:35:06,125 - MainThread - botocore.hooks - DEBUG - Event building-command-table.kafka_get-bootstrap-brokers: calling handler <function add_waiters at 0x10e277d80>
2023-03-20 15:35:06,125 - MainThread - botocore.hooks - DEBUG - Event building-command-table.kafka_get-bootstrap-brokers: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x10e32d210>>
2023-03-20 15:35:06,126 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.kafka.get-bootstrap-brokers.cluster-arn: calling handler <awscli.paramfile.URIArgumentHandler object at 0x10dbab7d0>
2023-03-20 15:35:06,126 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.kafka.get-bootstrap-brokers: calling handler <awscli.argprocess.ParamShorthandParser object at 0x10dc82cd0>
2023-03-20 15:35:06,126 - MainThread - awscli.arguments - DEBUG - Unpacked value of 'arn:aws:kafka:us-east-1:510**********:cluster/m1-plm-stg-npe-kafka/d5c1c50d-c3df-4b58-8883-d5fd969dd4f7-1' for parameter "cluster_arn": 'arn:aws:kafka:us-east-1:510**********:cluster/m1-plm-stg-npe-kafka/d5c1c50d-c3df-4b58-8883-d5fd969dd4f7-1'
2023-03-20 15:35:06,127 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.kafka.get-bootstrap-brokers.cli-input-json: calling handler <awscli.paramfile.URIArgumentHandler object at 0x10dbab7d0>
2023-03-20 15:35:06,127 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.kafka.get-bootstrap-brokers.cli-input-yaml: calling handler <awscli.paramfile.URIArgumentHandler object at 0x10dbab7d0>
2023-03-20 15:35:06,127 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.kafka.get-bootstrap-brokers.generate-cli-skeleton: calling handler <awscli.paramfile.URIArgumentHandler object at 0x10dbab7d0>
2023-03-20 15:35:06,127 - MainThread - botocore.hooks - DEBUG - Event calling-command.kafka.get-bootstrap-brokers: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x10e45e950>>
2023-03-20 15:35:06,130 - MainThread - botocore.hooks - DEBUG - Event calling-command.kafka.get-bootstrap-brokers: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x10e445490>>
2023-03-20 15:35:06,130 - MainThread - botocore.hooks - DEBUG - Event calling-command.kafka.get-bootstrap-brokers: calling handler <bound method GenerateCliSkeletonArgument.generate_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x10e446d10>>
2023-03-20 15:35:06,130 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2023-03-20 15:35:06,130 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2023-03-20 15:35:06,130 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2023-03-20 15:35:06,131 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: sso
2023-03-20 15:35:06,133 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2023-03-20 15:35:06,134 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2023-03-20 15:35:06,140 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/endpoints.json
2023-03-20 15:35:06,151 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x10caa05e0>
2023-03-20 15:35:06,245 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/kafka/2018-11-14/endpoint-rule-set-1.json
2023-03-20 15:35:06,248 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/partitions.json
2023-03-20 15:35:06,249 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.kafka: calling handler <function add_generate_presigned_url at 0x10c9f4d60>
2023-03-20 15:35:06,251 - MainThread - botocore.endpoint - DEBUG - Setting kafka timeout as (60, 60)
2023-03-20 15:35:06,251 - MainThread - botocore.regions - DEBUG - Calling endpoint provider with parameters: {'Region': 'us-east-1', 'UseDualStack': False, 'UseFIPS': False}
2023-03-20 15:35:06,252 - MainThread - botocore.regions - DEBUG - Endpoint provider result: https://kafka.us-east-1.amazonaws.com
2023-03-20 15:35:06,252 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.kafka.GetBootstrapBrokers: calling handler <function base64_decode_input_blobs at 0x10e2f53a0>
2023-03-20 15:35:06,252 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.kafka.GetBootstrapBrokers: calling handler <function generate_idempotent_uuid at 0x10caa2840>
2023-03-20 15:35:06,252 - MainThread - botocore.hooks - DEBUG - Event before-call.kafka.GetBootstrapBrokers: calling handler <function inject_api_version_header_if_needed at 0x10cac4360>
2023-03-20 15:35:06,252 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=GetBootstrapBrokers) with params: {'url_path': '/v1/clusters/arn%3Aaws%3Akafka%3Aus-east-1%3A510**********%3Acluster%2Fm1-plm-stg-npe-kafka%2Fd5c1c50d-c3df-4b58-8883-d5fd969dd4f7-1/bootstrap-brokers', 'query_string': {}, 'method': 'GET', 'headers': {'User-Agent': 'aws-cli/2.11.2 Python/3.11.2 Darwin/21.6.0 exe/x86_64 prompt/off command/kafka.get-bootstrap-brokers'}, 'body': b'', 'url': 'https://kafka.us-east-1.amazonaws.com/v1/clusters/arn%3Aaws%3Akafka%3Aus-east-1%3A510**********%3Acluster%2Fm1-plm-stg-npe-kafka%2Fd5c1c50d-c3df-4b58-8883-d5fd969dd4f7-1/bootstrap-brokers', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x10e88d590>, 'has_streaming_input': False, 'auth_type': None}}
2023-03-20 15:35:06,253 - MainThread - botocore.hooks - DEBUG - Event request-created.kafka.GetBootstrapBrokers: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x10e84a790>>
2023-03-20 15:35:06,254 - MainThread - botocore.hooks - DEBUG - Event choose-signer.kafka.GetBootstrapBrokers: calling handler <function set_operation_specific_signer at 0x10caa2700>
2023-03-20 15:35:06,254 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2023-03-20 15:35:06,254 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
GET
/v1/clusters/arn%253Aaws%253Akafka%253Aus-east-1%253A510**********%253Acluster%252Fm1-plm-stg-npe-kafka%252Fd5c1c50d-c3df-4b58-8883-d5fd969dd4f7-1/bootstrap-brokers

host:kafka.us-east-1.amazonaws.com
x-amz-date:20230320T100506Z

host;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2023-03-20 15:35:06,254 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20230320T100506Z
20230320/us-east-1/kafka/aws4_request
8becb28f7cce50331e810c282922d6d7e02fd711a6717390ba0fe3709aa82d25
2023-03-20 15:35:06,258 - MainThread - botocore.auth - DEBUG - Signature:
bf42b5d9f5be664b02c18754466129aa138be837af5a5438926f90e6049b51ae
2023-03-20 15:35:06,259 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://kafka.us-east-1.amazonaws.com/v1/clusters/arn%3Aaws%3Akafka%3Aus-east-1%3A510**********%3Acluster%2Fm1-plm-stg-npe-kafka%2Fd5c1c50d-c3df-4b58-8883-d5fd969dd4f7-1/bootstrap-brokers, headers={'User-Agent': b'aws-cli/2.11.2 Python/3.11.2 Darwin/21.6.0 exe/x86_64 prompt/off command/kafka.get-bootstrap-brokers', 'X-Amz-Date': b'20230320T100506Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=****/20230320/us-east-1/kafka/aws4_request, SignedHeaders=host;x-amz-date, Signature=**************************'}>
2023-03-20 15:35:06,259 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/awscli/botocore/cacert.pem
2023-03-20 15:35:06,262 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): kafka.us-east-1.amazonaws.com:443
2023-03-20 15:35:07,451 - MainThread - urllib3.connectionpool - DEBUG - https://kafka.us-east-1.amazonaws.com:443 "GET /v1/clusters/arn%3Aaws%3Akafka%3Aus-east-1%3A510%3Acluster%2Fm1-plm-stg-npe-kafka%2Fd5c1c50d-c3df-4b58-8883-d5fd969dd4f7-1/bootstrap-brokers HTTP/1.1" 200 476
2023-03-20 15:35:07,451 - MainThread - botocore.parsers - DEBUG - Response headers: {'Content-Type': 'application/json', 'Content-Length': '476', 'Connection': 'keep-alive', 'Date': 'Mon, 20 Mar 2023 10:05:07 GMT', 'x-amzn-RequestId': 'f0aabf6d-951f-4e6a-b22b-99519d8d5049', 'Access-Control-Allow-Origin': '', 'Access-Control-Allow-Headers': 'Authorization,Date,X-Amz-Date,X-Amz-Security-Token,X-Amz-Target,content-type,x-amz-content-sha256,x-amz-user-agent,x-amzn-platform-id,x-amzn-trace-id', 'x-amz-apigw-id': 'CExpCGgpIAMFhtA=', 'Access-Control-Allow-Methods': '', 'Access-Control-Expose-Headers': 'x-amzn-errortype,x-amzn-requestid,x-amzn-errormessage,x-amzn-trace-id,x-amz-apigw-id,date', 'X-Amzn-Trace-Id': 'Root=1-64182fd3-50896e7e2993d66f3d84f8f9', 'Access-Control-Max-Age': '86400', 'X-Cache': 'Miss from cloudfront', 'Via': '1.1 a1ace081afaa01eee6efbc2bb1e8d5ea.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'BOM78-P4', 'X-Amz-Cf-Id': 'wehMlb3_wT8JzS63kQxxtHLElJKjAfge2oIOeKBjEA7lDiAWw0MNWw=='}
2023-03-20 15:35:07,451 - MainThread - botocore.parsers - DEBUG - Response body:
b'{"bootstrapBrokerString":"b-1.m1-plm-stg-npe-kafka.78c8hp.c1.kafka.us-east-1.amazonaws.com:9092,b-3.m1-plm-stg-npe-kafka.78c8hp.c1.kafka.us-east-1.amazonaws.com:9092,b-2.m1-plm-stg-npe-kafka.78c8hp.c1.kafka.us-east-1.amazonaws.com:9092","bootstrapBrokerStringTls":"b-1.m1-plm-stg-npe-kafka.78c8hp.c1.kafka.us-east-1.amazonaws.com:9094,b-3.m1-plm-stg-npe-kafka.78c8hp.c1.kafka.us-east-1.amazonaws.com:9094,b-2.m1-plm-stg-npe-kafka.78c8hp.c1.kafka.us-east-1.amazonaws.com:9094"}'
2023-03-20 15:35:07,452 - MainThread - botocore.hooks - DEBUG - Event needs-retry.kafka.GetBootstrapBrokers: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x10e88f410>>
2023-03-20 15:35:07,452 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2023-03-20 15:35:07,452 - MainThread - botocore.hooks - DEBUG - Event after-call.kafka.GetBootstrapBrokers: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x10e84a390>>
2023-03-20 15:35:07,453 - MainThread - awscli.formatter - DEBUG - RequestId: f0aabf6d-951f-4e6a-b22b-99519d8d5049
{
"BootstrapBrokerString": "b-1.m1-plm-stg-npe-kafka.78c8hp.c1.kafka.us-east-1.amazonaws.com:9092,b-3.m1-plm-stg-npe-kafka.78c8hp.c1.kafka.us-east-1.amazonaws.com:9092,b-2.m1-plm-stg-npe-kafka.78c8hp.c1.kafka.us-east-1.amazonaws.com:9092",
"BootstrapBrokerStringTls": "b-1.m1-plm-stg-npe-kafka.78c8hp.c1.kafka.us-east-1.amazonaws.com:9094,b-3.m1-plm-stg-npe-kafka.78c8hp.c1.kafka.us-east-1.amazonaws.com:9094,b-2.m1-plm-stg-npe-kafka.78c8hp.c1.kafka.us-east-1.amazonaws.com:9094"
}
`

[tim-finnigan]

Thanks for following up and sharing your logs. I didn't see any issues in the logs, but upon further research I found that this issue has come up before in other places. I think this comment helps provide some context: hashicorp/terraform-provider-aws#19659 (comment).

As noted in the BootstrapBrokers documentation:

Operation ID: GetBootstrapBrokers

A list of brokers that a client can use to bootstrap. This list doesn't necessarily include all of the brokers in the cluster.

I'm not sure why the API was designed this way but can try to get more information. Regardless this behavior should be clarified and noted in the CLI command documentation. I will transfer this to our cross-SDK repo as the issue applies to other SDKs, and I'll reach out to the Kafka team for further escalation. Please let us know if you have any follow up questions.

[tim-finnigan]

P84110006

[eduardoasantana]

@tim-finnigan Our team is also running into this issue. Are you aware of any updates being made to the AWS SDK to allow you to return all broker connection strings, instead of just 3 from random AZs?

Similar to the Hashicorp issue you linked to, if this string updates (which it does every time you call the CLI or SDK to get brokers), Terraform will force the MSK connector to rebuild when nothing actually changed, and this can cause a ton of issues.

[tim-finnigan]

Hi @eduardoasantana I haven't heard back from the Kafka team regarding the documentation issue yet. I'm not aware of any updates made to the API — you can check the CHANGELOG of whichever SDK you're using to see changes in functionality by version. But I think the API returning 3 brokers is expected behavior based on this documentation: https://docs.aws.amazon.com/msk/latest/developerguide/msk-get-bootstrap-brokers.html.

The bootstrap brokers string should contain three brokers from across the Availability Zones in which your MSK cluster is deployed (unless only two brokers are available).

I can't speak to the design of the API as I don't have context on that, but hoping the Kafka documentation writes can provide more insight.

[jayteej]

Hello, I think there has been some confusion. The get-bootstrap-brokers api only returns 3 brokers by design. Specifically these brokers are a random selection of 3 for "bootstrapping".

The Kafka producer and consumer clients will auto discover other brokers in the cluster by making a getMetadata request to one of the bootstrap brokers. This metadata is periodically updated by the client and also updated in failure cases if a broker goes offline.

The only reason for having 3 is having redundancy when performing initial discovery.

[tim-finnigan]

Thanks for clarifying here, and I'll also note that API/command documentation has been updated: https://docs.aws.amazon.com/msk/1.0/apireference/clusters-clusterarn-bootstrap-brokers.html

This list doesn't necessarily include all of the brokers in the cluster.

[github-actions]

This issue is now closed.

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.