Failed to describe Kafka cluster configs when using KafkaAdminTopicConfigProvider #2153
Comments
|
can you share your properties file I m using against MSK with IAM auth its working fine I m also using without zookeeper config |
|
|
The configuration looks fine to me. Have you verified if your Cruise Control IAM role can connect to your MSK? I tried running your configuration against my MSK, and it worked. |
|
These are the MSK-related permissions attached to the IAM role used by the pod where Cruise Control is running: {
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"kafka:RebootBroker",
"kafka:ListScramSecrets",
"kafka:ListNodes",
"kafka:ListKafkaVersions",
"kafka:ListConfigurations",
"kafka:ListConfigurationRevisions",
"kafka:GetBootstrapBrokers",
"kafka:DescribeConfiguration",
"kafka:DescribeCluster",
"kafka-cluster:WriteDataIdempotently",
"kafka-cluster:DescribeCluster",
"kafka-cluster:Connect",
"kafka-cluster:AlterCluster"
],
"Effect": "Allow",
"Resource": "arn:aws:kafka:*:<aws_account>:cluster/<msk_cluster_name>/<msk_cluster_id>",
"Sid": "AllowMskAccessCluster"
},
{
"Action": [
"kafka-cluster:WriteData",
"kafka-cluster:ReadData",
"kafka-cluster:DescribeTransactionalId",
"kafka-cluster:DescribeTopicDynamicConfiguration",
"kafka-cluster:DescribeTopic",
"kafka-cluster:DeleteTopic",
"kafka-cluster:CreateTopic",
"kafka-cluster:AlterTransactionalId",
"kafka-cluster:AlterTopicDynamicConfiguration",
"kafka-cluster:AlterTopic"
],
"Effect": "Allow",
"Resource": [
"arn:aws:kafka:*:<aws_account>:topic/<msk_cluster_name>/<msk_cluster_id>/*",
"arn:aws:kafka:*:<aws_account>:topic/<msk_cluster_name>/*"
],
"Sid": "AllowMskAccessTopic"
},
{
"Action": [
"kafka-cluster:DescribeGroup",
"kafka-cluster:DeleteGroup",
"kafka-cluster:AlterGroup"
],
"Effect": "Allow",
"Resource": "arn:aws:kafka:*:<aws_account>:group/<msk_cluster_name>/<msk_cluster_id>/*",
"Sid": "AllowMskAccessGroup"
}
]
}{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"kafka:RebootBroker",
"kafka:ListScramSecrets",
"kafka:ListNodes",
"kafka:ListKafkaVersions",
"kafka:ListConfigurations",
"kafka:ListConfigurationRevisions",
"kafka:GetBootstrapBrokers",
"kafka:DescribeConfiguration",
"kafka:DescribeCluster",
"kafka-cluster:WriteDataIdempotently",
"kafka-cluster:DescribeCluster",
"kafka-cluster:Connect",
"kafka-cluster:AlterCluster"
],
"Effect": "Allow",
"Resource": "arn:aws:kafka:*:<aws_account>:cluster/<msk_cluster_name>/<msk_cluster_id>",
"Sid": "AllowMskAccessCluster"
},
{
"Action": [
"kafka-cluster:WriteData",
"kafka-cluster:ReadData",
"kafka-cluster:DescribeTransactionalId",
"kafka-cluster:DescribeTopicDynamicConfiguration",
"kafka-cluster:DescribeTopic",
"kafka-cluster:DeleteTopic",
"kafka-cluster:CreateTopic",
"kafka-cluster:AlterTransactionalId",
"kafka-cluster:AlterTopicDynamicConfiguration",
"kafka-cluster:AlterTopic"
],
"Effect": "Allow",
"Resource": [
"arn:aws:kafka:*:<aws_account>:topic/<msk_cluster_name>/<msk_cluster_id>/*",
"arn:aws:kafka:*:<aws_account>:topic/<msk_cluster_name>/*"
],
"Sid": "AllowMskAccessTopic"
},
{
"Action": [
"kafka-cluster:DescribeGroup",
"kafka-cluster:DeleteGroup",
"kafka-cluster:AlterGroup"
],
"Effect": "Allow",
"Resource": "arn:aws:kafka:*:<aws_account>:group/<msk_cluster_name>/<msk_cluster_id>/*",
"Sid": "AllowMskAccessGroup"
}
]
} |
|
Also in the Cruise Control pod logs I see: The MSK cluster is running Kafka 2.8.1 |
|
Related issues/docs: |
|
I m on kafka 3.6.1 unsure if we can configure kafka version for CC anywhere |
|
Thanks for confirming. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I am deploying CC 2.5.137 on EKS, and trying to connect it to MSK
Following the "run without zookeeper" instructions, I am using the kafka admin topic config provider (instead of the non-admin one used in the zookeper setup).
Cruise Control fails during startup with this exception:
I have been trying to debug this issue for quite some time, especially checking the IAM policies, to no avail.
The text was updated successfully, but these errors were encountered: