[Bug]: Changing KafkaTopic replicas and config simultaneously corrupts the topic

[vereshchagin-d]

Bug Description

Hello!
We noticed a strange behavior when we tried to change the replicas and config parameters of a topic simultaneously. After updating the manifest, the KafkaTopic object state got stuck in pending with the message "Replicas change failed, Request failed (500), Cluster model not ready."
Thanks in advance.

Steps to reproduce

1. Create topic

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  labels:
    strimzi.io/cluster: staging
  name: test
  namespace: staging
spec:
  partitions: 3
  replicas: 1
  topicName: test-topic

2. Update spec

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  labels:
    strimzi.io/cluster: staging
  name: test
  namespace: staging
spec:
  config:
    retention.ms: 600000
    segment.bytes: 3145728
    segment.ms: 600000
  partitions: 3
  replicas: 3
  topicName: test-topic

3. Got an error

Expected behavior

Number of replicas set to desired number.
Config applied.

Strimzi version

0.42.0

Kubernetes version

k8s 1.29.1

Installation method

Helm chart

Infrastructure

Cloud

Configuration files and logs

No response

Additional context

No response

[scholzj]

What does it mean that it got stuck? For how long for example? Hours? Days? Weeks? The best might be to share the full logs from the Topic Operator.

[vereshchagin-d]

Hi!

Yes, in my case, it took hours. For some reason, after the replicas and config were changed at 17:28, the operator wasn't able to complete the task, so the next day I had to revert the changes at 10:17.
"it got stuck" topic looked strange in Kafka UI

• it kminion wasn't able to produce messages (since it requires ack from all replicas)

2024-10-08 17:28:11.340	2024-10-08 14:28:11,16020 INFO  [LoopRunnable-0] ReplicasChangeHandler:176 - Replicas change pending, Topics: [kminion-end-to-end]
2024-10-08 17:28:12.343	2024-10-08 14:28:12,12913 INFO  [LoopRunnable-0] ReplicasChangeHandler:184 - Replicas change ongoing, Topics: [kminion-end-to-end]
2024-10-08 17:30:33.711	2024-10-08 14:30:33,46181 INFO  [LoopRunnable-0] ReplicasChangeHandler:176 - Task not found, Resetting the state, Topics: [kminion-end-to-end]
2024-10-08 17:30:33.962	2024-10-08 14:30:33,76668 INFO  [LoopRunnable-0] ReplicasChangeHandler:176 - Replicas change pending, Topics: [kminion-end-to-end]
2024-10-08 17:30:33.962	2024-10-08 14:30:33,91044 ERROR [LoopRunnable-0] ReplicasChangeHandler:198 - Replicas change failed, Request failed (500), Error processing POST request '/topic_configuration' due to: 'java.lang.IllegalStateException: All topics matching given pattern already have target replication factor. Requested topic pattern by replication factor: {3=kminion-end-to-end}.'., Topics: [kminion-end-to-end]
2024-10-08 17:32:35.512	2024-10-08 14:32:35,46505 INFO  [LoopRunnable-0] ReplicasChangeHandler:176 - Replicas change pending, Topics: [kminion-end-to-end]
2024-10-08 17:32:36.514	2024-10-08 14:32:36,41889 INFO  [LoopRunnable-0] ReplicasChangeHandler:184 - Replicas change ongoing, Topics: [kminion-end-to-end]
2024-10-09 10:17:53.523	2024-10-09 07:17:53,42703 INFO  [LoopRunnable-0] ReplicasChangeHandler:191 - Replicas change completed, Topics: [kminion-end-to-end]

I'm wondering if it could be related to segment changes, replica changes, or both?

[fvaleri]

Hi @vereshchagin-d. This is not a bug, Cruise Control takes some time to build its internal cluster model when it is started, or a new topic is created. During this period, it is expected to see the error message you are reporting, but eventually the topic will converge to the desired state, or give another type of error. Hope it helps.

[vereshchagin-d]

Thanks for the quick response!
I'll try to reproduce this again with timings and come back with more details (timeline + logs).

[vereshchagin-d]

Hello!

I've tried again with my local kind cluster. Is it okay that it takes approximately 10 minutes to scale replicas? (Created topic, checked that it's in the ready state, changed the number of replicas.) Can we speed this up somehow?

Thanks.

[fvaleri]

As I said, when you create a new topic, CC needs some time to collect metrics and build its internal cluster model, which is then used to perform the RF change. If you change the RF for an existing topic, the it will be much faster, depending on the amount of data it has to replicate/delete.

You also have to consider that CC executes one task at a time so, for example, if a cluster rebalance is ongoing, your RF change will have to wait in the queue for some time before being executed.

There isn't much tuning you can do on CC side, apart from making sure that you are not throttling. You should also make sure that the network between CC and brokers is fast, and CC has enough resources.