Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrade hang in group_metadata_migration when consumer group topic doesn't already exist #4469

Closed
jcsp opened this issue Apr 28, 2022 · 0 comments · Fixed by #4474
Closed

Upgrade hang in group_metadata_migration when consumer group topic doesn't already exist #4469

jcsp opened this issue Apr 28, 2022 · 0 comments · Fixed by #4474
Assignees
@VadimPlh
Labels
area/controller kind/bug Something isn't working
Milestone
v22.1.1

Comments

@jcsp
Copy link
Contributor

jcsp commented Apr 28, 2022

Found by @VadimPlh with the new ducktape upgrade test.

  • 3 nodes on 21.11.x
  • upgrade 1 node to 22.1.x
  • The upgraded node isn't controller leader
  • It enters group_metadata_migration::start and hits the "kafka_internal/group topic does not exists, activating" path
  • This call waits for activate_feature
  • activate_feature loops until the feature is active, but it cannot be activated because only the controller leader runs the feature_manager logic for activating features, and the controller leader is a 21.11.x node that doesn't have the code.
  • The node remains in 'booting' state indefinitely

I think the fix is probably simple, to spawn a background fiber with the activate feature call, so that its loop doesn't block the startup of redpanda.
@jcsp jcsp added kind/bug Something isn't working area/controller labels Apr 28, 2022
@jcsp jcsp added this to the v22.1.1 milestone Apr 28, 2022
VadimPlh added a commit to VadimPlh/redpanda that referenced this issue Apr 28, 2022
@VadimPlh
kafka/server: run activate_feature in background
da96d96 
Ufter upgrading node in cluster to 22.1.x version from 22.11.x
upgraded node isn't controller leade, it enters group_metadata_migration::start
and hits the "kafka_internal/group topic does not exists, activating" path
this call waits for activate_feature,activate_feature loops until the feature is active,
but it cannot be activated because only the controller leader runs the feature_manager
logic for activating features, and the controller leader is a 21.11.x node
that doesn't have the code. The node remains in 'booting' state indefinitely

Fixes: redpanda-data#4469
@VadimPlh VadimPlh mentioned this issue Apr 28, 2022
Merged
@VadimPlh VadimPlh self-assigned this Apr 28, 2022
VadimPlh added a commit to VadimPlh/redpanda that referenced this issue Apr 29, 2022
@VadimPlh
kafka/server: run activate_feature in background
62e4356 
Ufter upgrading node in cluster to 22.1.x version from 22.11.x
upgraded node isn't controller leade, it enters group_metadata_migration::start
and hits the "kafka_internal/group topic does not exists, activating" path
this call waits for activate_feature,activate_feature loops until the feature is active,
but it cannot be activated because only the controller leader runs the feature_manager
logic for activating features, and the controller leader is a 21.11.x node
that doesn't have the code. The node remains in 'booting' state indefinitely

Fixes: redpanda-data#4469
VadimPlh added a commit to VadimPlh/redpanda that referenced this issue Apr 29, 2022
@VadimPlh
kafka/server: run activate_feature in background
d42bf28 
Ufter upgrading node in cluster to 22.1.x version from 22.11.x
upgraded node isn't controller leade, it enters group_metadata_migration::start
and hits the "kafka_internal/group topic does not exists, activating" path
this call waits for activate_feature,activate_feature loops until the feature is active,
but it cannot be activated because only the controller leader runs the feature_manager
logic for activating features, and the controller leader is a 21.11.x node
that doesn't have the code. The node remains in 'booting' state indefinitely

Fixes: redpanda-data#4469
VadimPlh added a commit to VadimPlh/redpanda that referenced this issue May 1, 2022
@VadimPlh
kafka/server: run activate_feature in background
bdf83f5 
Ufter upgrading node in cluster to 22.1.x version from 22.11.x
upgraded node isn't controller leade, it enters group_metadata_migration::start
and hits the "kafka_internal/group topic does not exists, activating" path
this call waits for activate_feature,activate_feature loops until the feature is active,
but it cannot be activated because only the controller leader runs the feature_manager
logic for activating features, and the controller leader is a 21.11.x node
that doesn't have the code. The node remains in 'booting' state indefinitely

Fixes: redpanda-data#4469
VadimPlh added a commit to VadimPlh/redpanda that referenced this issue May 1, 2022
@VadimPlh
kafka/server: run activate_feature in background
e88a340 
Ufter upgrading node in cluster to 22.1.x version from 22.11.x
upgraded node isn't controller leade, it enters group_metadata_migration::start
and hits the "kafka_internal/group topic does not exists, activating" path
this call waits for activate_feature,activate_feature loops until the feature is active,
but it cannot be activated because only the controller leader runs the feature_manager
logic for activating features, and the controller leader is a 21.11.x node
that doesn't have the code. The node remains in 'booting' state indefinitely

Fixes: redpanda-data#4469
VadimPlh added a commit to VadimPlh/redpanda that referenced this issue May 4, 2022
@VadimPlh
kafka/server: run activate_feature in background
aa94f69 
Ufter upgrading node in cluster to 22.1.x version from 22.11.x
upgraded node isn't controller leade, it enters group_metadata_migration::start
and hits the "kafka_internal/group topic does not exists, activating" path
this call waits for activate_feature,activate_feature loops until the feature is active,
but it cannot be activated because only the controller leader runs the feature_manager
logic for activating features, and the controller leader is a 21.11.x node
that doesn't have the code. The node remains in 'booting' state indefinitely

Fixes: redpanda-data#4469
VadimPlh added a commit to VadimPlh/redpanda that referenced this issue May 4, 2022
@VadimPlh
kafka/server: run activate_feature in background
303d2d3 
Ufter upgrading node in cluster to 22.1.x version from 22.11.x
upgraded node isn't controller leade, it enters group_metadata_migration::start
and hits the "kafka_internal/group topic does not exists, activating" path
this call waits for activate_feature,activate_feature loops until the feature is active,
but it cannot be activated because only the controller leader runs the feature_manager
logic for activating features, and the controller leader is a 21.11.x node
that doesn't have the code. The node remains in 'booting' state indefinitely

Fixes: redpanda-data#4469
VadimPlh added a commit to VadimPlh/redpanda that referenced this issue May 4, 2022
@VadimPlh
kafka/server: run activate_feature in background
1e2ded3 
Ufter upgrading node in cluster to 22.1.x version from 22.11.x
upgraded node isn't controller leade, it enters group_metadata_migration::start
and hits the "kafka_internal/group topic does not exists, activating" path
this call waits for activate_feature,activate_feature loops until the feature is active,
but it cannot be activated because only the controller leader runs the feature_manager
logic for activating features, and the controller leader is a 21.11.x node
that doesn't have the code. The node remains in 'booting' state indefinitely

Fixes: redpanda-data#4469
VadimPlh added a commit to VadimPlh/redpanda that referenced this issue May 4, 2022
@VadimPlh
kafka/server: run activate_feature in background
b7fb5bd 
Ufter upgrading node in cluster to 22.1.x version from 22.11.x
upgraded node isn't controller leade, it enters group_metadata_migration::start
and hits the "kafka_internal/group topic does not exists, activating" path
this call waits for activate_feature,activate_feature loops until the feature is active,
but it cannot be activated because only the controller leader runs the feature_manager
logic for activating features, and the controller leader is a 21.11.x node
that doesn't have the code. The node remains in 'booting' state indefinitely

Fixes: redpanda-data#4469
vbotbuildovich pushed a commit to vbotbuildovich/redpanda that referenced this issue May 4, 2022
@VadimPlh @vbotbuildovich
kafka/server: run activate_feature in background
799d3ce 
Ufter upgrading node in cluster to 22.1.x version from 22.11.x
upgraded node isn't controller leade, it enters group_metadata_migration::start
and hits the "kafka_internal/group topic does not exists, activating" path
this call waits for activate_feature,activate_feature loops until the feature is active,
but it cannot be activated because only the controller leader runs the feature_manager
logic for activating features, and the controller leader is a 21.11.x node
that doesn't have the code. The node remains in 'booting' state indefinitely

Fixes: redpanda-data#4469
(cherry picked from commit b7fb5bd)
@vbotbuildovich vbotbuildovich mentioned this issue May 4, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@VadimPlh VadimPlh

Labels
area/controller kind/bug Something isn't working
Projects
None yet
Milestone
v22.1.1
Development

Successfully merging a pull request may close this issue.

kafka/server: run activate_feature in background VadimPlh/redpanda
2 participants
@jcsp @VadimPlh