Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

storage: reactor stalls during shutdown with 20k partitions #4550

Closed
jcsp opened this issue May 4, 2022 · 0 comments · Fixed by #4860
Closed

storage: reactor stalls during shutdown with 20k partitions #4550

jcsp opened this issue May 4, 2022 · 0 comments · Fixed by #4860
Assignees
Labels
area/storage kind/bug Something isn't working

Comments

@jcsp
Copy link
Contributor

jcsp commented May 4, 2022

Seen in 22.1 rc3, but almost certainly not a new issue. Using ManyPartitionsTest from #4529 on i3en.xlarge instances.

Happens after a log line like

34.222.181.103: INFO  2022-05-04 14:36:20,379 [shard 3] cluster - state_machine.cc:139 - State machine for ntp={kafka/scale_000000/7603} handles raft::offset_monitor::wait_aborted (offset monitor wait aborted)

Backtrace is like

 (inlined by) ~metric_groups_impl at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/metrics.cc:184
 (inlined by) ~metric_groups_impl at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/metrics.cc:182
~probe at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-085d2d03242ede921-1/redpanda/redpanda/vbuild/release/clang/../../../src/v/storage/probe.h:43
 (inlined by) ~disk_log_impl at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-085d2d03242ede921-1/redpanda/redpanda/vbuild/release/clang/../../../src/v/storage/disk_log_impl.cc:80
 (inlined by) ~shared_ptr_count_for at /vectorized/include/seastar/core/shared_ptr.hh:464
 (inlined by) ~shared_ptr_count_for at /vectorized/include/seastar/core/shared_ptr.hh:464

This is probably just a case of iterating through destroying thousands of logs, each of which has many sub-object (e.g. metrics) to destroy, and not doing any yields in there.

There's no associated functional issue: the shutdown completes cleanly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/storage kind/bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants