Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OOM due to 9 MB allocation in rm_stm #8507

Closed
travisdowns opened this issue Jan 30, 2023 · 14 comments
Closed

OOM due to 9 MB allocation in rm_stm #8507

travisdowns opened this issue Jan 30, 2023 · 14 comments
Assignees
@bharathv
Labels
area/raft area/transactions kind/bug Something isn't working

Comments

@travisdowns
Copy link
Member

Version & Environment

Redpanda version: 22.3.10

What went wrong?

Large allocation (9 MB) in rm_stm snapshot causes OOM.

What should have happened instead?

No OOM.

Additional information

Find the backtrace below:

operator new(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:2064
void* std::__1::__libcpp_operator_new<unsigned long>(unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:245
 (inlined by) std::__1::__libcpp_allocate(unsigned long, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:271
 (inlined by) std::__1::allocator<cluster::rm_stm::seq_entry>::allocate(unsigned long) at /vectorized/llvm/bin/../include/c++/v1/__memory/allocator.h:105
 (inlined by) std::__1::allocator_traits<std::__1::allocator<cluster::rm_stm::seq_entry> >::allocate(std::__1::allocator<cluster::rm_stm::seq_entry>&, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/__memory/allocator_traits.h:262
 (inlined by) std::__1::__split_buffer<cluster::rm_stm::seq_entry, std::__1::allocator<cluster::rm_stm::seq_entry>&>::__split_buffer(unsigned long, unsigned long, std::__1::allocator<cluster::rm_stm::seq_entry>&) at /vectorized/llvm/bin/../include/c++/v1/__split_buffer:306
 (inlined by) void std::__1::vector<cluster::rm_stm::seq_entry, std::__1::allocator<cluster::rm_stm::seq_entry> >::__push_back_slow_path<cluster::rm_stm::seq_entry>(cluster::rm_stm::seq_entry&&) at /vectorized/llvm/bin/../include/c++/v1/vector:1517
 (inlined by) std::__1::vector<cluster::rm_stm::seq_entry, std::__1::allocator<cluster::rm_stm::seq_entry> >::push_back(cluster::rm_stm::seq_entry&&) at /vectorized/llvm/bin/../include/c++/v1/vector:1549
 (inlined by) cluster::rm_stm::take_snapshot() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0fcef5c78516c756f-1/redpanda/redpanda/src/v/cluster/rm_stm.cc:2678
cluster::persisted_stm::do_make_snapshot() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0fcef5c78516c756f-1/redpanda/redpanda/src/v/cluster/persisted_stm.cc:133
cluster::persisted_stm::ensure_snapshot_exists(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::$_2::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0fcef5c78516c756f-1/redpanda/redpanda/src/v/cluster/persisted_stm.cc:168
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<cluster::persisted_stm::ensure_snapshot_exists(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::$_2::operator()() const::'lambda'()::operator()() const::'lambda'()>(cluster::persisted_stm::ensure_snapshot_exists(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::$_2::operator()() const::'lambda'()::operator()() const::'lambda'()&&) at /vectorized/include/seastar/core/future.hh:2149
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<cluster::persisted_stm::ensure_snapshot_exists(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::$_2::operator()() const::'lambda'()::operator()() const::'lambda'()>(cluster::persisted_stm::ensure_snapshot_exists(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::$_2::operator()() const::'lambda'()::operator()() const::'lambda'()&&, seastar::internal::monostate) at /vectorized/include/seastar/core/future.hh:1993
 (inlined by) seastar::future<void> seastar::future<void>::then_impl<cluster::persisted_stm::ensure_snapshot_exists(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::$_2::operator()() const::'lambda'()::operator()() const::'lambda'(), seastar::future<void> >(cluster::persisted_stm::ensure_snapshot_exists(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::$_2::operator()() const::'lambda'()::operator()() const::'lambda'()&&) at /vectorized/include/seastar/core/future.hh:1615
 (inlined by) seastar::internal::future_result<cluster::persisted_stm::ensure_snapshot_exists(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::$_2::operator()() const::'lambda'()::operator()() const::'lambda'(), void>::future_type seastar::internal::call_then_impl<seastar::future<void> >::run<cluster::persisted_stm::ensure_snapshot_exists(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::$_2::operator()() const::'lambda'()::operator()() const::'lambda'()>(seastar::future<void>&, cluster::persisted_stm::ensure_snapshot_exists(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::$_2::operator()() const::'lambda'()::operator()() const::'lambda'()&&) at /vectorized/include/seastar/core/future.hh:1248
 (inlined by) seastar::future<void> seastar::future<void>::then<cluster::persisted_stm::ensure_snapshot_exists(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::$_2::operator()() const::'lambda'()::operator()() const::'lambda'(), seastar::future<void> >(cluster::persisted_stm::ensure_snapshot_exists(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::$_2::operator()() const::'lambda'()::operator()() const::'lambda'()&&) at /vectorized/include/seastar/core/future.hh:1534
 (inlined by) cluster::persisted_stm::ensure_snapshot_exists(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::$_2::operator()() const::'lambda'()::operator()() const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0fcef5c78516c756f-1/redpanda/redpanda/src/v/cluster/persisted_stm.cc:161
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<cluster::persisted_stm::ensure_snapshot_exists(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::$_2::operator()() const::'lambda'()>(cluster::persisted_stm::ensure_snapshot_exists(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::$_2::operator()() const::'lambda'()&&) at /vectorized/include/seastar/core/future.hh:2149

The OOM occurs when filling a std::vector<seq_entry> in rm_stm::take_snapshot(). seq_entry is a 72 byte structure so this means a 9437184-byte allocation is a vector of 131072 of those. Is that a reasonable number of seq entries to be live on one shard?

I'm not sure.
@travisdowns travisdowns added kind/bug Something isn't working area/raft labels Jan 30, 2023
@dotnwat
Copy link
Member

dotnwat commented Jan 31, 2023

ping @bharathv @rystsov

@rystsov
Copy link
Contributor

rystsov commented Feb 6, 2023

The size of the snapshot depends on the client workload, what was the workload?

@rystsov rystsov self-assigned this Feb 6, 2023
@bharathv
Copy link
Contributor

bharathv commented Feb 6, 2023

This is same as https://redpandadata.slack.com/archives/C01ND4SVB6Z/p1675088864863619

This looks like a memory fragmentation issue..snapshot serialization/deserialization needs some fixing.

@rystsov
Copy link
Contributor

rystsov commented Feb 6, 2023

Let's discuss it during the stand up tomorrow. I may be able to take a look this week. The things I want to check are:

  • if we expire old pids and the associated cached seqs
  • if there are room for heuristic to expire the cache more agelessly

@travisdowns
Copy link
Member Author

travisdowns commented Feb 16, 2023 •
edited
Loading

See also slack for additional context: https://redpandadata.slack.com/archives/C01ND4SVB6Z/p1675088864863619

@kargh
Copy link

kargh commented Mar 6, 2023 

Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: ERROR 2023-03-06 08:20:50,481 [shard 32] seastar_memory - Dumping seastar memory diagnostics
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: Used memory:  601M
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: Free memory:  3137M
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: Total memory: 4G
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: Small pools:
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: objsz        spansz        usedobj        memory        unused        wst%
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 8        4K        67        52K        51K        98
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 10        4K        1        8K        8K        99
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 12        4K        3        24K        24K        99
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 14        4K        17        8K        8K        97
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 16        4K        37k        648K        70K        10
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 32        4K        4k        144K        5K        3
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 32        4K        5k        172K        8K        4
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 32        4K        1k        44K        6K        12
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 32        4K        11k        2M        1378K        80
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 48        4K        2k        80K        7K        8
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 48        4K        3k        1M        940K        87
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 64        4K        4k        296K        45K        15
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 64        4K        132k        9M        1019K        11
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 80        4K        4k        1M        822K        70
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 96        4K        3k        356K        30K        8
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 112        4K        319k        35M        592K        1
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 128        4K        26k        3M        215K        6
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 160        4K        3k        480K        36K        7
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 192        4K        130        76K        52K        67
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 224        4K        598        160K        29K        18
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 256        4K        688        712K        540K        75
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 320        8K        527        536K        371K        69
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 384        8K        436        240K        77K        31
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 448        4K        572        312K        62K        19
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 512        4K        67        232K        199K        85
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 640        16K        69        240K        197K        82
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 768        16K        502        592K        215K        36
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 896        8K        86        240K        165K        68
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 1024        4K        14        184K        170K        92
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 1280        32K        1k        3M        1923K        58
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 1536        32K        17        832K        806K        96
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 1792        16K        9        496K        480K        96
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 2048        8K        83        464K        298K        64
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 2560        64K        186        2M        1518K        76
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 3072        64K        21        2M        2M        96
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 3584        32K        11        896K        858K        95
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 4096        16K        199        2M        852K        51
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 5120        128K        79        3M        2M        85
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 6144        128K        21        2M        2M        93
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 7168        64K        63        3M        3M        87
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 8192        32K        4k        35M        7M        21
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 10240        64K        8        1M        1M        93
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 12288        64K        11        2M        2M        93
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 14336        128K        5        4M        4M        98
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 16384        64K        24k        377M        3M        0
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: Page spans:
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: index        size        free        used        spans
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 0        4K        5M        55M        15k
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 1        8K        8M        1M        1k
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 2        16K        13M        3M        1k
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 3        32K        45M        40M        3k
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 4        64K        94M        388M        8k
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 5        128K        193M        9M        2k
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 6        256K        341M        2M        1k
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 7        512K        575M        512K        1k
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 8        1M        826M        0B        826
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 9        2M        650M        6M        328
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 10        4M        300M        8M        77
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 11        8M        88M        8M        12
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 12        16M        0B        48M        3
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 13        32M        0B        32M        1
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 14        64M        0B        0B        0
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 15        128M        0B        0B        0
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 16        256M        0B        0B        0
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 17        512M        0B        0B        0
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 18        1G        0B        0B        0
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 19        2G        0B        0B        0
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 20        4G        0B        0B        0
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 21        8G        0B        0B        0
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 22        16G        0B        0B        0
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 23        32G        0B        0B        0
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 24        64G        0B        0B        0
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 25        128G        0B        0B        0
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 26        256G        0B        0B        0
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 27        512G        0B        0B        0
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 28        1T        0B        0B        0
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 29        2T        0B        0B        0
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 30        4T        0B        0B        0
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 31        8T        0B        0B        0
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: ERROR 2023-03-06 08:20:50,492 [shard 32] seastar - Failed to allocate 9437184 bytes
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: Aborting on shard 32.
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: Backtrace:
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 0x5bb46e6
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 0x5c180e2
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: /opt/redpanda/lib/libc.so.6+0x42abf
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: /opt/redpanda/lib/libc.so.6+0x92e3b
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: /opt/redpanda/lib/libc.so.6+0x42a15
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: /opt/redpanda/lib/libc.so.6+0x2c82e
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 0x5b275f6
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 0x5b35cf1
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 0x499d3bf
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 0x48b0903
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 0x48b8bee
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 0x48b1f12
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 0x48b1128
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 0x50e95c9
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 0x50e92c6
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 0x50e9df4
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 0x5bd281f
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 0x5bd64f7
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 0x5c19845
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: 0x5b745ef
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: /opt/redpanda/lib/libc.so.6+0x91016
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: /opt/redpanda/lib/libc.so.6+0x1166cf
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io systemd[1]: redpanda.service: main process exited, code=killed, status=6/ABRT
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io systemd[1]: Unit redpanda.service entered failed state.
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io systemd[1]: redpanda.service failed.
Mar 06 08:20:51 srv-01-04-404.iad1.trmr.io systemd[1]: redpanda.service holdoff time over, scheduling restart.
Mar 06 08:20:51 srv-01-04-404.iad1.trmr.io systemd[1]: Stopped Redpanda, the fastest queue in the West..
Mar 06 08:20:51 srv-01-04-404.iad1.trmr.io systemd[1]: Starting Redpanda, the fastest queue in the West....

Version: 23.1.1

We do NOT use transactions
We DO use idempotency

In the environment (that is having issues), we have a very large number of producers (~20k / node). This is unlikely to change anytime soon. In other environments where we have 2-3k producers per node we do not have any issues.

If I can provide any additional information that you might find helpful, please let me know.

@dotnwat
Copy link
Member

dotnwat commented Mar 6, 2023

thanks @kargh we are looking into this.

@dotnwat
Copy link
Member

dotnwat commented Mar 6, 2023 

Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]: Backtrace:
Mar 06 08:20:50 srv-01-04-404.iad1.trmr.io rpk[54429]:
[Backtrace #0]
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::'lambda'(seastar::frame)>(seastar::backtrace_buffer::append_backtrace()::'lambda'(seastar::frame)&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:59
 (inlined by) seastar::backtrace_buffer::append_backtrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:754
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:784
seastar::print_with_backtrace(char const*, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:796
 (inlined by) seastar::sigabrt_action() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3663
 (inlined by) void seastar::install_oneshot_signal_handler<6, (void (*)())(&seastar::sigabrt_action())>()::'lambda'(int, siginfo_t*, void*)::operator()(int, siginfo_t*, void*) const at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3639
 (inlined by) void seastar::install_oneshot_signal_handler<6, (void (*)())(&seastar::sigabrt_action())>()::'lambda'(int, siginfo_t*, void*)::__invoke(int, siginfo_t*, void*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3635
LLVMSymbolizer: error reading file: No such file or directory
?? at ??:0
?? at ??:0
?? at ??:0
?? at ??:0
seastar::memory::on_allocation_failure(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1821
 (inlined by) seastar::memory::allocate(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1410
operator new(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:2079
void* std::__1::__libcpp_operator_new<unsigned long>(unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:245
 (inlined by) std::__1::__libcpp_allocate(unsigned long, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:271
 (inlined by) std::__1::allocator<cluster::rm_stm::seq_entry>::allocate(unsigned long) at /vectorized/llvm/bin/../include/c++/v1/__memory/allocator.h:105
 (inlined by) std::__1::allocator_traits<std::__1::allocator<cluster::rm_stm::seq_entry>>::allocate(std::__1::allocator<cluster::rm_stm::seq_entry>&, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/__memory/allocator_traits.h:262
 (inlined by) std::__1::__split_buffer<cluster::rm_stm::seq_entry, std::__1::allocator<cluster::rm_stm::seq_entry>&>::__split_buffer(unsigned long, unsigned long, std::__1::allocator<cluster::rm_stm::seq_entry>&) at /vectorized/llvm/bin/../include/c++/v1/__split_buffer:306
 (inlined by) void std::__1::vector<cluster::rm_stm::seq_entry, std::__1::allocator<cluster::rm_stm::seq_entry>>::__push_back_slow_path<cluster::rm_stm::seq_entry>(cluster::rm_stm::seq_entry&&) at /vectorized/llvm/bin/../include/c++/v1/vector:1517
 (inlined by) std::__1::vector<cluster::rm_stm::seq_entry, std::__1::allocator<cluster::rm_stm::seq_entry>>::push_back(cluster::rm_stm::seq_entry&&) at /vectorized/llvm/bin/../include/c++/v1/vector:1549
 (inlined by) cluster::rm_stm::take_snapshot() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-06d29fac678da1dec-1/redpanda/redpanda/src/v/cluster/rm_stm.cc:2759
cluster::persisted_stm::do_make_snapshot() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-06d29fac678da1dec-1/redpanda/redpanda/src/v/cluster/persisted_stm.cc:140
cluster::persisted_stm::make_snapshot()::$_2::operator()() const::'lambda'()::operator()() const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-06d29fac678da1dec-1/redpanda/redpanda/src/v/cluster/persisted_stm.cc:154
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void>>::invoke<cluster::persisted_stm::make_snapshot()::$_2::operator()() const::'lambda'()>(cluster::persisted_stm::make_snapshot()::$_2::operator()() const::'lambda'()&&) at /vectorized/include/seastar/core/future.hh:2149
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void>>::invoke<cluster::persisted_stm::make_snapshot()::$_2::operator()() const::'lambda'()>(cluster::persisted_stm::make_snapshot()::$_2::operator()() const::'lambda'()&&, seastar::internal::monostate) at /vectorized/include/seastar/core/future.hh:1993
 (inlined by) seastar::future<void> seastar::future<void>::then_impl<cluster::persisted_stm::make_snapshot()::$_2::operator()() const::'lambda'(), seastar::future<void>>(cluster::persisted_stm::make_snapshot()::$_2::operator()() const::'lambda'()&&) at /vectorized/include/seastar/core/future.hh:1615
 (inlined by) seastar::internal::future_result<cluster::persisted_stm::make_snapshot()::$_2::operator()() const::'lambda'(), void>::future_type seastar::internal::call_then_impl<seastar::future<void>>::run<cluster::persisted_stm::make_snapshot()::$_2::operator()() const::'lambda'()>(seastar::future<void>&, cluster::persisted_stm::make_snapshot()::$_2::operator()() const::'lambda'()&&) at /vectorized/include/seastar/core/future.hh:1248
 (inlined by) seastar::future<void> seastar::future<void>::then<cluster::persisted_stm::make_snapshot()::$_2::operator()() const::'lambda'(), seastar::future<void>>(cluster::persisted_stm::make_snapshot()::$_2::operator()() const::'lambda'()&&) at /vectorized/include/seastar/core/future.hh:1534
 (inlined by) cluster::persisted_stm::make_snapshot()::$_2::operator()() const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-06d29fac678da1dec-1/redpanda/redpanda/src/v/cluster/persisted_stm.cc:154
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void>>::invoke<cluster::persisted_stm::make_snapshot()::$_2>(cluster::persisted_stm::make_snapshot()::$_2&&) at /vectorized/include/seastar/core/future.hh:2149
 (inlined by) auto seastar::futurize_invoke<cluster::persisted_stm::make_snapshot()::$_2>(cluster::persisted_stm::make_snapshot()::$_2&&) at /vectorized/include/seastar/core/future.hh:2180
 (inlined by) auto seastar::futurize<std::__1::invoke_result<cluster::persisted_stm::make_snapshot()::$_2>::type>::type seastar::with_semaphore<seastar::named_semaphore_exception_factory, cluster::persisted_stm::make_snapshot()::$_2, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, cluster::persisted_stm::make_snapshot()::$_2&&)::'lambda'(seastar::named_semaphore_exception_factory)::operator()<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>>(seastar::named_semaphore_exception_factory) at /vectorized/include/seastar/core/semaphore.hh:742
seastar::future<void> seastar::futurize<seastar::future<void>>::invoke<seastar::futurize<std::__1::invoke_result<cluster::persisted_stm::make_snapshot()::$_2>::type>::type seastar::with_semaphore<seastar::named_semaphore_exception_factory, cluster::persisted_stm::make_snapshot()::$_2, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, cluster::persisted_stm::make_snapshot()::$_2&&)::'lambda'(seastar::named_semaphore_exception_factory), seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>>(seastar::named_semaphore_exception_factory&&, seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>&&) at /vectorized/include/seastar/core/future.hh:2149
 (inlined by) cluster::persisted_stm::make_snapshot()::$_2 seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>>::then_impl<seastar::futurize<std::__1::invoke_result<cluster::persisted_stm::make_snapshot()::$_2>::type>::type seastar::with_semaphore<seastar::named_semaphore_exception_factory, cluster::persisted_stm::make_snapshot()::$_2, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, cluster::persisted_stm::make_snapshot()::$_2&&)::'lambda'(seastar::named_semaphore_exception_factory), seastar::future<void>>(seastar::named_semaphore_exception_factory&&) at /vectorized/include/seastar/core/future.hh:1615
 (inlined by) seastar::internal::future_result<seastar::named_semaphore_exception_factory, seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>>::future_type seastar::internal::call_then_impl<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>>>::run<seastar::futurize<std::__1::invoke_result<cluster::persisted_stm::make_snapshot()::$_2>::type>::type seastar::with_semaphore<seastar::named_semaphore_exception_factory, cluster::persisted_stm::make_snapshot()::$_2, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, cluster::persisted_stm::make_snapshot()::$_2&&)::'lambda'(seastar::named_semaphore_exception_factory)>(seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>>&, seastar::named_semaphore_exception_factory&&) at /vectorized/include/seastar/core/future.hh:1248
 (inlined by) cluster::persisted_stm::make_snapshot()::$_2 seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>>::then<seastar::futurize<std::__1::invoke_result<cluster::persisted_stm::make_snapshot()::$_2>::type>::type seastar::with_semaphore<seastar::named_semaphore_exception_factory, cluster::persisted_stm::make_snapshot()::$_2, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, cluster::persisted_stm::make_snapshot()::$_2&&)::'lambda'(seastar::named_semaphore_exception_factory), seastar::future<void>>(seastar::named_semaphore_exception_factory&&) at /vectorized/include/seastar/core/future.hh:1534
 (inlined by) seastar::futurize<std::__1::invoke_result<cluster::persisted_stm::make_snapshot()::$_2>::type>::type seastar::with_semaphore<seastar::named_semaphore_exception_factory, cluster::persisted_stm::make_snapshot()::$_2, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, cluster::persisted_stm::make_snapshot()::$_2&&) at /vectorized/include/seastar/core/semaphore.hh:741
 (inlined by) auto mutex::with<cluster::persisted_stm::make_snapshot()::$_2>(cluster::persisted_stm::make_snapshot()::$_2&&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-06d29fac678da1dec-1/redpanda/redpanda/src/v/utils/mutex.h:42
 (inlined by) cluster::persisted_stm::make_snapshot() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-06d29fac678da1dec-1/redpanda/redpanda/src/v/cluster/persisted_stm.cc:152
cluster::persisted_stm::make_snapshot_in_background()::$_4::operator()() const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-06d29fac678da1dec-1/redpanda/redpanda/src/v/cluster/persisted_stm.cc:148
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void>>::invoke<cluster::persisted_stm::make_snapshot_in_background()::$_4>(cluster::persisted_stm::make_snapshot_in_background()::$_4&&) at /vectorized/include/seastar/core/future.hh:2149
 (inlined by) auto seastar::futurize_invoke<cluster::persisted_stm::make_snapshot_in_background()::$_4>(cluster::persisted_stm::make_snapshot_in_background()::$_4&&) at /vectorized/include/seastar/core/future.hh:2180
 (inlined by) auto seastar::internal::invoke_func_with_gate<cluster::persisted_stm::make_snapshot_in_background()::$_4>(seastar::gate&, cluster::persisted_stm::make_snapshot_in_background()::$_4&&) at /vectorized/include/seastar/core/gate.hh:221
 (inlined by) auto seastar::try_with_gate<cluster::persisted_stm::make_snapshot_in_background()::$_4>(seastar::gate&, cluster::persisted_stm::make_snapshot_in_background()::$_4&&) at /vectorized/include/seastar/core/gate.hh:261
 (inlined by) auto ssx::spawn_with_gate_then<cluster::persisted_stm::make_snapshot_in_background()::$_4>(seastar::gate&, cluster::persisted_stm::make_snapshot_in_background()::$_4&&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-06d29fac678da1dec-1/redpanda/redpanda/src/v/ssx/future-util.h:282
 (inlined by) void ssx::spawn_with_gate<cluster::persisted_stm::make_snapshot_in_background()::$_4>(seastar::gate&, cluster::persisted_stm::make_snapshot_in_background()::$_4&&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-06d29fac678da1dec-1/redpanda/redpanda/src/v/ssx/future-util.h:301
 (inlined by) cluster::persisted_stm::make_snapshot_in_background() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-06d29fac678da1dec-1/redpanda/redpanda/src/v/cluster/persisted_stm.cc:148
storage::stm_manager::make_snapshot_in_background() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-06d29fac678da1dec-1/redpanda/redpanda/src/v/storage/types.h:136
 (inlined by) storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9::operator()(seastar::lw_shared_ptr<storage::segment>)::'lambda'()::operator()() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-06d29fac678da1dec-1/redpanda/redpanda/src/v/storage/disk_log_impl.cc:913
seastar::future<void> seastar::futurize<void>::invoke<storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9::operator()(seastar::lw_shared_ptr<storage::segment>)::'lambda'()>(storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9::operator()(seastar::lw_shared_ptr<storage::segment>)::'lambda'()&&) at /vectorized/include/seastar/core/future.hh:2146
 (inlined by) seastar::future<void> seastar::futurize<void>::invoke<storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9::operator()(seastar::lw_shared_ptr<storage::segment>)::'lambda'()>(storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9::operator()(seastar::lw_shared_ptr<storage::segment>)::'lambda'()&&, seastar::internal::monostate) at /vectorized/include/seastar/core/future.hh:1993
 (inlined by) seastar::future<void> seastar::future<void>::then_impl<storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9::operator()(seastar::lw_shared_ptr<storage::segment>)::'lambda'(), seastar::future<void>>(storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9::operator()(seastar::lw_shared_ptr<storage::segment>)::'lambda'()&&) at /vectorized/include/seastar/core/future.hh:1615
 (inlined by) seastar::internal::future_result<storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9::operator()(seastar::lw_shared_ptr<storage::segment>)::'lambda'(), void>::future_type seastar::internal::call_then_impl<seastar::future<void>>::run<storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9::operator()(seastar::lw_shared_ptr<storage::segment>)::'lambda'()>(seastar::future<void>&, storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9::operator()(seastar::lw_shared_ptr<storage::segment>)::'lambda'()&&) at /vectorized/include/seastar/core/future.hh:1248
 (inlined by) seastar::future<void> seastar::future<void>::then<storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9::operator()(seastar::lw_shared_ptr<storage::segment>)::'lambda'(), seastar::future<void>>(storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9::operator()(seastar::lw_shared_ptr<storage::segment>)::'lambda'()&&) at /vectorized/include/seastar/core/future.hh:1534
 (inlined by) storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9::operator()(seastar::lw_shared_ptr<storage::segment>) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-06d29fac678da1dec-1/redpanda/redpanda/src/v/storage/disk_log_impl.cc:905
decltype(static_cast<storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9&>(fp)(static_cast<seastar::lw_shared_ptr<storage::segment>>(fp0))) std::__1::__invoke<storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9&, seastar::lw_shared_ptr<storage::segment>>(storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9&, seastar::lw_shared_ptr<storage::segment>&&) at /vectorized/llvm/bin/../include/c++/v1/type_traits:3640
 (inlined by) std::__1::invoke_result<storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9&, seastar::lw_shared_ptr<storage::segment>>::type std::__1::invoke<storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9&, seastar::lw_shared_ptr<storage::segment>>(storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9&, seastar::lw_shared_ptr<storage::segment>&&) at /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:93
 (inlined by) auto seastar::internal::future_invoke<storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9&, seastar::lw_shared_ptr<storage::segment>>(storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9&, seastar::lw_shared_ptr<storage::segment>&&) at /vectorized/include/seastar/core/future.hh:1225
 (inlined by) seastar::future<void> seastar::future<seastar::lw_shared_ptr<storage::segment>>::then_impl_nrvo<storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9, seastar::future<void>>(storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9&, seastar::future_state<seastar::lw_shared_ptr<storage::segment>>&&)::operator()(seastar::internal::promise_base_with_type<void>&&, storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9&, seastar::future_state<seastar::lw_shared_ptr<storage::segment>>&&) const::'lambda'()::operator()() const at /vectorized/include/seastar/core/future.hh:1596
 (inlined by) void seastar::futurize<seastar::future<void>>::satisfy_with_result_of<seastar::future<void> seastar::future<seastar::lw_shared_ptr<storage::segment>>::then_impl_nrvo<storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9, seastar::future<void>>(storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9&, seastar::future_state<seastar::lw_shared_ptr<storage::segment>>&&)::operator()(seastar::internal::promise_base_with_type<void>&&, storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9&, seastar::future_state<seastar::lw_shared_ptr<storage::segment>>&&) const::'lambda'()>(seastar::internal::promise_base_with_type<void>&&, storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9&&) at /vectorized/include/seastar/core/future.hh:2134
 (inlined by) seastar::future<void> seastar::future<seastar::lw_shared_ptr<storage::segment>>::then_impl_nrvo<storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9, seastar::future<void>>(storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9&, seastar::future_state<seastar::lw_shared_ptr<storage::segment>>&&)::operator()(seastar::internal::promise_base_with_type<void>&&, storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9&, seastar::future_state<seastar::lw_shared_ptr<storage::segment>>&&) const at /vectorized/include/seastar/core/future.hh:1589
 (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<void>, storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9, seastar::future<void> seastar::future<seastar::lw_shared_ptr<storage::segment>>::then_impl_nrvo<storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9, seastar::future<void>>(storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, storage::disk_log_impl::new_segment(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true>>, seastar::io_priority_class)::$_9&, seastar::future_state<seastar::lw_shared_ptr<storage::segment>>&&), seastar::lw_shared_ptr<storage::segment>>::run_and_dispose() at /vectorized/include/seastar/core/future.hh:781
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2330
 (inlined by) seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2737
seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2906
seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95::operator()() const at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4140
 (inlined by) decltype(static_cast<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95&>(fp)()) std::__1::__invoke<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95&) at /vectorized/llvm/bin/../include/c++/v1/type_traits:3640
 (inlined by) void std::__1::__invoke_void_return_wrapper<void, true>::__call<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95&) at /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:61
 (inlined by) std::__1::__function::__alloc_func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95>, void ()>::operator()() at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:180
 (inlined by) std::__1::__function::__func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95>, void ()>::operator()() at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:354
std::__1::__function::__value_func<void ()>::operator()() const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:507
 (inlined by) std::__1::function<void ()>::operator()() const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:1184
 (inlined by) seastar::posix_thread::start_routine(void*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/posix.cc:73
?? at ??:0
?? at ??:0

@dotnwat
Copy link
Member

dotnwat commented Mar 7, 2023

Per out-of-band conversation:

  • Swap out std::vector for std::fragmented_vector
  • On-disk formats are identical for the two data structures
  • The on-disk vector is also populating an absl map, which should probably be swapped to a fragmented associative data structure as well assuming it is the number of elements that is the issue and both vector/map have similar value sizes.

@emaxerrno
Copy link
Contributor

Good insights here.

@travisdowns travisdowns mentioned this issue Mar 14, 2023
Closed
@travisdowns
Copy link
Member Author

@rystsov could you take a peek at #9408 and confirm if it's a duplicate?

The stack looks identical and I'd normally dupe it out right away but I wanted to check since the workload here was quite different: it occurs when one node is down for some time with a synthetic high throughput load and then comes back up. I don't think there are many distinct producers however.

@rystsov rystsov assigned bharathv and unassigned rystsov Mar 15, 2023
@rystsov
Copy link
Contributor

rystsov commented Mar 15, 2023

@bharathv is working on switching to fragmented_vector https://github.com/bharathv/redpanda/commits/switch_to_frag

@rystsov
Copy link
Contributor

rystsov commented Mar 16, 2023

@travisdowns dropped a comment there on how to confirm it

@bharathv
Copy link
Contributor

Fixed with #9484 #9616

Will do manual backports to 23.1.x due to conflicts.

@ztlpn ztlpn mentioned this issue Apr 5, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bharathv bharathv

Labels
area/raft area/transactions kind/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@emaxerrno @dotnwat @bharathv @rystsov @travisdowns @kargh