CI Failure (BadLogLines) in ManyPartitionsTest.test_many_partitions_compacted

[VadimPlh]

https://buildkite.com/redpanda/vtools/builds/5976#018632a2-593f-4308-93c6-0843200511b9

Module: rptest.scale_tests.many_partitions_test
Class:  ManyPartitionsTest
Method: test_many_partitions_compacted

test_id:    rptest.scale_tests.many_partitions_test.ManyPartitionsTest.test_many_partitions_compacted
status:     FAIL
run time:   22 minutes 34.654 seconds


<BadLogLines nodes=ip-172-31-1-206(2) example="ERROR 2023-02-08 20:56:05,422 [shard  0] seastar_memory - Dumping seastar memory diagnostics">
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 67, in wrapped
    self.redpanda.raise_on_bad_logs(allow_list=log_allow_list)
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1745, in raise_on_bad_logs
    raise BadLogLines(bad_lines)
rptest.services.utils.BadLogLines: <BadLogLines nodes=ip-172-31-1-206(2) example="ERROR 2023-02-08 20:56:05,422 [shard  0] seastar_memory - Dumping seastar memory diagnostics">

[andijcr]

the error: seastar - Exceptional future ignored: seastar::named_semaphore_timed_out (Semaphore timed out: raft/connected),

the end of the log:

1498581-WARN  2023-02-08 20:56:05,309 [shard 11] r/heartbeat - heartbeat_manager.cc:384 - Error dispatching heartbeats - seastar::named_semaphore_timed_out (Semaphore timed out: raft/connected)
1498582-INFO  2023-02-08 20:56:05,318 [shard  9] kafka - server.cc:139 - Disconnected 172.31.12.241:45012 (applying protocol, Connection reset by peer)
1498583-WARN  2023-02-08 20:56:05,320 [shard 11] seastar - Exceptional future ignored: seastar::named_semaphore_timed_out (Semaphore timed out: raft/connected), backtrace: 0x51ac377 0x4ef2653 0x22c602f 0x4f9d7db 0x4fa0643 0x4fd8c17 0x4f4b1bb /opt/redpanda/lib/libc.so.6+0x843b7 /opt/redpanda/lib/libc.so.6+0xef2db
1498584-   --------
1498585-   seastar::continuation<seastar::internal::promise_base_with_type<void>, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1::operator()() const::'lambda'(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >), seastar::futurize<seastar::future<void> >::type seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >::then_wrapped_nrvo<seastar::future<void>, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1::operator()() const::'lambda'(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)>(raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1::operator()() const::'lambda'(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1::operator()() const::'lambda'(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)&, seastar::future_state<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >&&), boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >
1498586-   --------
1498587-   seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1>(seastar::gate&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1&&)::'lambda'(), false>, seastar::futurize<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1>::type seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1>(seastar::gate&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1&&)::'lambda'(), false> >(seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1>(seastar::gate&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1&&)::'lambda'(), false>&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1>(seastar::gate&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1&&)::'lambda'(), false>&, seastar::future_state<seastar::internal::monostate>&&), void>
1498588-WARN  2023-02-08 20:56:05,320 [shard  4] r/heartbeat - heartbeat_manager.cc:384 - Error dispatching heartbeats - seastar::named_semaphore_timed_out (Semaphore timed out: raft/connected)
1498589-INFO  2023-02-08 20:56:05,342 [shard 12] kafka - server.cc:139 - Disconnected 172.31.4.182:50850 (applying protocol, Connection reset by peer)
1498590-INFO  2023-02-08 20:56:05,345 [shard  2] kafka - server.cc:139 - Disconnected 172.31.4.182:37242 (applying protocol, Connection reset by peer)
1498591-WARN  2023-02-08 20:56:05,347 [shard 11] seastar - Exceptional future ignored: seastar::named_semaphore_timed_out (Semaphore timed out: raft/connected), backtrace: 0x51ac377 0x4ef2653 0x22c602f 0x4f9d7db 0x4fa0643 0x4fd8c17 0x4f4b1bb /opt/redpanda/lib/libc.so.6+0x843b7 /opt/redpanda/lib/libc.so.6+0xef2db
1498592-   --------
1498593-   seastar::continuation<seastar::internal::promise_base_with_type<void>, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1::operator()() const::'lambda'(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >), seastar::futurize<seastar::future<void> >::type seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >::then_wrapped_nrvo<seastar::future<void>, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1::operator()() const::'lambda'(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)>(raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1::operator()() const::'lambda'(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1::operator()() const::'lambda'(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)&, seastar::future_state<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >&&), boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >
1498594-   --------
1498595-   seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1>(seastar::gate&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1&&)::'lambda'(), false>, seastar::futurize<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1>::type seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1>(seastar::gate&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1&&)::'lambda'(), false> >(seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1>(seastar::gate&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1&&)::'lambda'(), false>&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1>(seastar::gate&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1&&)::'lambda'(), false>&, seastar::future_state<seastar::internal::monostate>&&), void>
1498596-INFO  2023-02-08 20:56:05,352 [shard  3] kafka - server.cc:139 - Disconnected 172.31.0.186:45886 (applying protocol, Connection reset by peer)
1498597-INFO  2023-02-08 20:56:05,362 [shard 11] kafka - server.cc:139 - Disconnected 172.31.0.186:45544 (applying protocol, Connection reset by peer)
1498598-WARN  2023-02-08 20:56:05,384 [shard 12] r/heartbeat - heartbeat_manager.cc:384 - Error dispatching heartbeats - seastar::named_semaphore_timed_out (Semaphore timed out: raft/connected)
1498599-WARN  2023-02-08 20:56:05,384 [shard  6] seastar - Exceptional future ignored: seastar::named_semaphore_timed_out (Semaphore timed out: raft/connected), backtrace: 0x51ac377 0x4ef2653 0x22c602f 0x4f9d7db 0x4fa0643 0x4fd8c17 0x4f4b1bb /opt/redpanda/lib/libc.so.6+0x843b7 /opt/redpanda/lib/libc.so.6+0xef2db
1498600-   --------
1498601-   seastar::continuation<seastar::internal::promise_base_with_type<void>, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1::operator()() const::'lambda'(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >), seastar::futurize<seastar::future<void> >::type seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >::then_wrapped_nrvo<seastar::future<void>, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1::operator()() const::'lambda'(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)>(raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1::operator()() const::'lambda'(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1::operator()() const::'lambda'(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)&, seastar::future_state<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >&&), boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >
1498602-   --------
1498603-   seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1>(seastar::gate&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1&&)::'lambda'(), false>, seastar::futurize<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1>::type seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1>(seastar::gate&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1&&)::'lambda'(), false> >(seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1>(seastar::gate&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1&&)::'lambda'(), false>&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1>(seastar::gate&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1&&)::'lambda'(), false>&, seastar::future_state<seastar::internal::monostate>&&), void>
1498604-INFO  2023-02-08 20:56:05,412 [shard  6] kafka - server.cc:139 - Disconnected 172.31.12.241:34410 (applying protocol, Connection reset by peer)
1498605:ERROR 2023-02-08 20:56:05,422 [shard  0] seastar_memory - Dumping seastar memory diagnostics
1498606-Used memory:  4301M
1498607-Free memory:  1211M
1498608-Total memory: 5G
1498609-
1498610-Small pools:
1498611-objsz	spansz	usedobj	memory	unused	wst%
1498612-8	4K	2k	40K	24K	59
1498613-10	4K	1	8K	8K	99
1498614-12	4K	25	88K	88K	99
1498615-14	4K	56	24K	23K	96
1498616-16	4K	184k	3M	633K	18
1498617-32	4K	45k	1M	3K	0
1498618-32	4K	753k	24M	1354K	5
1498619-32	4K	5k	184K	20K	10
1498620-32	4K	63k	7M	5122K	72
1498621-48	4K	18k	3M	2528K	75
1498622-48	4K	115k	13M	7702K	58
1498623-64	4K	58k	4M	8K	0
1498624-64	4K	158k	10M	15K	0
1498625-80	4K	33k	3M	426K	14
1498626-96	4K	128k	12M	25K	0
1498627-112	4K	901k	98M	2249K	2
1498628-128	4K	7k	896K	24K	2
1498629-160	4K	11k	2M	827K	32
1498630-192	4K	2k	648K	180K	27
1498631-224	4K	7k	2M	18K	1
1498632-256	4K	20k	8M	3170K	38
1498633-320	8K	11k	4M	855K	20
1498634-384	8K	2k	784K	73K	9
1498635-448	4K	4k	2M	40K	2

[mmaslankaprv]

The root cause of this issue is really OOM error:

ERROR 2023-02-08 20:56:05,429 [shard  0] seastar - Failed to allocate 579840 bytes
Aborting on shard 0.
Backtrace:
  0x4f840a3
  0x4fd78f3
  linux-vdso.so.1+0x7db
  /opt/redpanda/lib/libc.so.6+0x86067
  /opt/redpanda/lib/libc.so.6+0x3e87f
  /opt/redpanda/lib/libc.so.6+0x2aef7
  0x4f00333
  0x4f0ead3
  0x430c14f
  0x430a86f
  0x4313b93
  0x433fbfb
  0x433af8f
  0x3949717
  0x395548f
  0x4f9d7db
  0x4fa0643
  0x4f9e727
  0x4ed47b3
  0x4ed2faf
  0x1c99493
  0x523d2bb
  /opt/redpanda/lib/libc.so.6+0x2b1c7
  /opt/redpanda/lib/libc.so.6+0x2b29f
  0x1c9402f

[mmaslankaprv]

This issue is a duplicate of #7405, but i will solve the not handled exceptional future as a part of this