Oversized 300K allocation during topic deletion

[travisdowns]

Version & Environment

Redpanda version: 23.2.17

What went wrong?

In the OOM described in #15607, the final allocation is almost 300K:

2023-12-14T13:29:59.838702085Z stderr F ERROR 2023-12-14 13:29:59,838 [shard 13] seastar - Failed to allocate 294920 bytes
2023-12-14T13:29:59.83871032Z stderr F Aborting on shard 13.

This exceeds the memory allocation limit of 128K. This wasn't the underlying problem in this OOM (more than 99% of memory was used), but regardless we should fix any allocation larger than 128K.

The allocation occurs in the topic_metadata_item::partitions map while adding a partition to the map in topic_table. The size makes sense: we have ~22,900 partitoins and assuming node hash map is an array of pointer-sized things (highly likely) we have 183K for that many elements: assuming a 2x growth strategy, this could allocate up to ~360K plus some headroom for load factor. So the 300K is completely explained.

The decoded stack is:

seastar::memory::on_allocation_failure(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:2054
seastar::memory::finish_allocation(void*, unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1579
 (inlined by) seastar::memory::allocate_slowpath(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1618
seastar::memory::allocate(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1629
 (inlined by) operator new(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:2318
void* std::__1::__libcpp_operator_new[abi:v160004]<unsigned long>(unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:266
 (inlined by) std::__1::__libcpp_allocate[abi:v160004](unsigned long, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:292
 (inlined by) std::__1::allocator<absl::lts_20220623::container_internal::AlignedType<8ul>>::allocate[abi:v160004](unsigned long) at /vectorized/llvm/bin/../include/c++/v1/__memory/allocator.h:115
 (inlined by) std::__1::allocator_traits<std::__1::allocator<absl::lts_20220623::container_internal::AlignedType<8ul>>>::allocate[abi:v160004](std::__1::allocator<absl::lts_20220623::container_internal::AlignedType<8ul>>&, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/__memory/allocator_traits.h:268
 (inlined by) void* absl::lts_20220623::container_internal::Allocate<8ul, std::__1::allocator<std::__1::pair<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>> const, cluster::topic_table::partition_meta>>>(std::__1::allocator<std::__1::pair<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>> const, cluster::topic_table::partition_meta>>*, unsigned long) at /vectorized/include/absl/container/internal/container_memory.h:64
 (inlined by) absl::lts_20220623::container_internal::raw_hash_set<absl::lts_20220623::container_internal::NodeHashMapPolicy<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>, cluster::topic_table::partition_meta>, absl::lts_20220623::hash_internal::Hash<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>> const, cluster::topic_table::partition_meta>>>::initialize_slots() at /vectorized/include/absl/container/internal/raw_hash_set.h:1937
 (inlined by) absl::lts_20220623::container_internal::raw_hash_set<absl::lts_20220623::container_internal::NodeHashMapPolicy<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>, cluster::topic_table::partition_meta>, absl::lts_20220623::hash_internal::Hash<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>> const, cluster::topic_table::partition_meta>>>::resize(unsigned long) at /vectorized/include/absl/container/internal/raw_hash_set.h:1978
 (inlined by) absl::lts_20220623::container_internal::raw_hash_set<absl::lts_20220623::container_internal::NodeHashMapPolicy<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>, cluster::topic_table::partition_meta>, absl::lts_20220623::hash_internal::Hash<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>> const, cluster::topic_table::partition_meta>>>::rehash_and_grow_if_necessary() at /vectorized/include/absl/container/internal/raw_hash_set.h:2126
absl::lts_20220623::container_internal::raw_hash_set<absl::lts_20220623::container_internal::NodeHashMapPolicy<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>, cluster::topic_table::partition_meta>, absl::lts_20220623::hash_internal::Hash<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>> const, cluster::topic_table::partition_meta>>>::prepare_insert(unsigned long) at /vectorized/include/absl/container/internal/raw_hash_set.h:2191
std::__1::pair<unsigned long, bool> absl::lts_20220623::container_internal::raw_hash_set<absl::lts_20220623::container_internal::NodeHashMapPolicy<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>, cluster::topic_table::partition_meta>, absl::lts_20220623::hash_internal::Hash<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>> const, cluster::topic_table::partition_meta>>>::find_or_prepare_insert<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>(detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>> const&) at /vectorized/include/absl/container/internal/raw_hash_set.h:2180
 (inlined by) std::__1::pair<absl::lts_20220623::container_internal::raw_hash_set<absl::lts_20220623::container_internal::NodeHashMapPolicy<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>, cluster::topic_table::partition_meta>, absl::lts_20220623::hash_internal::Hash<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>> const, cluster::topic_table::partition_meta>>>::iterator, bool> absl::lts_20220623::container_internal::raw_hash_set<absl::lts_20220623::container_internal::NodeHashMapPolicy<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>, cluster::topic_table::partition_meta>, absl::lts_20220623::hash_internal::Hash<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>> const, cluster::topic_table::partition_meta>>>::EmplaceDecomposable::operator()<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>, std::__1::piecewise_construct_t const&, std::__1::tuple<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>&>, std::__1::tuple<cluster::topic_table::partition_meta&&>>(detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>> const&, std::__1::piecewise_construct_t const&, std::__1::tuple<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>&>&&, std::__1::tuple<cluster::topic_table::partition_meta&&>&&) const at /vectorized/include/absl/container/internal/raw_hash_set.h:1863
 (inlined by) decltype(std::declval<absl::lts_20220623::container_internal::raw_hash_set<absl::lts_20220623::container_internal::NodeHashMapPolicy<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>, cluster::topic_table::partition_meta>, absl::lts_20220623::hash_internal::Hash<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>> const, cluster::topic_table::partition_meta>>>::EmplaceDecomposable>()(std::declval<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>& const&>(), std::__1::piecewise_construct, std::declval<std::__1::tuple<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>&>>(), std::declval<std::__1::tuple<cluster::topic_table::partition_meta&&>>())) absl::lts_20220623::container_internal::memory_internal::DecomposePairImpl<absl::lts_20220623::container_internal::raw_hash_set<absl::lts_20220623::container_internal::NodeHashMapPolicy<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>, cluster::topic_table::partition_meta>, absl::lts_20220623::hash_internal::Hash<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>> const, cluster::topic_table::partition_meta>>>::EmplaceDecomposable, detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>&, std::__1::tuple<cluster::topic_table::partition_meta&&>>(absl::lts_20220623::container_internal::raw_hash_set<absl::lts_20220623::container_internal::NodeHashMapPolicy<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>, cluster::topic_table::partition_meta>, absl::lts_20220623::hash_internal::Hash<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>> const, cluster::topic_table::partition_meta>>>::EmplaceDecomposable&&, std::__1::pair<std::__1::tuple<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>&>, std::__1::tuple<cluster::topic_table::partition_meta&&>>) at /vectorized/include/absl/container/internal/container_memory.h:139
 (inlined by) decltype(memory_internal::DecomposePairImpl(std::forward<absl::lts_20220623::container_internal::raw_hash_set<absl::lts_20220623::container_internal::NodeHashMapPolicy<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>, cluster::topic_table::partition_meta>, absl::lts_20220623::hash_internal::Hash<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>> const, cluster::topic_table::partition_meta>>>::EmplaceDecomposable>(fp), PairArgs(std::forward<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>&>(fp0), std::forward<cluster::topic_table::partition_meta>(fp0)))) absl::lts_20220623::container_internal::DecomposePair<absl::lts_20220623::container_internal::raw_hash_set<absl::lts_20220623::container_internal::NodeHashMapPolicy<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>, cluster::topic_table::partition_meta>, absl::lts_20220623::hash_internal::Hash<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>> const, cluster::topic_table::partition_meta>>>::EmplaceDecomposable, detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>&, cluster::topic_table::partition_meta>(absl::lts_20220623::container_internal::raw_hash_set<absl::lts_20220623::container_internal::NodeHashMapPolicy<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>, cluster::topic_table::partition_meta>, absl::lts_20220623::hash_internal::Hash<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>> const, cluster::topic_table::partition_meta>>>::EmplaceDecomposable&&, detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>&, cluster::topic_table::partition_meta&&) at /vectorized/include/absl/container/internal/container_memory.h:206
 (inlined by) decltype(absl::container_internal::DecomposePair(std::declval<absl::lts_20220623::container_internal::raw_hash_set<absl::lts_20220623::container_internal::NodeHashMapPolicy<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>, cluster::topic_table::partition_meta>, absl::lts_20220623::hash_internal::Hash<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>> const, cluster::topic_table::partition_meta>>>::EmplaceDecomposable>(), std::declval<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>&>(), std::declval<cluster::topic_table::partition_meta>())) absl::lts_20220623::container_internal::NodeHashMapPolicy<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>, cluster::topic_table::partition_meta>::apply<absl::lts_20220623::container_internal::raw_hash_set<absl::lts_20220623::container_internal::NodeHashMapPolicy<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>, cluster::topic_table::partition_meta>, absl::lts_20220623::hash_internal::Hash<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>> const, cluster::topic_table::partition_meta>>>::EmplaceDecomposable, detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>&, cluster::topic_table::partition_meta>(absl::lts_20220623::container_internal::raw_hash_set<absl::lts_20220623::container_internal::NodeHashMapPolicy<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>, cluster::topic_table::partition_meta>, absl::lts_20220623::hash_internal::Hash<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>> const, cluster::topic_table::partition_meta>>>::EmplaceDecomposable&&, detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>&, cluster::topic_table::partition_meta&&) at /vectorized/include/absl/container/node_hash_map.h:579
 (inlined by) decltype(absl::lts_20220623::container_internal::NodeHashMapPolicy<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>, cluster::topic_table::partition_meta>::apply(std::forward<absl::lts_20220623::container_internal::raw_hash_set<absl::lts_20220623::container_internal::NodeHashMapPolicy<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>, cluster::topic_table::partition_meta>, absl::lts_20220623::hash_internal::Hash<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>> const, cluster::topic_table::partition_meta>>>::EmplaceDecomposable>(fp), std::forward<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>&>(fp0), std::forward<cluster::topic_table::partition_meta>(fp0))) absl::lts_20220623::container_internal::hash_policy_traits<absl::lts_20220623::container_internal::NodeHashMapPolicy<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>, cluster::topic_table::partition_meta>, void>::apply<absl::lts_20220623::container_internal::raw_hash_set<absl::lts_20220623::container_internal::NodeHashMapPolicy<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>, cluster::topic_table::partition_meta>, absl::lts_20220623::hash_internal::Hash<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>> const, cluster::topic_table::partition_meta>>>::EmplaceDecomposable, detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>&, cluster::topic_table::partition_meta, absl::lts_20220623::container_internal::NodeHashMapPolicy<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>, cluster::topic_table::partition_meta>>(absl::lts_20220623::container_internal::raw_hash_set<absl::lts_20220623::container_internal::NodeHashMapPolicy<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>, cluster::topic_table::partition_meta>, absl::lts_20220623::hash_internal::Hash<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>> const, cluster::topic_table::partition_meta>>>::EmplaceDecomposable&&, detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>&, cluster::topic_table::partition_meta&&) at /vectorized/include/absl/container/internal/hash_policy_traits.h:170
 (inlined by) std::__1::pair<absl::lts_20220623::container_internal::raw_hash_set<absl::lts_20220623::container_internal::NodeHashMapPolicy<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>, cluster::topic_table::partition_meta>, absl::lts_20220623::hash_internal::Hash<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>> const, cluster::topic_table::partition_meta>>>::iterator, bool> absl::lts_20220623::container_internal::raw_hash_set<absl::lts_20220623::container_internal::NodeHashMapPolicy<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>, cluster::topic_table::partition_meta>, absl::lts_20220623::hash_internal::Hash<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>> const, cluster::topic_table::partition_meta>>>::emplace<detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>&, cluster::topic_table::partition_meta, 0>(detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true>>&, cluster::topic_table::partition_meta&&) at /vectorized/include/absl/container/internal/raw_hash_set.h:1485
 (inlined by) cluster::topic_table::apply(cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0bcab924475dfdff9-1/redpanda/redpanda/src/v/cluster/topic_table.cc:81
seastar::future<std::__1::error_code> cluster::do_apply<cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>>(unsigned int, cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>, seastar::sharded<cluster::topic_table>&, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>)::'lambda'(cluster::topic_table&)::operator()(cluster::topic_table&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0bcab924475dfdff9-1/redpanda/redpanda/src/v/cluster/topic_updates_dispatcher.cc:503
 (inlined by) decltype(std::declval<cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>>()(std::declval<cluster::topic_table&>())) std::__1::__invoke[abi:v160004]<seastar::future<std::__1::error_code> cluster::do_apply<cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>>(unsigned int, cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>, seastar::sharded<cluster::topic_table>&, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>)::'lambda'(cluster::topic_table&), cluster::topic_table&>(cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>&&, cluster::topic_table&) at /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:394
 (inlined by) decltype(auto) std::__1::__apply_tuple_impl[abi:v160004]<seastar::future<std::__1::error_code> cluster::do_apply<cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>>(unsigned int, cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>, seastar::sharded<cluster::topic_table>&, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>)::'lambda'(cluster::topic_table&), std::__1::tuple<cluster::topic_table&>, 0ul>(cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>&&, std::__1::tuple<cluster::topic_table&>&&, std::__1::__tuple_indices<0ul>) at /vectorized/llvm/bin/../include/c++/v1/tuple:1816
 (inlined by) decltype(auto) std::__1::apply[abi:v160004]<seastar::future<std::__1::error_code> cluster::do_apply<cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>>(unsigned int, cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>, seastar::sharded<cluster::topic_table>&, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>)::'lambda'(cluster::topic_table&), std::__1::tuple<cluster::topic_table&>>(cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>&&, std::__1::tuple<cluster::topic_table&>&&) at /vectorized/llvm/bin/../include/c++/v1/tuple:1825
 (inlined by) seastar::future<std::__1::error_code> seastar::sharded<cluster::topic_table>::invoke_on<seastar::future<std::__1::error_code> cluster::do_apply<cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>>(unsigned int, cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>, seastar::sharded<cluster::topic_table>&, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>)::'lambda'(cluster::topic_table&), seastar::future<std::__1::error_code>>(unsigned int, seastar::smp_submit_to_options, cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>&&)::'lambda'()::operator()() at /vectorized/include/seastar/core/sharded.hh:458
 (inlined by) seastar::future<std::__1::error_code> seastar::futurize<seastar::future<std::__1::error_code>>::invoke<seastar::future<std::__1::error_code> seastar::sharded<cluster::topic_table>::invoke_on<seastar::future<std::__1::error_code> cluster::do_apply<cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>>(unsigned int, cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>, seastar::sharded<cluster::topic_table>&, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>)::'lambda'(cluster::topic_table&), seastar::future<std::__1::error_code>>(unsigned int, seastar::smp_submit_to_options, cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>&&)::'lambda'()&>(cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>&&) at /vectorized/include/seastar/core/future.hh:1999
 (inlined by) seastar::smp_message_queue::async_work_item<seastar::future<std::__1::error_code> seastar::sharded<cluster::topic_table>::invoke_on<seastar::future<std::__1::error_code> cluster::do_apply<cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>>(unsigned int, cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>, seastar::sharded<cluster::topic_table>&, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>)::'lambda'(cluster::topic_table&), seastar::future<std::__1::error_code>>(unsigned int, seastar::smp_submit_to_options, cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>&&)::'lambda'()>::run_and_dispose() at /vectorized/include/seastar/core/smp.hh:240
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2660
 (inlined by) seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3123
seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3307
seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2::operator()() const at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4522
 (inlined by) decltype(std::declval<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&>()()) std::__1::__invoke[abi:v160004]<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&) at /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:394
 (inlined by) void std::__1::__invoke_void_return_wrapper<void, true>::__call<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&) at /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:487
 (inlined by) std::__1::__function::__alloc_func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2>, void ()>::operator()[abi:v160004]() at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:185
 (inlined by) std::__1::__function::__func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2>, void ()>::operator()() at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:356
std::__1::__function::__value_func<void ()>::operator()[abi:v160004]() const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:510
 (inlined by) std::__1::function<void ()>::operator()() const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:1156
 (inlined by) seastar::posix_thread::start_routine(void*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/posix.cc:73

What should have happened instead?

Allocatoins under the bound of 128K. I guess we need a different structure.

How to reproduce the issue?

1. Create 23,000 partitions in a cluster.

[travisdowns]

The top 4 stacks.

#1:

Adding a delete op to the topic_table_delta table.

Key stack:

 (inlined by) cluster::topic_table_delta& fragmented_vector<cluster::topic_table_delta, 8192ul>::emplace_back<model::ntp, cluster::partition_assignment&, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>&, cluster::topic_table_delta::op_type&>(model::ntp&&, cluster::partition_assignment&, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>&, cluster::topic_table_delta::op_type&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0bcab924475dfdff9-1/redpanda/redpanda/src/v/utils/fragmented_vector.h:129
 (inlined by) cluster::topic_table::do_local_delete(model::topic_namespace, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0bcab924475dfdff9-1/redpanda/redpanda/src/v/cluster/topic_table.cc:128
topic_table.cc:194 cluster::topic_table::apply(cluster::topic_lifecycle_transition, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0bcab924475dfdff9-1/redpanda/redpanda/src/v/cluster/topic_table.cc:194
topic_updates_dispatcher.cc:503 seastar::future<std::__1::error_code> cluster::do_apply<cluster::topic_lifecycle_transition>(unsigned int, cluster::topic_lifecycle_transition, seastar::sharded<cluster::topic_table>&, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>)::'lambda'(cluster::topic_table&)::operator()(cluster::topic_table&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0bcab924475dfdff9-1/redpanda/redpanda/src/v/cluster/topic_updates_dispatcher.cc:503

#2:

Very similar, same table I think with a slightly different command:

 (inlined by) cluster::topic_table_delta& fragmented_vector<cluster::topic_table_delta, 8192ul>::emplace_back<model::ntp, cluster::partition_assignment&, detail::base_named_type<long, model::model_offset_type, s
td::__1::integral_constant<bool, true>>&, cluster::topic_table_delta::op_type, std::__1::nullopt_t const&, absl::lts_20220623::flat_hash_map<detail::base_named_type<int, model::node_id_model_type, std::__1::inte
gral_constant<bool, true>>, detail::base_named_type<long, model::revision_id_model_type, std::__1::integral_constant<bool, true>>, absl::lts_20220623::hash_internal::Hash<detail::base_named_type<int, model::node
_id_model_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair
<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>> const, detail::base_named_type<long, model::revision_id_model_type, std::__1::integral_constant<bool, true>>>>>>(
model::ntp&&, cluster::partition_assignment&, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>&, cluster::topic_table_delta::op_type&&, std::__1::nullopt_t const&,
 absl::lts_20220623::flat_hash_map<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::revision_id_model_type, std::__1::integra
l_constant<bool, true>>, absl::lts_20220623::hash_internal::Hash<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int,
 model::node_id_model_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>> const, d
etail::base_named_type<long, model::revision_id_model_type, std::__1::integral_constant<bool, true>>>>>&&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0bcab924475dfdff9-1/redpanda/redpanda/src/
v/utils/fragmented_vector.h:129
 (inlined by) cluster::topic_table::apply(cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (c
luster::serde_opts)0>, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0bcab924475dfdff9-1/redpanda
/redpanda/src/v/cluster/topic_table.cc:86
topic_updates_dispatcher.cc:503 seastar::future<std::__1::error_code> cluster::do_apply<cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (
signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>>(unsigned int, cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signe
d char)0, (model::record_batch_type)6, (cluster::serde_opts)0>, seastar::sharded<cluster::topic_table>&, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>)::'lambda
'(cluster::topic_table&)::operator()(cluster::topic_table&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0bcab924475dfdff9-1/redpanda/redpanda/src/v/cluster/topic_updates_dispatcher.cc:503

#3:

Related to the above but weird.

 (inlined by) decltype(absl::lts_20220623::container_internal::FlatHashMapPolicy<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::revision_id_model_type, std::__1::integral_constant<bool, true>>>::value(std::__1::pair<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>> const, detail::base_named_type<long, model::revision_id_model_type, std::__1::integral_constant<bool, true>>>* std::__1::addressof[abi:v160004]<std::__1::pair<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>> const, detail::base_named_type<long, model::revision_id_model_type, std::__1::integral_constant<bool, true>>>>(std::__1::pair<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>> const, detail::base_named_type<long, model::revision_id_model_type, std::__1::integral_constant<bool, true>>>&)(decltype(std::__declval<std::__1::pair<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>> const, detail::base_named_type<long, model::revision_id_model_type, std::__1::integral_constant<bool, true>>>>(0)) std::__1::declval<std::__1::pair<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>> const, detail::base_named_type<long, model::revision_id_model_type, std::__1::integral_constant<bool, true>>>&>()()))) absl::lts_20220623::container_internal::raw_hash_map<absl::lts_20220623::container_internal::FlatHashMapPolicy<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::revision_id_model_type, std::__1::integral_constant<bool, true>>>, absl::lts_20220623::hash_internal::Hash<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>> const, detail::base_named_type<long, model::revision_id_model_type, std::__1::integral_constant<bool, true>>>>>::operator[]<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>, absl::lts_20220623::container_internal::FlatHashMapPolicy<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::revision_id_model_type, std::__1::integral_constant<bool, true>>>>(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>> const&) at /vectorized/include/absl/container/internal/raw_hash_map.h:169
 (inlined by) cluster::topic_table::apply(cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0bcab924475dfdff9-1/redpanda/redpanda/src/v/cluster/topic_table.cc:79

#4:

The chunk cache. Nothing to see here.