Use fragmented vector for metadata response #8469

fragmented_vector is a vector-like class which limits the maximum contiguous allocation required by breaking the elements up into chunks of a specific maximum size call the "fragment size". Currently, the vector places two restrictions on the fragment size: - It must be a power of two and one on the element size: - It must evenly divide the fragment size. It seems to be that neither of these is necessary, and they severely limit the element types that may be used: effectively, only elements whose size is a power of 2 can meet these requirements. The first restriction may be in place to ensure the fragment size lines exactly with seastar span sizes, and the second may be so that there is no internal waste within a fragment, but with many object sizes this waste is very small. We simply remove these restrictions for now.

This returns the approximate in-memory size of the vector in bytes. It is handy for methods which must estimate the in-memory size of a given request.

These operations were not implemented previously, but are necessary to satisfy the iterator concepts.

Previously, fragmented_vector only had const_iterator, but more general use will require also a non-const iterator. This change adds that capability, using a template iterator class parameterized on whether the iterator is const or not.

fragmented_vector<T>::iterator and const_iterator did not have a default constructor, but without that it does not satisfy the forward_iterator concept and some algorithms require that.

These operators move the iterators towards the random-access concept.

This change allows our fragmented vector to be assigned from std::vector so that it can interop with vector in the request/response path.

fragmented_vector::iterator is in principle a random access iterator so it should have less-than and other operators. Do it with spaceship.

So we can print it in tests.

Restrict the number of elements in a full fragment to the highest power of two that still satisfies the max_frag_bytes limits. This keeps the indexing for operator[] considerably similar because a shift is needed instead of a division-by-constant. While we're in there, add an accessor for the number of elements held in each fragment, allowing users to query this constexpr value.

Add several new tests for fragmented vector, covering: - Assignment from std::vector - clear() method - const vs non-const iterators - Fragment sizing - Iterator arithmetic and comparison - Using vector in std::sort Furthermore, a wrapper is created which checks the consistency of the vector on every operator->() deference: by using this widely in the test we fail fast when an inconsistency is discovered. We use an internals-accessing friend class to enable some of this, e.g., the consistency checking, without needing to make internals public.

This change lets the write_array methods of response_writer accept fragmented_vector in addition to std::vector.

Certain responses may be very large, such as a metadata response on a cluster with many partitions. Since int schemata we map lists in response types to std::vector, this may result in large contiguous allocations, which is a no-no in redpanda: we should limit such allocations to 128 K. To fix this, we add an enable_fragmentation_resistance list in the generator: array of types in this list will be implemented with a large_fragment_vector instead of std::vector: this type limits the contiguous memory allocated by making several smaller allocations if necessary.

Topic metadata is potentially large and slow to copy, so this change moves it in a couple of places we weren't moving it before which showed up in a large load test profile.

Now that we are using fragmented_vector, which is move-only, we need to use explicit moves in a few places and as a bonus this will be faster.

reserve() was being used on the std::vector of partition metadata to size it correctly, but now that we are using fragmented_vector this method no longer exists. It does not make sense for fragmented_vector since that class does not copy data to a new contiguous area as it grows, and always immediately allocates each fragment as its maximum size, so reserve would not change the allocation or copy pattern at all.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use fragmented vector for metadata response #8469

Use fragmented vector for metadata response #8469

Commits on Feb 1, 2023