-
Notifications
You must be signed in to change notification settings - Fork 589
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Oversized allocation: 1232896 bytes in admin_server::register_partition_routes
#11674
Comments
this one is interesting. the vector that is constructed in the admin server grows too large as a function of the number of partitions. however, if we replace it with a fragmented vector we still have the problem that seastar's json handling wants a vector to construct a json list. and even if we solved that problem, would we face the a challenge related to the size of what would ostensibly be a very large string being constructed some where? |
the bandaid solution here is to probably manually construct the json output and hope it either isn't too large or doesn't get linearized somewhere. other options beyond addressing the root limitations? |
Avoid oversized allocation by: * Collecting summaries into a fragmented_vector * Streaming the vector to an output_stream Thge overload of `ss::json::json_return_type` that provides an `ss::output_stream` requires a `std::function`, which is required to be copyable, so wrap the `fragmented_vector` in a simple container wrapper that holds the container by `ss::lw_shared_ptr`. Fixes redpanda-data#11674 Signed-off-by: Ben Pope <[email protected]>
Avoid oversized allocation by: * Collecting summaries into a fragmented_vector * Streaming the vector to an output_stream Thge overload of `ss::json::json_return_type` that provides an `ss::output_stream` requires a `std::function`, which is required to be copyable, so wrap the `fragmented_vector` in a simple container wrapper that holds the container by `ss::lw_shared_ptr`. Another optimisation is moving rather than copying during accumulation. Fixes redpanda-data#11674 Signed-off-by: Ben Pope <[email protected]>
Avoid oversized allocation by: * Collecting summaries into a fragmented_vector * Streaming the vector to an output_stream The overload of `ss::json::json_return_type` that provides an `ss::output_stream` requires a `std::function`, which is required to be copyable, so wrap the `fragmented_vector` in a simple container wrapper that holds the container by `ss::lw_shared_ptr`. Another optimisation is moving rather than copying during accumulation. Fixes redpanda-data#11674 Signed-off-by: Ben Pope <[email protected]>
Avoid oversized allocation by: * Collecting summaries into a fragmented_vector * Streaming the vector to an output_stream The overload of `ss::json::json_return_type` that provides an `ss::output_stream` requires a `std::function`, which is required to be copyable, so wrap the `fragmented_vector` in a simple container wrapper that holds the container by `ss::lw_shared_ptr`. Another optimisation is moving rather than copying during accumulation. Fixes redpanda-data#11674 Signed-off-by: Ben Pope <[email protected]> (cherry picked from commit d32b250) Conflicts: src/v/redpanda/admin_server.cc Due to 2cc38ee I took the opportunity to extract `reduce` and reuse it.
Avoid oversized allocation by: * Collecting summaries into a fragmented_vector * Streaming the vector to an output_stream The overload of `ss::json::json_return_type` that provides an `ss::output_stream` requires a `std::function`, which is required to be copyable, so wrap the `fragmented_vector` in a simple container wrapper that holds the container by `ss::lw_shared_ptr`. Another optimisation is moving rather than copying during accumulation. Fixes redpanda-data#11674 Signed-off-by: Ben Pope <[email protected]> (cherry picked from commit d32b250) Conflicts: src/v/redpanda/admin_server.cc Due to 2cc38ee I took the opportunity to extract `reduce` and reuse it.
Note that the location needs to be clarified.
Found via Brandon Allard's script.
Artifact URL: https://ci-artifacts.dev.vectorized.cloud:443/vtools/8213/0188e80f-cb8a-413f-a7a2-8741ea51e304/vbuild/output/ducktape-release-clang-amd64.tgz
Test details:
tmp/ducktape/results/2023-06-23--001/LargeControllerSnapshotTest/test_join_restart/169/RedpandaService-0-140553123898432/ip-172-31-2-199/redpanda.log
The text was updated successfully, but these errors were encountered: