Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: Support Raw String Option for HTTP Batch Mode #134

Open
bpb opened this issue Nov 13, 2024 · 0 comments
Open

Feature Request: Support Raw String Option for HTTP Batch Mode #134

bpb opened this issue Nov 13, 2024 · 0 comments

Comments

@bpb
Copy link

bpb commented Nov 13, 2024

In some use cases, apis require batched data to be submitted as a raw string rather than wrapped in a JSON array.

Currently the only way to send batched raw data strings is to use the single request mode and create a pre aggregated roll up payload as a string before sinking. This operation is a bit obtuse.

BodyPublisher publisher;
// By default, Java's BodyPublishers.ofByteArrays(elements) will just put Jsons
// into the HTTP body without any context.
// What we do here is we pack every Json/byteArray into Json Array hence '[' and ']'
// at the end, and we separate every element with comma.
elements.add(BATCH_START_BYTES);
for (HttpSinkRequestEntry entry : reqeustBatch) {
elements.add(entry.element);
elements.add(BATCH_ELEMENT_DELIM_BYTES);
}
elements.set(elements.size() - 1, BATCH_END_BYTES);
publisher = BodyPublishers.ofByteArrays(elements);

Please consider adding this functionality in the near future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant