Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

out_es: Fix buffer size when converted_size is equal to 0. #4414

Merged
merged 1 commit into from
Dec 9, 2021

Conversation

GuillaumeSmaha
Copy link
Contributor

In some case, ES_BULK_CHUNK is not enough to increase size of thebuffer

Fix #4412


Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • [N/A] Example configuration file for the change
  • [N/A] Debug log output from testing the change

Documentation

  • [N/A Documentation required for this feature

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

I didn't find a way to add a test because it seems already cover by the div0 test because the buffer size is 8000 which is superior to chunk size #define ES_BULK_CHUNK 4096

/* https://github.com/fluent/fluent-bit/issues/3905 */
void flb_test_div0()
{
int ret;
char record[8000];
char record_header[] = "[1448403340,{\"key\":\"";
flb_ctx_t *ctx;
int in_ffd;
int out_ffd;
int i;
/* Create context, flush every second (some checks omitted here) */
ctx = flb_create();
flb_service_set(ctx, "flush", "1", "grace", "1", NULL);
/* Lib input mode */
in_ffd = flb_input(ctx, (char *) "lib", NULL);
flb_input_set(ctx, in_ffd, "tag", "test", NULL);
/* Elasticsearch output */
out_ffd = flb_output(ctx, (char *) "es", NULL);
flb_output_set(ctx, out_ffd,
"match", "test",
NULL);
/* Override defaults of index and type */
flb_output_set(ctx, out_ffd,
NULL);
/* Enable test mode */
ret = flb_output_set_test(ctx, out_ffd, "formatter",
cb_check_nothing,
NULL, NULL);
/* Start */
ret = flb_start(ctx);
TEST_CHECK(ret == 0);
/* create json */
strncpy(&record[0], &record_header[0], strlen(record_header));
for(i=strlen(record_header); i<sizeof(record)-4; i++) {
record[i] = 'a';
}
record[sizeof(record)-4] = '"';
record[sizeof(record)-3] = '}';
record[sizeof(record)-2] = ']';
record[sizeof(record)-1] = '\0';
/* Ingest data sample */
flb_lib_push(ctx, in_ffd, (char *) &record[0], strlen(record));
sleep(2);
flb_stop(ctx);
flb_destroy(ctx);
}

@GuillaumeSmaha GuillaumeSmaha changed the title Fix buffer size when converted_size is equal to 0. ou_es: Fix buffer size when converted_size is equal to 0. Dec 9, 2021
@GuillaumeSmaha GuillaumeSmaha changed the title ou_es: Fix buffer size when converted_size is equal to 0. out_es: Fix buffer size when converted_size is equal to 0. Dec 9, 2021
In some case, ES_BULK_CHUNK is not enough to increase size of the
buffer

Fix fluent#4412

Signed-off-by: Guillaume Smaha <[email protected]>
@GuillaumeSmaha GuillaumeSmaha force-pushed the out_es_buffer_too_small branch from ccdcc30 to 8122f8e Compare December 9, 2021 14:44
@GuillaumeSmaha
Copy link
Contributor Author

@nokute78 I am made the PR as you requested

@edsiper edsiper merged commit 0a061fb into fluent:master Dec 9, 2021
edsiper pushed a commit that referenced this pull request Dec 9, 2021
In some case, ES_BULK_CHUNK is not enough to increase size of the
buffer

Fix #4412

Signed-off-by: Guillaume Smaha <[email protected]>
@GuillaumeSmaha GuillaumeSmaha deleted the out_es_buffer_too_small branch December 9, 2021 20:42
@nokute78
Copy link
Collaborator

@GuillaumeSmaha Thank you !

0Delta pushed a commit to 0Delta/fluent-bit that referenced this pull request Jan 20, 2022
In some case, ES_BULK_CHUNK is not enough to increase size of the
buffer

Fix fluent#4412

Signed-off-by: Guillaume Smaha <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

caught signal (SIGSEGV) with output elasticsearch
3 participants