Sync master #1
Merged
Sync master #1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ltiplex mode (#2777) this fix ensures that in non-multiplexing mode, if a task is in the retry state, the tasks behind it in the queue will not be started. Signed-off-by: zhanghjster <[email protected]>
Signed-off-by: Zhonghui Hu <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: davkor <[email protected]>
This is not on oss-fuzz yet, but the bug occurs when len is zero, and a write is performed on out[len-1]. In this case we write to the left of the allocated buffer. Signed-off-by: davkor <[email protected]>
oss-fuzz testcase 4907517888692224) Signed-off-by: davkor <[email protected]>
Signed-off-by: davkor <[email protected]>
Signed-off-by: davkor <[email protected]>
Signed-off-by: davkor <[email protected]>
Signed-off-by: davkor <[email protected]>
Signed-off-by: davkor <[email protected]>
Windows build is failing with the following error:
flb_network.c(243): error C2065: 'u_conn': undeclared identifier
This is because `u_conn` is not defined within the scope of this
function (net_connect_sync).
With this patch, master HEAD becomes compilable on Windows again.
Signed-off-by: Fujimoto Seiji <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
This patch is what is needed in Fluent Bit's side to allow Windows users to start using filesystem storage. With this and edsiper/chunkio/59 merged, I can confirm that Fluent Bit can persist input data on disks. Signed-off-by: Fujimoto Seiji <[email protected]>
Fluent Bit sometimes emits the following error on shutdown:
[error] [src\flb_utils.c:339 errno=0] No error
This is due to closing a socket is detected on Windows as a new
socket event.
Since the socket is already closed by the time the callback handler
dispatches, it results in the above error. Fix it.
Signed-off-by: Fujimoto Seiji <[email protected]>
* change error logs to warning and remove abortion of sending logs when warning occurs * error when level type is invalid Signed-off-by: Mohammad Ali Toufighi <[email protected]>
Signed-off-by: jevgenimarenkov <[email protected]>
The old behaviour was that it accepts time data only if the
format is specified exactly. If the format does not match, it
simply returns an error.
However, with the new 'Time_Strict' option, a time parser can
behave in "best effort" mode, and be permissive on heterogeneous
time data.
For example, suppose you set up the following time format:
Time_Format %Y-%m-%dT%H:%M:%S.%L
... and received the record like this:
{"time":"2020-11-10T11:11:26+00:00"}
With `Time_Strict` turned off, you'll get the following result:
"2020-11-10T11:11:26+00:00" => 2020-11-10T11:11:26.000000Z
Signed-off-by: Fujimoto Seiji <[email protected]>
Signed-off-by: Jeff Luo <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
This patch makes two significant changes: - replace murmur2 hashing algorithm with xxHash_64 - if a hash entry value less or equal zero, it's assumed that the caller want's to store the pointer reference of the value instead of holding a local memory allocation with a copy of the value. Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
…value Signed-off-by: Eduardo Silva <[email protected]>
When input plugins ingest data, the 'input chunk' interface is called since that's the responsible to store the incoming data. The critical piece of this routine is to find a candidate Chunk associated with the same tag that the new data is coming. One of the problems is that the lookup is linear across the Chunks using a linked list and in many scenarios most of the Chunks can be down losing the ability to make a match with the Tag (Chunk tags are part of the meta data), so doing a I/O to read the tag was mandatory. The problematic scenario: if destinations are down and we are buffering in the filesystem, the current approach might generate hundreds of small Chunks making the query expensive overtime: O(n). This patch introduce a new hash table to keep a reference to candidate chunks where data can be written, this hash table is associated to each input instance so the lookup for a candidate chunk is extremely faster, some 'perf' report numbers. == before (cummulative numbers: groups) == 87.20% flb_input_chunk_append_raw 46.64% input_chunk_get == after (cummulative numbers: groups) == 21.26% flb_input_chunk_append_raw 3.15% input_chunk_get The results demonstrate a ~92% improvement on finding a Chunk to append data, in addition it reduced significantly CPU and disk I/O usage. Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Wesley Pettit <[email protected]>
Prevent a segfault in the error path when the JSON format is used:
```
[2020/12/24 00:05:00] [ warn] [engine] failed to flush chunk '651791-1608761099.630731466.flb', retry in 6 seconds: task_id=4, input=dummy.0 > output=kafka.0
[2020/12/24 00:05:00] [ warn] [engine] failed to flush chunk '651791-1608761100.174611647.flb', retry in 7 seconds: task_id=16, input=dummy.0 > output=kafka.0
munmap_chunk(): invalid pointer
Thread 2 "flb-pipeline" received signal SIGABRT, Aborted.
--Type <RET> for more, q to quit, c to continue without paging--
[Switching to Thread 0x7ffff7885700 (LWP 651795)]
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
at /home/gstatkevicius/dev/fluent-bit/plugins/out_kafka/kafka.c:409
```
All of the paths besides `FLB_KAFKA_FMT_MSGP` operate on `flb_sds_t` so
the appropriate function to use here is `flb_sds_destroy`.
Signed-off-by: Giedrius Statkevičius <[email protected]>
This adds the first cut at fixing compile warnings on Windows. * Move the "/W<n>" flag to sub-component level, so that external libraries can set their own warning level. * Suppress several informational warnings (e.g. notification about inlinization of functions) * Define '_CRT_NONSTDC_NO_WARNINGS' to prevent warnings about POSIX function names. Signed-off-by: Fujimoto Seiji <[email protected]>
MSVC does not like us doing `n * 1.5` to increase an integer by
50%. It results in the following warning:
warning C4244: '=': conversion from 'double' to 'size_t',
possible loss of data
Replace it with an equivalent integer arithmetic expression.
Signed-off-by: Fujimoto Seiji <[email protected]>
This patch fixes the following two warnings: * warning C4020: 'pack_strings': too many actual parameters * warning C4477: '_snprintf' : format string '%u' requires an argument of type 'unsigned int', but variadic argument 4 has type 'time_t'. Signed-off-by: Fujimoto Seiji <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
This is a preparation for supporting older versions of OpenSSL. A new API flb_tls_init() is added so that we can initialize libssl safely. Since the method is supposed to be called once, we need to introduce this method into the booting sequence of Fluent Bit. Signed-off-by: Fujimoto Seiji <[email protected]>
This actually starts using the new API function, and hence makes flb_tls.c compatible with OpenSSL v1.0.2. Signed-off-by: Fujimoto Seiji <[email protected]>
Suppose we have the following configuration:
[OUTPUT]
Name http
...
tls On
tls.ca_file /var/cert/custom.crt
With the old code, `tls.ca_file` was simply ignored, because the old
code checked ca_path first, and just proceeded to load system certs
if ca_path was not provided.
Fix `tls_context_create()` to honor ca_path and ca_file properly.
Signed-off-by: Fujimoto Seiji <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
llhhbc
pushed a commit
that referenced
this pull request
Sep 15, 2021
output_thread patch #1 : pipe creation is platform dependent, in unixes it's achieved through the pipe syscall but in windows it's a socket pair created through libevent which means instead of read/write we need to use recv/send which was already abstracted through flb_pipe_(r/w). output_thread patch #2 : in windows the SOCKET data type is defined as an UINT_PTR which means in 64 bit operating systems it's an 8 byte number instead of 4. libevent abstracts this through the evutil_socket_t data type which in turn is abstracted using the flb_pipefd_t data type in fluent bit. The problem comes to play when calling mk_event_channel_create which receives 2 int pointers to return both pipe endpoints. This means there are 2 possible bugs (one of which was happening and the other is not really a concern I think) : Since mk_event_channel_create was only modifying the low part of the elements in the ch_parent_events array and the structure was not zeroed when allocated, the high parts of those 64 bit members contained garbage which caused any winsock calls to return error 10038 (not a socket). That's the reason why I added a memset call in line 421. There is still a possible issue with mk_event_channel_create which we should fix by moving our platform dependent data type to mk_lib and defining our flb local data types to those if need be (for consistency). Signed-off-by: Leonardo Alminana <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.