Plugin in_systemd: stalled reads

[manuelluis]

The systemd plugin is stalled reading events.

If you do in one shell: journalctl -f -o json and in other: fluent-bit -i systemd -prop=Read_From_Tail=true -p DB=journald.db -p tag='host.*' -o stdout -f 1 -v and you do for example sudo su - you don't see all the events in the fluent-bit output. If you restart the fluent-bit you see the missing events.

If you do the same without -p tag='host.*' it works.

The problem is in:

fluent-bit/plugins/in_systemd/systemd.c

Lines 168 to 169 in 23b6171

	if (mp_sbuf.size > 1024000 ||
	((last_tag_len != tag_len) || strncmp(last_tag, tag, tag_len) != 0)) {

When you hit he condition, it's easy to hit the condition of different tags, it will break the loop of sd_journal_next and in_systemd_collect returns FLB_SYSTEMD_MORE.

The problem is that there is remaining data to read but you will not read it until new data arrives.

When fluent-bit starts it works well because in_systemd_collect is called from in_systemd_collect_archive and controls the returns codes of in_systemd_collect and generates a event when there is more data to read, but the first time that in_systemd_collect reached the end of the journal systemd_collect_archive register in_systemd_collect with flb_input_set_collector_event and in_systemd_collect_archive it's not called any more.

I hope I have explained the problem.

[edsiper]

thanks for reporting the problem and the hints about root cause.

I did some tests and effectively there are missing records when breaking the loop (no matter the condition).

I've pushed a fix on GIT master, would you please rebuild the latest version and double check the new behavior ?

[manuelluis]

It still happens, for example with: max_entries=1

fluent-bit -i systemd -prop=Read_From_Tail=true -p max_entries=1 -p DB=journald.db -o stdout -f 1

[edsiper]

@manuelluis

would you please try the latest from GIT master ?, I did some changes and this time I was not able to reproduce the problem. can you double-check ?

[manuelluis]

Now works correctly.
Thank you.

[edsiper]

finally :), thanks for your help.