Auto_flush in multiline codec doesn't work with Kafka input plugin

[ciprianpascu]

No description provided.

[ciprianpascu]

Auto_flush doesn't happen, although set; configuration something like this:

input {
kafka {
topic_id => "test"
zk_connect => ...
consumer_threads => 1
codec => multiline {
auto_flush_interval => 3
pattern => "TRACE HIGH ACTIVE"
negate => true
what => previous
}
}
}

[jordansissel]

I wouldn't expect it to work well with kafka (based on my expectations of what multiline codec tries to solve). What are you trying to do?

[suyograo]

multiline is not supported in kafka because messages can come out of order in different partitions. You could use the message key to group all parts of multiline data into one partition, but this is not something we've tested or hear about.

[ciprianpascu]

Ok, that's really good to know! I was testing only with one Kafka partition and it worked pretty well, except that the last message from a burst was not handled, because of auto_flush not working. But, if there are issues when using multiple partitions, I will have to find another solution. Thanks!

[suyograo]

@ciprianpascu I don't know what your source of data is, but if it is files, you can use filebeat which has multiline functionality and can soon send to kafka directly (elastic/beats#942)

[suyograo]

Also, any beats in general in the future can send to Kafka

[TinLe]

According to this, elastic/beats#943, beats can not yet send direct to Kafka.

Has that been implemented and released?

[suyograo]

@TinLe its been implemented and merged to master in elastic/beats#942. Not yet released. I believe its targeted for the 5.0 release

[TinLe]

Nice! :)

[ciprianpascu]

My source of data is some processes sending logs, but now it seems I won't need the multiline codec, so auto_flush not needed either :).