No concurrency: Only opens single socket

[pranavmarla]

We're using this plugin (firehose) to send data directly to our Kinesis Firehose VPC endpoint, but the throughput seems fairly low (seems to max out at around 5 MB/s). After doing a packet capture, it appears as if this plugin only opens a single TCP socket/connection to the Firehose -- is there a way to get it to open multiple connections simultaneously, to increase throughput?

We tried using the new 'multiple workers' option mentioned in the documentation for the core plugin (kinesis_firehose), but it didn't seem to have any effect when used with this plugin, presumably because this plugin doesn't support it. If that's the case, and that feature would solve our throughput issue, do you have an estimate on when this plugin will support that 'multiple workers' feature?

[hossain-rayhan]

Hi @pranavmarla, thanks for opening this issue. From the README, it seems like we don't have concurrency support for this plugin. While the investigation will take time, can you share any specific reason you cannot use our core (kinesis_firehose) plugin?

[pranavmarla]

Thanks @hossain-rayhan. We would prefer to use this plugin instead since we previously experienced far worse performance (including outright crashing) in the core (kinesis_firehose) plugin -- see issue #3197 for details.

[zhonghui12]

We fixed the bug in our core plugin and we are working on other fix now. Close this one because we'd prefer customers to use our kinesis_firehose plugin: https://docs.fluentbit.io/manual/pipeline/outputs/firehose that has worker support for concurrency.