flb_aws_util: set max response buffer size for the http client

[hossain-rayhan]

This change will enable unlimited response buffer for aws http client. In this way we can mitigate the following error for all AWS plugins including kinesis_firehose and s3.

[ warn] [http_client] cannot increase buffer: current=4096 requested=36864 max=4096

Testing
Before we can approve your change; please submit the following in a comment:

• [ N/A] Example configuration file for the change
• [ X] Debug log output from testing the change

• [ N/A] Attached Valgrind output that shows no leaks or memory corruption was found

Documentation

• [ N/A] Documentation required for this feature

Debug log for this change

[2021/09/02 23:50:55] [debug] [aws_http_client] max response buffer size was increased to 0
[2021/09/02 23:50:55] [debug] [upstream] KA connection #83 to firehose.us-west-2.amazonaws.com:443 is now available
[2021/09/02 23:50:55] [debug] [output:kinesis_firehose:kinesis_firehose.0] PutRecordBatch http status=200
[2021/09/02 23:50:55] [debug] [output:kinesis_firehose:kinesis_firehose.0] Sent events to fluent-bit-perf-test
[2021/09/02 23:50:55] [debug] [output:kinesis_firehose:kinesis_firehose.0] Sending 500 records
[2021/09/02 23:50:55] [debug] [output:kinesis_firehose:kinesis_firehose.0] Sending log records to delivery stream fluent-bit-perf-test

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[PettitWesley]

Squash commits and check DCO

[krispraws]

Thank you!

[PettitWesley]

So there are two problems here:

1. this function returns -1 when it fails and that actually evaluates to true IIRC
2. also, fluent bit code style says you can't call a function inside an if condition

So you need to do:

ret = flb_http_buffer_size(c, FLB_MAX_AWS_RESP_BUFFER_SIZE);
if (ret == -1) {

[krispraws]

Use ret != 0 instead of ret == -1