Updating kafka logger to use the batch processor util

[sshniro]

Summary

Updating kafka logger to use the batch processor util.

Depends on #1349

[Akayeshmantha]

Hi @sshniro I think Kafka logger does not needs to use the batch processor since it already has the capability to push to the topic in an asynchronous way. Lua resty Kafka already has the feature already inbuilt I already mentioned that in the documentation file of the logger

The message will write to the buffer first. It will send to the Kafka server when the buffer exceeds the batch_num or every flush_time flush the buffer in Async mode

cc :- @membphis wdyt ?

[membphis]

@Akayeshmantha I agree with you. The current way is simpler and enough.

[sshniro]

Closing the PR as agreed to use the builtin buffer of the library.

[sshniro]

Reopening PR due to the max number of timers which could be executed within a period

[sshniro]

@membphis the following PR is ready for review.

[moonming]

@membphis please take a look

[membphis]

I have one question:

If there are no more new logs to be sent later, how to release the log_buffer object?
If the log_buffer is useless, how to delete it from buffers? If we don't delete it, it will make the buffers bigger.

we need to control the max count number of timer. the default value is 1024 for each worker process.

https://github.com/openresty/lua-nginx-module#lua_max_pending_timers

[sshniro]

Yes, @membphis I was also thinking about this problem,
For the memory problem:
we need to write a timer function which executes in all active loggers and finds stale buffer objects and remove the variable. Maybe it can run every 30 mins?

For the timer problem:
The buffer by default has an inactive timeout. If we don't send any logs for 30 seconds, by default it will send the existing logs and will not create any additional timers. Due to this fact, we won't be creating timers for inactive buffers. Only active buffers will have a timer running.

If we agree on the memory optimization technique, then I can send the fix in another PR.

A practical bottleneck is if a user configures like 1024 routes with a logger and actively using all the routes.
But I assume its not a practical scenario. To tackle this scenario we might have to have a global logger instead of a route based logger.

[sshniro]

I have created an issue to remove stale objects from memory:
#1494

The following issue is to create a global logger, to resolve the issue from having more than 1024 loggers running concurrently:
#1492

[sshniro]

@membphis any update on this?

[sshniro]

A PR has been created to remove stale objects from the buffer table
#1520

[membphis]

A PR has been created to remove stale objects from the buffer table
#1520

ok, got you. this PR has been merge first, then we can resolve it in PR #1520 .

I am still thinking about how to solve it in a simple way.

[sshniro]

We are now creating a batch processor per route id: Just to reduce the number of keys getting created how about we create only one batch processor, if the host port and timeout values are the same? @membphis

Because in reality, people will only have a single external log management platform.

So when a request comes:

local key = con.host .. conf.port .. conf.timeout .. conf.retry
local buffer = buffers[key]