Multiple consumers for the AsyncHandler

[antalk]

Hi,

We faced some issues where the queue of the AsyncHandler was full (102400 items ) and we ddint want to discard any logging resulting in a synchronous blocking loghandler which eventually slowed our system down to an almost halt.

Then we saw there is only 1 consumer :

    thread = threadFactory.newThread(new AsyncTask());

Would it be an idea to make that a configurable property and use a pool of threads to speed up consumption of the queue preveting it from becoming blocked too early ?

[jamezp]

You could lose order with more than one worker. That is a pretty full queue. Was I/O slow or was it just a lot of messages?

[raouldh]

This would indeed lose the order. But you could have a sequence number in your log message to be able to filter on that?

[antalk]

Ordering is maintained with the ExtLogRecord object, there is already a unique sequence number there. We use that number to format and send the log line to logstash over http and indeed yes. , this can take time coz some of our messages are over a megabyte. Hence the slowdown.

And then we had like 100+ lines per second, due to DEBUG level being on

[jamezp]

It's probably best to create a new handler. The idea behind this handler is it could be writing to a file where the order would matter. Writing to a log aggregator is less of a concern with the order the logs arrive.

[antalk]

will do ! thanks for the feedback

[pdoelam]

I've been exploring the AsyncHandler class within the org.jboss.logmanager as well and have also noticed this potential limitation.
One of the primary reasons for building a custom handler was to address the limitation mentioned above.
Making this parameter configurable in the standalone.xml would mean a lot of users like myself could leverage the native capabilities of the LogManager without having to build custom implementations.
Next to allowing the number of consumers for the AsyncHandler to be configurable I would also add a configurable timeout to the Blocked messages of this Blocking queue as well.
Expose this configuration in the standalone.xml for easy adjustments and we have more flexibility for users to tune the performance of the logging mechanism based on their system's workload => happy developers

[jamezp]

As mentioned above, a new handler would need to be created. This handler can be used with handlers that are writing to files where order matters.

Creating a new handler for WildFly/EAP would be more of a process really. However, not impossible at all.