Introduce a pulsar log4j2 appender #1316
Conversation
sijie
added
the
type/feature
|
NOTE: Ideally I would like to contribute this module as part of log4j2 - https://github.com/apache/logging-log4j2 . However we would like to use this module for pulsar-functions in #1314 , so make the initial contribution here. At the same time, I would send a PR to log4j2 as well. |
|
filed a jira issue to log4j2 to contribute this appender back to log4j2 : https://issues.apache.org/jira/browse/LOG4J2-2280 |
|
retest this please |
1 similar comment
|
retest this please |
pulsar-log4j2-appender/pom.xml
Outdated
| <scope>test</scope> | ||
| </dependency> | ||
| <!-- Required for AsyncLoggers --> | ||
| <dependency> |
There was a problem hiding this comment.
Let's add this in top level dependency management to fix the version there.
| public boolean releaseSub(final long timeout, final TimeUnit timeUnit) { | ||
| if (producer != null) { | ||
| try { | ||
| producer.close(); |
There was a problem hiding this comment.
should we use closeAsync().get(timeout, timeUnit) to respect the contract?
| producer = client.newProducer() | ||
| .topic(topic) | ||
| .producerName("pulsar-log4j2-appender-" + topic) | ||
| .enableBatching(true) |
There was a problem hiding this comment.
If the mode is syncSend, batching would just increase the latency (due to waiting for time budget) and decrease the throughput, since we only have 1 in-flight message.
| .topic(topic) | ||
| .producerName("pulsar-log4j2-appender-" + topic) | ||
| .enableBatching(true) | ||
| .batchingMaxPublishDelay(1, TimeUnit.MILLISECONDS) |
There was a problem hiding this comment.
If we use batching (with sendAsync()) we could increase this delay, since most likely it won't be latency sensitive.
|
retest this please |
|
@merlimat I have addressed all your comments. |
Motivation
when developing
pulsar-functions, it would be good to log the log messages generated by functions to pulsar, so the user can easily tail the log messages topic to get all the messages per functions.Modifications
This PR is to add a pulsar based appender.
Result
We are able to use pulsar log4j2 appender to log messages.