What is the minimal configuration ?

[jmdopereiro]

I'm just trying to start sending logs to ES. At the moment I'm facing the following error

ERROR StatusLogger No Elasticsearch client factory [JestHttp|ElasticsearchBulkProcessor] provided for AsyncBatchDelivery: clientObjectFactory
ERROR StatusLogger Could not create plugin of type class org.appenders.log4j2.elasticsearch.AsyncBatchDelivery for element AsyncBatchDelivery
org.apache.logging.log4j.core.config.ConfigurationException: Arguments given for element AsyncBatchDelivery are invalid

Please let me know if I should create an Issue instead of a discussion.

Thanks

Jose

[rfoltyns]

Mininal config depends on Elasticsearch version and log4j2-elasticsearch module that you're using.

Once you provide an index template and XML config from project's main page, it should work.

Have you tried log4j2-elasticsearch-examples yet? Could you post your current config?

In 1.5.1, required elements are:

• ClientObjectFactory implementation: JestHttp, JestBufferedHttp, HCHttp or ElasticsearchBulkProcessor
• Layout that produces JSON output: JacksonJsonLayout (recommended) or Log4j2 JsonLayout with compact=true
• Index template for appropriate Elasticsearch version: see indexTemplate-(version).json files here
• mappingType="index" if using Jest or HC for Elasticsearch <7.x. No mappingType if 7.x
• appenders-logging jar
• log4j2-api, log4j2-core (preferrably 2.13+), jackson-core, jackson-databind, jackson-annotations (preferrably 2.12+) jars for all modules
• httpasyncclient, httpclient, httpcore and httpcore-nio jars for HC and Jest
• appropriate version of elasticsearch jar for log4j2-elasticsearch(version)-bulkprocessor

Optional elements:

• If JestBufferedHttp or HCHttp, JacksonJsonLayout is mandatory and buffer pool configurations are mandatory for batches at ClientObjectFactory level and logs at JacksonJsonLayout level (PooledItemSourceFactory), see example
• (optional) netty-buffer jar if using pooled config
• chronicle-map jar is using failover with retry
• appenders-jackson-st jar if using JacksonJsonLayout.singlethread=true
• jackson-module-afterburner jar if using JacksonJsonLayout.afterburner=true
• bcprov-jdk15on and bcpkix-jdk15on if using XPack features with Jest or HC
• shield or x-pack-transport jar depending of Elasticsearch version if using log4j2-elasticsearch(version)-bulkprocessor
• elasticearch or transport jar depending of Elasticsearch version if using log4j2-elasticsearch(version)-bulkprocessor
• disruptor jar if using AsyncLogger
• lots of other transitive dependencies if you're not using Maven, Gradle or equivalent

[jmdopereiro]

I tried log4j2-elasticsearch-jest-springboot against my ES 6 and got some documents indexed there.

The log4j element in my application is

    <Elasticsearch name="elasticsearchAsyncBatch">
        <IndexName indexName="log4j2"/>
        <JacksonJsonLayout/>
        <AsyncBatchDelivery batchSize="5000" deliveryInterval="5000">
            <IndexTemplate name="log4j2" path="/Users/josemanueldopereiro/indexTemplate.json"/>
            <JestHttp serverUris="http://localhost:9200"/>
        </AsyncBatchDelivery>
    </Elasticsearch>

[jmdopereiro]

This are my dependencies at the moment (log4j2 coming from spring-boot-starter-log4j2:2.0.7.RELEASE):

log4j2-elasticsearch-core:1.4.0
io.searchbox:jest:6.3.1
log4j-api:2.10.0
log4j-core:2.10.0
jackson-core:2.9.7
jackson-databind:2.9.7
jackson-annotations:2.9.0

[rfoltyns]

I got this one working with ES 6.80 in jest-springboot example (notice JestHttp.mappingType=index ):

        <Elasticsearch name="elasticsearch">
            <IndexName indexName="log4j2"/>
            <JacksonJsonLayout/>
            <AsyncBatchDelivery batchSize="5000" deliveryInterval="500">
                <IndexTemplate name="log4j2" path="<MY_LOCAL_REPO_PATH>/log4j2-elasticsearch/log4j2-elasticsearch-jest/src/test/resources/indexTemplate-6.json"/>
                <JestHttp serverUris="http://localhost:9200" mappingType="index"/>
            </AsyncBatchDelivery>
        </Elasticsearch>

Apart from latest Log4j2 jars you're missing log4j2-elasticsearch-jest jar - without it Log4j2 has no idea what JestHttp is. log4j2-elasticsearch-core is a transitive dependency, if you're using Maven, Gradle or equivalent, once log4j2-elasticsearch-jest is declared, core should be pulled in automatically. Also, try the latest: log4j2-elasticsearch:1.5.1.

[jmdopereiro]

Thank you, I'm getting some progress, the problem now is related with log4j2 itself,

java.lang.NoClassDefFoundError: org/apache/logging/log4j/core/time/Instant

I assume that I need to get closer to 2.13+ as you suggested before.

[rfoltyns]

Yes, org/apache/logging/log4j/core/time/Instant was added in 2.11.0-rc1

[rfoltyns]

@jmdopereiro Any luck with upgraded Log4j2 jars?

[jmdopereiro]

I have to upgrade other libraries, I will keep you posted! Thanks! :-)