Support OpenSearch 2.x and Elasticsearch 8.x

[turesheim]

Description
OpenSearch 2.0 and Elasticsearch 8.0 har removed the option to specify types in requests and any call from log4j2-elasticsearch to mentioned instances will return something like this and fail:

2022-05-13 08:07:36,304 [I/O dispatcher 1] DEBUG - - http-outgoing-4 << "{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}"

Configuration
I don't think we need a particular configuration to recreate this.

Runtime (please complete the following information):

• OpenSearch 2.0RC1
• log4j2-elasticsearch-hc 1.5.2

Additional context

• https://www.elastic.co/guide/en/elasticsearch/reference/7.17/removal-of-types.html
• [Meta] Remove mapping types opensearch-project/OpenSearch#1940

[rfoltyns]

Option to remove _type will be added in 1.6.0.

As it may require comprehensive changes, it will most likely NOT be back-ported to 1.5.

[kamuffe]

Hi.

First of all thanks for this great project.
We already upgrated to ElasticSearch 8.4 and since then now logs are not sent to ElasticSearch anymore due to this issue.

Can you already weigh up when you have time to fix this issue?

Best,
Kamuffe

[rfoltyns]

Thank you for the good word. I'm glad that you found it useful.

I have the code working. Just wrapping up testing. I can actually push it today.

[rfoltyns]

I'll try to release it later this month

[kamuffe]

Thank you very much ;)

I really appreciate that :)

[rfoltyns]

Released in 1.6.0.

OpenSearch 2.0 support is limited to legacy Index Template API (no composable templates).

ISM is out-of-scope of this project - it will be a part of log4j2-opensearch project in the future. For now, you can get it by extending OperationFactoryDispatcher and ClientObjectFactory classes in a module you're using and adding appropriate Operation factory:

• ElasticsearchOperationFactory and HCHttpPlugin (or HCHttp for programmatic config) in HC module
• ElasticsearchOperationFactory and AHCHttpPlugin (or AHCHttp for programmatic config) in AHC module
• JestOperationFactoryDispatcher and (Buffered)JestHttpObjectFactory in Jest module

[turesheim]

Thanks for adding OpenSearch 2.0 support. It is very much appreciated! PS: We are using OpenSearch 2.3 with composable templates and this version of library. I'm not aware of anything broken. Could you elaborate?

[rfoltyns]

@turesheim Great news! 👍 I'm glad you got it working.

I tested it only against 2.0 tbh. I had trouble with the API - I don't remember why anymore. I need to revisit it then.

I'll happily make slight adjustments and accept PR for OpenSearch. However, given that it's forked from ES 7.10, API compatibility will unfortunately be only a side-effect rather than the goal of this project. This will obviously change once log4j2-opensearch come to life. I only have 2 hands though...

[kamuffe]

Hi.

Thank you for releasing this issue.
I can confirm, that the ES reporting with ElasticSearch 8.4.3 is working again.

Best,
Kamuffe