Fix broken compatibility with Elasticsearch 8.x/Opensearch 2.x

[ghost]

Request Type

Bug

Work Environment

Question	Answer
OS version (server)	Ubuntu
OS version (client)	20.04
Cortex version / git hash	3.1.6-withdeps
Package Type	Docker
Browser type & version	N/A

Problem Description

Hi, I identified problem with creating cortex database in Elasticsearch >=8.x and OpenSearch >= 2.x. Parameter include_type_name is removed from newest ES/OS versions (reference Moving from types to typeless APIs in Elasticsearch 7.0 for ES, and Remove mapping types #150 for OS).

Steps to Reproduce

1. Setup Opensearch 2.x or ElasticSearch 8.x.
2. Run clean cortex install.
3. Wait for cortex to setup, and click migrate database button.
4. See error message in logs.

Possible Solutions

Cortex uses elastic4play library, which uses elastic4s under the hood. Bumping elastic4s version from 7.17.2 to 8.x should enable compatibility with new Elasticsearch/Opensearch versions.

Complementary information

[error] o.e.d.DBConfiguration - ElasticSearch request failure: PUT:/cortex_6?include_type_name=false
StringEntity({"settings":{"index":{"number_of_shards":5,"number_of_replicas":1,"mapping.nested_fields.limit":100}},"mappings":{"date_detection":false,"numeric_detection":false,
...
"job":["dummy-job","report"],"sequence":["dummy-sequence"],"report":["artifact"],"audit":["dummy-audit"],"user":["dummy-user"],"dblist":["dummy-dblist"]}}}}},Some(application/json))
 => ElasticError(illegal_argument_exception,request [/cortex_6] contains unrecognized parameter: [include_type_name],None,None,None,List(ElasticError(illegal_argument_exception,request [/cortex_6] contains unrecognized parameter: [include_type_name],None,None,None,null,None,None,None,List())),None,None,None,List())

https://github.com/TheHive-Project/elastic4play/blob/86665bfe13a5cb34104482ebe49039d309f23f43/build.sbt#L46

[Linow974]

Hello, I'm facing the same problem, I can't use Cortex meanwhile :/

[sandervandegeijn]

Would be nice if this was fixed, we are moving to open search for our whole stack

[Linow974]

Would be nice if this was fixed, we are moving to open search for our whole stack

Same thing for us, we use Opensearch in its latest versions. To waiting the resolution, we will use an additional Opensearch node with an old version to continue using Cortex.

[TheMatrix97]

Same problem here..... :( Can we give priority to this issue?

Thanks!

[HolzmanoLagrene]

Same here!

[SysLunix]

problem also confirmed for me

[JordyEGNL]

Having the same issue here

2024-10-30 14:57:23,613 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-4 - ElasticSearch request failure: PUT /cortex_6?include_type_name=false
{"settings":{"index":{"number_of_shards":5,"number_of_replicas":1,"mapping.nested_fields.limit":100}},"mappings":{"date_detection":false,"numeric_detection":false,"properties":{"sequenceCounter":{"type":"long"},"updatedAt":{"type":"date","format":"epoch_millis||basic_date_time_no_millis"},"updatedBy":{"type":"keyword"},"createdAt":{"type":"date","format":"epoch_millis||basic_date_time_no_millis"},"createdBy":{"type":"keyword"},"operations":{"type":"binary"},"summary":{"type":"binary"},"full":{"type":"binary"},"dblist":{"type":"keyword"},"value":{"type":"keyword"},"type":{"type":"keyword"},"jobTimeout":{"type":"long"},"jobCache":{"type":"long"},"rateUnit":{"type":"keyword"},"rate":{"type":"long"},"baseConfig":{"type":"keyword"},"configuration":{"type":"binary"},"dataTypeList":{"type":"keyword"},"dockerImage":{"type":"text","fielddata":true},"command":{"type":"text","fielddata":true},"license":{"type":"text","fielddata":true},"url":{"type":"text","fielddata":true},"author":{"type":"text","fielddata":true},"description":{"type":"text","fielddata":true},"workerDefinitionId":{"type":"keyword"},"version":{"type":"keyword"},"name":{"type":"keyword"},"requestId":{"type":"keyword"},"rootId":{"type":"keyword"},"startDate":{"type":"date","format":"epoch_millis||basic_date_time_no_millis"},"base":{"type":"boolean"},"objectId":{"type":"keyword"},"objectType":{"type":"keyword"},"otherDetails":{"type":"text","fielddata":true},"details":{"type":"nested","properties":{"rate":{"type":"long"},"rateUnit":{"type":"keyword"},"organization":{"type":"keyword"},"name":{"type":"keyword"},"updatedAt":{"type":"date","format":"epoch_millis||basic_date_time_no_millis"},"tlp":{"type":"long"},"endDate":{"type":"date","format":"epoch_millis||basic_date_time_no_millis"},"errorMessage":{"type":"text","fielddata":true},"description":{"type":"text","fielddata":true},"_id":{"type":"keyword"},"label":{"type":"keyword"},"dataTypeList":{"type":"keyword"},"pap":{"type":"long"},"jobCache":{"type":"long"},"roles":{"type":"keyword"},"status":{"type":"keyword"},"message":{"type":"text","fielddata":true},"jobTimeout":{"type":"long"},"updatedBy":{"type":"keyword"},"startDate":{"type":"date","format":"epoch_millis||basic_date_time_no_millis"},"parameters":{"type":"binary"},"input":{"type":"binary"}}},"operation":{"type":"keyword"},"cacheTag":{"type":"keyword"},"label":{"type":"keyword"},"fromCache":{"type":"boolean"},"input":{"type":"binary"},"parameters":{"type":"binary"},"errorMessage":{"type":"text","fielddata":true},"message":{"type":"text","fielddata":true},"pap":{"type":"long"},"tlp":{"type":"long"},"attachment":{"type":"nested","properties":{"name":{"type":"keyword"},"hashes":{"type":"keyword"},"size":{"type":"long"},"contentType":{"type":"keyword"},"id":{"type":"keyword"}}},"data":{"type":"binary"},"dataType":{"type":"keyword"},"endDate":{"type":"date","format":"epoch_millis||basic_date_time_no_millis"},"status":{"type":"keyword"},"organization":{"type":"keyword"},"workerName":{"type":"keyword"},"workerId":{"type":"keyword"},"preferences":{"type":"binary"},"avatar":{"type":"binary"},"password":{"type":"keyword"},"roles":{"type":"keyword"},"key":{"type":"keyword"},"login":{"type":"keyword"},"tags":{"type":"keyword"},"config":{"type":"binary"},"binary":{"type":"binary"},"relations":{"type":"join","relations":{"organization":["worker","workerConfig","dummy-organization"],"data":["dummy-data"],"job":["report","dummy-job"],"sequence":["dummy-sequence"],"report":["artifact"],"audit":["dummy-audit"],"user":["dummy-user"],"dblist":["dummy-dblist"]}}}}}
 => ElasticError(illegal_argument_exception,request [/cortex_6] contains unrecognized parameter: [include_type_name],None,None,None,List(ElasticError(illegal_argument_exception,request [/cortex_6] contains unrecognized parameter: [include_type_name],None,None,None,null,None,None,None,List())),None,None,None,List())
2024-10-30 14:57:23,615 [ERROR] from org.elastic4play.services.MigrationSrv in application-akka.actor.default-dispatcher-4 - Migration fail
org.elastic4play.InternalError: Unknown error: ElasticError(illegal_argument_exception,request [/cortex_6] contains unrecognized parameter: [include_type_name],None,None,None,List(ElasticError(illegal_argument_exception,request [/cortex_6] contains unrecognized parameter: [include_type_name],None,None,None,null,None,None,None,List())),None,None,None,List())
        at org.elastic4play.database.DBConfiguration.$anonfun$execute$2(DBConfiguration.scala:158)
        at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
        at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
        at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63)
        at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
        at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
2024-10-30 14:57:23,616 [WARN] from org.thp.cortex.services.ErrorHandler in application-akka.actor.default-dispatcher-4 - POST /api/maintenance/migrate returned 500
2024-10-30 14:57:23,616 [INFO] from org.thp.cortex.services.AccessLogFilter in application-akka.actor.default-dispatcher-4 - 127.0.0.1 POST /api/maintenance/migrate took 490ms and returned 500 364 bytes
2024-10-30 14:58:19,799 [INFO] from org.thp.cortex.services.AccessLogFilter in application-akka.actor.default-dispatcher-10 - 127.0.0.1 GET /api/stream/yfAYwQi6Bv took 60034ms and returned 200 2 bytes
2024-10-30 14:58:19,825 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-10 - ElasticSearch request failure: POST /cortex_6/_search
{"query":{"match":{"relations":{"query":"user"}}},"size":0}
 => ElasticError(index_not_found_exception,no such index [cortex_6],Some(_na_),Some(cortex_6),None,List(ElasticError(index_not_found_exception,no such index [cortex_6],Some(_na_),Some(cortex_6),None,null,None,None,None,List())),None,None,None,List())
2024-10-30 14:59:19,845 [INFO] from org.thp.cortex.services.AccessLogFilter in application-akka.actor.default-dispatcher-10 - 127.0.0.1 GET /api/stream/yfAYwQi6Bv took 60024ms and returned 200 2 bytes
2024-10-30 14:59:20,124 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-4 - ElasticSearch request failure: POST /cortex_6/_search
{"query":{"match":{"relations":{"query":"user"}}},"size":0}
 => ElasticError(index_not_found_exception,no such index [cortex_6],Some(_na_),Some(cortex_6),None,List(ElasticError(index_not_found_exception,no such index [cortex_6],Some(_na_),Some(cortex_6),None,null,None,None,None,List())),None,None,None,List())
2024-10-30 15:00:20,145 [INFO] from org.thp.cortex.services.AccessLogFilter in application-akka.actor.default-dispatcher-4 - 127.0.0.1 GET /api/stream/yfAYwQi6Bv took 60280ms and returned 200 2 bytes
2024-10-30 15:00:20,175 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-4 - ElasticSearch request failure: POST /cortex_6/_search
{"query":{"match":{"relations":{"query":"user"}}},"size":0}
 => ElasticError(index_not_found_exception,no such index [cortex_6],Some(_na_),Some(cortex_6),None,List(ElasticError(index_not_found_exception,no such index [cortex_6],Some(_na_),Some(cortex_6),None,null,None,None,None,List())),None,None,None,List())
2024-10-30 15:01:20,195 [INFO] from org.thp.cortex.services.AccessLogFilter in application-akka.actor.default-dispatcher-5 - 127.0.0.1 GET /api/stream/yfAYwQi6Bv took 60031ms and returned 200 2 bytes
2024-10-30 15:01:20,220 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-7 - ElasticSearch request failure: POST /cortex_6/_search
{"query":{"match":{"relations":{"query":"user"}}},"size":0}
 => ElasticError(index_not_found_exception,no such index [cortex_6],Some(_na_),Some(cortex_6),None,List(ElasticError(index_not_found_exception,no such index [cortex_6],Some(_na_),Some(cortex_6),None,null,None,None,None,List())),None,None,None,List())
2024-10-30 15:02:20,235 [INFO] from org.thp.cortex.services.AccessLogFilter in application-akka.actor.default-dispatcher-7 - 127.0.0.1 GET /api/stream/yfAYwQi6Bv took 60019ms and returned 200 2 bytes
2024-10-30 15:02:20,262 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-4 - ElasticSearch request failure: POST /cortex_6/_search
{"query":{"match":{"relations":{"query":"user"}}},"size":0}
 => ElasticError(index_not_found_exception,no such index [cortex_6],Some(_na_),Some(cortex_6),None,List(ElasticError(index_not_found_exception,no such index [cortex_6],Some(_na_),Some(cortex_6),None,null,None,None,None,List())),None,None,None,List())
2024-10-30 15:03:20,275 [INFO] from org.thp.cortex.services.AccessLogFilter in application-akka.actor.default-dispatcher-4 - 127.0.0.1 GET /api/stream/yfAYwQi6Bv took 60019ms and returned 200 2 bytes
2024-10-30 15:03:20,301 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-7 - ElasticSearch request failure: POST /cortex_6/_search
{"query":{"match":{"relations":{"query":"user"}}},"size":0}
 => ElasticError(index_not_found_exception,no such index [cortex_6],Some(_na_),Some(cortex_6),None,List(ElasticError(index_not_found_exception,no such index [cortex_6],Some(_na_),Some(cortex_6),None,null,None,None,None,List())),None,None,None,List())