[FEATURE] Send search request query DSL json directly to search request

[graytaylor0]

Is your feature request related to a problem?

Data Prepper is going to support configurable queries that will utilize the opensearch java client for search requests.

The query DSL is being taken is as a json string, and the opensearch java client does not support passing a Map or json string to pass this query DSL directly.

However, the opensearch java client requires queries to be modeled using the opensearch java client classes

What solution would you like?

An option to pass the search request directly to the SearchRequest class instead of having to model it piece by piece

What alternatives have you considered?

modeling the entirety of a search request in Java in Data Prepper. This does not scale well

Do you have any additional context?

Add any other context or screenshots about the feature request here.

[wbeckler]

This looks like a duplicate of #377

Please take a look and close this if you think #377 does the job.

[graytaylor0]

Thanks @wbeckler.

yes that would do the job. Closing this issue

[graytaylor0]

Reopening as the real need is a similar feature supported by the Elasticsearch Java Client here (https://www.elastic.co/guide/en/elasticsearch/client/java-api-client/7.17/loading-json.html)

[SaranSundar]

I really like the description used in the es docs, i agree that my teams normal workflow is using kibana to make some custom query, then directly using that json in the high level rest client. I would like for that to still be possible in the future,

as the high level rest client gets deprecated there still needs to be a way to go from json request to any endpoint in kibana to being able to send that as a request in my java app.

[dblock]

For bookkeeping, in what way is #377 different? Also would love to see a PR ;)

[graytaylor0]

#377 Is a much larger feature add, and would require Data Prepper to use a completely different client implementation rather than just getting the json query support that is needed. Also due to the size of the change of #377, it looks like work has been paused on it for a couple of months now.

[brentam]

I wholeheartedly agree that we this feature should be prioritised.
Our team has encountered a significant challenge that is nudging us towards considering alternative clients. The issue arises when the internal builders of SearchRequest have a bug (referenced in Issue #780), leading to the generation of an incorrect request with invalid JSON.

In that case there is nothing we can do and the api is useless for us, since the deserialiser will produce the wrong query...

We are constructing the request body using a library and would like to be able to pass the Json body directly to the "search" endpoint.

Maybe we could have something like this
SearchRequest.Builder().index("someIndex").fromJson(theJson).build()

Please, correct me if I am wrong but that does not seem like a tall task, wouldn't we need to just modify the code that serialises the json prior to create a request and use the raw json passed?

[dblock]

Please, correct me if I am wrong but that does seem like a tall task, wouldn't we need to just modify the code that serialises the json prior to create a request and use the raw json passed?

I think so, I am not sure why the previous attempts haven't worked. Take it up @brentam!