[Feature Request] scripted_upsert not supported

[lucebert]

• Feature Request : Scripted_upsert functionality is not supported for the moment

Feature description

This issue is related with #538, the connector does not support for the moment scripted_upsert functionality

This functionality also requieres defining an empty upsert body

[andrewchen30]

Hi @lucebert

Is there any way I can use scripted_upsert to avoid document missing exception?

I am facing a problem that sometimes Spark streaming scriped_update will get document missing exception. I read some article then people said I should use upsert to replace update method. Is there any way I can use scripted_upsert to avoid document missing exception? Or I should try to ignore the exception by using a custom error handler.

Than you very much.

[lucebert]

Hi @PolarBearAndrew

You should use scripted_upsert but this is not available in Elasticsearch-hadoop until they merge this PR #1454.
There is a workaround solution which is to use the upsert query.

you have to put this parameter :

"es.write.operation" -> "upsert"
"es.update.script.params" -> <<my params>> // include all fields to update
"es.update.script.inline" -> <<my script>> //use params

This will work but the downside is that the query generated will have the full document in params and in the upsert, which is less performant.

[andrewchen30]

Hi @lucebert

Thank you very much 🙏, I will try another way to complete my work.

I hope your PR can be merge!