405 status code thrown due to missing delete by query plugin.

[TomonoriSoejima]

Issue description

A user stumbled upon 405 status code in their application log.

0.000 405 405 381 507 "DELETE http://10.212.133.38:80/index_query?q=* HTTP/1.1" "Jakarta Commons-HttpClient/3.0.1" - - arn:aws:xxxxx "Root=1-xxxx" "-" "-" 0 2023-03-07T06:18:57.581000Z "forward" "-" "-" "10.212.133.108:9200" "405" "-" "-"**

You can see "DELETE http://10.212.133.38:80/index_query?q=*, which results in 405 and this syntax is drawn from blocks below.

            try {
                if (resources.getResourceWrite().isTyped()) {
                    client.delete(resources.getResourceWrite().index() + "/" + resources.getResourceWrite().type() + "/_query?q=*");
                } else {
                    client.delete(resources.getResourceWrite().index() + "/_query?q=*");
                }
            } catch (EsHadoopInvalidRequest ehir) {
                log.info("Skipping delete by query as the plugin is not installed...");
            }

It mentions delete by query as the plugin but this doc https://www.elastic.co/guide/en/elasticsearch/plugins/current/plugins-delete-by-query.html says

The Delete-By-Query plugin has been removed in favor of a new Delete By Query API implementation in core.

As delete by query is now done by API, maybe we should adapt this code to do the same.

POST /my-index-000001/_delete_by_query
{
  "query": {
    "match": {
      "user.id": "elkbee"
    }
  }
}

[masseyke]

I agree that this isn't ideal (apparently the delete by query plugin was replaced in 5.0, so it's been gone for a while!), but are they experiencing any problems? It looks like the code attempts to use the delete by query plugin, and if it's not there it reverts to using a scroll-and-delete approach. So I assume it's working, but the 405 is concerning? And maybe we would get better performance using the delete by query API.

[masseyke]

It's probably also worth noting that this delete method is always going to be slow -- it acts as a SQL truncate in Elasticsearch (it deletes all the data without deleting the index), and Elasticsearch doesn't support any kind of efficient truncate. I'm going to put up a PR to stop using the long-defunct delete-by-query plugin, but I'd recommend not using this at any kind of scale. I believe it's only used via spark sql when SaveMode is set to Overwrite.