Update by query API #2230

martijnvg · 2012-09-03T16:37:38Z

The update by query API allows all documents that with the query to be updated with a script. This feature is experimental.
The update by query works a bit different than the delete by query. The update by query api translates the documents that match into bulk index / delete requests. After the bulk limit has been reached, the bulk requests created thus far will be executed. After the bulk requests have been executed the next batch of requests will be prepared and executed. This behavior continues until all documents that matched the query have been processed. The bulk size can be configured with the action.updatebyquery.bulk_size option in the elasticsearch configuration. For example:
action.updatebyquery.bulk_size=2500

Example usage

Index an example document:
curl -XPUT 'localhost:9200/twitter/tweet/1' -d '

{
   "text" : {
      "message" : "you know for search"
    }
}

Execute the following update by query command:
curl -XPOST 'localhost:9200/twitter/_update_by_query' -d '

{
    "query" : {
        "term" : {
            "message" : "you"
        }
    },
    "script" : "ctx._source.field1 += 1"
}

This will yield the following response:

{
  "ok" : true,
  "took" : 9,
  "total" : 1,
  "updated" : 1,
  "indices" : [ {
    "twitter" : { }
  } ]
}

By default no bulk item responses are included in the response. If there are bulk item responses included in the response, the bulk response items are grouped by index and shard. This can be controlled by the response option.

Options:

Additional general options in request body:

lang: The script language.
params: The script parameters.

Query string options:

replication: The replication type for the delete/index operation (sync or async).
consistency: The write consistency of the index/delete operation.
response: What bulk response items to include into the update by query response. This can be set to the following: none, failed and all. Defaults to none. Warning: all can result in out of memory errors when the query results in many hits.
routing : Sets the routing that will be used to route the document to the relevant shard.
timeout : Timeout waiting for a shard to become available.

This is issue originates from #1607

The text was updated successfully, but these errors were encountered:

mdojwa · 2012-12-08T09:37:33Z

Hi, when will it be released ?

martijnvg · 2012-12-10T09:46:42Z

Hi @mdojwa Not sure when this will be included into ES. The implementations works fine, but there is one thing that we are missing. An update by query request can take a long time to complete. It would be very help ful to cancel a running update by query request. This isn't implemented yet. Perhaps in the future we might have a process api, where one can see the current running requests and via this api requests can be cancelled.

ofavre · 2013-02-13T16:07:54Z

I've packaged it (pull request #2231) as a plugin: yakaz/elasticsearch-action-updatebyquery.
Have fun.

mdojwa · 2013-02-13T20:03:47Z

Hi, thanks for this one :)

ofavre · 2013-03-01T15:18:53Z

The plugin is now ported to ElasticSearch 0.90.0.Beta1!

neogenix · 2013-03-04T02:50:55Z

+1

Vineeth-Mohan · 2013-03-24T17:20:58Z

+1 to this feature. @ofavre - Just one question here. Will all the updates be atomic or sequential ? As in would there be a situation where say half of the documents in the query was updated and the rest was not in case of process crash or restart.

ofavre · 2013-03-24T18:01:38Z

I've just packaged the code, I didn't write it. But the pull request description states that it treats document in batch, hence there definitively can be cases where the documents are help updated.
For small changes, maybe ensuring that all the documents fit inside one batch can help. The batch size is controller by action.updatebyquery.bulk_size.

Vineeth-Mohan · 2013-03-25T02:15:19Z

@ofavre - Thanks Oliver. That answers my question perfectly.

The update by query API allows all documents that with the query to be updated with a script. This feature is experimental. The update by query works a bit different than the delete by query. The update by query api translates the documents that match into bulk index / delete requests. After the bulk limit has been reached, the bulk requests created thus far will be executed. After the bulk requests have been executed the next batch of requests will be prepared and executed. This behavior continues until all documents that matched the query have been processed. The bulk size can be configured with the *action.updatebyquery.bulk_size* option in the elasticsearch configuration. For example: `action.updatebyquery.bulk_size=2500` The commit relates to issue elastic#2230 Example usage ================================================= Index an example document: curl -XPUT 'localhost:9200/twitter/tweet/1' -d ' ``` { "text" : { "message" : "you know for search" } } ``` Execute the following update by query command: curl -XPOST 'localhost:9200/twitter/_update_by_query' -d ' ``` { "query" : { "term" : { "message" : "you" } }, "script" : "ctx._source.field1 += 1" } ``` This will yield the following response: ``` { "ok" : true, "took" : 9, "total" : 1, "updated" : 1, "indices" : [ { "twitter" : { } } ] } ``` By default no bulk item responses are included in the response. If there are bulk item responses included in the response, the bulk response items are grouped by index and shard. This can be controlled by the `response` option. Options ===================================================== Additional general options in request body: * `lang`: The script language. * `params`: The script parameters. Query string options ----------------------------------------------------- * `replication`: The replication type for the delete/index operation (sync or async). * `consistency`: The write consistency of the index/delete operation. * `response`: What bulk response items to include into the update by query response. This can be set to the following: `none`, `failed` and `all`. Defaults to none. Warning: `all` can result in out of memory errors when the query results in many hits. * `routing` : Sets the routing that will be used to route the document to the relevant shard. * `timeout` : Timeout waiting for a shard to become available.

parth-j-gandhi · 2014-06-03T18:47:19Z

This is a much needed feature for my current project which indexes hundreds of thousands of docs and updates them on a regular basis. Is this feature available via elasticsearch-py python client?

ofavre · 2014-06-04T08:15:36Z

There is no chance that this unofficial (or not yet official) feature contributed through a plugin is exposed through a mainstream client library.
And unless this client library lets you write arbitrary calls easily, you're good to go with urllib and the like.

1st · 2014-06-25T10:44:39Z

Hello guys!

As I understand, I can insert JSON data into index, query it, but can't update this JSON record? For example, I need to change structure of all JSON documents in one given collection - and I need to add extra field "is_moderated = true/false". Is it possible to do in current implementation of Elastic Search?

If its not possible now - please help me to find workaround for this common task. Thanks!

ofavre · 2014-06-27T09:21:38Z

You can use the Update API to update any single document.
To update several documents at once, the good news is the pull request associated to this issue has been packaged as a plugin : yakaz/elasticsearch-action-updatebyquery.
Using it you can run any script you like and modify the document structure, and it will be re-indexed.

xelllee · 2015-08-04T20:50:53Z

+1

IUnknownPtr · 2015-08-15T07:46:32Z

+1

abhas9 · 2015-08-16T05:52:27Z

+1

rtrujill007 · 2015-08-20T23:21:28Z

+1

chibenwa · 2015-08-24T00:11:35Z

+1

knoxxs · 2015-08-26T16:05:19Z

+1

ogorun · 2015-09-16T09:39:14Z

+1

clintongormley · 2015-09-19T17:15:50Z

Update by query should be implemented as part of the reindex API #492

davidchipping · 2015-10-13T09:52:03Z

+1

xpepermint · 2015-10-14T07:01:32Z

+1

sDaniel · 2015-10-14T09:45:06Z

Please stop to +vote spamming and let github know we need a voting feature.
isaacs/github#9

KurtPreston · 2015-11-25T21:19:44Z

+1

pavel-main · 2015-11-28T21:36:29Z

+1

nik9000 · 2016-03-01T15:24:35Z

_update_by_query has landed in master: c7c8bb3

I'm not going close this until it's backported to 2.3 though.

lukens · 2016-03-01T16:31:32Z

great work @nik9000!

eskibars · 2016-03-29T18:21:19Z

Closing, as the _update_by_query @nik9000 mentioned has been backported to 2.3. For those finding this later, see documentation at https://www.elastic.co/guide/en/elasticsearch/reference/2.x/docs-update-by-query.html

DaveVdE · 2016-06-17T09:22:54Z

Has the response query parameter been integrated with this feature too? I can't seem to get it to work.

nik9000 · 2016-06-17T12:06:10Z

No. This wasn't a port of the original, it was its own thing. The bulk
responses are only available for failures that abort the action. We talked
a lot about including more responses but decided that it wasn't worth it
given that they could consume too much memory.
On Jun 17, 2016 5:22 AM, "Dave Van den Eynde" [email protected]
wrote:

Has the response query parameter been integrated with this feature too? I
can't seem to get it to work.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2230 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AANLos0oaH6K9DI0bGcoAeVpTyoHe8yJks5qMmdzgaJpZM4AIsk8
.

ofavre pushed a commit to yakaz/elasticsearch-action-updatebyquery that referenced this issue Feb 13, 2013

Readme from elastic/elasticsearch#2230

4c5df16

ofavre mentioned this issue Sep 25, 2013

Upgrade for ES0.90.4 yakaz/elasticsearch-action-updatebyquery#5

Closed

martijnvg mentioned this issue Sep 25, 2013

Update API: update by query #1607

Closed

ofavre mentioned this issue Oct 25, 2013

Update query lock up yakaz/elasticsearch-action-updatebyquery#6

Closed

lukas-vlcek mentioned this issue Nov 26, 2013

Create a circuit breaker to prevent searches from bringing down a node #2929

Closed

consulthys mentioned this issue Mar 24, 2014

Leaking file handles? #5189

Closed

ofavre mentioned this issue Apr 8, 2014

Provide more context variables in update scripts #5724

Closed

clintongormley added the :Reindex API label Sep 19, 2015

nik9000 self-assigned this Nov 26, 2015

nik9000 mentioned this issue Dec 17, 2015

Basic reindex implementation #15125

Merged

nik9000 removed the stalled label Dec 19, 2015

eskibars mentioned this issue Jan 13, 2016

Reindex from _source by document ID or Query #492

Closed

nik9000 added v5.0.0-alpha1 v2.3.0 labels Mar 1, 2016

clintongormley removed v2.3.0 v5.0.0-alpha1 labels Mar 10, 2016

eskibars closed this as completed Mar 29, 2016

lcawl added :Distributed Indexing/CRUD A catch all label for issues around indexing, updating and getting a doc by id. Not search. and removed :Reindex API labels Feb 13, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update by query API #2230

Update by query API #2230

martijnvg commented Sep 3, 2012

mdojwa commented Dec 8, 2012

martijnvg commented Dec 10, 2012

ofavre commented Feb 13, 2013

mdojwa commented Feb 13, 2013

ofavre commented Mar 1, 2013

neogenix commented Mar 4, 2013

Vineeth-Mohan commented Mar 24, 2013

ofavre commented Mar 24, 2013

Vineeth-Mohan commented Mar 25, 2013

parth-j-gandhi commented Jun 3, 2014

ofavre commented Jun 4, 2014

1st commented Jun 25, 2014

ofavre commented Jun 27, 2014

xelllee commented Aug 4, 2015

IUnknownPtr commented Aug 15, 2015

abhas9 commented Aug 16, 2015

rtrujill007 commented Aug 20, 2015

chibenwa commented Aug 24, 2015

knoxxs commented Aug 26, 2015

ogorun commented Sep 16, 2015

clintongormley commented Sep 19, 2015

davidchipping commented Oct 13, 2015

xpepermint commented Oct 14, 2015

sDaniel commented Oct 14, 2015

KurtPreston commented Nov 25, 2015

pavel-main commented Nov 28, 2015

nik9000 commented Mar 1, 2016

lukens commented Mar 1, 2016

eskibars commented Mar 29, 2016

DaveVdE commented Jun 17, 2016

nik9000 commented Jun 17, 2016

Update by query API #2230

Update by query API #2230

Comments

martijnvg commented Sep 3, 2012

Example usage

Options:

Query string options:

mdojwa commented Dec 8, 2012

martijnvg commented Dec 10, 2012

ofavre commented Feb 13, 2013

mdojwa commented Feb 13, 2013

ofavre commented Mar 1, 2013

neogenix commented Mar 4, 2013

Vineeth-Mohan commented Mar 24, 2013

ofavre commented Mar 24, 2013

Vineeth-Mohan commented Mar 25, 2013

parth-j-gandhi commented Jun 3, 2014

ofavre commented Jun 4, 2014

1st commented Jun 25, 2014

ofavre commented Jun 27, 2014

xelllee commented Aug 4, 2015

IUnknownPtr commented Aug 15, 2015

abhas9 commented Aug 16, 2015

rtrujill007 commented Aug 20, 2015

chibenwa commented Aug 24, 2015

knoxxs commented Aug 26, 2015

ogorun commented Sep 16, 2015

clintongormley commented Sep 19, 2015

davidchipping commented Oct 13, 2015

xpepermint commented Oct 14, 2015

sDaniel commented Oct 14, 2015

KurtPreston commented Nov 25, 2015

pavel-main commented Nov 28, 2015

nik9000 commented Mar 1, 2016

lukens commented Mar 1, 2016

eskibars commented Mar 29, 2016

DaveVdE commented Jun 17, 2016

nik9000 commented Jun 17, 2016