mongodb to elasticsearch removal strategy

[richardwilly98]

Question from Martin
Hi Richard,

I am successfully using your mongo 2 elastic river plugin to power the backend of my latest web project. Thank you for taking the time to develop a great bit of code.

I wonder if I could just pick your brains for a second?

The documents I'm pushing into elastic do need to be removed once a certain flag is set.
I want to keep the records in mongo.
I did look at your 'filter' param in the config, but you said there was a limitation where it would not delete records once they were already in elastic.

Do you have any ideas on how I could accomplish this?

Many Thanks,
Martin

[richardwilly98]

Hi Martin,

Script filter should work in your scenario. It has not been released yet but it is available from version 1.6.0-SNAPSHOT available here [1].
It has not been clearly documented yet but take alook to my comments to this issue [2].
Also the unit test give you an idea on how to remove a document (see [3]).
I believe ctx.deleted = true should delete the document from the index.

Please let me know if it works or not..

[1] - https://github.com/richardwilly98/elasticsearch-river-mongodb/downloads
[2] - #41
[3] - https://github.com/richardwilly98/elasticsearch-river-mongodb/tree/master/src/test/java/test/elasticsearch/plugin/river/mongodb/script

Thanks,
Richard.

[martincharlesworth]

Hi Richard,

I can see that I can use the script param to delete a document but how can I add in the conditional to check a field which tells me whether it should be deleted or not?

Cheers,
Martin

[richardwilly98]

Hi Martin,

That's pure javascript.
So for example if your document has an attribute tobedeleted you can use it
to trigger the deletion in ES using the following snippet:
if (ctx.document.tobedeleted == true) { ctx.deleted = true; }

Again I have not tested deletion but it should work.

Thanks,
Richard.

On Thu, Nov 29, 2012 at 10:46 AM, mgcharl [email protected] wrote:

Hi Richard,

I can see that I can use the script param to delete a document but how can
I add in the conditional to check a field which tells me whether it should
be deleted or not?

Cheers,
Martin

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/42#issuecomment-10853028.

[martincharlesworth]

Hi Rich,

I got the elastic javascript plugin installed so the script runs but I get an error:

I am trying to check against a string:
"script": "if( ctx.document.state == 'CLOSED' ) { ctx.deleted = true; }"

but I get an error:

[org.elasticsearch.river.mongodb.MongoDBRiver$Indexer] failed to script process {id=50b79fc33c458821e2000002, operation=u, document={_id=50b79fc33c458821e2000002, title=foobar, state=CLOSED, updatedAt=null}}, ignoring

org.mozilla.javascript.EcmaError: ReferenceError: "CLOSED" is not defined. (Script1.js#1)

Any Ideas? I tried reading elasticsearch scripting docs but they do not give any javascript help.

Cheers,
Martin

[richardwilly98]

Hi Martin,

The javascript looks good. Can you please provide ES log?
I will take a look tomorrow.

Thanks,
Richard.
Sent via BlackBerry Bold

-----Original Message-----
From: mgcharl [email protected]
Date: Thu, 29 Nov 2012 09:59:58
To: richardwilly98/elasticsearch-river-mongodbelasticsearch-river-mongodb@noreply.github.com
Reply-To: richardwilly98/elasticsearch-river-mongodb [email protected]
Cc: Richard [email protected]
Subject: Re: [elasticsearch-river-mongodb] mongodb to elasticsearch removal
strategy (#42)

Hi Rich,

I got the elastic javascript plugin installed so the script runs but I get an error:

I am trying to check against a string:
"script": "if( ctx.document.state == 'CLOSED' ) { ctx.deleted = true; }"

but I get an error:

[org.elasticsearch.river.mongodb.MongoDBRiver$Indexer] failed to script process {id=50b79fc33c458821e2000002, operation=u, document={_id=50b79fc33c458821e2000002, title=foobar, state=CLOSED, updatedAt=null}}, ignoring

org.mozilla.javascript.EcmaError: ReferenceError: "CLOSED" is not defined. (Script1.js#1)

Any Ideas? I tried reading elasticsearch scripting docs but they do not give any javascript help.

Cheers,
Martin

---

Reply to this email directly or view it on GitHub:
#42 (comment)

[martincharlesworth]

Thanks Richard, unfortunately I have been tinkering since on my dev box and so the log traces are gone.

To replicate it would simply be trying to check any string value in javascript inside script param.

I'm afraid I don't have to time to reproduce right now but I will definitely come back to the issue soon.

Many Thanks,
Martin

[mzafer]

Hi Richard,

Thanks for the excellent plugin. I had the same need as this issue and see that it works fine. But in my case I have documents as parents- childs and when a parent is deleted in ES how can I cascade that delete to children too.

Thanks
Zafer

[richardwilly98]

Hi Zafer,

Could you please provide an example (via gists)? I will take a look at the options.

Thanks,
Richard.

[mzafer]

Hi Richard,

When I posted this comment I was not aware that elastic search does not provide a off the shelf feature to delete the children when ever a parent is deleted. There is a issue open for this in ES (elastic/elasticsearch#1954). But there is a work around discussed in elastic/elasticsearch#2705, I am yet to try if that works though. Is it possible to implement this workaround ( if it works ) in the river ?

I'll let you know if the work around works and create a gist as well in the next 2 -3 days.

Thanks

[mzafer]

I extended the scripts you have to test issue #64, updated two files and added a new one. The gist is at https://gist.github.com/mzafer/5542043

So now the question is more of "How do we change the script in the author river (_02_mongodb-river-author.json) to query for the books(with _parentId == author.id ) and delete them when the 'state' of author is changed to 'INACTIVE' ?" if it is possible in the first place. I look for some documentation on the es-javascript plugin but did not find any.

Thanks

[richardwilly98]

Hi,

It is currently not possible from the script but I will look at the options.

Thanks,
Richard.