Unable to take snapshots to S3 from AWS ES

[themac13]

Hi,

My AWS ES is version 2.3. Knowing that there are some issues with curator 4 for AWS, I have tried 3.5.0 and also 3.4.1 but snapshots taking does not work.

curator --port 80 --host x show snapshots --repository my-s3-bucket --all-snapshots works, it shows all the previous snapshots.

When I run

curator --port 80 --host x snapshot --repository my-s3-bucket --prefix curato-snapshot indices --time-unit days --older-than 7 --timestring '%Y.%m.%d' --prefix my-log-prefix

I get this:

2016-10-27 19:12:00,042 DEBUG urllib3.util.retry from_int:155 Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0)
2016-10-27 19:12:00,045 DEBUG urllib3.connectionpool _make_request:387 "GET /_snapshot/_status HTTP/1.1" 401 64
2016-10-27 19:12:00,045 WARNING elasticsearch log_request_fail:82 GET /_snapshot/_status [status:401 request:0.003s]
2016-10-27 19:12:00,045 DEBUG elasticsearch log_request_fail:90 > None
Traceback (most recent call last):
File "/usr/local/bin/curator", line 11, in
sys.exit(main())
File "/usr/local/lib/python2.7/site-packages/curator/curator.py", line 5, in main
cli( obj={ "filters": [] } )
File "/usr/local/lib/python2.7/site-packages/click/core.py", line 716, in call
return self.main(_args, *_kwargs)
File "/usr/local/lib/python2.7/site-packages/click/core.py", line 696, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python2.7/site-packages/click/core.py", line 889, in invoke
return ctx.invoke(self.callback, *_ctx.params)
File "/usr/local/lib/python2.7/site-packages/click/core.py", line 534, in invoke
return callback(_args, *_kwargs)
File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, *_kwargs)
File "/usr/local/lib/python2.7/site-packages/curator/cli/index_selection.py", line 172, in indices
retval = do_command(client, ctx.parent.info_name, working_list, ctx.parent.params, master_timeout)
File "/usr/local/lib/python2.7/site-packages/curator/cli/utils.py", line 250, in do_command
skip_repo_validation=params['skip_repo_validation'],
File "/usr/local/lib/python2.7/site-packages/curator/api/snapshot.py", line 49, in create_snapshot
in_progress = client.snapshot.status()['snapshots']
File "build/bdist.linux-x86_64/egg/elasticsearch/client/utils.py", line 69, in _wrapped
File "build/bdist.linux-x86_64/egg/elasticsearch/client/snapshot.py", line 149, in status
File "build/bdist.linux-x86_64/egg/elasticsearch/transport.py", line 307, in perform_request
File "build/bdist.linux-x86_64/egg/elasticsearch/connection/http_urllib3.py", line 93, in perform_request
File "build/bdist.linux-x86_64/egg/elasticsearch/connection/base.py", line 105, in _raise_error
elasticsearch.exceptions.AuthenticationException: TransportError(401, u'{"Message":"Your request: '/_snapshot/_status' is not allowed."}')

Am I missing something? Some flag? --dry_run does not produce any errors

[untergeek]

TransportError(401, u'{"Message":"Your request: \'/_snapshot/_status\' is not allowed."}')

It appears that the snapshot status endpoint is not available in AWS ES 2.x. This is outside the control of Curator. I don't recall anyone complaining about this in AWS ES 1.x, but I may just not have had any reports of it.

[untergeek]

I've added a note about this to the compatibility matrix in the README.

[yissachar]

@untergeek FWIW, I reported the same thing for AWS ES 1.x a while ago: #639. It would be good to add this to the compatibility matrix.

[untergeek]

@yissachar I'm sorry I didn't connect the dates with that. You're right, and I'll update the compatibility matrix.

[iain17]

Is there any alternative of doing this? I did find this repo: https://github.com/cldcvr/elasticsearch-s3-backup

Would it be wise calling curator afterwards to just delete a section of the data?

Also based from a little searching I found an answer from Amazon about this. Seems it is just a question of adding the right IAM policy rules: https://forums.aws.amazon.com/thread.jspa?messageID=697248 see HernanV@AWS his response.
Was this looked into at the time? A signed request should do it then?

Or is it better to just install elasticsearch-cloud-aws plugin? But that'll only work for elastic.co's hosted aws version right? Or is there some other way of doing this?

[yissachar]

@iain17 The problem is not with the IAM policy rules, but the fact that Curator makes certain API calls that AWS Elasticsearch has blocked off for whatever reason.

Our workaround was to manually write a script that performed the snapshot and deletes old indexes. We would have preferred to use Curator since it has many nice features and is more battle-tested, but until AWS opens up their endpoints our custom script works well enough.

[untergeek]

This is apparently still true as of #987, even with AWS ES 5.3