Error running against Elasticsearch 7

[d1rtym0nk3y]

I'm just trying praeco our, and spun up docker-compose, using our elasticsearh 7 staging cluster. I get the the following error

Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true.

I can see that various praeco_elastalert_* indexes have been created but they have no mappings. For example

GET /praeco_elastalert_status_error/_mapping
{
  "praeco_elastalert_status_error" : {
    "mappings" : { }
  }
}

I believe this would be related to the removal of document types in 7.0, references are here - https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html

Heres the full stacktrace

elastalert_1  | 17:08:38.485Z  INFO elastalert-server:
elastalert_1  |     ProcessController:  Elastic Version:7
elastalert_1  |     Mapping used for string:{'type': 'keyword'}
elastalert_1  |
elastalert_1  | 17:08:38.485Z ERROR elastalert-server:
elastalert_1  |     ProcessController:  Traceback (most recent call last):
elastalert_1  |       File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
elastalert_1  |         "__main__", fname, loader, pkg_name)
elastalert_1  |       File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
elastalert_1  |         exec code in run_globals
elastalert_1  |       File "/opt/elastalert/elastalert/create_index.py", line 275, in <module>
elastalert_1  |         main()
elastalert_1  |       File "/opt/elastalert/elastalert/create_index.py", line 252, in main
elastalert_1  |         es.indices.put_mapping(index=index, doc_type='elastalert', body=es_mapping)
elastalert_1  |       File "/usr/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/client/utils.py", line 76, in _wrapped
elastalert_1  |         return func(*args, params=params, **kwargs)
elastalert_1  |       File "/usr/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/client/indices.py", line 266, in put_mapping
elastalert_1  |         '_mapping', doc_type), params=params, body=body)
elastalert_1  |       File "/usr/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/transport.py", line 318, in perform_request
elastalert_1  |         status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
elastalert_1  |       File "/usr/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/connection/http_requests.py", line 90, in perform_request
elastalert_1  |         self._raise_error(response.status_code, raw_data)
elastalert_1  |       File "/usr/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/connection/base.py", line 125, in _raise_error
elastalert_1  |         raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elastalert_1  |     elasticsearch.exceptions.RequestError: RequestError(400, u'illegal_argument_exception', u'Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true.')
elastalert_1  |
elastalert_1  | 17:08:38.485Z ERROR elastalert-server: ProcessController:  Index create exited with code 1

[johnsusek]

Thanks for this report, I haven't had a chance to try everything on ES7 yet. This is an upstream issue with the elastalert api I think. If bitsensor hasn't added this yet upstream, I'll see what I can do

[d1rtym0nk3y]

It looks like this is fixed in yelp/elastalert
https://github.com/Yelp/elastalert/blob/master/elastalert/create_index.py#L61

[johnsusek]

Okay, I looked into this a little deeper, it looks like it is only merged into their beta branch right now, per Yelp/elastalert#2194 (comment). Our docker images are built from master. Once they get it merged in I will rebuild our images.

[d1rtym0nk3y]

I just tested with yelp/elastalert v0.2.0b2, and after removing the sed lines @ https://github.com/ServerCentral/elastalert-server/blob/master/Dockerfile#L22-L23 it starts and creates the indexes without error.

I now get a different error about failing to fetch mappings from my indexes when trying to create a rule, but i will investigate further and post a separate issue.

[majkiee]

Any updates on this issue?

[johnsusek]

@majkiee Elastalert still doesn't support ES7 in their master branch - it's just a beta right now. Once their ES7 support is out of beta I will add support for it to praeco.

[Shinerrs]

+1 Johnsusek, I would like to know how to resolve this matter either this beta is merged or a workaround would be great.

[kklmm]

Likewise. Cannot use praeco at all for the past 2 months. Tried building it earlier with all sorts of shenanigans and haven't been successful. @johnsusek can you provide a guide how to set it up for es7? workaround is fine, as long as it works. thank you.

[scott-oit]

@johnsusek Elastalert now supports ES7. Can you rebuild? I'm excited to try out Praeco!

[osherdp]

Is there an estimation for ES7 supported release?

[johnsusek]

I just released a new beta with es7 support. Give it a try and let me know if you find any issues!

[osherdp]

Hey, can you push the code of elastalert server beta which is ES7 compatible?
Thanks

[lianfulei]

我已经修复了这个问题，elastalert升级到0.2.4 就可以解决了
/usr/bin/elastalert-create-index

Elastic Version: 7.6.2
Reading Elastic 6 index mappings:
Reading index mapping 'es_mappings/6/silence.json'
Reading index mapping 'es_mappings/6/elastalert_status.json'
Reading index mapping 'es_mappings/6/elastalert.json'
Reading index mapping 'es_mappings/6/past_elastalert.json'
Reading index mapping 'es_mappings/6/elastalert_error.json'
New index elastalert_status created
Done!