Change doc_type and added support for Elasticsearch 6.6 and 7

[matsgoran]

• Pinned pip elasticsearch dependency >= 7.0.0 (update to ES 7 elastic/elasticsearch-py#911)
• Extracted Elasticsearch index mappings into external json-files
• Changed doc_type to '_doc' for Elasticsearch >= 6.2. Ref:
  • https://www.elastic.co/guide/en/elasticsearch/reference/6.0/removal-of-types.html
  • https://www.elastic.co/guide/en/elasticsearch/reference/6.2/indices-put-mapping.html
• Support subtle search API change starting with 6.6 where '_source_include' has been removed in favor of '_source_includes'. Ref:
  • https://www.elastic.co/guide/en/elasticsearch/reference/6.6/search-uri-request.html
• Added Travis tests for the following Elasticsearch versions:
  • 5.6.16
  • 6.0.1
  • 6.2.4
  • 6.3.2
  • 6.6.2
  • 7.0.0

The changes starting with elasticsearch-py 7 no longer accepts 'doc_type' as a search parameter. In order to offer backwards compatibility I've added a 'deprecated_search' which accepts this parameter. This function effectively works as an adapter between ElastAlert and elasticsearch-py since the Elasticsearch 7 API still accepts 'doc_type' (including some complaining about this being a deprecated feature). The function logs a warning informing that 'doc_type' will no longer work starting with Elasticsearch 8

Todos:

• Test and fix support for kibana 7

[caleb15]

Thanks! I'm guessing this would fix #2203 and #2204

[moetemad]

Is there a dockerized version of this available?

[christophetd]

We've made a Dockerized version: https://github.com/HacknowledgeCH/elastalert-docker

The image includes a temporary fix to force the elasticsearch dependency to be in a version <7.0.0, and we'll update it when this PR is merged.

[moetemad]

@christophetd Thank you. Unfortunately looks like the Dockerfile is not working. I get:
Connecting to github.com (192.30.253.113:443) wget: error getting response: Connection reset by peer
while trying to docker build.
Added a PR with the fix.

[jefflibby]

It looks like there are still issues with elastalert-create-index against Elasticsearch 7.0:

docker run --rm --entrypoint elastalert-create-index hacknowledge/elastalert --host 10.3.41.56 --port 9200 --index elastalert_status --no-ssl --username " " --password " " --url-prefix "" --old-index ""
Elastic Version:7
Mapping used for string:{'type': 'keyword'}
Traceback (most recent call last):
  File "/usr/local/bin/elastalert-create-index", line 11, in <module>
    load_entry_point('elastalert==0.1.39', 'console_scripts', 'elastalert-create-index')()
  File "/usr/local/lib/python2.7/site-packages/elastalert-0.1.39-py2.7.egg/elastalert/create_index.py", line 252, in main
    es.indices.put_mapping(index=index, doc_type='elastalert', body=es_mapping)
  File "/usr/local/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/client/utils.py", line 76, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/client/indices.py", line 266, in put_mapping
    '_mapping', doc_type), params=params, body=body)
  File "/usr/local/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/transport.py", line 318, in perform_request
    status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
  File "/usr/local/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/connection/http_requests.py", line 90, in perform_request
    self._raise_error(response.status_code, raw_data)
  File "/usr/local/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/connection/base.py", line 125, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.RequestError: RequestError(400, u'illegal_argument_exception', u'Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true.')

[matsgoran]

@jefflibby the current release of ElastAlert and elasticsearch-py 6.3.1 (https://github.com/elastic/elasticsearch-py) doesn't support ElasticSearch 7 due to breaking changes in the API. Ref. my comments in this PR, HacknowledgeCH/elastalert-docker@58fead0 , https://www.elastic.co/guide/en/elasticsearch/reference/7.0/indices-put-mapping.html

[eigengrau]

I gave this a quick spin. Due to the changed signature of create_index.py:main the elastalert-create-index script now fails:

Traceback (most recent call last):
  File "/usr/bin/elastalert-create-index", line 11, in <module>
    load_entry_point('elastalert', 'console_scripts', 'elastalert-create-index')()
TypeError: main() takes at least 2 arguments (0 given)

[issotina]

@jefflibby I had also the compatibility issue with elasticsearch 7.0. The problem is that before Elasticsearch 7.0.0, the mappings definition used to include a type name. Although specifying types in requests is now deprecated, a type can still be provided if the request parameter include_type_name is set.
refer to : https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-put-mapping.html

To solve this issue, you can clone and build my patch repository (https://github.com/geeckmc/elastalert.git) while waiting for my pull request to be accepted.

  $ pip uninstall elastalert
  $ git clone https://github.com/geeckmc/elastalert.git && cd elastalert
  $ pip install "setuptools>=11.3"
  $ pip install "elasticsearch>=7.0.0,<8.0.0"
  $ python setup.py install

hoping it helped you

[matsgoran]

I gave this a quick spin. Due to the changed signature of create_index.py:main the elastalert-create-index script now fails:

Traceback (most recent call last):
  File "/usr/bin/elastalert-create-index", line 11, in <module>
    load_entry_point('elastalert', 'console_scripts', 'elastalert-create-index')()
TypeError: main() takes at least 2 arguments (0 given)

@eigengrau Thanks for pointing this out. The main signature has been changed in order to fix this issue.

[jefflibby]

Sorry, I misunderstood. I thought the fix had already been applied to the docker image mentioned. I will try building from the PR branch.

[jefflibby]

elastalert-create-index verified locally with the latest from this branch, and I have alerts working against elasticsearch v7 now. Thanks for the help, I can now complete our POC.

[Qmando]

Hey all. I've merged this into the beta branch, and released a new version, 0.2.0b2. You'll have to use this for now if you want ES7 support.

There was a massive amount of merge conflicts, and I didn't do extensive testing on it, so please report any bugs you find!! Worth noting that there were some features that I had to remove/partially remove, like writeback_suffix. I'll see if it makes sense to merge these back in later.

I'll release this into a non beta release when I get time to merge in other changes and when it's had some time for people to report issues.