Add Elasticsearch 8 support + remove ES 6 support #744

ferozsalam · 2022-02-28T15:32:38Z

Description

A big diff that I would like as much attention as possible on!

The key underlying change is bumping elasticsearch-py to the latest 7.* version, 7.17.0.

This will - hopefully in a future diff - allow us to enable the ES 8 compatibility mode.

As a consequence of the version bump, ES 6 support is removed as discussed in #92 (comment). I took the opportunity to remove a lot of the ES-6 related code from the codebase.

I have tried to update as many unit tests as possible - as far as I'm aware I've only removed unit tests where they are no longer relevant because we don't support ES 5 or 6.

That said, this has involved some extensive refactoring. I have tested locally, both using the unit tests and an Elasticsearch instance, and things seem to be working, but I would appreciate at least one other person giving this a whirl before landing.

Checklist

I have reviewed the contributing guidelines.
I have included unit tests for my changes or additions.
I have successfully run make test-docker with my changes.
I have manually tested all relevant modes of the change in this PR.
I have updated the documentation.
I have updated the changelog.

Questions or Comments

This also drops support for Elasticsearch 6 (all versions)

And fix resulting unit tests

Didn't realise we were still intructing people to write rules in the ES4 style

nsano-rururu · 2022-02-28T16:22:53Z

Does changing elasticsearch-py to 7.17.0 mean to stop supporting opensearch? .. The connection of opensearch will be repelled from elasticsearch-py7.14.0.
https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/release-notes.html

7.14.0 (2021-08-02)
Added check that client is connected to an Elasticsearch cluster. If the client isn’t connected to a supported Elasticsearch cluster the UnsupportedProductError exception will be raised.

nsano-rururu · 2022-02-28T16:38:15Z

Do you not update?
https://github.com/jertel/elastalert2/blob/master/docs/source/running_elastalert.rst#as-a-python-package

Elasticsearch 6.x, 7.x.

nsano-rururu · 2022-02-28T16:49:53Z

Update setup.py as well as requirements.txt.

nsano-rururu · 2022-02-28T17:31:19Z

Users seem to want es8 support rather than refactoring. As a developer, I know you want to get rid of old code that you want to clean.

It's just my personal opinion, so you can ignore it. es8 support is the highest priority over erasing es5 and 6. Gradually erase old code after es8 support. For example, erase the es5 code six months later and erase the es6 code one year later.

nsano-rururu · 2022-02-28T17:41:31Z

I'm not sure that the es5 code is squeezed but the es_mapping 5 folders and files are still there. Forgot to erase it?

ferozsalam · 2022-02-28T20:27:57Z

Thanks for the review @nsano-rururu!

To address your points:

I haven't updated the documentation yet, I will do so soon
I forgot to remove the ES 5 mappings! Will do this as well 👍
About Elasticsearch 8 compatibility in general - elasticsearch-py 7.17.0 has an Elasticsearch 8 compatibility mode that can be activated by using an environment variable. My understanding is that if this environment variable is set, the library will handle all the doc_type conversion without having to change the code. So, I think it should be easy to enable ES 8 compatibility once this PR is landed.
I don't know what is best to do about maintaining Opensearch support. If we don't upgrade elasticsearch-py, it will become quite hard to add Elasticsearch 8 compatibility.

nsano-rururu · 2022-02-28T21:53:49Z

@ferozsalam
#92 (comment)

Abstracting the calls to the Elastic API away from the general ElastAlert 2 source code and into a new search.py class would give us the ability to put all the logic in that new class for choosing whether to use the opensearch-py library or the elasticsearch-py library. This might be easier said than done, but it would help isolate all of this complexity into one place.

If @ferozsalam can prove that ES8 compatibility can be had with the removal of doc_type and without switching to the new library then let's proceed with getting ES8 support into master without changing the Python library and without breaking ES7 or OpenSearch compatibility.

nsano-rururu · 2022-02-28T21:55:01Z

#92 (comment)

Mapping types and anything removed in 8.0 are still removed even when using "compatibility mode". However in client versions pre-7.16 will likely need to be more hands-on with the compatibility mode by settings HTTP headers yourself. In 7.16 I'm working towards getting the mode to be much easier to use.

nsano-rururu · 2022-02-28T21:58:56Z

OpenSearch client compatibility
https://opensearch.org/docs/clients/index/

Python Elasticsearch client 7.13.4

nsano-rururu · 2022-02-28T22:06:33Z

I don't know what is best to do about maintaining Opensearch support. If we don't upgrade elasticsearch-py, it will become quite hard to add Elasticsearch 8 compatibility.

Do not upgrade the version of elasticsearch-py.
Give up compatibility mode.
In case of elasticsearch 8, doc_type is deleted

ferozsalam · 2022-02-28T22:09:52Z

That seems to be the opposite of what was agreed here some months ago.

nsano-rururu · 2022-02-28T22:24:50Z

If you can handle all the complaints from users, you can only say that you can merge them. When you say so.

nsano-rururu · 2022-02-28T22:36:25Z

I don't know what is best to do about maintaining Opensearch support. If we don't upgrade elasticsearch-py, it will become quite hard to add Elasticsearch 8 compatibility.

Then why are you asking a question? I don't know what it means.

nsano-rururu · 2022-02-28T22:41:08Z

I think your idea is that opensearch support isn't considered from the beginning.

nsano-rururu · 2022-02-28T22:50:45Z

With the current pull request content, I think I have no choice but to announce that opensearch should continue to use the past version as well as es6.

nsano-rururu · 2022-02-28T22:55:38Z

If you merge as it is, the effect will be huge and you will get a lot of inquiries even if you write it in the changelog or put it on the FAQ, so it is realistic to add the correspondence table of elastalert2 version, elasticseach version, opensearch at the top of the readme. It may be. Do you hate that too?

nsano-rururu

e44dc75
Do you not update the document?

nsano-rururu · 2022-03-01T11:16:58Z

@ferozsalam

I installed it in the dockerfile of elastalert-server and tried it. There are a lot of warnings from elasticsearch-py, but is it working as expected?

elasticsearch 7.17
+
johnsusek/elastalert-server
https://github.com/johnsusek/elastalert-server

Dockerfile

FROM python:3.9-alpine3.15 as ea2
# ARG ELASTALERT_VERSION=2.3.0
# ENV ELASTALERT_VERSION=${ELASTALERT_VERSION}
# ARG ELASTALERT_URL=https://github.com/jertel/elastalert2/archive/refs/tags/$ELASTALERT_VERSION.zip
# ENV ELASTALERT_URL=${ELASTALERT_URL}
ENV ELASTALERT_HOME /opt/elastalert

RUN apk add --update --no-cache git

# WORKDIR /opt

# RUN apk add --update --no-cache wget && \
#     wget -O elastalert.zip "${ELASTALERT_URL}" && \
#     unzip elastalert.zip && \
#     rm elastalert.zip && \
#     mv e* "${ELASTALERT_HOME}"

RUN mkdir -p  /opt/elastalert_install_work && \
    cd /opt/elastalert_install_work && \
    git clone https://github.com/jertel/elastalert2 && \
    cd elastalert2 && \
    git fetch origin pull/744/head:es8 && \
    git checkout es8 && \
    cd ../ && \
    mv e* "${ELASTALERT_HOME}"

FROM node:16.13-alpine3.15 as install
ENV PATH /home/node/.local/bin:$PATH

RUN apk add --update --no-cache \
    ca-certificates \
    cargo \
    curl \
    gcc \
    libffi-dev \
    libmagic \
    make \
    musl-dev \
    openssl \
    openssl-dev \
    py3-pip \
    py3-wheel \
    python3 \
    python3-dev \
    tzdata

COPY --from=ea2 /opt/elastalert /opt/elastalert

WORKDIR /opt/elastalert-server
COPY . /opt/elastalert-server

RUN npm install --production --quiet

RUN pip3 install --no-cache-dir --upgrade pip==22.0.3

USER node

WORKDIR /opt/elastalert

RUN pip3 install --no-cache-dir cryptography --user
RUN pip3 install --no-cache-dir -r requirements.txt --user

FROM node:16.13-alpine3.15
LABEL maintainer="John Susek <[email protected]>"
ENV TZ Etc/UTC
ENV PATH /home/node/.local/bin:$PATH

RUN apk add --update --no-cache \
    ca-certificates \
    cargo \
    curl \
    gcc \
    libffi-dev \
    libmagic \
    make \
    musl-dev \
    openssl \
    openssl-dev \
    py3-pip \
    python3 \
    python3-dev \
    tzdata

COPY --from=install /opt/elastalert /opt/elastalert
COPY --from=install /home/node/.local/lib/python3.9/site-packages /home/node/.local/lib/python3.9/site-packages

WORKDIR /opt/elastalert-server

COPY --from=install /opt/elastalert-server ./

COPY config/elastalert.yaml /opt/elastalert/config.yaml
COPY config/config.json config/config.json
COPY rule_templates/ /opt/elastalert/rule_templates
COPY elastalert_modules/ /opt/elastalert/elastalert_modules

# Add default rules directory
# Set permission as unpriviledged user (1000:1000), compatible with Kubernetes
RUN mkdir -p /opt/elastalert/rules/ /opt/elastalert/server_data/tests/ \
    && chown -R node:node /opt

USER node

EXPOSE 3030

WORKDIR /opt/elastalert-server

ENTRYPOINT ["npm", "start"]

rule file

alert:
  - "slack"
alert_subject: "alert_subject: \"Test {} 123 aa☃\""
alert_subject_args:
  - "message"
alert_text: "Test {}  123 bb☃"
alert_text_args:
  - "message"
filter:
  - query:
      query_string:
        query: "message:Quit"
index: "mariadblog-*"
is_enabled: true
name: "a"
num_events: 2
realert:
  minutes: 5
slack_webhook_url: 'https://hooks.slack.com/services/xxxx'
slack_channel_override: "#abc"
slack_emoji_override: ":ghost:"
slack_msg_color: "danger"
slack_parse_override: "none"
slack_timeout: 10
slack_title_link: "http://127.0.0.1:8080/rules/a"
slack_username_override: "Praeco"
terms_size: 50
timeframe:
  minutes: 5
timestamp_field: "@timestamp"
timestamp_type: "iso"
type: "frequency"
use_strftime_index: false

docker log

10:58:24.377Z  INFO elastalert-server: Config:  No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json.
10:58:24.398Z  INFO elastalert-server: Config:  Proceeding to look for normal config file.
10:58:24.399Z  INFO elastalert-server: Config:  A config file was found in /opt/elastalert-server/config/config.json. Using that config.
10:58:24.500Z  INFO elastalert-server: Router:  Listening for GET request on /.
10:58:24.502Z  INFO elastalert-server: Router:  Listening for GET request on /status.
10:58:24.514Z  INFO elastalert-server: Router:  Listening for GET request on /status/errors.
10:58:24.516Z  INFO elastalert-server: Router:  Listening for GET request on /rules.
10:58:24.527Z  INFO elastalert-server: Router:  Listening for GET request on /rules/:id*.
10:58:24.535Z  INFO elastalert-server: Router:  Listening for POST request on /rules/:id*.
10:58:24.542Z  INFO elastalert-server: Router:  Listening for DELETE request on /rules/:id*.
10:58:24.549Z  INFO elastalert-server: Router:  Listening for GET request on /templates.
10:58:24.557Z  INFO elastalert-server: Router:  Listening for GET request on /templates/:id*.
10:58:24.562Z  INFO elastalert-server: Router:  Listening for POST request on /templates/:id*.
10:58:24.573Z  INFO elastalert-server: Router:  Listening for DELETE request on /templates/:id*.
10:58:24.579Z  INFO elastalert-server: Router:  Listening for PUT request on /folders/:type/:path*.
10:58:24.583Z  INFO elastalert-server: Router:  Listening for DELETE request on /folders/:type/:path*.
10:58:24.620Z  INFO elastalert-server: Router:  Listening for POST request on /test.
10:58:24.628Z  INFO elastalert-server: Router:  Listening for POST request on /silence/:path*.
10:58:24.633Z  INFO elastalert-server: Router:  Listening for GET request on /config.
10:58:24.635Z  INFO elastalert-server: Router:  Listening for POST request on /config.
10:58:24.653Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert.
10:58:24.654Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert_status.
10:58:24.655Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/silence.
10:58:24.657Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert_error.
10:58:24.658Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/past_elastalert.
10:58:24.658Z  INFO elastalert-server: Router:  Listening for GET request on /indices.
10:58:24.662Z  INFO elastalert-server: Router:  Listening for GET request on /mapping/:index.
10:58:24.663Z  INFO elastalert-server: Router:  Listening for POST request on /search/:index.
10:58:24.663Z  INFO elastalert-server: Router:  Listening for GET request on /config.
10:58:24.684Z  INFO elastalert-server: ProcessController:  Starting ElastAlert
10:58:24.685Z  INFO elastalert-server: ProcessController:  Creating index
10:58:34.746Z  INFO elastalert-server:
    ProcessController:  Reading Elastic 7 index mappings:
    Reading index mapping 'es_mappings/7/silence.json'
    Reading index mapping 'es_mappings/7/elastalert_status.json'
    Reading index mapping 'es_mappings/7/elastalert.json'
    Reading index mapping 'es_mappings/7/past_elastalert.json'
    Reading index mapping 'es_mappings/7/elastalert_error.json'
    New index praeco_elastalert_status created
    Done!
    
10:58:34.746Z ERROR elastalert-server:
    ProcessController:  /opt/elastalert/elastalert/create_index.py:39: DeprecationWarning: Using positional arguments for APIs is deprecated and will be disabled in 8.0.0. Instead use only keyword arguments for all APIs. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
      if es_index.exists(ea_index):
    /opt/elastalert/elastalert/create_index.py:57: DeprecationWarning: Using positional arguments for APIs is deprecated and will be disabled in 8.0.0. Instead use only keyword arguments for all APIs. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
      if es_index.exists(index_name):
    /opt/elastalert/elastalert/create_index.py:64: DeprecationWarning: Using positional arguments for APIs is deprecated and will be disabled in 8.0.0. Instead use only keyword arguments for all APIs. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
      es_index.create(index_name)
    /home/node/.local/lib/python3.9/site-packages/elasticsearch/connection/base.py:200: ElasticsearchWarning: Camel case format name dateOptionalTime is deprecated and will be removed in a future version. Use snake case name date_optional_time instead.
      warnings.warn(message, category=ElasticsearchWarning)
    /home/node/.local/lib/python3.9/site-packages/elasticsearch/connection/base.py:200: ElasticsearchWarning: [types removal] Using include_type_name in put mapping requests is deprecated. The parameter will be removed in the next major version.
      warnings.warn(message, category=ElasticsearchWarning)
    
10:58:34.746Z  INFO elastalert-server: ProcessController:  Index create exited with code 0
10:58:34.754Z  INFO elastalert-server: ProcessController:  Starting elastalert with arguments [none]
10:58:34.775Z  INFO elastalert-server: ProcessController:  Started Elastalert (PID: 377)
10:58:34.782Z  INFO elastalert-server: Server:  Server listening on port 3030
10:58:34.792Z  INFO elastalert-server: Server:  Websocket listening on port 3333
10:58:34.799Z  INFO elastalert-server: Server:  Server started
10:58:40.541Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
10:59:10.961Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
10:59:41.364Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
10:59:46.058Z ERROR elastalert-server:
    ProcessController:  WARNING:py.warnings:/opt/elastalert/elastalert/elastalert.py:1611: DeprecationWarning: The 'body' parameter is deprecated for the 'index' API and will be removed in a future version. Instead use the 'document' parameter. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
      res = self.writeback_es.index(index=index, body=body)
    
    
10:59:46.169Z ERROR elastalert-server:
    ProcessController:  WARNING:py.warnings:/home/node/.local/lib/python3.9/site-packages/elasticsearch/connection/base.py:200: ElasticsearchWarning: Camel case format name dateOptionalTime is deprecated and will be removed in a future version. Use snake case name date_optional_time instead.
      warnings.warn(message, category=ElasticsearchWarning)
    
    
11:00:12.052Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:00:38.115Z ERROR elastalert-server:
    ProcessController:  WARNING:py.warnings:/opt/elastalert/elastalert/elastalert.py:1631: DeprecationWarning: The 'body' parameter is deprecated for the 'search' API and will be removed in a future version. Instead use API parameters directly. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
      res = self.writeback_es.search(index=self.writeback_index, body=query, size=1000)
    
    
11:00:42.388Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:00:45.837Z ERROR elastalert-server:
    ProcessController:  WARNING:py.warnings:/opt/elastalert/elastalert/elastalert.py:371: DeprecationWarning: The 'body' parameter is deprecated for the 'search' API and will be removed in a future version. Instead use API parameters directly. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
      res = self.thread_data.current_es.search(
    
    
11:01:12.789Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:01:43.113Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:02:13.411Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:02:43.720Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:02:51.701Z ERROR elastalert-server:
    ProcessController:  WARNING:py.warnings:/opt/elastalert/elastalert/elastalert.py:1876: DeprecationWarning: The 'body' parameter is deprecated for the 'search' API and will be removed in a future version. Instead use API parameters directly. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
      res = self.writeback_es.search(index=index, size=1, body=query,
    
    
11:02:53.359Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/rules/:id'.
11:02:53.372Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:02:53.390Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/config'.
11:02:53.397Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/config'.
11:02:53.424Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/rules'.
11:03:14.289Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:03:30.577Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/rules/:id'.
11:03:44.919Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:04:15.349Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:04:46.035Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:04:57.605Z ERROR elastalert-server:
    ProcessController:  WARNING:py.warnings:/opt/elastalert/elastalert/elastalert.py:1611: DeprecationWarning: The 'body' parameter is deprecated for the 'index' API and will be removed in a future version. Instead use the 'document' parameter. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
      res = self.writeback_es.index(index=index, body=body)
    
    
11:05:16.366Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:05:38.115Z ERROR elastalert-server:
    ProcessController:  WARNING:py.warnings:/opt/elastalert/elastalert/elastalert.py:1631: DeprecationWarning: The 'body' parameter is deprecated for the 'search' API and will be removed in a future version. Instead use API parameters directly. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
      res = self.writeback_es.search(index=self.writeback_index, body=query, size=1000)
    
    
11:05:46.658Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:05:58.472Z ERROR elastalert-server:
    ProcessController:  WARNING:py.warnings:/opt/elastalert/elastalert/elastalert.py:371: DeprecationWarning: The 'body' parameter is deprecated for the 'search' API and will be removed in a future version. Instead use API parameters directly. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
      res = self.thread_data.current_es.search(
    
    
11:05:59.070Z ERROR elastalert-server:
    ProcessController:  WARNING:py.warnings:/opt/elastalert/elastalert/elastalert.py:1876: DeprecationWarning: The 'body' parameter is deprecated for the 'search' API and will be removed in a future version. Instead use API parameters directly. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
      res = self.writeback_es.search(index=index, size=1, body=query,
    
    
11:06:16.984Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:06:47.293Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:07:17.611Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:07:47.939Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:08:18.345Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:08:48.573Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:09:18.855Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:09:49.112Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:10:09.330Z ERROR elastalert-server:
    ProcessController:  WARNING:py.warnings:/opt/elastalert/elastalert/elastalert.py:1611: DeprecationWarning: The 'body' parameter is deprecated for the 'index' API and will be removed in a future version. Instead use the 'document' parameter. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
      res = self.writeback_es.index(index=index, body=body)
    
    
11:10:19.513Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.

nsano-rururu · 2022-03-01T12:09:20Z

I changed the elasticsearch-py version of setup.py to 7.17 and am running it.

elastalert-create-index

(py39) [CORP\sano@a-ngft53r34ong elastalert2]$ elastalert-create-index
Reading Elastic 7 index mappings:
Reading index mapping 'es_mappings/7/silence.json'
Reading index mapping 'es_mappings/7/elastalert_status.json'
Reading index mapping 'es_mappings/7/elastalert.json'
Reading index mapping 'es_mappings/7/past_elastalert.json'
Reading index mapping 'es_mappings/7/elastalert_error.json'
Deleting index elastalert_status_status.
Deleting index elastalert_status_error.
/home/sano/py39/lib/python3.9/site-packages/elasticsearch-7.17.0-py3.9.egg/elasticsearch/connection/base.py:200: ElasticsearchWarning: Camel case format name dateOptionalTime is deprecated and will be removed in a future version. Use snake case name date_optional_time instead.
  warnings.warn(message, category=ElasticsearchWarning)
/home/sano/py39/lib/python3.9/site-packages/elasticsearch-7.17.0-py3.9.egg/elasticsearch/connection/base.py:200: ElasticsearchWarning: [types removal] Using include_type_name in put mapping requests is deprecated. The parameter will be removed in the next major version.
  warnings.warn(message, category=ElasticsearchWarning)
New index elastalert_status created
Done!
(py39) [CORP\sano@a-ngft53r34ong elastalert2]$

elastalert-test-rule

Error in elastalert-test-rule

(py39) [CORP\sano@a-ngft53r34ong elastalert2]$ elastalert-test-rule rules/a.yaml
INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.
            To send them but remain verbose, use --verbose instead.
Error connecting to ElasticSearch:
AttributeError("type object 'ElastAlerter' has no attribute 'modify_rule_for_ES5'")
INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.
                To send them but remain verbose, use --verbose instead.
1 rules loaded
INFO:apscheduler.scheduler:Adding job tentatively -- it will be properly scheduled when the scheduler starts
INFO:elastalert:Queried rule a from 2022-03-01 20:48 JST to 2022-03-01 20:53 JST: 0 / 0 hits

Would have written the following documents to writeback index (default is elastalert_status):

elastalert_status - {'rule_name': 'a', 'endtime': datetime.datetime(2022, 3, 1, 11, 53, 55, 380438, tzinfo=tzutc()), 'starttime': datetime.datetime(2022, 3, 1, 11, 48, 52, 380438, tzinfo=tzutc()), 'matches': 0, 'hits': 0, '@timestamp': datetime.datetime(2022, 3, 1, 11, 53, 55, 575557, tzinfo=tzutc()), 'time_taken': 0.014517068862915039}

(py39) [CORP\sano@a-ngft53r34ong elastalert2]$

Running ElastAlert 2

(py39) [CORP\sano@a-ngft53r34ong elastalert2]$ python -m elastalert.elastalert --verbose --rule rules/a.yaml
1 rules loaded
INFO:elastalert:Starting up
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 9.99984 seconds
WARNING:elasticsearch:POST http://localhost:9200/elastalert_status_status/_search?_source_includes=endtime%2Crule_name&size=1 [status:404 request:0.016s]
ERROR:elastalert:Error querying for last run: NotFoundError(404, 'index_not_found_exception', 'no such index [elastalert_status_status]', elastalert_status_status, index_or_alias)
INFO:elastalert:Queried rule a from 2022-03-01 20:45 JST to 2022-03-01 21:00 JST: 3 / 3 hits
WARNING:elasticsearch:POST http://localhost:9200/elastalert_status_silence/_search?_source_includes=until%2Cexponent&size=1 [status:404 request:0.007s]
ERROR:elastalert:Error while querying for alert silence status: NotFoundError(404, 'index_not_found_exception', 'no such index [elastalert_status_silence]', elastalert_status_silence, index_or_alias)
WARNING:elasticsearch:POST http://localhost:9200/elastalert_status_silence/_search?_source_includes=until%2Cexponent&size=1 [status:404 request:0.008s]
ERROR:elastalert:Error while querying for alert silence status: NotFoundError(404, 'index_not_found_exception', 'no such index [elastalert_status_silence]', elastalert_status_silence, index_or_alias)
INFO:elastalert:Alert 'a' sent to Slack
WARNING:py.warnings:/home/sano/dkwork3/wk2/elastalert2/elastalert/elastalert.py:1611: DeprecationWarning: The 'body' parameter is deprecated for the 'index' API and will be removed in a future version. Instead use the 'document' parameter. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
  res = self.writeback_es.index(index=index, body=body)

INFO:elastalert:Ran a from 2022-03-01 20:45 JST to 2022-03-01 21:00 JST: 3 query hits (0 already seen), 1 matches, 1 alerts sent
INFO:elastalert:a range 900
WARNING:py.warnings:/home/sano/dkwork3/wk2/elastalert2/elastalert/elastalert.py:1631: DeprecationWarning: The 'body' parameter is deprecated for the 'search' API and will be removed in a future version. Instead use API parameters directly. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
  res = self.writeback_es.search(index=self.writeback_index, body=query, size=1000)

INFO:elastalert:Background configuration change check run at 2022-03-01 21:00 JST
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 9.997979 seconds
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2022-03-01 21:00 JST
WARNING:py.warnings:/home/sano/dkwork3/wk2/elastalert2/elastalert/elastalert.py:371: DeprecationWarning: The 'body' parameter is deprecated for the 'search' API and will be removed in a future version. Instead use API parameters directly. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
  res = self.thread_data.current_es.search(

INFO:elastalert:Background configuration change check run at 2022-03-01 21:06 JST
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2022-03-01 21:06 JST
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 9.999773 seconds
INFO:elastalert:Queried rule a from 2022-03-01 21:01 JST to 2022-03-01 21:07 JST: 12 / 12 hits
INFO:elastalert:Alert 'a' sent to Slack
WARNING:py.warnings:/home/sano/dkwork3/wk2/elastalert2/elastalert/elastalert.py:1611: DeprecationWarning: The 'body' parameter is deprecated for the 'index' API and will be removed in a future version. Instead use the 'document' parameter. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
  res = self.writeback_es.index(index=index, body=body)

WARNING:py.warnings:/home/sano/dkwork3/wk2/elastalert2/elastalert/elastalert.py:1876: DeprecationWarning: The 'body' parameter is deprecated for the 'search' API and will be removed in a future version. Instead use API parameters directly. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
  res = self.writeback_es.search(index=index, size=1, body=query,

INFO:elastalert:Ignoring match for silenced rule a
INFO:elastalert:Ran a from 2022-03-01 21:01 JST to 2022-03-01 21:07 JST: 12 query hits (9 already seen), 2 matches, 1 alerts sent
INFO:elastalert:a range 358
WARNING:py.warnings:/home/sano/dkwork3/wk2/elastalert2/elastalert/elastalert.py:1631: DeprecationWarning: The 'body' parameter is deprecated for the 'search' API and will be removed in a future version. Instead use API parameters directly. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
  res = self.writeback_es.search(index=self.writeback_index, body=query, size=1000)

INFO:elastalert:Background configuration change check run at 2022-03-01 21:07 JST
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2022-03-01 21:07 JST
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 9.999797 seconds
WARNING:py.warnings:/home/sano/dkwork3/wk2/elastalert2/elastalert/elastalert.py:371: DeprecationWarning: The 'body' parameter is deprecated for the 'search' API and will be removed in a future version. Instead use API parameters directly. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
  res = self.thread_data.current_es.search(

INFO:elastalert:Queried rule a from 2022-03-01 21:01 JST to 2022-03-01 21:07 JST: 12 / 12 hits
INFO:elastalert:Ran a from 2022-03-01 21:01 JST to 2022-03-01 21:07 JST: 12 query hits (12 already seen), 0 matches, 0 alerts sent
INFO:elastalert:a range 370
INFO:elastalert:Background configuration change check run at 2022-03-01 21:07 JST
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2022-03-01 21:07 JST
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 9.999757 seconds
INFO:elastalert:Background configuration change check run at 2022-03-01 21:07 JST
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2022-03-01 21:07 JST
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 9.999779 seconds
INFO:elastalert:Queried rule a from 2022-03-01 21:01 JST to 2022-03-01 21:07 JST: 12 / 12 hits
INFO:elastalert:Ran a from 2022-03-01 21:01 JST to 2022-03-01 21:07 JST: 12 query hits (12 already seen), 0 matches, 0 alerts sent
INFO:elastalert:a range 383
INFO:elastalert:Background configuration change check run at 2022-03-01 21:07 JST
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2022-03-01 21:07 JST
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 9.999748 seconds
INFO:elastalert:Queried rule a from 2022-03-01 21:01 JST to 2022-03-01 21:07 JST: 12 / 12 hits
INFO:elastalert:Ran a from 2022-03-01 21:01 JST to 2022-03-01 21:07 JST: 12 query hits (12 already seen), 0 matches, 0 alerts sent
INFO:elastalert:a range 396
INFO:elastalert:Background configuration change check run at 2022-03-01 21:07 JST
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2022-03-01 21:07 JST
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 9.999725 seconds
INFO:elastalert:Queried rule a from 2022-03-01 21:01 JST to 2022-03-01 21:07 JST: 12 / 12 hits
INFO:elastalert:Ran a from 2022-03-01 21:01 JST to 2022-03-01 21:07 JST: 12 query hits (12 already seen), 0 matches, 0 alerts sent
INFO:elastalert:a range 409
INFO:elastalert:Background configuration change check run at 2022-03-01 21:07 JST
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2022-03-01 21:07 JST
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 9.999805 seconds

nsano-rururu · 2022-03-01T12:09:49Z

make test-docker has no test failures.

nsano-rururu · 2022-03-03T10:34:57Z

Isn't it okay to remove the description of elasticsearch2 below?

docs/source/recipes/writing_filters.rst

For Elasticsearch 2.X, any of the filters can be embedded in ``not``, ``and``, and ``or``::

    filter:
    - or:
        - term:
            field: "value"
        - wildcard:
            field: "foo*bar"
        - and:
            - not:
                term:
                  field: "value"
            - not:
                term:
                  _type: "something"

nsano-rururu · 2022-03-03T10:44:49Z

It might be better to add that _type is no longer elasticsearch8

examples/rules/example_percentage_match.yaml

L12

   _type: http_request

examples/rules/example_new_term.yaml

L50

    _type: "login_event"

jertel · 2022-03-03T16:12:07Z

I've updated the upgrade notes with a warning about deleting ElastAlert indices before starting the 8.x upgrade process. More information can be found on this Elastic 8 upgrade bug here: elastic/elasticsearch#84199

nsano-rururu · 2022-03-03T19:22:53Z

kibana discover will not support kibana 8.0 with its current implementation. It seems that it needs to be fixed.

Error message

Cannot retrieve search results

[1:478] unknown field [query]

nsano-rururu · 2022-03-03T19:27:28Z

By the way, it seems that from kibana 8.0 it has changed to data view instead of the name index pattern. The setting screen is also data views. Therefore, it is necessary to add that the description of the index pattern ID of the document should also specify the data view id in kibana 8.0.

jertel · 2022-03-03T22:28:50Z

I just tested Discover URLs with ES 8 + Kibana 8 and it worked without any changes. I don't have any ElastAlert 2 error logs. Can you give me more information about the error? Was it printed in the ElastAlert 2 output? Or in Kibana error log?

Here's the rule configuration I used:

alert_text_type: alert_text_only
alert_text: |-
  Women's Clothing order placed.
  Details of the event:
          - currency: {0}
          - email: {1}
          - products.manufacturer: {2}
          - geoip.continent_name: {3}
          - taxful_total_price: {4}
          - url: {5}

alert_text_args:
  - currency
  - email
  - products[0]manufacturer
  - geoip.continent_name
  - taxful_total_price
  - kibana_discover_url

generate_kibana_discover_url: true
shorten_kibana_discover_url: true
kibana_url: "http://172.17.0.1:5601/"
kibana_discover_app_url: "app/discover#/"
kibana_discover_index_pattern_id: "*"
kibana_discover_version: "8.0"

Here's the Slack message:

jertel · 2022-03-03T22:29:42Z

By the way, it seems that from kibana 8.0 it has changed to data view instead of the name index pattern. The setting screen is also data views. Therefore, it is necessary to add that the description of the index pattern ID of the document should also specify the data view id in kibana 8.0.

This has been pushed. Thanks for reporting this.

nsano-rururu · 2022-03-03T23:13:55Z

you give me more information about the error? Was it printed in the ElastAlert 2 output? Or in Kibana error log?

There are no elastalert2 errors. I get an error when I display the url generated by elastalert2 in a browser.
By the way, shorten_kibana_discover_url is not enabled in my environment.

alert:
  - "slack"
alert_subject: "alert_subject: \"Test {} 123 aa☃\""
alert_subject_args:
  - "message"
alert_text: "Test {}  123 bb☃"
alert_text_args:
  - "message"
filter:
  - query:
      query_string:
        query: "message:Quit"
generate_kibana_discover_url: true
import: "BaseRule.config"
index: "mariadblog-*"
is_enabled: true
kibana_discover_app_url: "http://localhost:5601/app/discover#/"
kibana_discover_from_timedelta:
  minutes: 10
kibana_discover_index_pattern_id: "mariadblog"
kibana_discover_to_timedelta:
  minutes: 10
kibana_discover_version: "8.0"
match_enhancements: []
name: "a"
num_events: 2
realert:
  minutes: 5
slack_attach_kibana_discover_url: true
slack_channel_override: "#abc"
slack_emoji_override: ":ghost:"
slack_kibana_discover_color: "#ec4b98"
slack_kibana_discover_title: "Discover in Kibana"
slack_msg_color: "danger"
slack_parse_override: "none"
slack_timeout: 10
slack_title_link: "http://127.0.0.1:8080/rules/a"
slack_username_override: "Praeco"
slack_webhook_url: 'https://hooks.slack.com/services/xxxxxxxxxxx'
terms_size: 50
timeframe:
  minutes: 5
timestamp_field: "@timestamp"
timestamp_type: "iso"
type: "frequency"
use_strftime_index: false

jertel · 2022-03-04T00:27:37Z

Ok, I see it now. Since I didn't have an actual index pattern ID defined it defaulted to an existing view which allowed it to return results. When I specify a real pattern ID it will fail as you mentioned.

…breaks 8.x when left in

jertel · 2022-03-04T02:22:16Z

The Kibana 8.0 discover URL fix is now pushed. A new Docker image is also pushed for tag pr744.

Since we've dropped 6.x and earlier support I suggest in a future PR that the kibana_version logic and related schema property be removed. There's no point in spending time updating that list when there's no differences between the versions.

nsano-rururu · 2022-03-04T03:05:23Z

Since we've dropped 6.x and earlier support I suggest in a future PR that the kibana_version logic and related schema property be removed. There's no point in spending time updating that list when there's no differences between the versions.

I should have judged by looking at kibana_version about the shortened URL. If the api specifications do not change when es7 support is completely discontinued, it can be deleted, but it cannot be said that it will definitely change. I think that it is a parameter prepared to clarify which version is supported, so I think that it is better not to delete it.

nsano-rururu · 2022-03-04T03:07:40Z

look
Yelp/elastalert#2474 (comment)

jertel · 2022-03-04T03:10:26Z

Since we've dropped 6.x and earlier support I suggest in a future PR that the kibana_version logic and related schema property be removed. There's no point in spending time updating that list when there's no differences between the versions.

I should have judged by looking at kibana_version about the shortened URL. If the api specifications do not change when es7 support is completely discontinued, it can be deleted, but it cannot be said that it will definitely change. I think that it is a parameter prepared to clarify which version is supported, so I think that it is better not to delete it.

That's fine with me to keep it. I know you are the one that typically does the version update, so if you don't mind that maintenance work let's continue with it as-is.

nsano-rururu · 2022-03-04T03:38:43Z

Is the fix that kibana Discover supports kibana 8 without affecting the behavior of kibana 7.x?

nsano-rururu · 2022-03-04T13:08:49Z

kibana discover has confirmed that it does not affect the operation in kibana 7.17.

jertel · 2022-03-04T13:29:55Z

I tested the fix against both 7 and 8, both worked fine.

nsano-rururu · 2022-03-04T15:35:30Z

generate_kibana_link

generate_kibana_link doesn't seem to work for quite some time, so you can remove it. There was a bug report in yelp / elastalert issue at kibana 6. The developer has replied that it doesn't support kibana 6.
Yelp/elastalert#437 (comment)

This feature is only for Kibana 3, https://elastalert.readthedocs.io/en/latest/ruletypes.html?highlight=generate_kibana_link#generate-kibana-link, so it hasn't received much attention from me recently.

I see there is a slight bug, when using Elasticsearch 5 or 6 with this (would that even work with Kibana 3?), it will break if you are using query string filters.

The documentation says it's for kibana 3 only.
https://github.com/jertel/elastalert2/blob/master/docs/source/ruletypes.rst#generate_kibana_link

generate_kibana_link: This option is for Kibana 3 only. If true, ElastAlert 2 will generate a temporary Kibana dashboard and include a link to it in alerts. The dashboard consists of an events over time graph and a table with include fields selected in the table. If the rule uses query_key, the dashboard will also contain a filter for the query_key of the alert. The dashboard schema will be uploaded to the kibana-int index as a temporary dashboard. (Optional, boolean, default False)

use_kibana_dashboard and use_kibana4_dashboard

use_kibana_dashboard also replies that the developer doesn't support kibana 6.
Yelp/elastalert#1840 (comment)

There's no support or integration with the newer versions of Kibana. Triggered alerts are written into elasticsearch, with the (default) type/index elastalert_status, as are other events (each query made, errors, pending aggregated alerts, silence stashes, etc). So, you should be able to view them in Kibana, but you have to set up your dashboard manually.

The documentation states that use_kibana_dashboard is for kibana 3 and use_kibana4_dashboard is for kibana 4.

https://github.com/jertel/elastalert2/blob/master/docs/source/ruletypes.rst#use_kibana_dashboard

use_kibana_dashboard: The name of a Kibana 3 dashboard to link to. Instead of generating a dashboard from a template, ElastAlert 2 can use an existing dashboard. It will set the time range on the dashboard to around the match time, upload it as a temporary dashboard, add a filter to the query_key of the alert if applicable, and put the url to the dashboard in the alert. (Optional, string, no default)

https://github.com/jertel/elastalert2/blob/master/docs/source/ruletypes.rst#use_kibana4_dashboard

use_kibana4_dashboard: A link to a Kibana 4 dashboard. For example, "https://kibana.example.com/#/dashboard/My-Dashboard". This will set the time setting on the dashboard from the match time minus the timeframe, to 10 minutes after the match time. Note that this does not support filtering by query_key like Kibana 3. This value can use $VAR and ${VAR} references to expand environment variables.

jertel · 2022-03-04T17:16:12Z

Thanks @nsano-rururu. All references to Kibana dashboard integration are now removed. Docker image with tag pr744 is pushed with these changes.

nsano-rururu · 2022-03-04T17:29:23Z

https://github.com/jertel/elastalert2/blob/master/chart/elastalert2/values.yaml#L82

  # generate_kibana_link: true

ferozsalam added 8 commits February 27, 2022 16:55

Bump elasticsearch-py to 7.17.0

bcf7733

This also drops support for Elasticsearch 6 (all versions)

Remove support for Elasticsearch 6 mappings and query structures

cec18c4

Remove unnecessary else block

d724722

Remove ES 6-related code

00592d8

And fix resulting unit tests

Remove ES 5-related code

ee52133

Remvoe more leftover ES5 and ES6 code

1a24a68

Remove further legacy code

b8caa5c

Reinstate query mangling

e44dc75

Didn't realise we were still intructing people to write rules in the ES4 style

ferozsalam requested review from jertel and nsano-rururu February 28, 2022 15:32

nsano-rururu requested changes Feb 28, 2022

View reviewed changes

ferozsalam and others added 2 commits March 3, 2022 11:37

Update examples

4d96105

Clarify ES 8 upgrade procedure to avoid users breaking their ES clusters

7ce2c40

jertel changed the title ~~Bump elasticsearch-py to 7.17.0 + remove ES 6 support~~ Add Elasticsearch 8 support + remove ES 6 support Mar 3, 2022

Grammar fixes

de1260f

Add note about 'data view' terminology

b32eac4

Remove unnecessary query wrapper from filters. Not needed in 7.x and …

9c5f6aa

…breaks 8.x when left in

Removed obsolete kibana dashboard integration

39c6001

Removed obsolete kibana dashboard integration

e69829e

jertel approved these changes Mar 5, 2022

View reviewed changes

jertel merged commit 61777fa into jertel:master Mar 5, 2022

github-actions bot locked as resolved and limited conversation to collaborators Feb 19, 2024

Add Elasticsearch 8 support + remove ES 6 support #744

Add Elasticsearch 8 support + remove ES 6 support #744

Conversation

ferozsalam commented Feb 28, 2022 • edited by jertel Loading

Description

Checklist

Questions or Comments

nsano-rururu commented Feb 28, 2022 • edited Loading

nsano-rururu commented Feb 28, 2022

nsano-rururu commented Feb 28, 2022

nsano-rururu commented Feb 28, 2022 • edited Loading

nsano-rururu commented Feb 28, 2022

ferozsalam commented Feb 28, 2022 • edited Loading

nsano-rururu commented Feb 28, 2022

nsano-rururu commented Feb 28, 2022

nsano-rururu commented Feb 28, 2022

nsano-rururu commented Feb 28, 2022

ferozsalam commented Feb 28, 2022

nsano-rururu commented Feb 28, 2022

nsano-rururu commented Feb 28, 2022

nsano-rururu commented Feb 28, 2022

nsano-rururu commented Feb 28, 2022

nsano-rururu commented Feb 28, 2022

nsano-rururu left a comment • edited Loading

Choose a reason for hiding this comment

nsano-rururu commented Mar 1, 2022

nsano-rururu commented Mar 1, 2022 • edited Loading

nsano-rururu commented Mar 1, 2022

nsano-rururu commented Mar 3, 2022

nsano-rururu commented Mar 3, 2022

jertel commented Mar 3, 2022 • edited Loading

nsano-rururu commented Mar 3, 2022

nsano-rururu commented Mar 3, 2022

jertel commented Mar 3, 2022 • edited Loading

jertel commented Mar 3, 2022

nsano-rururu commented Mar 3, 2022

jertel commented Mar 4, 2022

jertel commented Mar 4, 2022

nsano-rururu commented Mar 4, 2022

nsano-rururu commented Mar 4, 2022

jertel commented Mar 4, 2022

nsano-rururu commented Mar 4, 2022

nsano-rururu commented Mar 4, 2022

jertel commented Mar 4, 2022

nsano-rururu commented Mar 4, 2022

jertel commented Mar 4, 2022

nsano-rururu commented Mar 4, 2022

ferozsalam commented Feb 28, 2022 •

edited by jertel

Loading

nsano-rururu commented Feb 28, 2022 •

edited

Loading

nsano-rururu commented Feb 28, 2022 •

edited

Loading

ferozsalam commented Feb 28, 2022 •

edited

Loading

nsano-rururu left a comment •

edited

Loading

nsano-rururu commented Mar 1, 2022 •

edited

Loading

jertel commented Mar 3, 2022 •

edited

Loading

jertel commented Mar 3, 2022 •

edited

Loading