About the description of the documentation of Loading Filters Directly From Kibana 3

[nsano-rururu]

I ran it with kibana7.14, but since it hits only 0, it may not be working properly.
The documentation is incorrect.

filter download_dashboard AttributeError: 'dict' object has no attribute 'insert'
Yelp/elastalert#1481

Thanks. This solved the problem. I think we should update documentation, because we are missing - term in examples from

current

filter:
  download_dashboard: "My Dashboard Name"

after

filter:
  - term:
      download_dashboard: "My Dashboard Name"

rule file

alert:
  - "slack"
alert_subject: "Test {} 123 aa"
alert_subject_args:
  - "message"
  - "@timestamp"
alert_text: "Test {}  123 bb"
alert_text_args:
  - "message"
filter:
  - term:
      download_dashboard: "My Dashboard Name"
index: "mariadblog-*"
is_enabled: true
name: "a"
num_events: 2
realert:
  minutes: 5
slack_channel_override: "#abc"
slack_emoji_override: ":ghost:"
slack_msg_color: "danger"
slack_timeout: 10
slack_title_link: "http://127.0.0.1:8080/rules/a"
slack_username_override: "elastalert"
terms_size: 50
timeframe:
  minutes: 5
timestamp_field: "@timestamp"
timestamp_type: "iso"
type: "frequency"
use_strftime_index: false
slack_webhook_url: 'https://hooks.slack.com/services/xxxxxx'

log

(myvenv) [CORP\sano@a-ngft53r34ong elastalert2]$ python -m elastalert.elastalert --verbose --rule a.yaml
1 rules loaded
INFO:elastalert:Starting up
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 59.999928 seconds
INFO:elastalert:Queried rule a from 2021-09-03 01:58 JST to 2021-09-03 01:58 JST: 0 / 0 hits
INFO:elastalert:Ran a from 2021-09-03 01:58 JST to 2021-09-03 01:58 JST: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:a range 20
INFO:elastalert:Background configuration change check run at 2021-09-03 01:59 JST
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2021-09-03 01:59 JST
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 59.999824 seconds
INFO:elastalert:Queried rule a from 2021-09-03 01:58 JST to 2021-09-03 01:59 JST: 0 / 0 hits
INFO:elastalert:Ran a from 2021-09-03 01:58 JST to 2021-09-03 01:59 JST: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:a range 83

---

elastalert-rule-from-kibana did not work properly.

Looking at the source code, elasticsearch 6 seems to be compatible, but it may be a pattern that has not been confirmed to work with only the branch of processing. I think it's a bug from the time of the original yelp / elastalert.
https://github.com/jertel/elastalert2/blob/master/elastalert/rule_from_kibana.py

Elasticsearch 6.8.18, kibana 6.8.18

(myvenv) [CORP\sano@a-ngft53r34ong elastalert2]$ elastalert-rule-from-kibana
Elasticsearch host: localhost
Elasticsearch port: 9200
Dashboard name: My Dashboard Name
Method for querying Elasticsearch[GET]: 
Elastic Version:6.8.18
WARNING:elasticsearch:GET http://localhost:9200/kibana-int/dashboard/_search?_source_includes=dashboard [status:404 request:0.002s]
Traceback (most recent call last):
  File "/home/sano/myvenv/bin/elastalert-rule-from-kibana", line 11, in <module>
    load_entry_point('elastalert2==2.2.1', 'console_scripts', 'elastalert-rule-from-kibana')()
  File "/home/sano/myvenv/lib64/python3.7/site-packages/elastalert2-2.2.1-py3.7.egg/elastalert/rule_from_kibana.py", line 26, in main
    res = es.deprecated_search(index='kibana-int', doc_type='dashboard', body=query, _source_includes=['dashboard'])
  File "/home/sano/myvenv/lib64/python3.7/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/home/sano/myvenv/lib64/python3.7/site-packages/elastalert2-2.2.1-py3.7.egg/elastalert/__init__.py", line 255, in deprecated_search
    "GET", _make_path(index, doc_type, "_search"), params=params, body=body
  File "/home/sano/myvenv/lib64/python3.7/site-packages/elasticsearch/transport.py", line 318, in perform_request
    status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
  File "/home/sano/myvenv/lib64/python3.7/site-packages/elasticsearch/connection/http_requests.py", line 91, in perform_request
    self._raise_error(response.status_code, raw_data)
  File "/home/sano/myvenv/lib64/python3.7/site-packages/elasticsearch/connection/base.py", line 131, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.NotFoundError: NotFoundError(404, 'index_not_found_exception', 'no such index', kibana-int, index_or_alias)

Elasticsearch 7.14, kibana 7.14

(myvenv) [CORP\sano@a-ngft53r34ong elastalert2]$ elastalert-rule-from-kibana
Elasticsearch host: localhost
Elasticsearch port: 9200
Dashboard name: My Dashboard Name
Method for querying Elasticsearch[GET]: 
Elastic Version:7.14.0
WARNING:elasticsearch:GET http://localhost:9200/kibana-int/dashboard/_search?_source_includes=dashboard [status:404 request:0.003s]
Traceback (most recent call last):
  File "/home/sano/myvenv/bin/elastalert-rule-from-kibana", line 11, in <module>
    load_entry_point('elastalert2==2.2.1', 'console_scripts', 'elastalert-rule-from-kibana')()
  File "/home/sano/myvenv/lib64/python3.7/site-packages/elastalert2-2.2.1-py3.7.egg/elastalert/rule_from_kibana.py", line 26, in main
    res = es.deprecated_search(index='kibana-int', doc_type='dashboard', body=query, _source_includes=['dashboard'])
  File "/home/sano/myvenv/lib64/python3.7/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/home/sano/myvenv/lib64/python3.7/site-packages/elastalert2-2.2.1-py3.7.egg/elastalert/__init__.py", line 255, in deprecated_search
    "GET", _make_path(index, doc_type, "_search"), params=params, body=body
  File "/home/sano/myvenv/lib64/python3.7/site-packages/elasticsearch/transport.py", line 318, in perform_request
    status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
  File "/home/sano/myvenv/lib64/python3.7/site-packages/elasticsearch/connection/http_requests.py", line 91, in perform_request
    self._raise_error(response.status_code, raw_data)
  File "/home/sano/myvenv/lib64/python3.7/site-packages/elasticsearch/connection/base.py", line 131, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.NotFoundError: NotFoundError(404, 'index_not_found_exception', 'no such index [kibana-int]', kibana-int, index_or_alias)

[nsano-rururu]

I feel that there is no index called kibana-int in elasticsearch7.

[nsano-rururu]

Do you want to discontinue this function? I think it is necessary to consider whether to modify it so that it works with elasticsearch 6 and elasticsearch 7.

[jertel]

I've never used that feature. Perhaps try modifying code to use the .kibana* indices and if still doesn't work then abandon the feature. I don't want to waste our time supporting a feature no one uses.

[nsano-rururu]

@jertel

The result has changed a little, but it still doesn't work. We are planning to issue a pull request to remove the function.
Also, the following settings are also related, so they may be abolished at the same time.

delete setting (investigating)

• generate_kibana_link
• kibana_url
• use_kibana_dashboard
• use_kibana4_dashboard
• kibana4_start_timedelta
• kibana4_end_timedelta

log

(myvenv) [CORP\sano@a-ngft53r34ong elastalert2]$ elastalert-rule-from-kibana
Elasticsearch host: localhost
Elasticsearch port: 9200
Dashboard name: MyDashboard
Method for querying Elasticsearch[GET]: 
Elastic Version:7.14.0
No dashboard MyDashboard found
(myvenv) [CORP\sano@a-ngft53r34ong elastalert2]$ python -m elastalert.elastalert --verbose --rule a.yaml
1 rules loaded
INFO:elastalert:Starting up
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 59.999923 seconds
INFO:elastalert:Queried rule a from 2021-09-03 08:24 JST to 2021-09-03 08:32 JST: 0 / 0 hits
INFO:elastalert:Ran a from 2021-09-03 08:24 JST to 2021-09-03 08:32 JST: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:a range 455
INFO:elastalert:Background configuration change check run at 2021-09-03 08:33 JST
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2021-09-03 08:33 JST
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 59.999767 seconds
INFO:elastalert:Queried rule a from 2021-09-03 08:24 JST to 2021-09-03 08:33 JST: 0 / 0 hits
INFO:elastalert:Ran a from 2021-09-03 08:24 JST to 2021-09-03 08:33 JST: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:a range 515
INFO:elastalert:Background configuration change check run at 2021-09-03 08:34 JST
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2021-09-03 08:34 JST
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 59.999794 seconds
INFO:elastalert:Queried rule a from 2021-09-03 08:24 JST to 2021-09-03 08:34 JST: 0 / 0 hits
INFO:elastalert:Ran a from 2021-09-03 08:24 JST to 2021-09-03 08:34 JST: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:a range 580

[jertel]

I spent some time this morning looking at what it would take to update the code to support kibana dashboards again. It's not a massive amount of work, but it is going to take some effort. I stopped a the point where I discovered the dashboard filters are now stored in a different part of the JSON. The current ElastAlert2 release expects to find filters in JSON.services.filter.list, yet it appears that filters are now stored in JSON.kibanaSavedObjectMeta.searchSourceJSON which has a value of an embedded JSON. Ex:

          "dashboard": {
            "hits": 0,
            "description": "",
            "title": "My Dashboard",
            "timeRestore": false,
            "version": 1,
            "panelsJSON": "[{... lots of panel JSON ...}]",
            "optionsJSON": "{\"hidePanelTitles\":false,\"useMargins\":true}",
            "kibanaSavedObjectMeta": {
              "searchSourceJSON": "{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[{\"meta\":{\"alias\":null,\"negate\":false,\"disabled\":false,\"type\":\"phrase\",\"key\":\"some.field\",\"params\":{\"query\":\"some_search_value\"},\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index\"},\"query\":{\"match_phrase\":{\"agent.ephemeral_id\":\"some_search_value\"}},\"$state\":{\"store\":\"appState\"}}]}"
            }

I moved the download_dashboard key into the top level of the rule. Moving it out of the filter eliminates the problem with how the filters are now expected to be an array of maps.

Here's the diff of changes I had to make to get this far:

Based on the facts that this hasn't worked in quite some time, and that the cost to get it working again is relatively high I agree that the kibana dashboard integration should be removed.