Skip to content

Commit

Permalink
Merge master
Browse files Browse the repository at this point in the history
  • Loading branch information
Florian GAULTIER committed Oct 21, 2019
2 parents 7589d33 + 325f1df commit cfcec50
Show file tree
Hide file tree
Showing 40 changed files with 3,737 additions and 1,461 deletions.
18 changes: 9 additions & 9 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
language: python
python:
- '2.7'
- '3.6'
env:
- TOXENV=docs
- TOXENV=py27
- TOXENV=py36
install:
- pip install tox
- >
Expand All @@ -19,16 +19,16 @@ script:
make test-elasticsearch
else
make test
fi
fi
jobs:
include:
- stage: 'Elasticsearch test'
env: TOXENV=py27 ES_VERSION=7.0.0-linux-x86_64
- env: TOXENV=py27 ES_VERSION=6.6.2
- env: TOXENV=py27 ES_VERSION=6.3.2
- env: TOXENV=py27 ES_VERSION=6.2.4
- env: TOXENV=py27 ES_VERSION=6.0.1
- env: TOXENV=py27 ES_VERSION=5.6.16
env: TOXENV=py36 ES_VERSION=7.0.0-linux-x86_64
- env: TOXENV=py36 ES_VERSION=6.6.2
- env: TOXENV=py36 ES_VERSION=6.3.2
- env: TOXENV=py36 ES_VERSION=6.2.4
- env: TOXENV=py36 ES_VERSION=6.0.1
- env: TOXENV=py36 ES_VERSION=5.6.16

deploy:
provider: pypi
Expand Down
4 changes: 2 additions & 2 deletions Dockerfile-test
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
FROM ubuntu:latest

RUN apt-get update && apt-get upgrade -y
RUN apt-get -y install build-essential python-setuptools python2.7 python2.7-dev libssl-dev git tox python-pip
RUN apt-get -y install build-essential python3.6 python3.6-dev python3-pip libssl-dev git

WORKDIR /home/elastalert

ADD requirements*.txt ./
RUN pip install -r requirements-dev.txt
RUN pip3 install -r requirements-dev.txt
22 changes: 17 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,5 @@
Note: If you're using Elasticsearch 7, you'll need to install a beta release of Elastalert: `pip install "elastalert>=0.2.0b"`
Recent changes: As of Elastalert 0.2.0, you must use Python 3.6. Python 2 will not longer be supported.


[![Stories in Ready](https://badge.waffle.io/Yelp/elastalert.png?label=ready&title=Ready)](https://waffle.io/Yelp/elastalert)
[![Stories in In Progress](https://badge.waffle.io/Yelp/elastalert.png?label=in%20progress&title=In%20Progress)](https://waffle.io/Yelp/elastalert)
[![Build Status](https://travis-ci.org/Yelp/elastalert.svg)](https://travis-ci.org/Yelp/elastalert)
[![Join the chat at https://gitter.im/Yelp/elastalert](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/Yelp/elastalert?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

Expand Down Expand Up @@ -74,8 +71,23 @@ In addition to this basic usage, there are many other features that make alerts
To get started, check out `Running ElastAlert For The First Time` in the [documentation](http://elastalert.readthedocs.org).

## Running ElastAlert
You can either install the latest released version of ElastAlert using pip:

```pip install elastalert```

or you can clone the ElastAlert repository for the most recent changes:

```git clone https://github.com/Yelp/elastalert.git```

Install the module:

```pip install "setuptools>=11.3"```

```python setup.py install```

The following invocation can be used to run ElastAlert after installing

``$ python elastalert/elastalert.py [--debug] [--verbose] [--start <timestamp>] [--end <timestamp>] [--rule <filename.yaml>] [--config <filename.yaml>]``
``$ elastalert [--debug] [--verbose] [--start <timestamp>] [--end <timestamp>] [--rule <filename.yaml>] [--config <filename.yaml>]``

``--debug`` will print additional information to the screen as well as suppresses alerts and instead prints the alert body. Not compatible with `--verbose`.

Expand Down
17 changes: 17 additions & 0 deletions changelog.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,22 @@
# Change Log

# v0.2.1

### Fixed
- Fixed an AttributeError introduced in 0.2.0

# v0.2.0

- Switched to Python 3

### Added
- Add rule loader class for customized rule loading
- Added thread based rules and limit_execution
- Run_every can now be customized per rule

### Fixed
- Various small fixes

# v0.1.39

### Added
Expand Down
1 change: 1 addition & 0 deletions config.yaml.example
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ es_port: 9200
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status
writeback_alias: elastalert_alerts

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
Expand Down
6 changes: 4 additions & 2 deletions docs/source/elastalert.rst
Original file line number Diff line number Diff line change
Expand Up @@ -133,9 +133,12 @@ The environment variable ``ES_USE_SSL`` will override this field.

``es_conn_timeout``: Optional; sets timeout for connecting to and reading from ``es_host``; defaults to ``20``.

``rules_loader``: Optional; sets the loader class to be used by ElastAlert to retrieve rules and hashes.
Defaults to ``FileRulesLoader`` if not set.

``rules_folder``: The name of the folder which contains rule configuration files. ElastAlert will load all
files in this folder, and all subdirectories, that end in .yaml. If the contents of this folder change, ElastAlert will load, reload
or remove rules based on their respective config files.
or remove rules based on their respective config files. (only required when using ``FileRulesLoader``).

``scan_subdirectories``: Optional; Sets whether or not ElastAlert should recursively descend the rules directory - ``true`` or ``false``. The default is ``true``

Expand Down Expand Up @@ -200,7 +203,6 @@ The default value is ``.raw`` for Elasticsearch 2 and ``.keyword`` for Elasticse

``skip_invalid``: If ``True``, skip invalid files instead of exiting.

=======
Logging
-------

Expand Down
1 change: 1 addition & 0 deletions docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ Contents:
recipes/adding_alerts
recipes/writing_filters
recipes/adding_enhancements
recipes/adding_loaders
recipes/signing_requests

Indices and Tables
Expand Down
85 changes: 85 additions & 0 deletions docs/source/recipes/adding_loaders.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
.. _loaders:

Rules Loaders
========================

RulesLoaders are subclasses of ``RulesLoader``, found in ``elastalert/loaders.py``. They are used to
gather rules for a particular source. Your RulesLoader needs to implement three member functions, and
will look something like this:

.. code-block:: python
class AwesomeNewRulesLoader(RulesLoader):
def get_names(self, conf, use_rule=None):
...
def get_hashes(self, conf, use_rule=None):
...
def get_yaml(self, rule):
...
You can import loaders by specifying the type as ``module.file.RulesLoaderName``, where module is the name of a
python module, and file is the name of the python file containing a ``RulesLoader`` subclass named ``RulesLoaderName``.

Example
-------

As an example loader, let's retrieve rules from a database rather than from the local file system. First, create a
modules folder for the loader in the ElastAlert directory.

.. code-block:: console
$ mkdir elastalert_modules
$ cd elastalert_modules
$ touch __init__.py
Now, in a file named ``mongo_loader.py``, add

.. code-block:: python
from pymongo import MongoClient
from elastalert.loaders import RulesLoader
import yaml
class MongoRulesLoader(RulesLoader):
def __init__(self, conf):
super(MongoRulesLoader, self).__init__(conf)
self.client = MongoClient(conf['mongo_url'])
self.db = self.client[conf['mongo_db']]
self.cache = {}
def get_names(self, conf, use_rule=None):
if use_rule:
return [use_rule]
rules = []
self.cache = {}
for rule in self.db.rules.find():
self.cache[rule['name']] = yaml.load(rule['yaml'])
rules.append(rule['name'])
return rules
def get_hashes(self, conf, use_rule=None):
if use_rule:
return [use_rule]
hashes = {}
self.cache = {}
for rule in self.db.rules.find():
self.cache[rule['name']] = rule['yaml']
hashes[rule['name']] = rule['hash']
return hashes
def get_yaml(self, rule):
if rule in self.cache:
return self.cache[rule]
self.cache[rule] = yaml.load(self.db.rules.find_one({'name': rule})['yaml'])
return self.cache[rule]
Finally, you need to specify in your ElastAlert configuration file that MongoRulesLoader should be used instead of the
default FileRulesLoader, so in your ``elastalert.conf`` file::

rules_loader: "elastalert_modules.mongo_loader.MongoRulesLoader"

2 changes: 1 addition & 1 deletion docs/source/recipes/writing_filters.rst
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ a field that appears to have the value "foo bar", unless it is not analyzed. Con
matching on analyzed fields, use query_string. See https://www.elastic.co/guide/en/elasticsearch/guide/current/term-vs-full-text.html

`terms <https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-terms-query.html>`_
*****
*****************************************************************************************************



Expand Down
Loading

0 comments on commit cfcec50

Please sign in to comment.