[RFC] Rules loader

[pdscopes]

I've been wanting to have greater control over how rules are loaded. Specifically, being able to load them from places other than a local directory; for example, a mongo database.

This PR creates a means to do just that. It uses a similar methodology as Enhancements and Alerters in that there is a base class RulesLoader which is specified in the config.yaml under the rules_loader property. You can use the format module.file.ClassName to use your own loader. The default is called FileRulesLoader and this behaves exactly like the current way. If you don't make a change you will see no difference in the way that EA works.

RulesLoaders are subclasses of RulesLoader, found in elastalert/loaders.py. They are used to
gather rules for a particular source. Your RulesLoader needs to implement three member functions, and will look something like this:

class AwesomeNewRulesLoader(RulesLoader):
    def get_names(self, conf, use_rule=None):
        ...
    def get_hashes(self, conf, use_rule=None):
        ...
    def get_yaml(self, rule):
        ...

You can import loaders by specifying the type as module.file.RulesLoaderName, where module is the name of a python module, and file is the name of the python file containing a RulesLoader subclass named RulesLoaderName.

As an example loader, let's retrieve rules from a database rather than from the local file system. First, create a
modules folder for the loader in the ElastAlert directory.

    $ mkdir elastalert_modules
    $ cd elastalert_modules
    $ touch __init__.py

Now, in a file named mongo_loader.py, add

from pymongo import MongoClient
from elastalert.loaders import RulesLoader

class MongoRulesLoader(RulesLoader):
    def __init__(self, conf):
        super(MongoRulesLoader, self).__init__(conf)
        self.client = MongoClient(conf['mongo_url'])
        self.db     = client[conf['mongo_db']]

    def get_names(self, conf, use_rule=None):
        if use_rule:
            return [use_rule]

        rules = []
        self.cache = {}
        for rule in self.db.rules.find():
            self.cache[rule.name] = rule.yaml
            rules.append(rule.yaml)

        return rules


    def get_hashes(self, conf, use_rule=None):
        if use_rule:
            return [use_rule]

        hashes = {}
        self.cache = {}
        for rule in self.db.rules.find():
            self.cache[rule.name] = rule.yaml
            hashes[rule.name] = rule.hash

        return hashes

    def get_yaml(self, rule):
        if rule in self.cache:
            return self.cache[rule]

        self.cache[rule] = self.db.rules.find_one({'name': rule}).yaml
        return self.cache[rule]

Finally, you need to specify in your ElastAlert configuration file that MongoRulesLoader should be used instead of the
default FileRulesLoader, so in your elastalert.conf file:

    rules_loader: "elastalert_modules.mongo_loader.MongoRulesLoader"

[phil-hachey]

Awesome! I've been looking for something like this, but loading the rules from Elasticsearch itself. I'll see if I can write a RuleLoader for my purposes

[pdscopes]

@phil-hachey, actually that's a really good idea and should be included in the pull request.

[pdscopes]

I've just updated this to resolve the conflicts that were introduced. What is the likelihood of this being merged?

[pdscopes]

Closing this PR as a copy of this has been merged into the beta branch. Thanks @Qmando.