Option skip_invalid not working for schema validation

[missaeldenadai]

Hi,

If a rule file is invalid due to schema validation, it is not being skipped and hence our ElastAlert2 is not deploying.

My rule containing the schema error (the error is on metric_agg_type: percentile):

  payment_send_spike_latency: |-
    ---
    name: some_name
    type: spike_aggregation
    index: index_name
    filter:
      - query:
          some_query
    timeframe:
      hours: 1
    metric_agg_key: transaction.duration.us
    metric_agg_type: percentile
    percentile_range: 95
    spike_type: "up"
    spike_height: 1.5
    min_doc_count: 10
    alert: opsgenie
    realert:
      minutes: 60

The stack trace:

Index elastalert already exists. Skipping index creation.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/elastalert/loaders.py", line 299, in load_options
    self.rule_schema.validate(rule)
  File "/usr/local/lib/python3.11/site-packages/jsonschema/validators.py", line 430, in validate
    raise error
jsonschema.exceptions.ValidationError: <file content>, 'rule_file': '/opt/elastalert/rules/payment_send_spike_latency.yaml'} is not valid under any of the given schemas

Failed validating 'oneOf' in schema:
<the schema>
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/elastalert", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/elastalert/elastalert.py", line 1868, in main
    client = ElastAlerter(args)
             ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/elastalert/elastalert.py", line 127, in __init__
    self.rules = self.rules_loader.load(self.conf, self.args)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/elastalert/loaders.py", line 176, in load
    raise EAException('Error loading file %s: %s' % (rule_file, e))
elastalert.util.EAException: Error loading file /opt/elastalert/rules/payment_send_spike_latency.yaml: Invalid Rule file: /opt/elastalert/rules/payment_send_spike_latency.yaml
<file_content> is not valid under any of the given schemas

Failed validating 'oneOf' in schema:
<the schema>

Does skip_invalid work for schema validation?

[jertel]

From what I'm seeing that skip_invalid setting isn't used. I searched all history that I could find and cannot find a reference to that setting. It likely needs to be removed from the documentation.

[jertel]

Thanks for that thorough investigation @nsano-rururu. We'll need to get that functionality re-added to the project. Created issue #1336.

[nsano-rururu]

A refactoring was done in the beta branch that moved the process of loading rules from config.py to a separate file, loders.py.
It appears that the skip_invalid implementation added to config.py was removed because changes added in master were not included in loders.py when merging to master.

[nsano-rururu]

The pull request below moves some processing from config.py to loaders.py and has been merged into the beta branch.

Rules loader #2030
Yelp/elastalert#2030

postscript

Completely missing a review. I overlooked that you forgot to move the skip_invalid processing.

[nsano-rururu]

https://github.com/jertel/elastalert2/blob/master/elastalert/loaders.py#L233

current

        rule = self.load_yaml(filename)

fix

        try:
            rule = self.load_yaml(filename)
        except Exception as e:
            if (conf.get('skip_invalid')):
                logging.error(e)
                return False
            else:
                raise e