Option skip_invalid not working for schema validation

[jertel]

Discussed in #1333

^{Originally posted by missaeldenadai December 4, 2023}
Hi,

If a rule file is invalid due to schema validation, it is not being skipped and hence our ElastAlert2 is not deploying.

My rule containing the schema error (the error is on metric_agg_type: percentile):

  payment_send_spike_latency: |-
    ---
    name: some_name
    type: spike_aggregation
    index: index_name
    filter:
      - query:
          some_query
    timeframe:
      hours: 1
    metric_agg_key: transaction.duration.us
    metric_agg_type: percentile
    percentile_range: 95
    spike_type: "up"
    spike_height: 1.5
    min_doc_count: 10
    alert: opsgenie
    realert:
      minutes: 60

The stack trace:

Index elastalert already exists. Skipping index creation.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/elastalert/loaders.py", line 299, in load_options
    self.rule_schema.validate(rule)
  File "/usr/local/lib/python3.11/site-packages/jsonschema/validators.py", line 430, in validate
    raise error
jsonschema.exceptions.ValidationError: <file content>, 'rule_file': '/opt/elastalert/rules/payment_send_spike_latency.yaml'} is not valid under any of the given schemas

Failed validating 'oneOf' in schema:
<the schema>
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/elastalert", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/elastalert/elastalert.py", line 1868, in main
    client = ElastAlerter(args)
             ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/elastalert/elastalert.py", line 127, in __init__
    self.rules = self.rules_loader.load(self.conf, self.args)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/elastalert/loaders.py", line 176, in load
    raise EAException('Error loading file %s: %s' % (rule_file, e))
elastalert.util.EAException: Error loading file /opt/elastalert/rules/payment_send_spike_latency.yaml: Invalid Rule file: /opt/elastalert/rules/payment_send_spike_latency.yaml
<file_content> is not valid under any of the given schemas

Failed validating 'oneOf' in schema:
<the schema>

Does skip_invalid work for schema validation?