Add configuration to skip specific ingestion errors

[stacimc]

Problem

WordPress/openverse-catalog#650 added dagrun conf options to make a provider script skip ingestion errors. This is currently all-or-nothing; when skip_ingestion_errors is true, all errors (except for AirflowExceptions) are caught, allowing ingestion to continue, and then re-raised at the end of ingestion.

In reality, it's likely that we'll want to skip specific errors, but we may still want ingestion to be halted if an unexpected error is thrown.

Description

We can update the conf option to be similar to the silenced_slack_alerts configuration (#654), which allows matching an error predicate. Instead of a boolean value, skip_ingestion_errors could be a list of strings to match on.

Like silenced_slack_alerts, we should make sure this allows us to match specific error message text and error types (ie adding 'KeyError' to the list should allow us to skip all KeyErrors). It may be nice to make sure there's still a way to easily skip all errors if that's truly what we want.

Implementation

• 🙋 I would be interested in implementing this feature.

[stacimc]

Resolved in WordPress/openverse-catalog#654

[stacimc]

I mistakenly closed this thinking it was referring to skipping Slack notifications for specific errors. This issue is for skipping the errors themselves during ingestion.