Raise an error when Airbyte sync job fails to mark the task run as Failed

[anna-geller]

Summary

Airbyte task currently checks for the job status here:

prefect/src/prefect/tasks/airbyte/airbyte.py

Line 271 in 01705d3

while job_status not in [self.JOB_STATUS_FAILED, self.JOB_STATUS_SUCCEEDED]:

However, when the job status is failed, it only logs the error rather than raising an exception:

prefect/src/prefect/tasks/airbyte/airbyte.py

Line 280 in 01705d3

self.logger.error(f"Job {job_id} failed.")

Changes

Raising an exception to mark the task run as Failed.

Importance

Fixes a bug flagged by a community user in this Slack thread.

Checklist

This PR:

• [] adds new tests (if appropriate)
• adds a change file in the changes/ directory (if appropriate)
• [] updates docstrings for any new functions or function arguments, including docs/outline.toml for API reference docs (if appropriate)

[zanieb]

Let's add a changelog entry for this, it's a significant change in behavior.

[anna-geller]

Let's add a changelog entry for this, it's a significant change in behavior.

I added changes/pr5362.yml already - or do you mean something else?

[zanieb]

Sorry I guess I missed that :)

[b4stien]

Don't we want the same behaviour for cancelled jobs? Otherwise we're stuck in an infinite loop.

[anna-geller]

Don't we want the same behaviour for cancelled jobs? Otherwise we're stuck in an infinite loop.

I see. If you would be willing to contribute a PR for it, we would take time to review and QA this behavior together. Is this common that someone cancels Airbyte sync jobs? I imagined it as a data replication mechanism you wouldn't want to cancel to avoid some inconsistent database state, but I'm not that familiar with Airbyte

[b4stien]

Is this common that someone cancels Airbyte sync jobs?

There is a button to cancel one in Airbyte's interface, so it can happen. Not that frequently I'd agree, but disproportionately during test/setup to have a fast feedback loop.

I imagined it as a data replication mechanism you wouldn't want to cancel to avoid some inconsistent database state

Depending on the mode you chose the next synchronization will overwrite previous data, so there won't be any inconsistency.

I'd see if I have time to draft a PR, thanks.