Errors handling

[karolisg]

Errors handling

In the current implementation dozer process dies if the orchestrator receives any error. This can lead to bad user experience when app dies on minor error. To improve experience we need to handle some error cases and have some recoverable errors. So, we can group errors into 3 groups handled, recoverable and unrecoverable.

Unrecoverable

At the moment all errors, which are returned to orchestrator are unrecoverable and they are handled in panic hook . We can reduce such errors by handling all (or most) API errors without crashing and having some recoverable errors in connectors.

Recoverable

Some of errors in connectors can be recoverable, which means that we can retry connection or data read step. Some examples of recoverable errors can be:

• Connector cannot connect to database (network error)
• All publication slots are used (postgres usecase)

By handling and retrying such errors we should consider:

• How many retries we can do
• What messages are printed in logs
• How frequently we can retry

Handled errors (API crate)

In current implementation, dozer process crashes on any incorrect REST/gRPC request to dozer APIs. To avoid this, we need to refactor request handling and in error case try to return response with particular HTTP error code

[karolisg]

@chubei Is there any scenario when we can handle cache errors without app crash?
@mediuminvader Same question for sql stuff, do we have any error, which can be handled instead of crashing app?

[mediuminvader]

For SQL pipeline I would consider if the error is happening on building stage, in that case we can stop di application without any prolems, but we should try to handle as much errors as possible on processing stage.
My opinion here is: after we start serving the APIs, we should raise the problem to the user, but we should try to keep the APIs running as much as possible.

[karolisg]

@chubei You mentioned PrimaryKeyNotFound and PrimaryKeyExists errors. We have at least two possible handle scenarios:

• Expect insert and update to work like upsert.
• Ignore operation if primary key error is returned.

Not sure which way is better, because in most cases those two errors happen when there is issue in primary key definition, data ingestion or schema mapping.

[chubei]

@v3g42 wanted the behaviour to be configurable.

For PrimaryKeyExists, we can have 3 different options:

• Ignore new value and log error.
• Override old value and log error.
• Stop processing and quit.(In case user wants to make sure it doesn't happen)

For PrimaryKeyNotFound, we can have two options:

• Ignore and log error.
• Stop processing and quit. (In case user wants to make sure it doesn't happen)

I don't know if the "Stop processing" option makes sense though.

[v3g42]

@v3g42 wanted the behaviour to be configurable.

For PrimaryKeyExists, we can have 3 different options:

• Ignore new value and log error.
• Override old value and log error.
• Stop processing and quit.(In case user wants to make sure it doesn't happen)

For PrimaryKeyNotFound, we can have two options:

• Ignore and log error.
• Stop processing and quit. (In case user wants to make sure it doesn't happen)

I don't know if the "Stop processing" option makes sense though.

We can take inspiration in a way DBs typically handle this.
https://www.postgresql.org/docs/current/sql-insert.html

For example Postgres has on_conflict for insert and update.

[karolisg]

I am thinking about such structure where developer can control conflict resolution for all 3 operations.

Example yaml

  - name: airports
    path: /bookings/airports
    table_name: airports
    index:
      primary_key:
        - airport_code
    conflict_resolution:
        on_insert: update
        on_update: upsert
        on_delete: panic

Insert

on_insert	nothing (default)	Ignore insert when record with same PK exists
on_insert	update	Update existing record
on_insert	panic	Stop processing and quit

Update

on_update	nothing (default)	Ignore update when old doesnt exist
on_update	upsert	Insert if missing old one
on_update	panic	Stop processing and quit

Delete

on_delete	nothing (default)	Ignore delete when old doesnt exist
on_delete	panic	Stop processing and quit

In all cases it would log error and what following action will be done. Something like
WARN Record(id #1) insert failed, error: PrimaryKeyExists. Trying update

[chubei]

In my opinion, for app, componnets should handle recoverable errors by themselves. All errors that returned from Source::start, Processor::process and Sink::process should be unrecoverable and crash the app.

[v3g42]

In my opinion, for app, componnets should handle recoverable errors by themselves. All errors that returned from Source::start, Processor::process and Sink::process should be unrecoverable and crash the app.

Agreed.

Then we should define how Processor::process and Sink::process` should behave.

Processor::process

• Default behavior -> log::warn!
• In the future, we can implement a global error counter

Sink::process

• Implement on_conflict behavior for endpoints, meaning caches
• By default log:warn!