dozer migrate

[chubei]

Migration

State

The Dozer state is defined as following:

• Pipeline nodes (sources, processors and sinks), each with a unique name, its input and output schemas, checkpoint and internal state.
• Caches, each with a unique name and internal state.
• Endpoint mapping, from endpoint name to sink names. If there exists more than one sink names, exactly one of them is marked Write.
• Sink mapping, from sink name to cache name.

Name Calculation

Source name is the connection name from configuration file.
Processor names are generated and unique.
Sink name is the Write sink name if it exists, otherwise generated and unique.
Cache name is the cache name from sink mapping if it exists, otherwise generated and unique.

Migration Check

On startup, Dozer calculates all names, and checks if pipeline is consistent (See #763 for definition of consistency).

If consistent, Dozer further checks:

• If all sources can start from the checkpoint.
• If all sinks can start from the checkpoint.

If any of the check fails, migration is needed.

Migration Execution

If migration is requested, Dozer do the following:

• Delete the state of the source nodes.
• For each endpoint name, generate a new sink name and mark it Write.

After these actions, the whole pipeline has no state and is consistent.

Dozer API serves all sinks of all endpoints. Dozer App writes to Write sink.

API Path Calculation

For any endpoint, the first sink is served under the API path specified in configuration file.

Following sinks are served under versioned API path.

For example, if the configured API path is /users, the first sink is served under /users, and the second sink is served under /v2/users.

Replace on Catchup

If requested, Dozer points the first sink to the cache of the Write sink, when the Write sink has processed enough number of records. This is the blue green cache mechanism which is already implemented.

@getdozer/dozer-dev

[mediuminvader]

Could make sense to have versioned endpoints like /v1/users /v2/users and serve under /users the current stable one?

[chubei]

I see. It makes sense to have an extra endpoint. Good idea!

[v3g42]

Looks good. As discussed, and for others' benefit we can also introduce a way to switch versions either through configuration or manually calling an internal grpc endpoint.

Current blue green cache implementation is one of the supported automatic switching mechanisms.

[chubei]

See #1488