rdam

RDBMS Data Movement Framework

Proposed Features

Transfer data from RDBMS to local fs or distributed fs
Parallel data read
HA
Restart / Retry ability
Containerized
Incremental Data load
Historic load
Default Parquet
Db , Table , Column selection
Scaling hint
Logging
In transit PII encryption/decryption
Extensibility for new DBs
Data Tagging
Data filtering
Custom query
Error Handling (silent/ loud / default values)
Quarantine Data (based on criteria and corrup data)
Authentication Wrapper
Preemptive connection pool

Info: CDC SCD1..6 WAL