rdam

RDBMS Data Movement Framework

Proposed Features

1. Transfer data from RDBMS to local fs or distributed fs
2. Parallel data read
3. HA
4. Restart / Retry ability
5. Containerized
6. Incremental Data load
7. Historic load
8. Default Parquet
9. Db , Table , Column selection
10. Scaling hint
11. Logging
12. In transit PII encryption/decryption
13. Extensibility for new DBs
14. Data Tagging
15. Data filtering
16. Custom query
17. Error Handling (silent/ loud / default values)
18. Quarantine Data (based on criteria and corrup data)
19. Authentication Wrapper
20. Preemptive connection pool

Info: CDC SCD1..6 WAL