[Umbrella] InLong offline synchronization feature

[aloyszhang]

Motivation

Currently, InLong provides real-time data synchronization based on the Flink engine, which has the advantage of low latency. Compared to real-time synchronization, offline data synchronization(not supported yet) pays more attention to synchronization throughput and efficiency.

To enhance the usage scenarios of InLong, we plan to add support for offline data synchronization capability in InLong. The implementation is based on the Flink computing engine uniformly. Real-time synchronization tasks run in the manner of Flink stream tasks, while offline synchronization runs in the manner of Flink batch tasks. This approach can ensure the consistency of real-time and offline synchronization tasks' code as much as possible, reducing maintenance costs.

Solution

The offline synchronization feature of the InLong dataset integration provides sources and sinks for processing data, corresponding to data sources and destinations, and combines with the scheduling system to synchronize full or incremental data from the data source to the data target.

InLong supports scheduling offline synchronization tasks by setting specific trigger times(including year, month, day, hour, and minute) through the scheduling system.

Offline synchronization tasks are created by the Manager (including scheduling information), and the specific data synchronization logic is implemented through the InLong Sort module.

Logical Architecture

Key Competency

Job Configuration: Support Wizard Mode(Configuration through page wizard) and OpenAPI mode.

Scheduling Configuration: Support Wizard Mode(Configuration through page wizard) and OpenAPI mode

Job Type: Support Periodic Incremental Synchronization and Periodic Full Synchronization

Scheduling: Built-in simple periodic scheduling capability, complex capabilities such as task dependencies are supported by third-party scheduling systems.

Data Source: RMDB, Message Queue and Big data storage(Hive,StarRocks,Iceberg etc.)

Data Sink: RMDB, Message Queue and Big data storage(Hive,StarRocks,Iceberg etc.)

Compute Engine: Flink

Offline Job Operation and Maintenance: Job start,stop and running status monitoring

Special Handling: Dirty Data Processing Capability

Data Flow Architecture

1. The user creates an offline synchronization task.
2. The manager saves task information and scheduling information in the DB.
3. After task approval, the offline synchronization task information is encapsulated.
4. Register scheduling information with the scheduling system; InLong has a built-in simple scheduling solution (Quartz), while complete scheduling capabilities rely on third-party scheduling systems (DolphinScheduler, US, etc.).
5. The scheduling system regularly generates scheduling instances.
6. For the initial run, the manager constructs a Flink batch job.
7. Submit the Flink batch job to the Flink cluster.

Task list

new dev branch

Since this is a big feature for InLong, so, create a new branch for development, and after development and testing are completed, merge it back to master.

• create new dev branch https://github.com/apache/inlong/tree/dev-offline-sync

Manager

Offline Synchronization Task Management: Definition and Management of Offline Synchronization Tasks

• [Feature][Manager] Support offline synchronization task definition #9781
• [Feature][Manager] Support offline data sync management #9813
  • Page wizard mode
  • OpenAPI mode（already covered by existed OpenAPI）
• [Feature][Manager] Support submit flink job for offline data sync #9862
• [Improve][Manager] refactor code for building and submitting flink job #10353

Scheduling Management: Scheduling task definition, scheduling instance definition, scheduling task management (CRUD)

• [Feature][Manager] Support schedule information management for offline sync #10247
  • Page wizard mode
  • OpenAPI mode
• [INLONG-10436][Manager] Move schedule configuration from stream to group #10437
• [Improve][Manager] Combine schedule state transition with group operations #10360
• [Bug][Manager] Authentication failed when submitting Flink job by quartz schedule engine #10566
• Support for periodic scheduling capability
  • Scheduling instance definition
  • [Feature][Manager] Add interface for schedule client and engine #10395
  • [Feature][Manager] Support configrations for bounded source #10561
  • Plugin-based scheduling framework support
    • [Feature][Manager] Support build-in schedule base on quartz #10396
    • [Feature][Manager] Support built-in schedule engine trigger submitting of Flink batch job  #10514
    • DolphinScheduler, US, etc.

Offline Task Submission

• [Feature][Manager] Support schedule instance callback to submit Flink batch job #10459

Offline Task Operation and Maintenance

• Start (task submission), stop
• Retrieve running status
• Task logs, exceptions

Sort

Flink Task Encapsulation: Add support for Flink environment in batch mode

Flink Batch Capability Support

• [Feature][Sort] Add dependencies for flink 1.18 #10054
• [Feature][Sort] Support flink-connector-pulsar based on flink 1.18 #10053
• [Feature][Sort] Support flink-connector-jdbc based on flink 1.18 #10055
• [Feature][Sort] Support audit metrics for sort-connector-pulsar-1.18 #10069
• [Feature][Manager] Support new manager plugin for flink 1.18 #10056
• [Feature][Distribution] Support assembling flink 1.18 dependencies and connectors #10057
• [Feature][Sort] Add json format for 1.18 #10091
• [Feature][Sort] Support bounded pulsar source #10560

InLong Component

Other for not specified component

Are you willing to submit PR?

• Yes, I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct