Connectors unit tests infrastructure

[karolisg]

Almost every connector needs external database, so we need to create infrastructure for every connector. That infrastructure and test cases should consider several aspects:

• Database container
• Initial data
• Valid/invalid configurations
• Data cleanup (if needed) after tests

Current capabilities

We currently have implementation for e2e tests to be executed on buildkite. This approach creates external database infrastructure in docker container and run dozer app. After running, it compares the expected result/error with the actual result.

For connectors unit tests, we can reuse dozer-tests module parts, which allows to create and run docker commands. In that case every connector unit tests cases can control what infrastructure they need and connect to it.

# Example of test case with wrapper function
fn start_docker<T>(test: T)
    where
        T: FnOnce() + panic::UnwindSafe,
    {
        create_docker_compose_for_local_runner();
        let container = run_docker_compose(
            &docker_compose.path,
            &docker_compose.connections_healthy_service_name,
        );
        
        let result = panic::catch_unwind(|| {
            test();
        });

        drop(container);

        assert!(result.is_ok())
    }

#[test]
#[serial]
fn postgres_can_connect_and_get_data() -> IngestionIterator {
    let config = load_config(PG_CONFIG_PATH);
    let (ingestor, iterator) = Ingestor::initialize_channel(IngestionConfig::default());

    thread::spawn(move || {
        let connector = get_connector(config).unwrap();
        connector.start(None, &ingestor, tables).unwrap();
    });

    let mut client = TestPostgresClient::new(&config);
    client.insert_rows(&table_name, 10);

    let mut i = 1;
    while i < 10 {
        let op = iterator.next();
        if let Some((_, Operation::Insert { new })) = op {
            assert_eq!(new.values.get(0).unwrap(), &Field::Int(i));
            i += 1;
        }
    }
}

Infrastructure needed for connectors

Every connector has different needs of running services. Below is short list what containers every connector might need for unit test cases.

Postgres	- Empty database container
Eth
Kafka (debezium)	- Kafka container - zookeeper container - debezium connector - schema registry container (if needed) - Postgres (or any other, which has proper data types mapping) container
Object_store	- Simple FS container - AWS S3 Mock container (if any suitable exists)
Snowflake		We cannot use docker container for having isolated version of data. To run unit tests, we need to connect real snowflake database, create data there, run test and cleanup after that

[karolisg]

PR for wrapper function #980

[chubei]

Should we close this discussion?