Databricks integration

[karolisg]

Is your feature request related to a problem? Please describe.
Allow users to use data from databricks in ingestion.

Describe the solution you'd like
Implement new connector based on the current connector interface.

[karolisg]

I made some eploration. Couple things to mention:

• Created delta live tables in aws s3 bucket
  https://s3.console.aws.amazon.com/s3/buckets/db-14a8f8bfc0464631dcfbe0539db38e7b-s3-root-bucket?region=us-west-2&prefix=oregon-prod/61718142707882/user/hive/warehouse/&showversions=false

• There are two libraries, which can be used, but has some limitations:

  • https://docs.rs/dbfs-client/latest/dbfs_client/ No implementation of changes stream
  • https://github.com/delta-io/delta-rs Latest version, which supports polling of changes is not released yet (Implement polling for table updates. (#413) delta-io/delta-rs#550 and Cut 0.5 Rust crate delta-io/delta-rs#480 )

[karolisg]

let client = DbfsClient::new("https://dbc-beef84de-82d1.cloud.databricks.com","dapi938c7b3a990ade61bff831cf0f731190");
let tables = client.list("/user/hive/warehouse").await.unwrap();

let mut lines = client.read("/user/hive/warehouse/squirrel_data").unwrap().lines();
while let Some(Ok(line)) = lines.next().await {
    println!("Line is `{:?}`", line);
}