publish dozer crates to crates.io?

[universalmind303]

Wondering if there are plans to publish various dozer components to crates.io? Is the plan to have this as an extensible system similar to datafusion?

It'd be nice to have a public crate dozer that exposes various components of the repo in a well organized way.

Ideally something like this would be nice

use dozer::prelude::*;

#[tokio::main]
async fn main() {
    let pg = PostgresConfig {
        user,
        password,
        host,
        port,
        database,
    };

    let cfg = Some(ConnectionConfig::Postgres(pg));

    let conn = Connection {
        config: cfg,
        name: "test".to_string(),
    };
    let conn = dozer::build_connector(conn).unwrap();

    let (tbl_info, schemas) = conn.list_all_schemas().await.unwrap();
    let ingestor = build_custom_ingestor_somehow();
    conn.start(ingestor, tbl_info).await?;
    Ok(())
}

[v3g42]

Hey @universalmind303. Definitely, something we are considering. Our hands are a little occupied right now but this is in consideration.

Another way we were considering is to expose similar functionality even in python. We started creating a few python libraries already.

Basically the code you wrote could be done using python and get the speed of rust similar to https://www.pola.rs/

Would love for contributors to participate and see if there is interest from the community.