-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
sharing functionality between delta-*-rs / unitycatalog-rs and others #15
Comments
this sounds like a very fun project - I would love to be involved with the |
@roeap I actually had a very similar idea — I was planning on extending object_store's trait with a custom |
We're also interesting in a |
@roeap I agree with the solidity of the HTTP code in the object_store crate but would disagree with the credential handling logic 😆 I like the idea here, so long as it's a lot of async traits 😺 delta-rs was fundamentally broken on AWS for literal years 😱 recently because of the complexity of cloud-interactions we were not handling; fortunately the @abhiaagarwal that change also contains the From my perspective pursuing a cloud-client crate without dependencies on cloud's native crates (what object_store tries to do) is not likely going to be successful. I cannot speak for the Azure crates, but the AWS SDK crates are the standard interface, supported and maintained, by AWS for interaction with AWS. What it sounds like you're wanting is something like jclouds for Rust, which is an ambitious project but definitely a useful one! Just please don't try to avoid cloud-specific dependencies like object_store does |
Motivations
Whenever we need to communicate with some sort of (cloud) service, authentication is usually a requirement. The means of authentication of course depend on the cloud / identity provider used for the API. Especially in cases like unity (or delta sharing for that matter) this often depends the cloud a service is running in. This usually means especially client libraries often either defer the question of acquiring auth tokens to the user, or depend on the various cloud SDKs to provide more convenience.
Back when we wrote
object_store
for the arrow ecosystem, we therefore went ahead and implemented the auth mechanics for various cloud providers and oAuth flows with minimal dependency footprint. This evolved into what I believe to be a production-grade http client tailored to cloud services. Since then I found myself regularly hoisting that code into different projects - like delta-sharing.So we are trying to solve essentially the same problem in a lot of places ..
Solution
create a shared "cloud-client" crate that can be used to build higher level clients in various projects.
Alternatives
Not do this.
Additional context
The text was updated successfully, but these errors were encountered: