Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sharing functionality between delta-*-rs / unitycatalog-rs and others #15

Open
roeap opened this issue Jul 7, 2024 · 4 comments
Open
Labels
enhancement New feature or request

Comments

@roeap
Copy link

roeap commented Jul 7, 2024

Motivations

Whenever we need to communicate with some sort of (cloud) service, authentication is usually a requirement. The means of authentication of course depend on the cloud / identity provider used for the API. Especially in cases like unity (or delta sharing for that matter) this often depends the cloud a service is running in. This usually means especially client libraries often either defer the question of acquiring auth tokens to the user, or depend on the various cloud SDKs to provide more convenience.

Back when we wrote object_store for the arrow ecosystem, we therefore went ahead and implemented the auth mechanics for various cloud providers and oAuth flows with minimal dependency footprint. This evolved into what I believe to be a production-grade http client tailored to cloud services. Since then I found myself regularly hoisting that code into different projects - like delta-sharing.

So we are trying to solve essentially the same problem in a lot of places ..

  • Would you like to implement this feature? [y]

Solution

create a shared "cloud-client" crate that can be used to build higher level clients in various projects.

Alternatives

Not do this.

Additional context

@roeap roeap added the enhancement New feature or request label Jul 7, 2024
@abrassel
Copy link
Collaborator

abrassel commented Jul 8, 2024

this sounds like a very fun project - I would love to be involved with the cloud-client crate. It seems particularly useful to delta-kernel-rs, another project I've been doing some work on.

@abhiaagarwal
Copy link
Collaborator

@roeap I actually had a very similar idea — I was planning on extending object_store's trait with a custom TemporaryCredentialProvider that would live as a subcrate of this project to handle the logic of dispensing temp credentials per cloud provider (as mentioned on #13). Maybe that could be folded into a larger cloud-client crate, there's obviously a demand for it!

@Jeadie
Copy link

Jeadie commented Sep 19, 2024

@roeap I actually had a very similar idea — I was planning on extending object_store's trait with a custom TemporaryCredentialProvider that would live as a subcrate of this project to handle the logic of dispensing temp credentials per cloud provider (as mentioned on #13). Maybe that could be folded into a larger cloud-client crate, there's obviously a demand for it!

We're also interesting in a TemporaryCredentialProvider like paradigm in object-store. For us, we are needing to do some OAuth2.0 temporary credentials for some new ObjectStore types we're building.

@rtyler
Copy link
Collaborator

rtyler commented Sep 21, 2024

@roeap I agree with the solidity of the HTTP code in the object_store crate but would disagree with the credential handling logic 😆 I like the idea here, so long as it's a lot of async traits 😺

delta-rs was fundamentally broken on AWS for literal years 😱 recently because of the complexity of cloud-interactions we were not handling; fortunately the aws-sdk crates from AWS have simplified that. Have a look at the recently merged delta-io/delta-rs#2887 for some updates to our credentials.rs.

@abhiaagarwal that change also contains the AwsCredential implementation to glue temporary credentials into object_store. That sort of code is certainly worth bringing out.

From my perspective pursuing a cloud-client crate without dependencies on cloud's native crates (what object_store tries to do) is not likely going to be successful. I cannot speak for the Azure crates, but the AWS SDK crates are the standard interface, supported and maintained, by AWS for interaction with AWS.

What it sounds like you're wanting is something like jclouds for Rust, which is an ambitious project but definitely a useful one! Just please don't try to avoid cloud-specific dependencies like object_store does

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

5 participants