URL Registered with ObjectStore registry is different from url in DeltaScan

[Blajda]

Environment

Delta-rs version: latest

Binding: rust

---

Bug

What happened:
I loaded a Delta table backed by S3 storage and then registered it with Datafusion. I then performed a select which failed since ObjectStore was unable to get data from the backend.

The error provided by ObjectStore showed it tried to get from "https://amazon.com/path" where I configured Objectstore to use "http://localhost/"

What you expected to happen:
That when a table is registered with Datafusion it uses the same underlying configuration the registered table had.

How to reproduce it:

More details:
Further investigation showed that the URL passed to the ObjectStore registry is different from the URL used to get the ObjectStore. Since the correct url is not used it will create a new DeltaTable instance with defaults.

See rust/src/delta_datafusion.rs:404 and rust/src/delta_datafusion.rs:309

[roeap]

Thanks for the report! We'll have to look a bit deeper here. The Url passed to the object store registry is actually just an invention of delta-rs since we require out object store to be rooted at the table root and to avoid collisions with other stores that may be registered to the "raw" object store url.

However somehwere urls get mixed up ... :). Do you by any chance have a quick repro example we can use for debugging.

We do some integration testing with datafusion and S§, maybe we can find some differences.

[Blajda]

This defect slips through the integration tests since the Object Store configurations are exported to the environment hence when the "incorrect" url is obtained from the registry it rebuilds from that new URL plus the environment.

In my use case I configured the underlying storage by passing in a HashMap with all the configurations values hence the default S3 environment variables are not exposed causing the defaults to be used.

I can help by modifying the integration tests to instead maybe use a prefix of DELTA_RS_INT_ and then read the configuration into a HashMap that's passed to the correct builders. It would help with detecting any other implicit environments defects that exists.

If you have any other approaches I'd like to hear them.

[roeap]

@Blajda - sorry for the late reply. With the latest commits on main there are some updates to make more consistent use of the options map to create object stores in various places, which may already cover this bug. Would it be possible for you to confirm this?

Of course having integration tests cover this scenario would be great!

While we worked on making configuration more consistent, there is one piece still missing, which is configuration for the S3 lock client. Right now this also updates the environment. However as long as the configuration in the map is complete, we should in most places no longer take any values from the environment.

[Blajda]

Hi @roeap
Yes it seems like this issue is resolved. I was able to use query with Datafusion using main.
While checking this fix I encountered another issues so I'll open a new ticket.

Thanks for all the work.