fix: avoid some allocations in DeltaStorageHandler

[roeap]

Description

Fixes an error in the DeltaFileSystemHandler, when reading file metadata from remote storages. Due to an inconsistency between the behaviour object stores when invoking list operations on a path that points to a file, we incorrectly returned an Directory type for files in case of object stores. The bug only surfaced when using pyarrow < 9, since we used the call only when getting the file size, which we avoid when using more recent pyarrow versions.

@tustvold - I seem to vaguely remember discussing this at some point, but am not sure anymore. Is this something we should look into in object-store?

Update: validated locally, that the upstream fixes will fix the linked issue, so the main reason for this PR is resolved elsewhere. There are some changes included which safe us some allocation (admittedly very few), but hopefully an improvement anyhow.

Related Issue(s)

closes #1109

Documentation

[tustvold]

I seem to vaguely remember discussing this at some point, but am not sure anymore. Is this something we should look into in object-store?

object_store should not return directories in listing responses, it follows an object store / KV model and not a filesystem model. If one of the implementations is returning directories that is a bug. I'm not sure if this is what is going on here though?

[roeap]

From what I could tell happend. If you have a store with a single object file at the root. Using list (or list with prefix) like

path = Path::from("file")
store.list(Some(path))

I think the remote stores will return a list containing that single object, whereas LocalFileSystem will return an empty list.

[tustvold]

Both should return an empty list, that may be a bug, I'll take a look

Edit: filed a bug report upstream, thank you for reporting - apache/arrow-rs#3712

[roeap]

turning this back into a draft, since the upstream fixes are in the works ...