fix(writer): retry storage.put on temporary network errors

[qinix]

Description

Read-only operations are retried by [ObjectStore] internally. However, PUT/DELETE operations
are not retried even thought they are technically idempotent. [ObjectStore] does not retry
those operations because having preconditions may produce different results for the same
request. PUT/DELETE operations without preconditions are idempotent and can be retried.
Unfortunately, [ObjectStore]'s retry mechanism only works on HTTP request level, thus there
is no way to distinguish whether a request has preconditions or not.

This PR provides additional methods for working with [ObjectStore] that automatically retry
unconditional operations when they fail, and use these methods in writer to retry on temporary
network errors.

Related Issue(s)

See

• https://datatracker.ietf.org/doc/html/rfc7231#section-4.2.2
• Retry Safe/Read-Only Requests on Timeout apache/arrow-rs#5278

[rtyler]

The code largely looks good to me @qinix. I have reviewed what object_store::Error has , and since this is only retrying on the Generic variant, I'm not sure what types of failure conditions you're seeing in the real world?

[qinix]

I encountered the following error (reformatted for readability):

called `Result::unwrap()` on an `Err` value: ObjectStore { 
    source: Generic { 
        store: "S3", 
        source: Reqwest { 
            retries: 0, 
            max_retries: 10, 
            elapsed: 30.000858021s, 
            retry_timeout: 180s, 
            source: reqwest::Error {
                kind: Request,
                url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("REDACTED")), port: None, path: "/table/REDACTED.zstd.parquet", query: None, fragment: None },
                source: TimedOut
            }
        }
    }
}

Inside the Generic error it was object_store::client::retry::Error. I can't downcast to this concrete type as it's not publicly exported (the mod object_store::client is private).

I dislike the way of retrying Generic error, however it's impossible to know the concrete error without modifying object_store to export the concrete error or doing the err.to_string().contains(...) trick.

Besides of Generic error, other variants are returned from server side in normal logic, thus those should not be retried.