Per call objstore Upload/Get options

[pracucci]

Is your proposal related to a problem?

Cortex uses on Thanos objstore clients. Cortex needs to build per-tenant S3 encryption key support and would like to avoid to create a new S3 client for each tenant: having a single S3 client allows to reuse S3 connections.

Describe the solution you'd like

The Thanos S3 client allows to config the SSEConfig at client creation time. In order to reuse the same client for different tenants, we would need to be able to pass a custom SSEConfig to each single Get() and Upload() call.

A solution would be introducing the ability to pass options to Get() and Upload(). To not introduce breaking changes in the objstore.Bucket signature we could leverage on the context.Context.

An idea could be having a WithOptions(objstore.Bucket) objstore.Bucket generic function to wrap a objstore.Bucket with options which are then injected in the context by the options wrapper. The underlying client implementation (eg. s3.go) would be changed to be able to read such options and configure the Get/Upload operation accordingly.

If a bucket doesn't support options (eg. SSE encryption is only supported by S3) such options would be ignored.

Describe alternatives you've considered

• Create an S3 client for each tenant (not an option, we have a large number of tenants managed for each process, we can't afford to re-create S3 connections for each of them)
• Fork S3 client in Cortex (still an option, but not desired)

Additional context

• Slack thread
• Encryption in Thanos objstore clients is currently supported only by S3

[bwplotka]

Just to continue our discussions from Slack.

An idea could be having a WithOptions(objstore.Bucket) objstore.Bucket generic function to wrap a objstore.Bucket with options which are then injected in the context by the options wrapper. The underlying client implementation (eg. s3.go) would be changed to be able to read such options and configure the Get/Upload operation accordingly.
If a bucket doesn't support options (eg. SSE encryption is only supported by S3) such options would be ignored.

I love this direction. Ideally, it would be awesome if the implementation (fact that it is S3 or not) does not leak from objstore at all.

Wonder only what's the difference between WithOptions(objstore.Bucket) and some Wrapper struct in objstore package that can use context passed values (private to objstore)

[pracucci]

We discussed this in the 2021-02-11 contributor office hour and decided to start simple, with an exposed context key defined in pkg/objstore/s3/ which allows to override the SSE config.