[v2] s3 Blob Store #841
[v2] s3 Blob Store #841
Conversation
| func (b *BlobStore) StoreBlob(ctx context.Context, blobKey string, data []byte) error { | ||
| err := b.s3Client.UploadObject(ctx, b.bucketName, blobKey, data) | ||
| if err != nil { | ||
| b.logger.Errorf("failed to upload blob in bucket %s: %v", b.bucketName, err) | ||
| return err | ||
| } | ||
| return nil | ||
| } | ||
|
|
||
| // GetBlob retrieves a blob from the blob store | ||
| func (b *BlobStore) GetBlob(ctx context.Context, blobKey string) ([]byte, error) { | ||
| data, err := b.s3Client.DownloadObject(ctx, b.bucketName, blobKey) | ||
| if err != nil { | ||
| b.logger.Errorf("failed to download blob from bucket %s: %v", b.bucketName, err) | ||
| return nil, err | ||
| } | ||
| return data, nil | ||
| } |
There was a problem hiding this comment.
When we eventually merge the fragmented uploading utility, we will want to revisit this code. A prerequisite to using fragmented uploads will be to store extra metadata for each blob (i.e. fragment size + blob size needed to download without round trips).
No need to wait to merge this PR. The fact that blobs are fragmeneted in S3 should be invisible to things utilizing this API. As long as we don't deploy this code to production environments, a switchover to the fragmented strategy won't be that tricky.
There was a problem hiding this comment.
Not sure if we want the fragmented uploading in the blob store. This is not the relay store that stores chunks, but the one that stores whole blobs. I think this simple wrapper can serve the blob store in the long run too?
There was a problem hiding this comment.
I'm ok waiting to have the discussion for how we want to store blobs. It's mostly a question as to whether we want to try and improve latency for really big blobs. Since we don't really support super big blobs currently, it might be something we can avoid dealing with for the MVP.
Why are these changes needed?
Thin wrapper around s3 for blob storage
Checks