You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Following up on work from #10260, we changed the buffering logic to set some upper limits on the buffer size while buffering a stream before invoking the upload operations using StreamTransferManager.
Therefore, the amount of data being transferred is not unknown anymore (fixed to 200MB), and we don't need to configure an optimal number of part sizes as described here:
Server allows up to 10,000 parts to be uploaded for a single object, and each part must be identified by a unique number from 1 to 10,000.
Therefore the maximum amount of data that can be written to a stream is 10000/numStreams * partSize.
The total object size can be at most 5 TB, so there is no reason to set this higher than 525MB.
Describe the solution you’d like
Deprecate and remove PART_SIZE_MB fields from connectors based on StreamTransferManager
The text was updated successfully, but these errors were encountered:
Tell us about the problem you're trying to solve
Following up on work from #10260, we changed the buffering logic to set some upper limits on the buffer size while buffering a stream before invoking the upload operations using
StreamTransferManager
.Therefore, the amount of data being transferred is not unknown anymore (fixed to 200MB), and we don't need to configure an optimal number of part sizes as described here:
Describe the solution you’d like
Deprecate and remove
PART_SIZE_MB
fields from connectors based onStreamTransferManager
The text was updated successfully, but these errors were encountered: