Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cloud Storage: provide option to disableGzipContent on InputStream variants #36

Closed
nblair opened this issue Dec 18, 2019 · 2 comments · Fixed by #82
Closed

Cloud Storage: provide option to disableGzipContent on InputStream variants #36

nblair opened this issue Dec 18, 2019 · 2 comments · Fixed by #82
Assignees
@frankyn
Labels
api: storage Issues related to the googleapis/java-storage API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@nblair
Copy link
Contributor

nblair commented Dec 18, 2019

Is your feature request related to a problem? Please describe.

I have a use case that necessitates using the Storage/Bucket write method variants with InputStream arguments. The inputs for my use case to store in Cloud Storage are:

  • predominantly already compressed,
  • have a wide range of file sizes, from a few KB to GB+,
  • we don't always know the size in advance.

Describe the solution you'd like

Given that our content is already compressed, I would prefer to avoid spending the CPU time on compressing the content again en route to the Bucket.

Describe alternatives you've considered

We have used the byte[] variants, with BlobTargetOption.disableGzipContent and the Compose request. This is suitable but leaves us with a tuning challenge:

  • Buffering these streams into memory (byte[] chunks) requires additional heap space be available
  • Compose is limited to only 32 chunks. If the file content we intend to store is far larger than 32 * bufferSize, we will upload 31 small sized chunks and 1 large chunk that we have to use the InputStream variant for anyways, and pay the additional overhead of gzip compression.

Additional context

I have done some exploration and it appears that the values on the BlobWriteOption (used on InputStream variants) and BlobTargetOption (used on byte[] variants) enums both are translated into StorageRpc.Option. I have a small contribution to offer in the form of a pull request to follow.
@nblair
Copy link
Contributor Author

nblair commented Jan 7, 2020

Hello - I'm curious if you have any feedback or questions about this idea? Adding this feature would provide a benefit for my use case. Thanks in advance!

@chingor13 chingor13 transferred this issue from googleapis/google-cloud-java Jan 7, 2020
@chingor13 chingor13 added the type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. label Jan 7, 2020
@nblair
Copy link
Contributor Author

nblair commented Jan 23, 2020

Just saw the project has relocated - I'll take a look at porting the changes over to this new project.

@nblair nblair mentioned this issue Jan 23, 2020
Merged
nblair added a commit to nblair/java-storage that referenced this issue Jan 23, 2020
@nblair
feat: disableGzipContent option on create with InputStream (googleapi…
d0ef86f 
…s#36)

Previously, only the methods to create blobs that take a byte[] argument offer the option to disable gzip compression; the methods that accept an InputStream argument do not. This is due to the BlobWriteOption enum missing a matching constant for BlobTargetOption.IF_DISABLE_GZIP_CONTENT.

This change set adds a matching IF_DISABLE_GZIP_CONTENT constant to BlobWriteOption including the correct translation to StorageRpc.Option. The net result is that the Storage create functions that accept an InputStream now offer the option to disable gzip compression.
nblair added a commit to nblair/java-storage that referenced this issue Jan 29, 2020
@nblair
feat: disableGzipContent option on create with InputStream (googleapi…
183d0c6 
…s#36)

Previously, only the methods to create blobs that take a byte[] argument offer the option to disable gzip compression; the methods that accept an InputStream argument do not. This is due to the BlobWriteOption enum missing a matching constant for BlobTargetOption.IF_DISABLE_GZIP_CONTENT.

This change set adds a matching IF_DISABLE_GZIP_CONTENT constant to BlobWriteOption including the correct translation to StorageRpc.Option. The net result is that the Storage create functions that accept an InputStream now offer the option to disable gzip compression.
@google-cloud-label-sync google-cloud-label-sync bot added the api: storage Issues related to the googleapis/java-storage API. label Jan 29, 2020
frankyn pushed a commit that referenced this issue Feb 14, 2020
@nblair
feat: disableGzipContent option on create with InputStream (#36) (#82)
65d3739 
Previously, only the methods to create blobs that take a byte[] argument offer the option to disable gzip compression; the methods that accept an InputStream argument do not. This is due to the BlobWriteOption enum missing a matching constant for BlobTargetOption.IF_DISABLE_GZIP_CONTENT.

This change set adds a matching IF_DISABLE_GZIP_CONTENT constant to BlobWriteOption including the correct translation to StorageRpc.Option. The net result is that the Storage create functions that accept an InputStream now offer the option to disable gzip compression.
@yoshi-automation yoshi-automation mentioned this issue Feb 21, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@frankyn frankyn

Labels
api: storage Issues related to the googleapis/java-storage API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

add disableGzipContent option for create from InputStream nblair/java-storage
3 participants
@chingor13 @nblair @frankyn