Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support streaming uploads of file-like objects #1909

Closed
linar-jether opened this issue Jun 26, 2016 · 5 comments
Closed

Support streaming uploads of file-like objects #1909

linar-jether opened this issue Jun 26, 2016 · 5 comments
Assignees
Labels
api: storage Issues related to the Cloud Storage API. type: question Request for information or clarification. Not an issue.

Comments

@linar-jether
Copy link
Contributor

How can we upload a file-like object (such as an http response, with an unknown content-length) without writing the file to disk or storing the entire file in-memory?

Seems like the upload_from_file option accepts a file-like object but enforces the size parameter (which isn't available in case of streaming uploads).

@daspecster
Copy link
Contributor

@linar-jether, quickly searching around it appears to be a question with some tough answers.

One possible solution could be to chunk the stream that you're collecting into StringIO objects(which can be used like files and you could set the length to your own specs) and then pass those to upload_from_file. That will use a limited amount of member as long as you can upload as fast as the data is streaming in.

See: https://docs.python.org/2/library/stringio.html

@tseaver tseaver added type: question Request for information or clarification. Not an issue. api: storage Issues related to the Cloud Storage API. labels Jun 26, 2016
@linar-jether
Copy link
Contributor Author

Did you mean composite objects? is there support for this in the library?

@linar-jether
Copy link
Contributor Author

linar-jether commented Jun 27, 2016

Tried to fix the upload_from_file method to support streaming for file-like objects without a defined size

@daspecster
Copy link
Contributor

@linar-jether, not composite objects, just trying to find a workaround.

@tseaver
Copy link
Contributor

tseaver commented Jul 14, 2016

Via #1914

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: storage Issues related to the Cloud Storage API. type: question Request for information or clarification. Not an issue.
Projects
None yet
Development

No branches or pull requests

3 participants