storage: resumable uploads

[stephenplusplus]

Big files take big time. Big time means big chances for failure. Currently, if something goes wrong during an upload at 95%, the user is forced to re-upload starting from 0%. Using the resumable upload capabilities of the storage API, we should be able to handle this for our users.

It's not exactly trivial, however. My current idea for a solution involves storing a configuration file on the user/server's drive, where we can store tokens/state of uploads. Yeoman happens to have a tool to enable this for us, https://github.com/yeoman/configstore

Unless there are any objections, or more ideas for solutions, I will try to have a PR this week with this functionality.

[ryanseys]

What strategy will be taken if writing to disk is not possible due to
permission issues or some other arbitrary reason?

[stephenplusplus]

I guess fallback to non-resumable behavior. What would make the most sense?

On Monday, November 10, 2014, Ryan Seys [email protected] wrote:

What strategy will be taken if writing to disk is not possible due to
permission issues or some other arbitrary reason?

—
Reply to this email directly or view it on GitHub
#298 (comment)
.

[ryanseys]

Exposing the id and other necessary information used to me as a
developer may be nice so I can store it wherever I want (maybe in data
store? ;) )

On Monday, November 10, 2014, Stephen Sawchuk [email protected]
wrote:

I guess fallback to non-resumable behavior. What would make the most
sense?

On Monday, November 10, 2014, Ryan Seys <[email protected]
javascript:_e(%7B%7D,'cvml','[email protected]');> wrote:

What strategy will be taken if writing to disk is not possible due to
permission issues or some other arbitrary reason?

—
Reply to this email directly or view it on GitHub
<
https://github.com/GoogleCloudPlatform/gcloud-node/issues/298#issuecomment-62487064>

.

—
Reply to this email directly or view it on GitHub
#298 (comment)
.

[silvolu]

The boto approach might be interesting: the library will automatically retry x times on failures; a tracker file is optional and can be used to save/read the state.

[stephenplusplus]

That boto approach makes sense. We should be able to do the same thing. If a tracker file is provided, we first read it to see if there's already data to use to resume the upload. If not, we store the state of the upload in it and are able to re-use it the next time through.

• does the tracker file get written to if the upload is successful?
• is there a standard format the tracker file is written in?

[silvolu]

does the tracker file get written to if the upload is successful?

If the option is passed the file is written, and it's then removed if the upload is successful or if any non-retryable exception is raised.

is there a standard format the tracker file is written in?

Looks like is just a line in the tracker file that contains the URI.

[stephenplusplus]

SGTM 👍

[stephenplusplus]

After thinking more on this, I think it might be beneficial to still attempt to handle resumable uploads automatically, without a tracker file having to be provided:

fs.createReadStream("./photos.zip")
  .pipe(file.createWriteStream())
  .on("error", callback)

bucket.upload("./photos.zip", callback)

• Start resumable upload
• Store resumable ID in a config file (using yeoman/configstore)
• If it succeeds, remove id from file
• If it fails, emit error/callback with resumable ID / message that says "try again to resume"

Handling the streaming upload should be possible by counting bytes: https://cloud.google.com/storage/docs/concepts-techniques#unknownresumables

** edit: can't read. JSON docs: https://cloud.google.com/storage/docs/json_api/v1/how-tos/upload#resumable **

[silvolu]

After thinking more on this, I think it might be beneficial to still attempt to handle resumable uploads automatically, without a tracker file having to be provided:

Yeah, as mentioned the tracker file is optional.

Store resumable ID in a config file (using yeoman/configstore)

How is this different from using a tracker file? Can't we just store the ID in memory and on failure emit it?

[stephenplusplus]

Yeah, as mentioned the tracker file is optional ... How is this different from using a tracker file? Can't we just store the ID in memory and on failure emit it?

By defaulting to "we'll store it in a file for you" but allowing "give us a file to store it in", I think that would create a confusing situation where it's not clear when it makes sense to provide your own file. The file is useless outside of gcloud-node, as far as I know. After playing with this today, I can't even think of a good reason to return the resumable ID to the user -- with it alone, it's not very handy. If they don't use it with gcloud-node, they need to somehow assemble their own request (authenticate the headers, read the last byte sent, discard unused bytes from the retried upload data stream, etc). I'd rather just handle everything, and emit an error as we currently do - if the upload fails for any reason. The user will try again, and we'll automatically pick up where we left off, without them having to think about it.

The configstore is preferred because the user won't even need to know there is a config file.

[silvolu]

SGTM.