Thumbnails example -Error: The uploaded data did not match the data from the server.

[goodcourage-jimmy]

Hello,

I use Thumbnails example, and sometimes (little) I'm getting this error when uploading files from Storage :

Error: The uploaded data did not match the data from the server. As a precaution, the file has been deleted. To be sure the content is the same, you should try uploading the file again.
    at /user_code/node_modules/@google-cloud/storage/src/file.js:1058:21
    at /user_code/node_modules/@google-cloud/storage/node_modules/@google-cloud/common/src/service-object.js:155:5
    at Object.handleResp (/user_code/node_modules/@google-cloud/storage/node_modules/@google-cloud/common/src/util.js:135:3)
    at /user_code/node_modules/@google-cloud/storage/node_modules/@google-cloud/common/src/util.js:465:12
    at Request.onResponse [as _callback] (/user_code/node_modules/@google-cloud/storage/node_modules/retry-request/index.js:133:7)
    at Request.self.callback (/user_code/node_modules/@google-cloud/storage/node_modules/request/request.js:188:22)
    at emitTwo (events.js:106:13)
    at Request.emit (events.js:191:7)
    at Request.<anonymous> (/user_code/node_modules/@google-cloud/storage/node_modules/request/request.js:1171:10)
    at emitOne (events.js:96:13)

Here is some code snippet, for the deletion part :

index.js

exports.generateThumbnail = functions.storage.object().onChange(event => {
  return thumbnail(event)
})

thumbnail

const gcs = require('@google-cloud/storage')();
const spawn = require('child-process-promise').spawn;

function thumbnail(event) {

    const object = event.data;
    const fileBucket = object.bucket;
    const filePath = object.name;
    const contentType = object.contentType;
    const resourceState = object.resourceState;

    if (!contentType.startsWith('image/')) {
        return;
    }

    const fileName = filePath.split('/').pop();
    if (fileName.startsWith('thumb_')) {
        return;
    }

    if (resourceState === 'not_exists') {
        return;
    }

    const bucket = gcs.bucket(fileBucket);
    const tempFilePath = `/tmp/${fileName}`;
    return bucket.file(filePath).download({
        destination: tempFilePath
    }).then(() => {
        return spawn('convert', [tempFilePath, '-thumbnail', '200x200>', tempFilePath]).then(() => {
            const thumbFilePath = filePath.replace(/(\/)?([^\/]*)$/, `$1thumb_$2`);
            return bucket.upload(tempFilePath, {
                destination: thumbFilePath
            });
        });
    });
}
module.exports = thumbnail;

[nicolasgarnier]

It looks like sometimes validation of the uploaded image fails.

It may be failing for good reasons but, if you want you can disable validation by passing validation: false to the bucket.upload call:

return bucket.upload(tempFilePath, {
  destination: thumbFilePath,
  validation: false
});

The alternative is to catch the error and retry the upload. Maybe with something like this:

// Try one upload with validation and if it fails another one without validation.
return bucket.upload(tempFilePath, {
  destination: thumbFilePath
}).catch(e => {
  return bucket.upload(tempFilePath, {
    destination: thumbFilePath,
    validation: false
});

And of course you could do something more sophisticated with multiple retry and exponential backoff etc... :)

If some of you could try this in some cases where these errors usually happen and repport that would be great :)

[PierBover]

It looks like sometimes validation of the uploaded image fails.

I'm getting this problem too and it seems a very erratic behavior.

Where is the appropriate place to report this bug?

[joelgetaction]

I'm also seeing this and it's very frustrating! Whenever I update an image I get this issue ...

[nicolasgarnier]

does using validation: false fixes the issue?

Looking at this issue: googleapis/google-cloud-node#2604 it seems that disabling resumable upload could help too.

Can you try passing resumable: false instead of validation: false and see if it works?

Re: the correct place to report this bug would be on https://github.com/googleapis/google-cloud-node/. If resumable uploads are the issue you could add onto googleapis/google-cloud-node#2604