Storage: small file uploads are randomly returning md5 errors #2604

gcochard · 2017-09-12T17:35:13Z

Getting intermittent errors with codes FILE_NO_UPLOAD_DELETE and FILE_NO_UPLOAD when I am uploading a large amount of files in parallel.

Environment details

OS: Mac OS 10.12.6
Node.js version: 8.3.0
npm version: 5.4.1
google-cloud-node version: 0.56.0
google-cloud/storage version: 1.2.0

Steps to reproduce

This gist is a complete, self-contained example
https://gist.github.com/gcochard/5dd10b16911a57f3bad2b71bbbab833e

I have verified that the MD5sum of the file is correct before uploading, and that it is incorrect after uploading. It appears that the MD5sum returned is of the file minus the last 3 bytes. I will investigate further and see if this is consistent behavior.

The text was updated successfully, but these errors were encountered:

stephenplusplus · 2017-09-12T17:47:30Z

Can you try disabling resumable uploads?:

- .pipe(targetFile.createWriteStream({ validation: 'md5' }))
+ .pipe(targetFile.createWriteStream({ validation: 'md5', resumable: false }))

storage: add warning for resumable uploads #1600

gcochard · 2017-09-12T18:12:14Z

With resumable: false, it does not reproduce within 10k files uploaded. I will let it run longer to see if it reproduces.

Forgot the stack trace (note that I modified the code to print the MD5 sums):

{ Error: The uploaded data did not match the data from the server. As a precaution, we attempted to delete the file, but it was not successful. To be sure the content is the same, you should try removing the file manually, then uploading the file again.
                                            
The delete attempt failed with this message:
                                                                                       
  Not Found MD5 expected: x1YQGmcVcup+gG5jHr2P5Q== MD5 actual: +XikEmgOezNtunVxjBFjPQ==                                  
    at /Users/gcochard/gcs-upload-timeout/node_modules/google-cloud/node_modules/@google-cloud/storage/src/file.js:952:21                  
    at Object.handleResp (/Users/gcochard/gcs-upload-timeout/node_modules/google-cloud/node_modules/@google-cloud/common/src/util.js:135:3)             
    at /Users/gcochard/gcs-upload-timeout/node_modules/google-cloud/node_modules/@google-cloud/common/src/util.js:465:12                         
    at Request.onResponse [as _callback] (/Users/gcochard/gcs-upload-timeout/node_modules/google-cloud/node_modules/retry-request/index.js:160:7)
    at Request.self.callback (/Users/gcochard/gcs-upload-timeout/node_modules/google-cloud/node_modules/request/request.js:188:22)                    
    at emitTwo (events.js:125:13)                                                                                                      
    at Request.emit (events.js:213:7)                                                                                            
    at Request.<anonymous> (/Users/gcochard/gcs-upload-timeout/node_modules/google-cloud/node_modules/request/request.js:1171:10)
    at emitOne (events.js:115:13)                                                                                                     
    at Request.emit (events.js:210:7) 
  code: 'FILE_NO_UPLOAD_DELETE',          
  errors:        
   [ { ApiError                                                                                                                                         
         at Object.parseHttpRespBody (/Users/gcochard/gcs-upload-timeout/node_modules/google-cloud/node_modules/@google-cloud/common/src/util.js:192:30)
         at Object.handleResp (/Users/gcochard/gcs-upload-timeout/node_modules/google-cloud/node_modules/@google-cloud/common/src/util.js:132:18)
         at /Users/gcochard/gcs-upload-timeout/node_modules/google-cloud/node_modules/@google-cloud/common/src/util.js:465:12                         
         at Request.onResponse [as _callback] (/Users/gcochard/gcs-upload-timeout/node_modules/google-cloud/node_modules/retry-request/index.js:160:7)
         at Request.self.callback (/Users/gcochard/gcs-upload-timeout/node_modules/google-cloud/node_modules/request/request.js:188:22)
         at emitTwo (events.js:125:13)                                                                                          
         at Request.emit (events.js:213:7)                                                                                            
         at Request.<anonymous> (/Users/gcochard/gcs-upload-timeout/node_modules/google-cloud/node_modules/request/request.js:1171:10)
         at emitOne (events.js:115:13)                                 
         at Request.emit (events.js:210:7)
       code: 404,      
       errors: [Array],    
       response: undefined,                                                 
       message: 'Not Found' } ] }

gcochard · 2017-09-12T18:29:44Z

Ran with 250k files and still no repro. Seems like resumable uploads are the source of the issue.

gcochard · 2017-09-12T19:05:38Z

Reproduced with fast-crc32c installed, so that is not the cause. Didn't expect it to be though.

It seems like the second chunk of the file is not uploaded when I reproduce, as the actual MD5 is the same (the base file) and I add a counter to the end of each chunk: https://gist.github.com/gcochard/5dd10b16911a57f3bad2b71bbbab833e#file-poc-gcs-upload-bug-js-L38

Is there an issue with the resumable upload module that would cause this? Am I doing it wrong?

gcochard · 2017-09-12T23:03:29Z

Digging further into this, the gcs-resumable-upload module seems to be the culprit. I am trying to capture some context from that module while reproducing the error, but it seems to disappear when I modify the module locally. Will report back more tomorrow.

Fixes googleapis#2604, googleapis#2611

Fixes #2604, #2611

jbdemonte · 2019-05-17T07:29:45Z

Just to as some feedback regarding resumable: false
We were facing some errors while uploading some small files (<200ko), kinda:

2019-05-13T08:42:23.817Z - error: [9e86d100-0a3a-413b-a4d8-8e8f58c8b37c] Error: The uploaded data did not match the data from the server. As a precaution, the file has been deleted. To be sure the content is the same, you should try uploading the file again.
    at delete (/var/www/node/prod/node_modules/@google-cloud/storage/build/src/file.js:1316:35)
    at Util.handleResp (/var/www/node/prod/node_modules/@google-cloud/common/build/src/util.js:142:9)
    at retryRequest (/var/www/node/prod/node_modules/@google-cloud/common/build/src/util.js:423:22)
    at onResponse (/var/www/node/prod/node_modules/retry-request/index.js:200:7)
    at /var/www/node/prod/node_modules/teeny-request/build/src/index.js:208:17
    at bound (domain.js:396:14)

After digging a lot, we finally ends on this issue and tried the same resumable: false.

Here are our results:

no problem at all on around 1.5 million of upload
7 times faster
CPU usage 60% => 2%

Some uploads randomly return checksum errors. These can be mitigated using resumable: false (optionally validation: false) See - googleapis/google-cloud-node#2604 - firebase/functions-samples#140 Example error: ```$ ../../node_modules/@finn-no/cdn-uploader/index.js --credentials ${CDN_UPLOADER_CREDENTIALS} --project-id foo-storage --bucket-name foo-assets --app-prefix foo-web/_next build/next /foo/node_modules/@google-cloud/storage/src/file.js:1093 const error = new Error(message); ^ Error: The uploaded data did not match the data from the server. As a precaution, the file has been deleted. To be sure the content is the same, you should try uploading the file again. at /foo/node_modules/@google-cloud/storage/src/file.js:1093:23 at Object.handleResp (/foo/node_modules/@google-cloud/common/src/util.js:134:3) at /foo/node_modules/@google-cloud/common/src/util.js:496:12 at Request.onResponse [as _callback] (/foo/node_modules/retry-request/index.js:198:7) at Request.self.callback (/foo/node_modules/request/request.js:185:22)```

stephenplusplus added the api: storage Issues related to the Cloud Storage API. label Sep 12, 2017

gcochard mentioned this issue Sep 16, 2017

Use bucket/file for configStore key googleapis/gcs-resumable-upload#23

Merged

gcochard added a commit to gcochard/google-cloud-node that referenced this issue Sep 21, 2017

Update gcs-resumable-upload

d26965c

Fixes googleapis#2604, googleapis#2611

gcochard mentioned this issue Sep 21, 2017

Update gcs-resumable-upload #2624

Merged

stephenplusplus closed this as completed in #2624 Sep 21, 2017

stephenplusplus pushed a commit that referenced this issue Sep 21, 2017

Update gcs-resumable-upload (#2624)

5690043

Fixes #2604, #2611

This was referenced Sep 22, 2017

update gcs-resumable-upload googleapis/nodejs-storage#33

Merged

Release v1.3.1 googleapis/nodejs-storage#40

Merged

Error : Not Found google cloud storage googleapis/nodejs-storage#25

Closed

nicolasgarnier mentioned this issue Oct 5, 2018

Thumbnails example -Error: The uploaded data did not match the data from the server. firebase/functions-samples#140

Closed

JustinBeckwith assigned stephenplusplus Feb 1, 2021

henrik242 mentioned this issue Jun 14, 2021

Add validation and resumable options finn-no/cdn-uploader#69

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Storage: small file uploads are randomly returning md5 errors #2604

Storage: small file uploads are randomly returning md5 errors #2604

gcochard commented Sep 12, 2017 •

edited

Loading

stephenplusplus commented Sep 12, 2017

gcochard commented Sep 12, 2017

gcochard commented Sep 12, 2017 •

edited

Loading

gcochard commented Sep 12, 2017 •

edited

Loading

gcochard commented Sep 12, 2017 •

edited

Loading

jbdemonte commented May 17, 2019

Storage: small file uploads are randomly returning md5 errors #2604

Storage: small file uploads are randomly returning md5 errors #2604

Comments

gcochard commented Sep 12, 2017 • edited Loading

Environment details

Steps to reproduce

stephenplusplus commented Sep 12, 2017

gcochard commented Sep 12, 2017

gcochard commented Sep 12, 2017 • edited Loading

gcochard commented Sep 12, 2017 • edited Loading

gcochard commented Sep 12, 2017 • edited Loading

jbdemonte commented May 17, 2019

gcochard commented Sep 12, 2017 •

edited

Loading

gcochard commented Sep 12, 2017 •

edited

Loading

gcochard commented Sep 12, 2017 •

edited

Loading

gcochard commented Sep 12, 2017 •

edited

Loading