The downloaded data did not match the data from the server

[beshkenadze]

Hey,
Any files that I get from the bucket "pubsite_prod_rev_", gets error code: CONTENT_DOWNLOAD_MISMATCH.

bucket.file("reviews/reviews_com.sample.android_201409.csv").download({
  destination: './reviews.cvs'
},
function(err, content){
  console.log(err.code);//CONTENT_DOWNLOAD_MISMATCH
});

[beshkenadze]

:(

[jgeewax]

Any thougths @stephenplusplus ?

[stephenplusplus]

@beshkenadze sorry that it's taken so long to get on this. It's hard to say what could be causing this or if it's related to #651.

The error is being returned because either an MD5 or CRC32c validation check isn't passing. In other words, the data you've received isn't matching the data that's stored in the bucket. You can workaround this by disabling validation:

bucket.file("reviews/reviews_com.sample.android_201409.csv").download({
  destination: './reviews.cvs',
  validation: false
},
function(err, content){
  // No more mismatch error (hopefully)
});

If you want to do some debugging, I put up a branch you can swap out your gcloud dependency for. It will just do a little console.log-ing to help see what's going on:

$ npm install --save stephenplusplus/gcloud-node#spp--654

[beshkenadze]

Hey @stephenplusplus,
How is getting of the hash from a local file?
Play Сloud gives files compressed in GZ format.
Possible hashes are calculated from the extracted file?

[jgeewax]

Not sure I understand the question @beshkenadze ...

To give some background (sorry if you already know this, just adding for context):

• Play is uploading files to a bucket that they own
• Play then shares access to the bucket with your Google account (not your project's service accounts)
• When you download a file, gcloud-node does some validation to make sure that the data wasn't tampered with along the way (see https://github.com/GoogleCloudPlatform/gcloud-node/blob/d4d3d06b466834fd8a19771e75ff5cd9f89013fa/lib/storage/file.js#L491)
• If the validation fails (ie, MD5 or CRC32 don't match what the file should have), we raise the error you're seeing (see https://github.com/GoogleCloudPlatform/gcloud-node/blob/d4d3d06b466834fd8a19771e75ff5cd9f89013fa/lib/storage/file.js#L548)

I don't see anything that would indicate that we're looking at the uncompressed file, as we're treating the data as nothing more than bytes and ignoring the file type all together.

It could be possible that Play (when uploading the data) is somehow bypassing the part where they set the CRC32 and MD5 hash for the file (which would cause this error to happen on all Play-uploaded files). Could you tell us what you get back in the headers that start with x-goog-hash when you GET the files from GCS?

/cc @stephenplusplus

[beshkenadze]

Now find the real file and will try to show an example.

[stephenplusplus]

Could you tell us what you get back in the headers that start with x-goog-hash when you GET the files from GCS?

That's what this will do:

$ npm install --save stephenplusplus/gcloud-node#spp--654

[beshkenadze]

@stephenplusplus version ("version": "0.8.1") to old :)

[stephenplusplus]

That branch (spp--654) is tracking master: https://github.com/stephenplusplus/gcloud-node/tree/spp--654

[beshkenadze]

I used request-debug and got this:

{
  response: {
    debugId: 1,
    headers: {
      'x-guploader-uploadid': 'XXXX',
      expires: 'Mon, 20 Jul 2015 13:40:35 GMT',
      date: 'Mon, 20 Jul 2015 13:40:35 GMT',
      'cache-control': 'private, max-age=0',
      'last-modified': 'Sun, 19 Jul 2015 18:53:59 GMT',
      etag: 'W/"XXXX"',
      'x-goog-generation': '1437332039288000',
      'x-goog-metageneration': '1',
      'x-goog-stored-content-encoding': 'gzip',
      'x-goog-stored-content-length': '5939',
      'content-type': 'text/csv; charset=utf-16le',
      'x-goog-hash': 'crc32c=66rJzQ==, md5=2T/NKanU9vTItoiF7+tMAA==',
      'x-goog-storage-class': 'STANDARD',
      vary: 'Accept-Encoding',
      'content-length': '24148',
      server: 'UploadServer',
      'alternate-protocol': '443:quic,p=1',
      connection: 'close'
    },
    statusCode: 200
  }
}

[stephenplusplus]

Nice :) Using my branch will show the hashes that are being built locally as well.

[beshkenadze]

@stephenplusplus ;)

Headers: { 'x-guploader-uploadid': 'XXXX',
  expires: 'Mon, 20 Jul 2015 13:46:35 GMT',
  date: 'Mon, 20 Jul 2015 13:46:35 GMT',
  'cache-control': 'private, max-age=0',
  'last-modified': 'Sun, 19 Jul 2015 18:53:59 GMT',
  etag: 'W/"XXXX"',
  'x-goog-generation': '1437332039288000',
  'x-goog-metageneration': '1',
  'x-goog-stored-content-encoding': 'gzip',
  'x-goog-stored-content-length': '5939',
  'content-type': 'text/csv; charset=utf-16le',
  'x-goog-hash': 'crc32c=66rJzQ==, md5=2T/NKanU9vTItoiF7+tMAA==',
  'x-goog-storage-class': 'STANDARD',
  vary: 'Accept-Encoding',
  'content-length': '24148',
  server: 'UploadServer',
  'alternate-protocol': '443:quic,p=1',
  connection: 'close' }

Local CRC32c Hash: Fw==
Local MD5 Hash: Hwt6cw9joXTy4EOtQqh0pg==
crypto.js:126
  return this._handle.digest(outputEncoding);
                      ^
Error: Not initialized
    at Error (native)

[stephenplusplus]

Those don't match even a little bit! Like you pointed out @beshkenadze, I think we're running into issues because request automatically decodes the file as it's being downloaded, resulting in different hashes. I can't think of a great solution immediately for how we can work around this, other than:

1. shut off the auto-decoding for all downloads (don't think we want this),
2. ignore the hash mismatch if we see the file was gzip'd in the response headers,
3. branch off from the request download stream, and run the calculation on the native http.IncomingMessage response stream (which won't do the decoding)

[jgeewax]

Can we run a quick test of uploading a zipped file, and seeing that the verification fails ? I'd feel much more confident if this was happening on a non-magic bucket (the one in question here is actually owned by Google Play, so it might be doing something wonky).

[stephenplusplus]

Yep, just confirmed it still happens.

[jgeewax]

OK cool - so that's going to be tricky. It means anytime anyone downloads a file that's zipped will get a content mismatch error (when really, there is no mismatch).

[stephenplusplus]

I think idea no. 3 is probably the best way to handle this. I'll take a stab at it and PR soon.

[beshkenadze]

👍

[adielmil]

Hi ,
Issue resolved? it happens to me today more than once.
I'm running on Google App engine flex env.
@google-cloud/storage version: 1.5.1

[stephenplusplus]

@adielmil Could you post a new issue on https://github.com/googleapis/nodejs-storage? Be sure to fill out the issue template and provide sample code/files that we can use to reproduce.

[kirillgroshkov]

Started happening to us too (also with a .gz file). Piece of logs:

ApiError: Multiple errors occurred during the request. Please see the `errors` array for complete details.

    1. Bad Request
    2. Metadata part is too large.


    at new ApiError (/root/repo/node_modules/@google-cloud/common/build/src/util.js:73:15)
    at Util.parseHttpRespMessage (/root/repo/node_modules/@google-cloud/common/build/src/util.js:175:41)
    at Util.handleResp (/root/repo/node_modules/@google-cloud/common/build/src/util.js:149:76)
    at /root/repo/node_modules/@google-cloud/common/build/src/util.js:477:22
    at onResponse (/root/repo/node_modules/retry-request/index.js:228:7)
    at /root/repo/node_modules/teeny-request/src/index.ts:244:13
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  code: 400,
  errors: [],
  response: PassThrough {

...

statusMessage: 'Bad Request',
    request: {
      agent: [Agent],
      headers: [Object],
      href: 'https://storage.googleapis.com/upload/storage/v1/b/our-bucket-id/o?uploadType=multipart&name=incremental%2FUserAchievements.ndjson.gz'
    },
    body: 'Metadata part is too large.',
    headers: {
      'alt-svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"',
      'content-length': '27',
      'content-type': 'text/plain; charset=utf-8',
      date: 'Wed, 22 Sep 2021 10:17:23 GMT',
      server: 'UploadServer',
      'x-guploader-uploadid': 'ADPycdvEW5Od42f4mkWWh8AJtU2Pm2qcEBgw2rXrv5_GC6iOu72CaG9gOIjFLQX3LJuwTh4yiizP2Sc18Vito9g9gsHb96lSHg'
    },

[acSpock]

Just to throw another angle on this issue. I'm currently seeing this in firebase storage emulators when developing locally. So far, not in production.

[JarnoRFB]

@acSpock Hit the same problem using firebase emulator. What helped me was to use file.download({validation: false}) as described here https://www.reddit.com/r/Firebase/comments/nhilth/help_in_new_firebase_storage_emulator/

[LB-Digital]

@acSpock Hit the same problem using firebase emulator. What helped me was to use file.download({validation: false}) as described here https://www.reddit.com/r/Firebase/comments/nhilth/help_in_new_firebase_storage_emulator/

Perhaps a slightly better approach would be...

bucket.file(filePath).download({
    validation: !process.env.FUNCTIONS_EMULATOR,
});

So then you keep the validation in prod. In addition, if you're using TypeScript, you can satisfy the type checking by following the suggestion in this SO post: https://stackoverflow.com/a/53981706/6506026

[JarnoRFB]

@LB-Digital is FUNCTIONS_EMULATOR an environment variable you expect to be set by default when running with the emulator? Because in tests I execute with firebase emulators:exec it seems not to be set.

[ElBouhaliMohamed]

@JarnoRFB yes you need it for firebase admin to know youre running local emulator

[johnnyoshika]

I'm running into this error when using the emulator as well