buffer file uploads work but streams do not

[bchr02]

In the below examples, buffer file uploads work but streams do not. In the streams example I get a Error: socket hang up error.

Because of the third party API in which I am uploading to, the headers must be sent as outlined in the examples, however I do not believe this to be causing the issue. My hunch is that the read stream is not ending. Any help would greatly be appreciated. Thank you.

Here is the Stream version:

var needle = require('needle');
var fs = require("fs");

var stream = fs.createReadStream(__dirname + '/test.pdf');

var options = {
    multipart: false,
    headers: {
        'Content-Type': 'application/pdf;name=test.pdf',
    }
}

needle.post('http://192.168.1.200:95/Insert', stream, options, function(err, resp, body) {
    if (err) {
        return console.error('upload failed:', err);
    }
    console.log('Upload successful!  Server responded with:', body);
});

Here is the Buffer version:

var needle = require('needle');
var fs = require("fs");

var buffer = fs.readFileSync(__dirname + '/test.pdf');

var options = {
    multipart: false,
    headers: {
        'Content-Type': 'application/pdf;name=test.pdf',
    }
}

needle.post('http://192.168.1.200:95/Insert', buffer, options, function(err, resp, body) {
    if (err) {
        return console.error('upload failed:', err);
    }
    console.log('Upload successful!  Server responded with:', body);
});

[tomas]

Thanks for the heads up @bchr02. I'll take a look in a minute and let you know what I find. What version of Node are you running this in?

[bchr02]

6.9.4 LTS 64bit Windows

Thanks

[tomas]

I think it has to do with the receiving side. I just tested the following code and it works without a problem:

var http   = require('http'),
    needle = require('..'),
    fs     = require('fs');

http.createServer(function(req, res) {
  var ws = fs.createWriteStream('./out');

  ws.on('close', function() {
     res.end('Thanks.');
     console.log('File size:', fs.statSync('./out').size);
  });

  req.pipe(ws);
}).listen(5000);

var options = {
  multipart: false,
  headers: {
    'Content-Type': 'application/pdf;name=test.pdf',
  }
}

var stream = fs.createReadStream(__dirname + '/test.pdf');

needle.post('localhost:5000', stream, options, function(err, resp, body) {
  if (err) {
    return console.error('upload failed:', err);
  }
  console.log('Upload successful!  Server responded with:', body);
});

var buffer = fs.readFileSync(__dirname + '/test.pdf');

needle.post('localhost:5000', buffer, options, function(err, resp, body) {
  if (err) {
    return console.error('upload failed:', err);
  }
  console.log('Upload successful!  Server responded with:', body);
});

What kind of server is processing the request?

[bchr02]

What kind of server is processing the request?

Not exactly sure. It's a proprietary image server. Our ERP vendor uses it to store images from their ERP client software. The software that I am writing needs to be able to leverage off this proprietary image server and send requests the same way as the ERP client software does.

I inspected the HTTP communication that the ERP client software is sending and have successfully been able to utilize the image server using Buffers but not using Streams. From an abstraction stand point I am not sure why there should be any difference when using Buffers over Streams?

[bchr02]

I was able to successfully run your test script.

If in fact it is the receiving server that is causing the issue that would have to mean that needle is not sending the POST in exactly the same way, no?

[bchr02]

If in fact it is the receiving server that is causing the issue that would have to mean that needle is not sending the POST in exactly the same way, no?

and if that's the case perhaps there is an issue with needle or one if's dependencies.

[bchr02]

and if that's the case perhaps there is an issue with needle or one if's dependencies.

actually I believe node's http module sends file streams as chunks and not within the payload. Therefore the issue would be with the receiveing server, like you said. 😄

[bchr02]

The strange thing is that using Unirest I was able to get it working with Streams however for some reason Unirest would change the content-type so I opened Kong/unirest-nodejs#102 and then came across needle.

[tomas]

Since the error you're getting is a socket hang up error you can try running your test with open_timeout: 0, read_timeout: 0 } and see what happens. That error usually occurs when the open or read timeouts is triggered, which provokes needle to abort the request.

[bchr02]

Since the error you're getting is a socket hang up error you can try running your test with open_timeout: 0, read_timeout: 0 } and see what happens. That error usually occurs when the open or read timeouts is triggered, which provokes needle to abort the request.

nope that didn't work. But thanks for the suggestion.

I forgot to mention that while using Streams although I get the socket hang error everything else works. In other words the image server successfully copies the file to the where it needs to go and does the few other tasks that it does. The only issue is that the body of needle's callback response is supposed to provide the image key which I need but instead I get the error. I even tried removing the the error to see if maybe the body would contain the response but it does not.

[bchr02]

I started searching around the request issues and can see many others are having similar errors. One person in particular recommended using bhttp which he explained "uses streams2".

To my amazement I gave bhttp a try and it worked perfectly. Here is the code:

var fs = require("fs");
var bhttp = require("bhttp");

var stream = fs.createReadStream(__dirname + '/code128.pdf');

var options = {
    multipart: false,
    headers: {
        'content-type': 'application/pdf;name=code128.pdf'
    }
};

bhttp.post("http://192.168.1.200:95/Insert", stream, options, function(err, response) {
    console.log("Response from server:", response.body.toString());
});

Regardless, I want to thank you for your help. I greatly appreciate it. You can close this issue unless you would like to try to continue to resolve it at which point I wouldn't mind testing it out for you. Let me know.

[tomas]

Sure @bchr02, always glad to help (if I can).

It's hard for me to debug the issue without being able to reproduce it locally. I even tried the example code in the request issue you linked, but needle did run the code with no errors. ;)

My guess is that this has something to do either with the 'content-length' header or the 'content-transfer-encoding' one. It looks like the server is closing the request before the client is done sending the data, so by setting one of these two headers we should be able to prevent that behaviour.

If possible, try running your code with these options:

var options = {
  headers: {
    'Content-Type': 'application/pdf;name=test.pdf',
    'Content-Transfer-Encoding': 'chunked' 
  }
}

Or by prefetching the length of the stream (it's not what you would normally do, but just to see whether this is the problem or not):

var options = {
  headers: {
    'Content-Type': 'application/pdf;name=test.pdf',
    'Content-Length': fs.statSync(__dirname + '/test.pdf').size
  }
}

And let me know how it goes!

[bchr02]

Adding 'Content-Length': fs.statSync(__dirname + '/test.pdf').size worked! I no longer get the error and I get a response back. I'm still confused as to why and what is responsible. Is this a bug?

[tomas]

That's interesting. So setting the Content-Transfer-Encoding header didn't work, then?

[bchr02]

that's correct.

[bchr02]

@tomas Thank you!