Not clear how to leave an IPFS node ready for new requests

[davidweisss]

• Code:
• Repo: https://github.com/DNAvid/DNA-IDstorage/blob/master/storageServer.js
• Hash with implementation
  03157b4
• Version:
  "ipfs": "^0.25.4"
• Platform:
  Linux dnacoinstorage 4.10.0-32-generic IPFS Repo JavaScript Implementation #36-Ubuntu SMP Tue Aug 8 12:10:06 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
• Subsystem: unknown

Type: Question

Severity: Critical

Description:

I am working out of the example 1.js to provide a hosted service to encrypt upload files to ipfs. I removed the part in async/series where the process is exited.
I want the node to keep processing future requests.
The upload works great for the first upload but then hangs when a new request is received.
I've tried to stop the node then start it again, but there is something I must be missing because it doesn't work. Or this is not the way it is supposed to be used? Thanks for any insight as I've run out of ideas besides going deep in the source.

Steps to reproduce the error:

Upload one file, then try once again. Server hangs indefinitely.
It's on my hosted server: https://dnacoinstorage.com, please contact me if you'd like me to demo the issue.

[dryajov]

At first glance I can see that you are not terminating the series sequence properly in https://github.com/DNAvid/DNA-IDstorage/blob/ecd7a6291370e642d18509912089291782fdbbf8/storageServer.js#L75, which requires a callback after the array of steps (https://caolan.github.io/async/docs.html#series). I would add the terminating callback and move the res.end there, something like -

   series([..., ...], res.end)

IPFS code base and examples rely heavily on the async library, but IPFS also supports a promise based interface, so doing ipfs.files.add().then().catch() should also work.

[davidweisss]

Thanks a lot for your help. I've been trying what you suggested, which improved the example, but still no success.
I may be having trouble understanding how the scope of the object created by "new IPFS()" works. If I make it global, I can't use it inside my form parser.
If I create the new object inside the form parser, then after one successful run, the object's isOnline is always false, and / since ipfs is undefined. Creating a new ipfs object at each file upload event doesn't solve the issue.
I looked a bit into the source constructor for debugging:
DNAvid/DNA-IDstorage@b2f6f6a

Then I get these errors:

Error: Stream ended prematurely
    at series (/home/davidweisss/DNA-IDstorage/node_modules/libp2p-secio/src/handshake/index.js:21:15)
    at /home/davidweisss/DNA-IDstorage/node_modules/async/internal/parallel.js:39:9
    at /home/davidweisss/DNA-IDstorage/node_modules/async/internal/once.js:12:16
    at iterateeCallback (/home/davidweisss/DNA-IDstorage/node_modules/async/internal/eachOfLimit.js:44:17)
    at /home/davidweisss/DNA-IDstorage/node_modules/async/internal/onlyOnce.js:12:16
    at /home/davidweisss/DNA-IDstorage/node_modules/async/internal/parallel.js:36:13
    at waterfall (/home/davidweisss/DNA-IDstorage/node_modules/libp2p-secio/src/handshake/propose.js:29:14)
    at /home/davidweisss/DNA-IDstorage/node_modules/async/internal/once.js:12:16
    at next (/home/davidweisss/DNA-IDstorage/node_modules/async/waterfall.js:21:29)
    at /home/davidweisss/DNA-IDstorage/node_modules/async/internal/onlyOnce.js:12:16

Maybe I have to stop and restart the ipfs instance?

Any pointers help, as I'm really eager to get this to work.

I may be able to run it as a back-end service and spawn and terminate a node process each time but this seems suboptimal.

[haadcode]

@davidweisss I believe I can help you here.

In the code linked above, the IPFS instance gets created here https://github.com/DNAvid/DNA-IDstorage/blob/b2f6f6aaf12f8c522536e74b814fd9dab3bb13d3/storageServer.js#L34. However, in order to make sure IPFS has properly started, one has to wait for the the "ready" event, eg. ipfsNode.on('ready', () => /* adding files code goes here */). That should fix your problem.

If you want to separate IPFS from your http-backend process, you can use js-ipfs-api to connect to a running go-ipfs daemon (replace ipfs with ipfs-api on line https://github.com/DNAvid/DNA-IDstorage/blob/b2f6f6aaf12f8c522536e74b814fd9dab3bb13d3/storageServer.js#L12).

I would also recommend to start the IPFS instance outside your endpoints. Eg. create IPFS instance here and put the http.createServer() part inside ipfsNode.on('ready', () => http.createServer(...))

Let us know if that doesn't work or if you have any questions!

[dryajov]

@davidweisss - here is what you're trying to do reworked a little bit:

var express = require('express')
var formidable = require('formidable')
var fs = require('fs')
// key not synched on git
var https = require('https')
var http = require('http')
var fs = require('fs')
var crypto = require('crypto')
var series = require('async/series')
var waterfall = require('async/series')
var IPFS = require('ipfs')

var https_options = {
  //   key: key,
  //   cert: cert
}

// var key = fs.readFileSync(process.cwd() + '/selfCert.key')
// var cert = fs.readFileSync(process.cwd() + '/selfCert.crt')

var PORT = 9090
var HOST = '0.0.0.0'

// Nodejs encryption of buffers
var privateKey = new Buffer('my secret')

var ipfsNode = new IPFS()

ipfsNode.on('ready', (err) => {
  if (err) {
    console.error(err)
    process.exit(1)
  }

  var server = http.createServer(function (req, res) {
    if (req.url == '/fileupload' && req.method == 'POST') {
      var form = new formidable.IncomingForm()

      // form parsing
      form.parse(req, function (err, fields, files) {
        console.log('\nNode info on property isOnline:', ipfsNode.isOnline())
        // create read stream to encrypt and send to ipfs
        var rstream = fs
          .createReadStream(files.upload.path)
          .pipe(crypto.createCipher('aes-256-cbc', privateKey))
        let fileMultihash

        ipfsNode.files.add(rstream,
          (err, result) => {
            if (err) { return }

            res.write(
              '<h1> Your information is now encrypted in the interplanetary file system</h1>' +
              '<h2>Your data is immortalized, and encrypted <a href="https://ipfs.io/ipfs/' + result[0].hash + '">here</a>. This is the Multi hash of your file:' + fileMultihash + '</h2>' +
              '<h2> Only you have its password. If shared with this location, it gives access to your data. Share wisely.</h2>',
              res.end.bind(res))
          })
      })
    } else {
      // Upload form
      res.writeHead(200, {'Content-Type': 'text/html'})
      res.write('<h1> Store and send encrypted content to anyone</h1>')
      res.write('<h2>Encrypt and store in always-available, non-erasable peer-to-peer storage <br> Access and share with keys on www.</h2>')
      res.write('<form action="fileupload" method="post" enctype="multipart/form-data">')
      res.write('<input type="file" name="filetoupload"><br>')
      res.write('<input type="submit">')
      res.write('</form>')
      res.write('<p>secure https server encrypting your info then uploading it to permanent peer to peer storage network (ipfs). <br> </br> Gives you a web adress (ipfs) and a password to decrypt your data</p>')
      res.write('<p>Only you, and whomever you share the key with, has access</p>')
      res.write('<p>* this service is used by https://dnavid.com through its API to upload personal DNA information while retaining control. <p>')
      return res.end()
    }
  }).on('listening', (err) => {
    if (err) {
      console.error(err)
      process.exit(1)
    }

    console.log('HTTPS Server listening on %s:%s', HOST, PORT)
  }).listen(PORT, HOST)
})

I removed all the ssl parts for simplicity.

The main issue was that you were re-using the read-write stream returned by createCipher, the stream has to be created for each file.

In addition, and as suggested by @haadcode, in a node application you probably want to use an external IPFS instance through the ipfs-api module.

[davidweisss]

That's so helpful. Thanks!
Gonna give it a rip this evening.

[davidweisss]

It is now working, thank you! I didn't think of creating the server as a callback to IPFS instance 'ready' event. Clever!

Re the external api instance, for security and portability, I am aiming ideally for a simple code base that doesn't touch the file system of the server where it runs, streaming the file to ipfs then deleting all content and credentials, by piping from client to RAM to remote ipfs server.

But I confess I haven't gotten into the detail of how ipfs 'swarms' (?) work.

Perhaps I need a separate server to 'pin' the content. Probably I just need to reference it by ip and port in the 'new IPFS()' options?

My plan is to externalize the actual pinning and distribution with filecoin markets, but afaik it's still on the works, right?

(The 'stream ended prematurely' errors are still appearing, not critical as the server doesn't crash.)

Thanks again for the effort to answer outside the scope strictly of js-ipfs.

[davidweisss]

I'm closing this issue, as the proof-of-concept is now working, Thanks!!