Set up interop and benchmark tests

[daviddias]

I want to set up interop and benchmark tests that are run pre-release of js-ipfs (or any time a dev wants), while the unit tests are run in the CI always.

I also would love to make sure that benchmark tests produce something parseable (text or json) so that we can generate graphs in the future and that aegir-release adds them to a benchmarks folder with the date and hour of the release, so that we can graph it evolving over time.

This is the first step towards a more robust implementation js-ipfs. The second step will be 'mad network topologies configurations' on https://github.com/ipfs/ipfs-interop-stress-tests.

@dignifiedquire what is the best way to achieve this with aegir, 'this' being:

• setting up npm test only running the unit tests
• set up the benchmarks to generate something that gets added with the release process

[dignifiedquire]

Not sure yet how to do this, lets first figure out what we are using for interop and benchmark tests.

[daviddias]

Think of this interop tests as 'unit interop tests', just making sure that the wire protocols are not mangled, that the flows are still intact, etc. These are here in order to help the development.

The 'heavy interop tests', also known as stress tests will have their own repo -- https://github.com/ipfs/ipfs-interop-stress-tests --, and should be continuously testing everything. This will always run before any release.

With regards to benchmark, it can be an afterthought. I would like to have some benchmarking happening ASAP, I'm afraid that ipfs-interop-stress-tests with benchmarks will take a while. Also, there are things that are benchmarks within a single implementation, like importing/exporting files, bitswap exchange blocks, etc.

[dignifiedquire]

I think the simplest solution for that would be using: http://npmjs.com/ipfs-daemon to

1. init
   1. go -> js
      • init repo, with go, and add a file
      • start daemon, with js, on the go repo and cat the file
   2. js -> go
      • init repo, with js, and add a file
      • start daemon, with cat, on the go repo and cat the file
2. connection
   • spawn a js and a go daemon
   • connect them
   • add a file on each
   • cat the file on the other one

[dignifiedquire]

@diasdavid I just pushed a set of basic interop tests for go and js (node). They pass fine with go-ipfs master + ipfs/js-ipfsd-ctl#140

[dignifiedquire]

I've also added repo tests, but they are not passing, so there is some incompat here. With that, all the tests I proposed above are at least implemented.

[daviddias]

I've also added repo tests, but they are not passing, so there is some incompat here.

that is because go decided to use CIDS for repo keys now

But as for the rest, <3 you are wonderful @dignifiedquire

[daviddias]

@dignifiedquire tests are failing in this PR (even with deps updated) silently:

npm test


> [email protected] test /home/ubuntu/js-ipfs
> gulp test

[07:41:46] Using gulpfile ~/js-ipfs/gulpfile.js
[07:41:46] Starting 'test'...
[07:41:46] Starting 'test:node'...
[07:41:46] Starting 'libnode:start'...
Starting at /tmp/ipfs-test-081107
Starting at /tmp/ipfs-test-727825
Starting at /tmp/ipfs-test-078677
Starting at /tmp/ipfs-test-348420
Swarm listening on /ip4/127.0.0.1/tcp/20012/ws
Swarm listening on /ip4/127.0.0.1/tcp/10012
npm ERR! Test failed.  See above for more details.

npm test returned exit code 1

Testing it locally yielded the same results