feat: load files/dirs from hamt shards

[achingbrain]

Use the HAMT support and IPFS overlay build in to the mfs related files commands to enable loading files from HAMT shards.

Fixes ipfs/js-ipfs#1963
Depends on ipfs-inactive/js-ipfs-mfs#48 and a more recent version of ipfs/ipfsd-ctl

[achingbrain]

I think this also resolves #14, though I took a slightly different approach.

[lidel]

For some reason /ipfs/bafybeiemxf5abjwjbikoz4mc3a3dla6ual3jsgpdr4cjr3oz3evfyavhwq/ does not load for me (just hangs)

I was able to load Wikipedia from /ipfs/bafybeiemxf5abjwjbikoz4mc3a3dla6ual3jsgpdr4cjr3oz3evfyavhwq/wiki/ tho :)

Unrelated note about index.html

Below is not related to this PR, just something I noticed

Instead of returning index.html as payload for wiki/, it returns a redirect to wiki/index.html, which makes ugly URLs and breaks interop with go-ipfs.

this PR

$ curl -I -X GET http://127.0.0.1:9090/ipfs/bafybeiemxf5abjwjbikoz4mc3a3dla6ual3jsgpdr4cjr3oz3evfyavhwq/wiki/
HTTP/1.1 302 Found
location: /ipfs/bafybeiemxf5abjwjbikoz4mc3a3dla6ual3jsgpdr4cjr3oz3evfyavhwq/wiki/index.html

go-ipfs 0.4.19

$ curl -I -X GET http://127.0.0.1:8080/ipfs/bafybeiemxf5abjwjbikoz4mc3a3dla6ual3jsgpdr4cjr3oz3evfyavhwq/wiki/
HTTP/1.1 200 OK 

[ HTML PAYLOAD from index.html ]

I believe we should not redirect, and keep wiki/, just like go-ipfs does.

[vasco-santos]

@achingbrain Can you rebase this PR with master branch?

The travis CI was not configured yet in this repo and I just added it

[lidel]

LGTM, issues I described before were not related to this PR.

I re-tested it with in Brave (ipfs/ipfs-companion#716) with QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1mXWo6uco and works as expected now, wikipedia loads fine :)

[lidel]

@achingbrain let's push this over the finish line: mind rebasing on master to add green CI and get it merged? If you don't have bandwidth, want me to help with it?

@vasco-santos I opened #22 to track Sharding in IPNS without blocking this PR (which adds sharding support in /ipfs/ paths)

[vasco-santos]

@achingbrain I rebased the PR and some tests are failing. Are you able to have a look?

@lidel thanks for opening the issue for tracking sharding in IPNS :)

[lidel]

I've re-run Travis and the PR one passed, but the Branch one did not 🤔 🙃

[lidel]

@vasco-santos I re-tested locally and tests fail with Uncaught Error: Invalid version, must be a number equal to 1 or 0 because this PR depends on ipfs-inactive/js-ipfs-mfs#48 which is a dependency of ipfs-unixfs.

AFAIK @achingbrain is working on new release of ipfs-unixfs, so when that lands this PR needs to switch to new release and tests will pass.

[vasco-santos]

Thanks for your testing @lidel ! Let's wait for it then

[vasco-santos]

Can you rebase this PR? I think that rebasing this PR and using temporarily [email protected] should get CI green

[vasco-santos]

LGTM!