Skip to content
This repository has been archived by the owner on Feb 12, 2024. It is now read-only.

fix: enable preload on MFS commands that accept IPFS paths #2355

Merged
merged 3 commits into from
Aug 23, 2019

Conversation

alanshaw
Copy link
Member

@alanshaw alanshaw commented Aug 14, 2019

@achingbrain please could you verify these are the methods that accept IPFS paths? Also, do they accept CID instances or CID strings or "cid paths" as I've allowed here or is it only IPFS paths? Verified myself - it is IPFS paths (Strings) and CID instances.

TODO:

  • Add tests

fixes #2335

@@ -32,15 +34,37 @@ module.exports = self => {
repoOwner: self._options.repoOwner
})

const withPreload = fn => (...args) => {
const cids = args.reduce((cids, p) => {
if (isIpfs.ipfsPath(p)) {
Copy link
Member

@lidel lidel Aug 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC /ipns/ paths are not supported by files.stat.

That is why HTTP gateway resolves /ipns//ipfs/ before passing it to stat:

// The resolver from ipfs-http-response supports only immutable /ipfs/ for now,
// so we convert /ipns/ to /ipfs/ before passing it to the resolver ¯\_(ツ)_/¯
// This could be removed if a solution proposed in
// https://github.com/ipfs/js-ipfs-http-response/issues/22 lands upstream
let ipfsPath = decodeURI(path.startsWith('/ipns/')
? await ipfs.name.resolve(path, { recursive: true })
: path)

That being said, is it possible /ipns/ paths will be supported in the future, as suggested in ipfs/js-ipfs-http-response#22? If so, a future-proof check would be:

Suggested change
if (isIpfs.ipfsPath(p)) {
if (isIpfs.path(p)) {

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'd also have to add the resolve step to this PR if we wanted to "future-proof" the check...right now it would end up sending Peer IDs to the preload nodes...

Copy link
Member

@lidel lidel Aug 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, preload call for /ipns/ path will work fine. Check with:

$ curl "https://node0.preload.ipfs.io/api/v0/refs?arg=/ipns/docs.ipfs.io&r=true"

Above is DNSLink, however I think preload of real IPNS (with peerid) will be actually beneficial by making the key available in more places.

IPNSsupport does not need to land/block this PR. Let's merge is asap.

License: MIT
Signed-off-by: Alan Shaw <[email protected]>
@alanshaw alanshaw marked this pull request as ready for review August 21, 2019 11:02
@alanshaw
Copy link
Member Author

Note: Tests are all passing on CI. The bundlesize check is currently failing due to a permissions error.

Copy link
Member

@lidel lidel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, works as expected.

Sidenote: do we know what will be the impact this change on default preload nodes? Watching logs when running DEBUG="*preload*" jsipfs daemon and then opening http://127.0.0.1:9090/ipns/tr.wikipedia-on-ipfs.org/wiki/Mars.html is pretty intense.

@@ -32,15 +34,37 @@ module.exports = self => {
repoOwner: self._options.repoOwner
})

const withPreload = fn => (...args) => {
const cids = args.reduce((cids, p) => {
if (isIpfs.ipfsPath(p)) {
Copy link
Member

@lidel lidel Aug 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, preload call for /ipns/ path will work fine. Check with:

$ curl "https://node0.preload.ipfs.io/api/v0/refs?arg=/ipns/docs.ipfs.io&r=true"

Above is DNSLink, however I think preload of real IPNS (with peerid) will be actually beneficial by making the key available in more places.

IPNSsupport does not need to land/block this PR. Let's merge is asap.

@lidel
Copy link
Member

lidel commented Aug 21, 2019

While testing this PR in Brave I noticed multiple requests for the same CID blocking browser's request queue.

Potential mitigations:

  • cache/skip subsequent preload calls of the same CID for some time
    (probably the best way to handle this)
  • our preload nodes (*.preload.ipfs.io) should return HTTP headers to enable time-based http caching in web browser and/or cache result at Nginx level for a few minutes to decrease load on go-ipfs

Copy link
Member

@lidel lidel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As noted on IRC we have a problem of preloads turned out to be too eager:

Opening

/ipns/tr.wikipedia-on-ipfs.org/wiki/Mars.html

triggers preload of DNSLink root (QmT5NvUtoM5nWFfrQdVrFtvGfKFmG7AHE8P34isapyhCxX) instead of Mars.html alone

License: MIT
Signed-off-by: Alan Shaw <[email protected]>
@alanshaw alanshaw requested a review from lidel August 21, 2019 17:19
Copy link
Member

@lidel lidel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested fixup with Brave, preload works as expected now 👍

FYSA: preload of sharded paths such as /ipns/tr.wikipedia-on-ipfs.org/I/m/Mars_Hubble.jpg returns ERROR 500 but that should get fixed when preload nodes are updated to go-ipfs with the fix from ipfs/kubo#6601

@alanshaw alanshaw merged commit 0e0d1dd into master Aug 23, 2019
@alanshaw alanshaw deleted the fix/enable-preload-on-mfs branch August 23, 2019 09:21
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Enable preload on MFS commands that accept IPFS paths
2 participants