IPNS resolution in the browser

[jsonsivar]

Hello - thanks again for your work on js-ipfs. Excited to be using and testing it, and hopefully contribute. Currently I'm trying to use the in-browser node to resolve an IPNS address and it is failing. I've looked at the existing issues similar to this that are now closed but still unable to get it working. Details below:

js-ipfs version: 0.41.2
go-ipfs version: 0.4.23
browser version: Chrome 80.0.3987.132

In-browser node setup:

const node = await window.Ipfs.create({
  config: {
    dht: {
      enabled: true,
    },
  }
});
  
await node.swarm.connect("/dns4/node1.mydomain.com/tcp/8001/wss/ipfs/QmZRgRJUxUgakETE76vvLGGCdzusdPtsTWD7THv8aJy1iJ");
await node.swarm.connect("/dns4/node2.mydomain.com/tcp/8001/wss/ipfs/Qmcbo9iTcFoqXMfL7vGapt4TnSB6QPvk16aBE87cDB4LFZ");

The 2 nodes I connect to above are go-ipfs nodes on version 0.4.23 and ran with the --enable-namesys-pubsub flag. They are able to publish and resolve IPNS addresses among themselves fine, it's just the browser node that is unable to do so.

So basically when I try

await node.name.resolve('/ipns/QmZRgRJUxUgakETE76vvLGGCdzusdPtsTWD7THv8aJy1iJ');

I get the following error:

I will keep digging into this and post more info as I find it. For now, it's failing here and when I print routingKey.toBuffer().toString() I get "/ipns/� ��d��z?�ꃑ�Y��U���U��,}�87�x1��". It jumped out since part of it is string encoded and the rest is in binary but haven't looked elsewhere yet to see if this is expected.

[lidel]

Your js-ipfs node running in browser is unable to resolve /ipns/{libp2p-key} to an IPNS record due to a gap in peer routing.

I believe the key problem is that experimental DHT in JS is not ready for production use yet (iirc js-ipfs postponed DHT work until DHT changes land around go-ipfs 0.6)

For now, use delegated routing modules which enable js-ipfs nodes to query DHT using remote go-ipfs node. Try setting it up as noted in https://github.com/ipfs/js-ipfs/tree/master/packages/ipfs#configuring-delegate-routers. It should solve the peer routing problem, just make sure you also have preload nodes set up as well, because your browser node won't be able to dial TCP multiaddrs to fetch the content.

(cc #2093)

[lidel]

Hm.. I take that back, seems that right now IPNS resolution is somehow hardcoded to use DHT directly:

• js-ipfs/packages/ipfs-core/src/ipns/routing/config.js

  Lines 30 to 36 in d13d15f

  	// DHT should not be added as routing if we are offline or it is disabled
  	if (get(options, 'offline') || !get(options, 'libp2p.config.dht.enabled', false)) {
  	const offlineDatastore = new OfflineDatastore(repo)
  	ipnsStores.push(offlineDatastore)
  	} else {
  	ipnsStores.push(libp2p._dht)
  	}

We probably could fix this up so delegated routing is used instead, enabling use in the browser.

cc @vasco-santos to think about this in Q2

[kumavis]

@lidel one issue here is that the libp2p api changed here libp2p/js-libp2p#480
libp2p.dht is now libp2p._dht

having some typedoc validation would be good to catch these sort of things
or perhaps having ipnsStores.push validate the interface its being passed in

[vasco-santos]

Bear in mind that the goal for js-libp2p users is to use libp2p.contentRouting for this, instead of using the .dht directly. That was the reason to go with the private dht property, since the DHT should be used for content routing if libp2p is configured that way, otherwise other contentRouting modules should be used.

[kumavis]

@vasco-santos makes sense
this is the quick fix #2996

[kumavis]

@vasco-santos seems like the current state of content-routing is
the specified read/write interface (provide, findProviders) https://github.com/libp2p/js-libp2p-interfaces/tree/afc2aa6891c2f71b0c62aabccdca25666a62da71/src/content-routing#api
but also a wrapper around the dht (put, get, getMany) (is there a name for this interface? datastore?)
is there an issue tracking ipns/dht/content-routing work?
I'm interested in helping but unsure where we are and dont want to conflict

[kumavis]

@vasco-santos I guess I'm confused because the content-routing interface spec seems to imply that it only provides providers. But libp2p.contentRouting exposes a (datastore?) put/get/getMany interface

also confused because ipfs doesn't seem to use contentRouting at all

[vasco-santos]

You make a valid point here, which I also hit sometime ago when refactoring the DHT. With that in mind, we cannot use delegated routing for the time being in the browsers, since they do not implement {put, get}, until you figure out what you are pointing here.

Ideally, I think that {put, get} should be part of the contentRouting interface as well. We do not have a naming for it, in the dht code I named it content-fetching but I am not happy with that name. What do you folks think?

[kumavis]

the name content-fetching makes sense to me next to content-routing, and its function is intuitive except maybe for the put case

a question:
is the content-fetching implementation unique to the content-routing implementation, or can it just be a generic pipeline of getProviders -> request data from providers

starting with get

yes, but no
dht's get uses bestRecord selection with dht.selectors
https://github.com/libp2p/js-libp2p-kad-dht/blob/a94f4d43ce7ab638b8d05e3af3d94b98d23c2fdf/src/content-fetching/index.js#L151
but by default its just a generic peer selector https://github.com/libp2p/js-libp2p-kad-dht/blob/a94f4d43ce7ab638b8d05e3af3d94b98d23c2fdf/src/index.js#L143

yesish because of desired side effects
after bestRecord selection we send our correction records
https://github.com/libp2p/js-libp2p-kad-dht/blob/a94f4d43ce7ab638b8d05e3af3d94b98d23c2fdf/src/content-fetching/index.js#L166

now for getMany

yes, as an optimization
because we make a special query for getting values
https://github.com/libp2p/js-libp2p-kad-dht/blob/a94f4d43ce7ab638b8d05e3af3d94b98d23c2fdf/src/content-fetching/index.js#L234
and exit the query path based on how many values we've seen
https://github.com/libp2p/js-libp2p-kad-dht/blob/a94f4d43ce7ab638b8d05e3af3d94b98d23c2fdf/src/content-fetching/index.js#L254-L257
which is ultimately backed by a different network call
https://github.com/libp2p/js-libp2p-kad-dht/blob/f0fb2122caa10cf38cb7a5c06a0de1cc56d5830b/src/index.js#L521
than when just finding providers
https://github.com/libp2p/js-libp2p-kad-dht/blob/a94f4d43ce7ab638b8d05e3af3d94b98d23c2fdf/src/content-routing/index.js#L25

you could seemingly use the generic pipeline just fine but would result in either

1. extra network requests to found providers to actually get the data
2. or extra network io if you asked for the value but didnt actually need it

ok I answered my own question thank you 😁

sorry for the thread spam

[vasco-santos]

@kumavis thanks for exposing your thoughts here, this is valuable!

I would like to see us decoupling the dht in the future into the contentRouting and peerRouting, so that we can integrate it in the libp2p, like the delegated modules, as well as other future implementations. However, we need to fill the gap of the put, get, getMany! Since contentRouting modules might not need to implement the latest methods, it will probably make sense to create this new interface called content-fecthing, or anything better named. This would be what would be used for the IPNS here.

But we get to the same issue, since we intended to unblock IPNS resolution in the browser, which is not possible now with the delegated-content-routing, unless it provides the put and get. The other options, is to have a new delegated module for put and get if it makes sense.

cc @jacobheun since we will probably spend some time on the content routing side of things next quarter, this is something that I feel that we need to solve.

[autonome]

@vasco-santos Is this picture different now that we've shipped new content routing to the world and also shipped delegate nodes in js?

[vasco-santos]

However, we need to fill the gap of the put, get, getMany!

Delegate nodes don't do this. They allow findPeer, provide and findProviders.

The conversation here was going on a way to create the same concept for put and get, which are needed for IPNS. However, with the latest discussions and progress made, the way will be to use the DHT as client mode.

So, in theory this should work now if a browser node enables DHT client mode as explained in https://github.com/ipfs/blog/blob/master/content/post/104-js-ipfs-0.48.md#-dht-configuration

However, this still uses the incomplete and not stable DHT implementation for the time being. I would say that it is ready for experimenting with.

[rysiekpl]

I am trying to get inbrowser IPNS to work, and can't seem to. Tried with:

ipfs = await self.Ipfs.create({
            libp2p: {
                config: {
                    dht: {
                        enabled: true,
                        clientMode: true
                    }
                }
            }
        });

...and...

ipfs = await self.Ipfs.create({
            config: {
                dht: {
                    enabled: true,
                }
            }
        });

...and using both. Each time IPNS resolution (ipfs.name.resolve()) just fails with:

Error: "record requested for <CID> was not found in the network"

Would love some pointers what to try.

[vasco-santos]

@rysiekpl
I recommend that you use the ipns over pubsub for the time being, while we work on improving the content routing and discoverability in the browser context.
This example should help you out: https://github.com/ipfs-examples/js-ipfs-examples/blob/master/examples/browser-ipns-publish/
Let me know if you hit any issues

[rysiekpl]

Thanks for the link. Sorry, I should have been more clear: in the browser I am only trying to get the data from IPNS/IPFS, not publish it. And one crucial requirement is that it needs to work without me running a dedicated IPFS node that the browser code connects to.

I am trying to build a censorship circumvention tool using ServiceWorkers and IPFS. The idea is that the ServiceWorker code fetches the content from IPFS if a regular HTTPS fetch() fails for whatever reason (using IPNS to figure out what are the IPFS addresses of the freshest content):
https://samizdat.is/

So, requiring that the ServiceWorker code connects to an IPFS node running on the (potentially censored) host is not really workable. Using a list of public nodes however - that could work.

[DougAnderson444]

@rysiekpl have you checked out IPFS http client? You can connect to a remote ipfs node then do core API actions like await ipfs.resolve(name) and for await (const file of ipfs.get(cid)). Questions like this should be take over to https://discuss.ipfs.io/

[lidel]

@rysiekpl thank you for mentioning your project (https://samizdat.is / 0xacab.org/rysiek/samizdat/). It is a good example of how resolving IPNS in the browser context is much more important than publishing (cc @Gozala @autonome)

ps. if you want additional, HTTP-based resolution fallback there is a list of public gateways here, most expose /ipns and a subset of /api/v0 but ymmv

[rysiekpl]

@rysiekpl have you checked out IPFS http client? You can connect to a remote ipfs node then do core API actions like await ipfs.resolve(name) and for await (const file of ipfs.get(cid)).

Yes, I looked at it. But in my case it does not seem to provide any benefit over just using IPFS gateways (which is now implemented as a Samizdat plugin), apart from standardizing the API between using gateways and embedded js-ipfs node.

Since the embedded node does not seem to have ipns.name.resolve() working at all, it kind of doesn't make sense for me to switch to it now, other than prototyping. I've been burnt by IPFS API changes before, so until some equivalent of ipns.name.resolve() works in the browser, it doesn't make sense for me to dive into it, I feel.

Questions like this should be take over to https://discuss.ipfs.io/

Eh, Yet Another Forum... There's a cost to joining any new community like that, and currently I would prefer to avoid that, since this issue seems to be related exactly to the problem I am having. Unless I am missing something?

@rysiekpl thank you for mentioning your project (https://samizdat.is / 0xacab.org/rysiek/samizdat/). It is a good example of how resolving IPNS in the browser context is much more important than publishing (cc @Gozala @autonome)

Yeah, sounds about right.

ps. if you want additional, HTTP-based resolution fallback there is a list of public gateways here, most expose /ipns and a subset of /api/v0 but ymmv

Already using five hard-coded gateways in the gateway-ipns plugin. At the current stage (PoC being developed into a beta) this should be enough.

Anyway, my takeaway from this is that IPNS name resolution in the browser (using the js-ipfs node, not piggy-backing on gateways) is not something that is implemented yet. Correct?

[Weedshaker]

Just wanted to enter this conversation for testing purposes, once some solution becomes usable. For the time being I still get: An Error occured! Error: record requested for Qm... was not found in the network
I think ipns is the most important feature of ipfs and I hope that it will find its way into the js-ipfs without workarounds. Also, ipns on go is kinda slow, I really wish that this thing becomes faster, only ipns will allow advanced ipfs use cases and working around this with a centralized service is no option for the https://peerweb.site nor myself.

[rysiekpl]

Just checking in, any movement on this? I am going back to hacking on Samizdat and having IPNS resolution working directly and reliably in the browser would be a huge boon for the project.

Thanks!

[lidel]

Hi all. Some stuff landed, some got improved. My understanding of the current situation is that we should be able to close the gap and make IPNS fully functional in the browser out-of-the-box (fully functional in default configuration, no need to opt-in anything), if at least one of below lands:

• (A) pubsub is enabled by default in both js-ipfs and go-ipfs (right now it is opt-in experiment in go)
  • tracked Enable pubsub by default kubo#6621
• (B) js-ipfs uses Delegates for IPNS publishing/resolution when in the browser context
  • tracked in feat: use preload nodes for delegated IPNS operations #3722 ← poking at this is probably the low-hanging fruit here

I believe (B) is easier to accomplish, as it does not depend on pubsub in browser and go-ipfs, and we already have Delegates in js-ipfs/src/runtime/config-browser.js#L12-L16 and the remaining work is to wire things up and extend delegate modules if necessary.

[rysiekpl]

Hey hey, any news on this?

[BigLep]

2022-03-18 conversation: this just needs someone to review. Everyone has been fully booked on other things. Potential candidates are:

• @lidel
• @achingbrain
• @vasco-santos

[rysiekpl]

Ping again? Is there anything a person not deeply involved in the project can do to help get this fixed?

[tinytb]

We're focusing on other work such as Helia and don't have the bandwidth to pick this up in the short term. For others to help get this fixed:

• Check out the repo
• Replicate the problem:
  • Was the record in the network? Was it in peers that the browser couldn't connect to?
  • If the record is on a node that only has quic or tcp, then you won't be able to connect
  • In the browser, you can query the DHT, and turn on logging to confirm that your node is querying nodes that are closer to the one holding the record
  • If you can't dial peers that hold the record, then you'll get the error message in the original post. In that case, the code is working as intended. If the browser is trying to find an IPNS record but can't connect to the nodes that have it, then the error is the expected behavior.
  • The solution would be to use delegated content routing so that Kubo can do the queries and supply the record
• Address / fix the problem
• Add tests to ensure no regression

[SgtPooki]

js-ipfs is being deprecated in favor of Helia. You can #4336 and read the migration guide.

Please feel to reopen with any comments by 2023-06-02. We will do a final pass on reopened issues afterward (see #4336).

Assigning to @achingbrain to answer whether this issue is already resolved in Helia, or if this issue needs to be migrated to that repo!

[achingbrain]

@helia/ipns uses the libp2p ContentRouting interface to resolve content (including the get operations necessary to support IPNS) so there's nothing further to do on that front, the question is more what Content Routers do you plug into it.

The @libp2p/delegated-content-routing module is an implementation of this interface which should let browser nodes resolve IPNS names from Kubo nodes.

I don't think there's a spec yet for delegated IPNS resolution in the same way there is for resolving content providers via IPNI or HTTP Routing V1 but when there is an implementation will unlock this functionality against more than just Kubo nodes.

[BigLep]

I don't think there's a spec yet for delegated IPNS resolution in the same way there is for resolving content providers via IPNI or HTTP Routing V1

There is an active IPIP that I expect will close out in the next ~month once an implementation of it is made in Boxo: ipfs/specs#379

[achingbrain]

I was just about to link to the same one!