test: add ipns tests #26

vasco-santos · 2018-07-07T12:08:09Z

In the context of the js-ipfs PR for the IPNS working locally implementation, I made this PR.

PR's needed:

js-ipfs-repo#173
js-ipns#4
js-ipfs#1400
Release [email protected] with IPNS and replace the package.json dependency (actually pointing to master)
js-ipfs#157

test/ipns.js

Stebalien · 2018-07-24T03:43:37Z

I am having strange problems with this test. I need to have two go daemons running, in order to be able to publish an IPNS record. Otherwise, I receive Error: failed to find any peer in table.

You'll need to start the daemon in offline mode for that to work (pass --offline).

Stebalien · 2018-07-24T03:44:58Z

There is a problem with the repo in this test. I have also added a test for simulating this in the repo tests.
Basically, If we try to reuse to same repo previously used in JS (after running init) using a non-disposable daemon, the start fails with Error: required Datastore.Spec entry missing from config file. I tried to add the datastore config to the spawn method (in the repo test, but I am not sure if it is correct) but I couldn’t figure out the problem here.

The repos are currently not compatible. Go supports complex datastore configs and, as far as I know, javascript doesn't (yet).

Stebalien · 2018-07-24T03:49:37Z

In this test, I can use the same repo (id is equal for both), but cannot resolve the record. I am not sure if it is a problem of the JS implementation, or the datastore. I would like to debug the GO part, but is is running in an EXEC file. Any suggestion for debugging in the go side?

How are you storing the IPNS record currently? go-ipfs does will store the IPNS record in a libp2p "record" (https://github.com/libp2p/go-libp2p-record/blob/master/pb/record.proto) under "/" + base32("/ipns/" + binaryPeerID).

test/ipns.js

vasco-santos · 2018-07-24T11:15:32Z

@Stebalien

The repos are currently not compatible. Go supports complex datastore configs and, as far as I know, javascript doesn't (yet).

Yes, I already addressed that problem in ipfs/interop#29 and I am waiting for feedback.

How are you storing the IPNS record currently? go-ipfs does will store the IPNS record in a libp2p "record" (https://github.com/libp2p/go-libp2p-record/blob/master/pb/record.proto) under "/" + base32("/ipns/" + binaryPeerID).

Regarding this, I tried to reach you yesterday for discussing this.

I have been debugging the go implementation and I think that I understood the problem. Basically, as go-ipfs have a local + dht implementation, after publishing a record, you will have 3 distinct entries in your datastore. For instance, consider the following ones:

/F5UXA3TTF4JCA4L3WCGADRZNGPJYSNG5OBLYSSDFQKRNQZHCIMH24IZRO7MOH7K6
/F5YGWLYSEBYXXMEMAHDS2M6TRE2N24CXRFEGLAVC3BSOEQYPVYRTC56Y4P6V4
/ipns/CIQHC65QRQA4OLJT2OETJXLQK6EUQZMCULMGJYSDB6XCGMLX3DR72XQ

As I am doing the IPNS local implementation first, I only have the last one on js repo.

Accordingly, when I run the following command in go:

Ipfs name resolve —local

I expect that it only cares about the last entry and it does not happen in fact. At first, it tries to get the PublicKey here (https://github.com/ipfs/go-ipfs/blob/master/namesys/routing.go#L76), which fails. I commented that block, since it only checks if the public key exists, and proceeded, in order to understand if it was the only problem. However, I got another problem in here (https://github.com/ipfs/go-ipfs/blob/master/namesys/routing.go#L86).

Basically, both problems try to get from the datastore what was stored in the routing context: https://github.com/ipfs/go-ipfs/blob/master/namesys/publisher.go#L225

As it was the routing context, I didn’t implement that function, but it will be quick to add it if it is supposed. This way, I only want to ask you if this is in fact the correct approach, because if we run locally, we will immediately have the public key and the local record. Am I right, or missing something?

Moreover, I found another interop problem that I will work on today. I only noticed during this debug that you also publish an IPNS record during the init command, and I also have to add that part to JS pull request. Any more hidden usages of IPNS?

Stebalien · 2018-07-24T17:13:45Z

As I am doing the IPNS local implementation first, I only have the last one on js repo.

That last one isn't actually a part of the "routing" system, we put raw IPNS records there for republishing (both to remember the last-used sequence number and to remember which records we need to periodically republish).

The first record in that list is the IPNS record as stored in the routing system. That's the one we care about here.

The second record is the public key record. Now that we (a) embed the public key in the IPNS record and (b) don't sign the outer "routing" record, that's less important.

Stebalien · 2018-07-24T17:15:45Z

I commented that block, since it only checks if the public key exists, and proceeded, in order to understand if it was the only problem.

See the comment above that line. We need the public key to validate the record. Note: once everyone starts just embedding the public key in their IPNS record, we won't need that. Really, we might just want to not fail if that lookup fails and try getting the IPNS record anyways.

Stebalien · 2018-07-24T17:18:30Z

I only noticed during this debug that you also publish an IPNS record during the init command, and I also have to add that part to JS pull request.

We publish the empty directory but that's not really part of the IPNS spec. That's just something that go-ipfs does. We probably do this because we support mounting user's IPNS entry using fuse (under /ipns/local). I wouldn't say that's a strictly necessary feature.

Any more hidden usages of IPNS?

You'll have to make sure to republish IPNS entries once every ~4hr.

vasco-santos · 2018-08-08T11:08:49Z

PR was refactored, and the interop was also achieved!! 🙌

The first message of the PR was updated with the list of PRs needed for merging this one.

@Stebalien

A working version of **IPNS working locally** is here! 🚀 😎 💪 Steps: - [x] Create a new repo (**js-ipns**) like it was made with go in the last week ([go-ipns](https://github.com/ipfs/go-ipns)) and port the related code from this PR to there - [x] Resolve IPNS names in publish, in order to verify if they exist (as it is being done for regular files) before being published - [x] Handle remaining parameters in publish and resolve (except ttl). - [x] Test interface core spec [interface-ipfs-core#327](ipfs-inactive/interface-js-ipfs-core#327) - [x] Test interoperability with go. [interop#26](ipfs/interop#26) - [x] Integrate logging - [x] Write unit tests - [x] Add support for the lifetime with nanoseconds precision - [x] Add Cache - [x] Add initializeKeyspace - [x] Republish Some notes, regarding the previous steps: - There is an optional parameter not implemented in this PR, which is `ttl`, since it is still experimental, we can add it in a separate PR. Finally, thanks @Stebalien for your help and time answering all my questions regarding the IPNS implementation in GO. Moreover, since there are no specs, and not that much documentation, I have been writing a document with all the IPNS workflow. It is a WIP by now, but it is currently available [here](ipfs/specs#184). Related PRs: - [x] [js-ipns#4](ipfs/js-ipns#4) - [x] [js-ipfs-repo#173](ipfs/js-ipfs-repo#173) - [x] [js-ipfs#1496](#1496) - [x] [interface-ipfs-core#327](ipfs-inactive/interface-js-ipfs-core#327) - [x] enable `interface-ipfs-core` tests for IPNS in `js-ipfs`

daviddias · 2018-09-15T13:44:35Z

package.json

@@ -51,7 +51,7 @@
    "form-data": "^2.3.2",
    "go-ipfs-dep": "~0.4.17",
    "hat": "0.0.3",
-    "ipfs": "~0.31.6",
+    "ipfs": "github:ipfs/js-ipfs",


This has been released, you can :D

Yeah! Just found a problem blocking the interop to work 😞

js-ipfs#1570

~~@vasco-santos ipfs/js-ipfs#1570 now merged and released, can this now be updated so we can merge?~~ ignore, I just noticed you already updated. I'm going to review and re-run the tests

alanshaw · 2018-10-17T08:46:51Z

test/ipns.js

+    .spawn({
+      repoPath: dir,
+      disposable: false,
+      initOptions: { bits: 512 }


How come no args: ['--offline'] for the js daemon?

alanshaw · 2018-10-17T08:54:47Z

test/ipns.js

+  const stopPublisherAndStartResolverDaemon = (callback) => {
+    if (sameDaemon) {
+      return callback()
+    }


I'd move this check out of this function and into your series below and pass the two daemons in instead. I think it's unexpected for this function to not do any stopping or starting.

test/ipns.js

alanshaw · 2018-10-17T09:05:16Z

test/ipns.js

+    this.timeout(120 * 1000)
+    const dir = path.join(os.tmpdir(), hat())
+
+    parallel([


This has to be series for now, ipfsd-ctl will occasionally error for parallel spawns because of races for ports.

alanshaw · 2018-10-17T09:10:04Z

test/ipns.js

+  ], callback)
+}
+
+describe('ipns', () => {


This could be named better - these are IPNS repo interop tests, when I first started reviewing I had incorrectly assumed they were testing that IPNS records published on js could be resolved on go and vice versa over the network.

I changed it to: ipns locally using the same repo across implementations. What do you think?

alanshaw · 2018-10-17T09:11:53Z

test/ipns.js

+    this.timeout(160 * 1000)
+    const dir = path.join(os.tmpdir(), hat())
+
+    parallel([


Also series :(

ghost assigned vasco-santos Jul 7, 2018

ghost added the status/in-progress In progress label Jul 7, 2018

vasco-santos force-pushed the add-ipns-tests branch from 792473e to 97f10fd Compare July 7, 2018 12:09

vasco-santos commented Jul 7, 2018

View reviewed changes

test/ipns.js Show resolved Hide resolved

test/ipns.js Show resolved Hide resolved

test/ipns.js Show resolved Hide resolved

vasco-santos requested review from daviddias, Stebalien and whyrusleeping July 7, 2018 12:17

vasco-santos force-pushed the add-ipns-tests branch from 4b43129 to 835f166 Compare July 17, 2018 13:45

This was referenced Jul 21, 2018

Error: required Datastore.Spec entry missing form config file ipfs/js-ipfs-repo#163

Closed

fix: interoperability with go repo ipfs/js-ipfs-repo#173

Merged

Peer default config must have default Datastore config ipfs/js-ipfs#1460

Closed

Stebalien reviewed Jul 24, 2018

View reviewed changes

test/ipns.js Show resolved Hide resolved

test/ipns.js Show resolved Hide resolved

vasco-santos mentioned this pull request Jul 24, 2018

feat: ipns locally ipfs/js-ipfs#1400

Merged

16 tasks

vasco-santos force-pushed the add-ipns-tests branch 5 times, most recently from ade1db4 to 4483871 Compare August 8, 2018 11:03

vasco-santos force-pushed the add-ipns-tests branch from 4483871 to 47d521c Compare August 8, 2018 11:12

vasco-santos requested a review from alanshaw August 8, 2018 11:17

vasco-santos force-pushed the add-ipns-tests branch from 47d521c to 8b5d67e Compare August 30, 2018 13:33

vasco-santos changed the title ~~WIP: add ipns tests~~ test: add ipns tests Aug 30, 2018

vasco-santos force-pushed the add-ipns-tests branch from 8b5d67e to 248c4a2 Compare August 30, 2018 13:38

vasco-santos changed the title ~~test: add ipns tests~~ WIP: test: add ipns tests Aug 30, 2018

vasco-santos force-pushed the add-ipns-tests branch 2 times, most recently from 45b91f8 to 962ee23 Compare August 30, 2018 13:54

daviddias reviewed Sep 15, 2018

View reviewed changes

vasco-santos mentioned this pull request Sep 17, 2018

fix: revert libp2p records being signed for ipns ipfs/js-ipfs#1570

Merged

vasco-santos force-pushed the add-ipns-tests branch 4 times, most recently from 1a2504d to c374b7f Compare October 1, 2018 09:02

vasco-santos changed the title ~~WIP: test: add ipns tests~~ test: add ipns tests Oct 3, 2018

alanshaw suggested changes Oct 17, 2018

View reviewed changes

vasco-santos force-pushed the add-ipns-tests branch 3 times, most recently from 664da27 to 992cdbf Compare October 17, 2018 16:53

vasco-santos added 7 commits October 24, 2018 10:19

test: add ipns tests

9188ad4

fix: removed test

2fc9bbc

fix: ipns go -> js interoperable

ec32e72

chore: refactor

32f1714

fix: updated ipfs version

cae1689

chore: update js-ipfs version

65ef477

fix: code review

634e937

vasco-santos force-pushed the add-ipns-tests branch from 992cdbf to 634e937 Compare October 24, 2018 09:21

alanshaw approved these changes Oct 24, 2018

View reviewed changes

alanshaw merged commit 1c691e8 into master Oct 24, 2018

alanshaw deleted the add-ipns-tests branch October 24, 2018 10:34

ghost removed the status/in-progress In progress label Oct 24, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: add ipns tests #26

test: add ipns tests #26

vasco-santos commented Jul 7, 2018 •

edited

Loading

Stebalien commented Jul 24, 2018

Stebalien commented Jul 24, 2018

Stebalien commented Jul 24, 2018

vasco-santos commented Jul 24, 2018

Stebalien commented Jul 24, 2018

Stebalien commented Jul 24, 2018

Stebalien commented Jul 24, 2018

vasco-santos commented Aug 8, 2018 •

edited

Loading

daviddias Sep 15, 2018

vasco-santos Sep 17, 2018

alanshaw Oct 17, 2018 •

edited

Loading

alanshaw Oct 17, 2018

alanshaw Oct 17, 2018

vasco-santos Oct 17, 2018

alanshaw Oct 17, 2018

alanshaw Oct 17, 2018

vasco-santos Oct 17, 2018

alanshaw Oct 17, 2018

test: add ipns tests #26

test: add ipns tests #26

Conversation

vasco-santos commented Jul 7, 2018 • edited Loading

Stebalien commented Jul 24, 2018

Stebalien commented Jul 24, 2018

Stebalien commented Jul 24, 2018

vasco-santos commented Jul 24, 2018

Stebalien commented Jul 24, 2018

Stebalien commented Jul 24, 2018

Stebalien commented Jul 24, 2018

vasco-santos commented Aug 8, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alanshaw Oct 17, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vasco-santos commented Jul 7, 2018 •

edited

Loading

vasco-santos commented Aug 8, 2018 •

edited

Loading

alanshaw Oct 17, 2018 •

edited

Loading