False links are shown on the map

[zielmicha]

Either a bug in collection util or someone sends incorrect links on purpose.

[nsapa]

This node
http://h.fc00.org/#fccc:64de:a7a9:54ec:4812:b5b0:20a7:fbd8
only have one peer:
http://h.fc00.org/#fc87:f729:c42b:8a74:9e05:916f:7b80:6df4

I checked what sendGraph.py send ( http://pastebin.com/mq2TgJVH ) and it's correct.
Maybe you should store in the database whom added what.

[AwesomePatrol]

My Guess is that not only physical connections are shown/sent (from cjdns's README.md):

Routers have a responsibility to "keep in touch" with other routers that are physically close by and numerically near to their address.

My node has only 3 peers and even after I regenrated cjdroute.conf, restarted, etc. more peers appear on the map after a while.

[zielmicha]

It's not the case - each node typically contacts about 150 nodes. (and it
worked correctly before, even when there were more peers in the routing
table)

wt., 18.08.2015 o 00:25 użytkownik Aleksander Mistewicz <
[email protected]> napisał:

My Guess is that not only physical connections are shown/sent (from
cjdns's README.md):

Routers have a responsibility to "keep in touch" with other routers that
are physically close by and numerically near to their address.

My node has only 3 peers and even after I regenrated cjdroute.conf,
restarted, etc. more peers appear on the map
http://h.fc00.org/#fc06:6aac:efc3:9032:58b6:4d68:0ca0:dd05 after a
while.

—
Reply to this email directly or view it on GitHub
#4 (comment).

[AwesomePatrol]

It gets crazy.

Nodes: 469
Links: 4105

Try adding to database who add whom and reseting. Maybe accept graphs sent via hyperboria only?

[interfect]

My nodes have a bunch of fake links as well. Perhaps links should not be believed unless they come from several places.

[zielmicha]

I've added origin information to database. I believe that incorrent information was sent by ams.smash-net.org. Ping @Weuxel

@interfect Your approach is not resistant to sybil attacks.

[Weuxel]

Incorrect in what way? I am using your sendGraph.py script.

[interfect]

Nothing that crowdsources the data can be immune to Sybil attacks, can it?
Unless you add in an element like proof of work or signatures from the
involved nodes.

On Wed, Sep 2, 2015 at 10:38 AM, Norman Rieß [email protected]
wrote:

Incorrect in what way? I am using your sendGraph.py script.

—
Reply to this email directly or view it on GitHub
#4 (comment).

[zielmicha]

And now I got false links from other sources. This may be a bug in collection script.

[Kubuxu]

Confirming, h.n1.kubuxu.ovh should be linked to and only to h.vs1.kubuxu.ovh it is linked to two other.

[kyrias]

I've made a customized version of sendGraph.py that instead of using getLink to find the links it issues a getPeers against each node in the nodestore.This should theoretically prevent the false links, since the remote node should just reply to the getPeers call with its actual peers, but since it has to query each node instead of just getting info already known by cjdroute it's quite a lot slower.

The question is why cjdroute has false links in its tables though, or if it's some bug in the API causing it to give incorrect data by accident. It would be interesting to get a dump of the full nodestore and all of the links from a node that's submitted false links.

It would probably be a good idea to try to parallelize it somewhat though, to run multiple queries at once, but haven't done any work on that yet.

[kyrias]

For reference, it takes around 30 seconds on my laptop, with 130 nodes and it finds 226 edges.

[kyrias]

Another interesting question: Does every submission from the nodes sending false links contain those false links, or is it a once-off thing?

[zielmicha]

It seems that the errors are only for nodes that are far away (> 3 hops?). I don't know if every submission returns errors, but I guess most do.

[zielmicha]

@kyrias Can you share your modified script?

[kyrias]

Current version is available here, though I've only tested it with Python 3 and it requires hyperboria/python-cjdns.

[Kubuxu]

There are three options IMHO:

1. Try fixing cjdns (might be not worth it as routing algs will change in close future).
2. Use getPeers (quite slower).
   3 Implement some sort of filtering algorithm so if link is reported once or twice it is not shown.

It could be done by keeping track of how frequently on average links are reported ie. if each report tells us about 5% links in the network then if link is reported in only 0.5% of reports don't show it.

It might not be easy but either way will be beneficial.

[zielmicha]

I've updated the sendGraph.py script based on @Kubuxu sGNG.py. fc00.org will now reject submissions from old versions of the script, we'll see if new false links will be introduced.

[kyrias]

Well, mine, but hmm.. Would be nice if there was a simple way to test if a certain link exists.

Edit: Or, if we save the submission IP and timestamp of every submitted link, not necessarily in the DB, it might be easier to detect false links semi-automatically, hmm...

[Kubuxu]

Not mine but thanks. :p

also now you do not give option to add cjdns path by default. It would be nice to have it back.

[kyrias]

It's useless right now because it won't work with the python library in the cjdns repo, and for a git clone of the python-cjdns repo we should just let pip handle it instead.

[Kubuxu]

Ok, I just didn't know that cjdns lib is on pip.

[kyrias]

@zielmicha Hmm, I see 7 nodes without any links now.. Are they old nodes that are still there but their old links have expired, or are they new nodes that have submitted their node without any links?

[zielmicha]

I've removed all old nodes. If node doesn't respond to get_peers request it may be left without any links (get_peers returns empty set).

[kyrias]

Hmm, true, so basically, the empty nodes are cases where node A has node B in its nodestore, possibly it has it added as a peer. But node B happens to be down, but it's still in the NodeStore for node A. So when node A tries to submit data to fc00 it will add node B to its list of known nodes, but not add any links, since when querying node B for its peers it didn't respond...

This does begs a question though, if a node submitted doesn't have a link, should we drop the node instead of inserting it into the db?

---

Another thought, since we now only add links that have been confirmed by a remote node, maybe the timeout values for the graph generation should be adjusted somewhat.

A node timeout of 3 hours doesn't sound too bad, maybe up it to 4-5 in case some remote node doesn't submit data more often, since I've noticed that right now the number of nodes goes down a lot occasionally.

But I feel that the link timeout of 7 days is a bit high now that all links are guaranteed to be links that a remote node says that it is peered with. Lowering it to maybe around 1 day should make it fine if the link happens to be down for a little while, but makes the map more accurate overall.

Thoughts?

[zielmicha]

I've changed edge timeout to 1 day and node timeout to 5 hours.