Expire and republish records received via PUT_VALUE and ADD_PROVIDER

[raulk]

We:

1. Store such data permanently, thus deviating from Kademlia and creating weakness in our system for various reasons.
2. Don't rebalance/republish these records, allowing these records to disappear from the network as nodes leave and join.

See discussion in https://discuss.libp2p.io/t/dht-hourly-key-value-republish/93/4.

[vyzo]

this will be a problem for ipns, as the original publisher may be offline.
cc @Stebalien

[raulk]

Note that IPNS does republish the name record every 4 hours:

https://github.com/ipfs/go-ipfs/blob/42e191c0170a8e818c3302dab9b5afbf968df405/namesys/republisher/repub.go#L27

But the way I see it, this is to deal with network churn, not because the DHT peers are actively pruning the record. Even if the IPNS record carries the expiration, all the DHT sees is a []byte. Maybe via a Validator it validates a record is not expired, but this would happen only reactively: when processing a PUT_VALUE or a GET_VALUE request.

[dirkmc]

I want to make sure I understand the reason that Kademlia requires a node to republish its key/value pairs every hour. Is it so that as peers enter and leave the DHT the key/value is rebalanced onto the nodes closest to the key? If that's the case then it should make IPNS work better, because it will be more likely to find 16 copies of the key/value pair.

[raulk]

@dirkmc in addition to that, it helps keep storage footprint in check -- although by itself it's insufficient to deter DoS attacks exploiting that vector.

[Stebalien]

I want to make sure I understand the reason that Kademlia requires a node to republish its key/value pairs every hour.

The requirement comes from the fact that the network isn't exactly stable.

We should definitely be throwing away old records, we even make sure to record the time received so we can do this. The main issue is that we simply don't sweep the datastore.

[raulk]

@dirkmc if you're up for making a contribution, we can guide you through it ;-)

[dirkmc]

Sure, I can do so - I've been working on the Javascript DHT so I will do the same there

[jacobheun]

Should we also do the same for ADD_PROVIDER? It seems like it will suffer from the same issues if not republished.

[vyzo]

provider records are already collected after 24hs, and changing that would be a difficult change to deploy.

[Stebalien]

Closing #354 in favor of this:

There are two parts to republishing:

1. The original publisher should republish periodically (4-12 hours) and records should expire every 8-24 hours.
2. Nodes that have received a record should republish hourly.

See section 2.5 of https://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia-lncs.pdf.

[aarshkshah1992]

@Stebalien Do we plan to implement this anytime soon ?

[Stebalien]

I'd like to run a change like this against the real network for a while before merging it. Luckily, some of the test infrastructure we're working on will allow us to do that.

So yeah, go ahead! Just be warned that there may be a long road to merging it.

[aarshkshah1992]

@Stebalien When you say test infrastructure, are you talking about the testlab network ?

[Stebalien]

Yes.

[aarshkshah1992]

@Stebalien @raulk

Please can you assign this to me ?

There are two parts to republishing:

1. The original publisher should republish periodically (4-12 hours)

So, after a successful PutValue call to the network, the dht instance that is the original publisher will start a go-routine that keeps calling PutValue for that key every 4-12 hours ?
I see the paper mentions an interval of 24 hours or more. Why do we choose 4-12 hours ?

2. Records should expire every 8-24 hours.

The Kad paper simply states that they require an expiry interval of 24 hours for file sharing records but we ALREADY expire provider records after 24 hours. This issue is for the PutValue records & we currently have a MaxRecordAge of 36 hours. Why do we choose an interval of 8-24 ?

3. Nodes that have received a record should republish hourly.

See section 2.5 of https://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia-lncs.pdf.

If we choose a scan interval randomly (say between 60-75), the first peer that fires a republish resets the received time on the record on all other peers that are still among the K closest to the key. This should minimise the number of re-publish calls(This optimisation is mentioned in the paper).

[aarshkshah1992]

TODO

• Expire non-provider records in the datastore that are older than 12 hours.

• Original publisher should republish every 6 hours.

• Nodes that have received a record should republish every 60-75 minutes.

• Test with libp2p-testlab

[Stebalien]

I see the paper mentions an interval of 24 hours or more. Why do we choose 4-12 hours?

You're right, 24 hours should be fine. The 4-12 hours was necessary because we don't currently rebalance.

The Kad paper simply states that they require an expiry interval of 24 hours for file sharing records but we ALREADY expire provider records after 24 hours. This issue is for the PutValue records & we currently have a MaxRecordAge of 36 hours. Why do we choose an interval of 8-24 ?

That's also a good point. File sharing records make sense (really, the client should choose but we can handle that later) but we IPNS records can last longer (36 hours is reasonable).

If we choose a scan interval randomly (say between 60-75), the first peer that fires a republish resets the received time on the record on all other peers that are still among the K closest to the key. This should minimise the number of re-publish calls(This optimisation is mentioned in the paper).

SGTM.

[aarshkshah1992]

@bigs @Stebalien @raulk

This PR needs some love. Please take a look when you can :)

[Stebalien]

While implementing this feature, we've discovered a chain of dependencies. In reverse order:

1. Implement this feature.
2. To safely implement this feature, we need to ensure that a peer can't send out provider records on behalf of other peers. At the moment, peers only accept provider records from the peer providing the content. However, once we start supporting re-publishing, peers will need to re-publish records from other peers. To be secure, those records need to be signed by the providing peer.
3. To start ensure these signed records aren't too large, we need to switch to more space efficient signature algorithm (ed25519) by default. RSA keys & signatures are at least 256 bytes. Ed25519 keys & signatures are ~32 bytes.
4. Ed25519 peer IDs inline the public key in the per ID, instead of hashing it. Unfortunately, this makes them slightly larger (4 bytes) than RSA peer IDs (the current default). When encoded in multibase-base32, they encode to 65 bytes, which is 2 bytes over the maximum subdomain size. This breaks go-ipfs subdomain gateway feature where go-ipfs puts peer IDs in subdomains for IPNS support: https://bafz...ipns.io.
5. To shrink the peer ID sizes in subdomains, we've decided to switch to encoding them with base36 by default (in subdomains, in go-ipfs). When encoded with base36, ed25519 IPNS keys now just barely fit in a single subdomain.
6. To switch to base36, we need to define a base36 multibase, implement it in multiple languages, and ideally include this implementation in at least one go-ipfs and js-ipfs release before we start using it.

[dokterbob]

@Stebalien Thanks for the summery. If I read it well, considering the current state of affairs (at least go-ipfs), it seems that republishing is blocked by the lack of signatures for provider records. Is this correct?

Is there a current plan/spec/issue to implement signed provider records?

[Stebalien]

In reverse order, we've now finished steps 6-3. Unfortunately, the current go-ipfs team is tiny. Fortunately, this is now changing. Unfortunately, people currently moving back to the team are working towards other goals at the moment (@aarshkshah1992 will be working on NAT traversal, packet orientation, and connection establishment speed, for example).