feat: update IPNS specification

[lidel]

Closes #314

This PR aims to document implementable IPNS Record creation and validation.

I've documented IPNS record creation and verification based on what latest Kubo and JS-IPFS do, but without v1 signatures (as we want to remove support for them anyway).

Would appreciate additional 👁️ from implementers and people working with IPNS.

TODO

• Describe IPNS Name
  • explain what libp2p-key is and how tis protobuf looks like
  • explain when inlining makes sense
  • document String representation as CID
• Describe IPNS Record protobuf and DAG-CBOR part
  • extensible record data in strict DAG-CBOR
  • signatures v1 creation
  • signatures v2: creation and validation
• Describe IPNS Keys section
• reference DHT and PubSub specs
• backward-compatibility policy
• go over additional asks listed in Update IPNS specification #314

Open questions

• move everything to ipns/ ?
• is it ok to only require Ed25519?
  • it is MUST, we also have RSA which is 'SHOULD'
• is it ok to state that IpnsEntry.data DAG-CBOR (to avoid duplicating strictness spec from https://ipld.io/specs/codecs/dag-cbor/spec/)
  • yes, go-ipns uses dag-cbor from https://github.com/ipld/go-ipld-prime internally
• is it ok to simply not document v1 signatures?
  • yes, this is legacy which we want to go away (only relevant to Kubo and JS-IPFS)
• if we dont support v1, do we need to duplicate value, validity, validityType, sequence, and ttl (as IpnsEntry.* and in CBOR ar IpnsEntry.data)?
  • yes, for now we need to keep requirement of duplicated values – see rational in [DO NOT MERGE] poc: allow pure V2 records go-ipns#42

cc @2color @mrodriguez3313 @TMoMoreau @rvagg @vasco-santos

[Winterhuman]

Just to confirm, the spec states "Before parsing protobuf, confirm IpnsEntry size is less than 2 MiB", however, will IPFS implementations (specifically Kubo) likely only support smaller IPNS records? Is there a minimum max size of IPNS record that all implementations must support?

[lidel]

@Winterhuman afaik there is no explicit max size of IPNS record defined anywhere, but for sure big record wil lbreak something.
That is why I added this explicit number to this spec (so we can implement the same thing across implementations).

The only reason I picked 2 MiB is to be in the ballpark of the bitswap block limit (ipfs/kubo#8968)

[Winterhuman]

Ah I see! I do think a minimum max size is needed, it'd be nice to have a safe amount of data that can be inlined into IPNS records (by Value and/or by data) without any risk of being rejected by specific implementations, even like 4KiB or something would be good just so it's a definitive spec-defined size

[lidel]

Pushed some changes, expanding separate sections about "IPNS Name" and "IPNS Keys", formally decoupling IPNS spec from PeerIDs.

[aschmahmann]

😍 at this writeup. I didn't read through things too precisely for correctness, but left a few comments that might be useful.

[aschmahmann]

As mentioned in #273 (comment) 2MiB seems quite large (and won't work with IPNS over PubSub). If data is ever super large it could probably just point at a CID or another IPNS record instead.

Some ways it might be reasonable to establish a magic number here if we don't want to just spitball a number (which is also doable)

• What do records in the network currently look like? (e.g. use Hydras to analyze stored records)
• Look at something like a "maximum CID size" and add some more to it (e.g. double it). The indexer folks could probably give some insight here as would the related issues on this topic. Blocking on people committing to a maximum CID size seems unwise since that could take a while/not happen so we'd have to guesstimate here
• What's the longest path we've seen show up on ipfs.io
• Max public key size
  • Probably a 4096 RSA key, but you could ask the libp2p folks.
  • If people have really large keys we could either change the protocol to go back to having people find them separately from the records, or keep their values smaller to tradeoff against their key size and push towards smaller key types

---

Aside: it might also be worth considering switching the wording here to be similar to how Bitswap has block sizes specified.

specs/BITSWAP.md

Line 46 in c889e0b

Bitswap implementations must support sending and receiving individual blocks of sizes less than or equal to 2MiB. Handling blocks larger than 2MiB is not recommended so as to keep compatibility with implementations which only support up to 2MiB.

It's a bit of a different take, but could be useful to document (even just on GitHub) why we chose one approach over the other.

[lidel]

Things are more complicated: the introduction of the extensible IpnsEntry.data CBOR field means some people and services WILL start putting additional metadata there, and that means all bets and estimates will be off.

Still, we need to create some arbitrary headroom. It is easier to increase the limit than to decrease it.
I've switched to max to 10 kiB as it is closer to realistic needs, and adjusted the language to match bitswap one (see 1baf069).

Will try to gather some data from hydras, but in the meantime, @alanshaw / @vasco-santos do you have any metrics around record sizes on w3name? or thoughts on this matter?

[lidel]

Some quick numbers from hydras (records that are currently valid)

Total DynamoDB Records: 256697
  ipns: 12287
  pk:   244410

DynamoDB Record Value Sizes:
  avg:    299.62269913555673
  std:    45.88040769545226
  median: 299
  max:    2033
  min:    36

PK Records Key Types:
  rsa:       244403
  ed25519:   3
  secp256k1: 0
  ecdsa:     4

And just the extensible data field

IPNS 'IpnsEntry.Data' Size:
  avg:    139.27228998453873
  std:    9.81409166725822
  median: 141
  max:    182
  min:    91

10 KiB sounds ok, leaving some room for growth and experimentation without being blocked by implementations that follow this spec, but lmk if we should adjust.

[aschmahmann]

Seems mostly reasonable as this copies from the libp2p peerID spec, however RSA keys might still be very common since they used to be the default in implementations like kubo (formerly go-ipfs) and js-ipfs. This is the same as in the libp2p world, but given we're not just referencing that spec directly figured I'd call it out here in case anyone comes looking.

It'd be nice if we could post some numbers here (on GitHub) coming from the Hydra nodes on the distribution of key types justifying that Ed25519 covers most usage.

[lidel]

Will try to get % usage of Ed25519, RSA, Secp256k1 and ECDSA and get back to this with numbers.

[lidel]

Some quick numbers from hydras (records that are currently valid)

Total DynamoDB Records: 256697
  ipns: 12287
  pk:   244410

DynamoDB Record Value Sizes:
  avg:    299.62269913555673
  std:    45.88040769545226
  median: 299
  max:    2033
  min:    36
IPNS Records Signatures:
  v1:     12287
  v2:     6892
  with pk: 913
PK Records Key Types:
  rsa:       244403
  ed25519:   3
  secp256k1: 0
  ecdsa:     4

ed25519 is used by default and inlined, "3" is probably someone writing own IPNS tool and embedding it the same way RSA - can be ignored.

“with pk” means IPNS records with IpnsEntry.pubKey present.
RSA is dominating here, and used in ~7% of cases.

I think Ed25519 being ~93% and RSA ~7% is a good argument to keep Ed25519 as MUST and change RSA to SHOULD (same as in PeerID spec).

I adjusted language in 41612da

[lidel]

Some DHT stats

We've looked at some stats from Hydra nodes (a small sample of recently valid records):

IPNS Records Signatures:
  v1:       9513
  v2:       5821
  with pk:  733 (RSA)
  nil data: 3692 (V1-only)

Every V2 record is also V1 (V1-only records have no data field, so their count is in "nil data").
This means for the sample size of 9513 we have ~61% of V2+V1 and ~38% V1-only.
We also looked at other time-ranges and V2 is growing, V1-only is decreasing – seems to follow the usual slow update curve we also see for Kubo itself.

On this upgrade strategy

It is reasonable to expect both clients and publishers will update over time,
but the act of updating a publisher should not introduce any record resolution regressions for legacy nodes.

To make this clear, I've documented backward compatibility policy in b926e8c

• A legacy publisher MUST always be able to update to the latest implementation
  of this specification without breaking record resolution for legacy consumers.
• A legacy consumer MUST always be able to resolve IPNS name, even when publisher
  updated to the latest implementation of this specification.

This policy ensures the spec is aligned with existing code that was created to ensure IPNS implementations in GO and JS are backward-compatible from the perspective of legacy consumers:

• both Kubo and js-ipfs produce V1+V2 records
  • This allows legacy go-ipfs <0.9.0 node to correctly resolve IPNS records published by modern Kubo 0.15+
  • Maintaining this contract is important, I've added it to this spec
• Old go-ipfs <0.9.0 node publishing legacy RSA+V1-only records should be able to upgrade to Kubo 0.16 and start publishing V1+V2 records, allowing for V2 resolution on modern clients (but not break V1-only for legacy ones).

[paulgmiller]

does "resolve IPNS" name mean they should be able to validate the signature? Or just that they can read the record?
Isn't that already not the case since supporting all Key types isn't required there's already consumers that can't validate all signtures? And new key types (or signing prefixes) would invalidate old consumers?

[lidel]

does "resolve IPNS" name mean they should be able to validate the signature?

@paulgmiller Yes, IPNS resolution includes signature verification step: records that have invalid signatureV2 are ignored by implementations like Kubo or js-ipfs.

And new key types (or signing prefixes) would invalidate old consumers?

The use case we are protecting here is preexisting IPNS client resolving preexisting IPNS name for mission-critical updates.

In that case, the key type will not change, records are published with the same key, and the old signature must be present next to the new one to ensure the name still resolves for the legacy client.

An example scenario is as follows:

• IoT device implemented minimal support for RSA and V1, and uses it for polling for new firmware updates.
• It retrieves record with V1+V2 signatures, validates V1 and fetches new firmware
• The new firmware has support for IPNS V2, and uses a new IPNS name for polling for updates (requires V2 signatures, and can also use a different key type like Ed25519).

[lidel]

Merging since the hardened IPNS record verification shipped in:

• go-ipns v0.3.0
  • included in Kubo 0.16 (Release Notes)
• js-ipns v3.0.0
  • will be included in next JS-IPFS (tracked in chore: release master js-ipfs#4221)