From bb256a7adcaf071bc59512e8f9e21b040f21cfbb Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Tue, 25 Jul 2023 14:59:43 +0200 Subject: [PATCH 1/7] ipip-428: Allowing V2-Only Records in IPNS initial draft for early feedback --- src/ipips/ipip-0428.md | 163 +++++++++++++++++++++++++++++++++++++ src/ipns/ipns-record.md | 172 +++++++++++++++++++++++++++++++++------- 2 files changed, 307 insertions(+), 28 deletions(-) create mode 100644 src/ipips/ipip-0428.md diff --git a/src/ipips/ipip-0428.md b/src/ipips/ipip-0428.md new file mode 100644 index 000000000..ac5beedad --- /dev/null +++ b/src/ipips/ipip-0428.md @@ -0,0 +1,163 @@ +--- +title: "IPIP-0428: Allowing V2-Only Records in IPNS" +date: 2023-07-24 +ipip: proposal +editors: + - name: Marcin Rataj + github: lidel + url: https://lidel.org/ + affiliation: + name: Protocol Labs + url: https://protocol.ai/ + - name: Henrique Dias + github: hacdias + url: https://hacdias.com/ + affiliation: + name: Protocol Labs + url: https://protocol.ai/ +relatedIssues: + - https://github.com/ipfs/specs/issues/376 +order: 428 +tags: ['ipips'] +--- + +## Summary + +Introduce support for creation and validation of compact, V2-only IPNS Records. + +## Motivation + +IPNS record creation and validation is overly complex due to the legacy of +decisions made in 2021. + +The "V1+V2" record creation and validation was reverse-engineered and documented +the current state in [ipfs/specs#319](https://github.com/ipfs/specs/pull/319), +and created a base for specifications to improve upon. + +A quick refresher on how IPNS Record lifecycle works today (2023 Q2): + +- Record Creation produces both V1 and V2 signatures, and the record has + duplicated values in both top level protobuf AND `data` CBOR field. + +- Record Validation only cares about V2 signature, but still requires fields + related to V1 to be always present in a record and match values from CBOR in + `data` field, for the record to be considered valid. + +We've been producing and expecting these hybrid V1+V2 records [since 2021](https://github.com/ipfs/js-ipns/pull/121). + +An unfortunate result is that all mandatory values MUST be duplicated, even +when both ends use a modern client that only cares about `signatureV2` that +guards CBOR field, otherwise the record will not be valid. + +What this IPIP aims to improve is allow implementations to produce lean, +V2-only IPNS records and ensure clients will interpret them as valid IPNS. + +## Detailed design + +Finish V2 story by making V2-Only records possible, namely: + +- creation: document and default to lean V2-Only records, keep V1+V2 as legacy + backward-compatible variant + +- validation: adjust the algorithm to stop requiring V1 fields when there is no + `signatureV1`. + +For details, see the updated :cite[ipns-record] specification. + +## Design rationale + +For modern IPNS, the outer `IpnsEntry` protobuf should effectively only have +two required fields: `data` and its `signatureV2`, and such record, as long +signature is valid, should be accepted as valid. + +At the same time, we don't want to break existig software, especially software +and hardware devices which use IPNS for pulling updates. + +We can get to that future in two steps: + +1. Reference implementations (go-ipfs, js-ipfs) will keep producing V1+V2 + records as backward-compatible default, but we adjust validation algorithm + to allow V2-only records, and support creation of such records as opt-in. + - Namely, only check/require fields to be duplicated in top level protobuf IF + `signatureV1` is present in the `IpnsEntry` protobuf. + - IF there is no `signatureV1`, the V1 record would be invalid anyway. + - IF there is no `signatureV1` but `signatureV2` and `data` fields + are present and valid, the V2-only record should be considered valid. + - This will allow people to build V2-only systems that produce records that + are considered valid. + +2. At some point in the future, e.g. when we see the majority of the public + swarm supports V2-Only records, libraries like go-ipns/js-ipfs and + implementations like Kubo stop producing V1+V2 and switch to publishing + V2-only records that are protobuf with only two fields: Data + CBOR+signatureV2 + + +### User benefit + +- End users: the main benefit for end user is smaller size of IPNS record and + less complex creation/validation of modern V2-only records. + +- Developers interested in IPNS: by making IPNS Record creation as simple as + "create DAG-CBOR with these fields, and sign it", and validation to + "signatureV2 should match the DAG-CBOR value and key". we've removed surface + for bugs, making it easier to reason about for use in greenfield projects. + +- IPFS ecosystem: lowering the complexity related to IPNS record creation and + validation makes it more likely for third-party interoperable IPNS + implementations to happen. + +### Compatibility + +- This is backward-compatible, we adjust validation logic to allow V2-only + records, but all V1+V2 records that are being used in the wild today are + still valid + +- V2-only rollout is not part of this IPIP. + - Our suggestion is to default to creating V1+V2 records for now, keeping + backward-compatibility with the old IPNS clients. + + - Creation of V2-only records should be introduced as an explicit opt-in. It + is up to implementations to decide when it is feasible to default to + creating V2-only records on IPNS publish. + +### Security + +- `IpnsEntry.signatureV1` (protobuf field) is parsed only by legacy clients, modern ones ignore this value + +It is highly advised to implement validation conformance tests using the fixtures +included at the end of this IPIP. + +### Alternatives + +Describe alternate designs that were considered and related work. + +1. Just switch to V2-only as the new default! + + - No, this would be a breaking change. We have to do this in two steps, + because we've rushed the way V2 was introduced in 2021, and STILL require + field copying, even when `signatureV1` is missing. So technically there was + never "V2", it was more like "V1.5". Only with this IPIP, we finally + adjust validation to only care about CBOR values when there is no `signatureV1`. + +2. Why keeping the outer protobuf envelope? Could we make IPNS DAG-CBOR-only? + + - Due to how long it takes for decentralized network nodes to upgrade, we prefer evolution rather than revolution. + - Protobuf is an useful envelope for two reasons: + 1. ensures the opaque V2-only record can be passed and stored in existing infrastructure + 2. allows us to evolve IPNS record ("V3") in the future without impacting existing infrastructure + +## Test fixtures + +TODO: IPNS record that is valid for 100 years and + +1. V1-only → record invalid +1. V1+V2 (both signatures valid) → record valid +1. V1+V2 (both signatures valid, but 'value' is different in V1) → record invalid +1. V1+V2 (only V1 valid) → record invalid +1. V1+V2 (only V2 valid, V1 missing signature) → record valid (because we do V1 validation only when signatureV1 is present in protobuf) +1. V2-only (only V2 valid) → record valid + +### Copyright + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). diff --git a/src/ipns/ipns-record.md b/src/ipns/ipns-record.md index 8edfe7293..5be866659 100644 --- a/src/ipns/ipns-record.md +++ b/src/ipns/ipns-record.md @@ -8,16 +8,31 @@ maturity: reliable editors: - name: Vasco Santos github: vasco-santos + affiliation: + name: Protocol Labs + url: https://protocol.ai/ - name: Steve Allen github: Stebalien + affiliation: + name: Protocol Labs + url: https://protocol.ai/ - name: Marcin Rataj github: lidel url: https://lidel.org/ + affiliation: + name: Protocol Labs + url: https://protocol.ai/ - name: Henrique Dias github: hacdias url: https://hacdias.com/ + affiliation: + name: Protocol Labs + url: https://protocol.ai/ - name: Gus Eggert github: guseggert + affiliation: + name: Protocol Labs + url: https://protocol.ai/ tags: ['ipns'] order: 0 --- @@ -116,31 +131,38 @@ and refer to IPNS addresses as `/ipns/{ipns-name}` (or `/ipns/{libp2p-key}`). A logical :dfn[IPNS Record] is a data structure containing the following fields: - **Value** (bytes) - - It can be any path, such as a `/ipns/{ipns-key}` path to another IPNS record, a [DNSLink](https://dnslink.dev/) path (`/ipns/example.com`) or an immutable IPFS path (`/ipfs/baf...`). - - Implementations MUST include this value in both `IpnsEntry.value` and inside the DAG-CBOR document in `IpnsEntry.data[Value]`. + - It can be any content path, such as a `/ipns/{ipns-key}` path to another IPNS record, a [DNSLink](https://dnslink.dev/) path (`/ipns/example.com`) or an immutable IPFS path (`/ipfs/baf...`). + - Implementations MUST include this value inside the DAG-CBOR document in `IpnsEntry.data[Value]`. + - **Validity Type** (uint64) - Defines the conditions under which the record is valid. - The only supported value is `0`, which indicates the `validity` field contains the expiration date after which the IPNS record becomes invalid. - - Implementations MUST support `validityType = 0` and include this value in both `IpnsEntry.validityType` and inside the DAG-CBOR document at `IpnsEntry.data[ValidityType]`. + - Implementations MUST support `ValidityType = 0` and include this value inside the DAG-CBOR document at `IpnsEntry.data[ValidityType]`. + - **Validity** (bytes) - - When `validityType = 0` + - When `ValidityType = 0` - Expiration date of the record with nanoseconds precision. Expiration time should match the publishing medium's window. - For example, IPNS records published on the DHT should have an expiration time set to within 48 hours after publication. Setting the expiration time to longer than 48 hours will not have any effect, as DHT peers only keep records for up to 48 hours. - Represented as an ASCII string that follows notation from :cite[rfc3339] (`1970-01-01T00:00:00.000000001Z`). - - Implementations MUST include this value in both `IpnsEntry.validity` and inside the DAG-CBOR document at `IpnsEntry.data[Validity]`. + - Implementations MUST include this value inside the DAG-CBOR document at `IpnsEntry.data[Validity]`. + - **Sequence** (uint64) - Represents the current version of the record (starts at 0). - - Implementations MUST include this value in both `IpnsEntry.sequence` and inside the DAG-CBOR document at `IpnsEntry.data[Sequence]`. + - Implementations MUST include this value in inside the DAG-CBOR document at `IpnsEntry.data[Sequence]`. + - **TTL** (uint64) - A hint for how long the record should be cached before going back to, for instance the DHT, in order to check if it has been updated. - - Implementations MUST include this value in both `IpnsEntry.ttl` and inside the DAG-CBOR document at `IpnsEntry.data[TTL]`. + - Implementations MUST include this value inside the DAG-CBOR document at `IpnsEntry.data[TTL]`. + - **Public Key** (bytes) - Public key used to sign this record. - If public key is small enough to fit in IPNS name (e.g., Ed25519 keys inlined using `identity` multihash), `IpnsEntry.pubKey` field is redundant and MAY be skipped to save space. - The public key MUST be included if it cannot be extracted from the IPNS name (e.g., legacy RSA keys). Implementers MUST follow key serialization defined in [PeerID specs](https://github.com/libp2p/specs/blob/master/peer-ids/peer-ids.md#key-types). + - **Signature** (bytes) - Provides the cryptographic proof that the IPNS record was created by the owner of the private key. - Implementations MUST include this value in `IpnsEntry.signatureV2` and follow signature creation and verification as described in [Record Creation](#record-creation) and [Record Verification](#record-verification). + - **Extensible Data** (DAG-CBOR) - Extensible record data in [DAG-CBOR](https://ipld.io/specs/codecs/dag-cbor/spec/) format. - The default set of fields can be augmented with additional information. @@ -168,42 +190,46 @@ message IpnsEntry { EOL = 0; } - // deserialized copy of data[Value] + // legacy V1 copy of data[Value] optional bytes value = 1; - // legacy field, verify 'signatureV2' instead + // legacy V1 field, verify 'signatureV2' instead optional bytes signatureV1 = 2; - // deserialized copies of data[ValidityType] and data[Validity] + // legacy V1 copies of data[ValidityType] and data[Validity] optional ValidityType validityType = 3; optional bytes validity = 4; - // deserialized copy of data[Sequence] + // legacy V1 copy of data[Sequence] optional uint64 sequence = 5; - // record TTL in nanoseconds, a deserialized copy of data[TTL] + // legacy V1 copy copy of data[TTL] optional uint64 ttl = 6; - // in order for nodes to properly validate a record upon receipt, they need the public - // key associated with it. For old RSA keys, its easiest if we just send this as part of - // the record itself. For newer Ed25519 keys, the public key can be embedded in the + // Optional Public Key to be used for signature verification. + // Used for big keys such as old RSA keys. Including the public key as part of + // the record itself makes it verifiable in offline mode, without any additional lookup. + // For newer Ed25519 keys, the public key is small enough that it can be embedded in the // IPNS Name itself, making this field unnecessary. optional bytes pubKey = 7; - // the signature of the IPNS record + // (mandatory V2) signature of the IPNS record optional bytes signatureV2 = 8; - // extensible record data in DAG-CBOR format + // (mandatory V2) extensible record data in DAG-CBOR format optional bytes data = 9; } ``` -:::issue +:::warning -For legacy reasons, some values must be stored in both `IpnsEntry` protobuf **and** `IpnsEntry.data` CBOR. -This should not be ignored, as it impacts interoperability with old software. +The `optional` keyword in Protocol Buffers (protobufs) indicates a field isn't +required for message exchange. However, even if a field is marked `optional` in +protobuf `message` syntax, an application such as IPNS may require it, making +it mandatory at the application level. -Opt-in lean IPNS Records are discussed in [ipfs/specs#376](https://github.com/ipfs/specs/issues/376). +Thus, despite protobuf rules, developers must heed application-level +requirements present in [Record Creation](#record-creation) section. ::: @@ -259,9 +285,91 @@ Finally, the network nodes may also republish their records, so that the records ### Record Creation -IPNS record MUST be serialized as `IpnsEntry` protobuf, and `IpnsEntry.data` MUST be signed using the private key. +IPNS record MUST be serialized as `IpnsEntry` protobuf, and `IpnsEntry.data` +MUST be signed (`IpnsEntry.signatureV2`) using the private key. + Creating a new IPNS record MUST follow the below steps: +1. Create `IpnsEntry` protobuf + +2. Create a DAG-CBOR document with values for `Value`, `Validity`, + `ValidityType`, `Sequence`, and `TTL` + + - Following [DAG-CBOR specification](https://ipld.io/specs/codecs/dag-cbor/spec/) is paramount. + The CBOR bytes will be used for signing and the serialized form must be + deterministic. + + - If you are updating an existing record, remember to increase values in + `sequence` and `validity` + +3. Store DAG-CBOR in `IpnsEntry.data`. + + - If you want to store additional metadata in the record, add it under + unique keys at `IpnsEntry.data`. + + - The order of fields impacts signature verification. If you are using an + alternative CBOR implementation, make sure the CBOR field order follows + :cite[rfc7049] sorting rules: length and then bytewise. The order of + fields impacts signature verification. + +4. If your public key can't be inlined inside the IPNS Name, include a + serialized copy in `IpnsEntry.pubKey` + + - This step SHOULD be skipped for Ed25519, and any other key types that are + small enough (32 bytes) to be inlined inside of [IPNS Name](#ipns-name) itself. + +5. Create `IpnsEntry.signatureV2` + + - Create bytes for signing by concatenating `ipns-signature:` prefix (bytes + in hex: `69706e732d7369676e61747572653a`) with raw CBOR bytes from + `IpnsEntry.data` + + - Sign concatenated bytes from the previous step using the private key, and + store the signature in `IpnsEntry.signatureV2` + +7. Confirm that the serialized `IpnsEntry` bytes sum to less than or equal to + [the size limit](#record-size-limit). + + +Created `IpnsEntry` protobuf includes signed `data` CBOR and optional public key: + +```protobuf +message IpnsEntry { + optional bytes pubKey = 7; + optional bytes signatureV2 = 8; + optional bytes data = 9; +} +``` + +The `IpnsEntry.data` CBOR document includes key-value pairs for `Value`, +`Validity`, `ValidityType`, `Sequence` and `TTL`. +Keys are sorted and serialized in order that follows +the [DAG-CBOR specification](https://ipld.io/specs/codecs/dag-cbor/spec/): + +```json +// IpnsEntry.data +{ + Sequence: … + TTL: … + Validity: … + ValidityType: … + Value: … +} +``` + +#### Record Creation: V1+V2 with Legacy V1 Signature + +:::warning + +Fields related to `signatureV1` has been deprecated since 2021. +V1 signatures are no longer used during record validation. + +However it may be necessary to create a V1+V2 record that allows legacy +software to use IPNS to upgrade itself to the latest version which supports V2 +signatures. In such case, follow the steps below. + +::: + 1. Create `IpnsEntry` and set `value`, `validity`, `validityType`, `sequence`, and `ttl` - If you are updating an existing record, remember to increase values in `sequence` and `validity` 2. Create a DAG-CBOR document with the same values for `Value`, `Validity`, `ValidityType`, `Sequence`, and `TTL` @@ -281,7 +389,7 @@ Creating a new IPNS record MUST follow the below steps: ### Record Verification -Implementations MUST resolve IPNS Names using only verified records. +Implementations MUST resolve IPNS Names only using verified records. Record's data and signature verification MUST be implemented as outlined below, and fail on the first error. 1. Before parsing the protobuf, confirm that the serialized `IpnsEntry` bytes sum to less than or equal to [the size limit](#record-size-limit). @@ -293,16 +401,24 @@ Record's data and signature verification MUST be implemented as outlined below, - Confirm Multihash type is `identity` - Unmarshall public key from Multihash digest 4. Deserialize `IpnsEntry.data` as a DAG-CBOR document -5. Confirm values in `IpnsEntry` protobuf match deserialized ones from `IpnsEntry.data`: +5. Create bytes for signature verification by concatenating `ipns-signature:` prefix (bytes in hex: `69706e732d7369676e61747572653a`) with raw CBOR bytes from `IpnsEntry.data` +6. Verify the signature in `IpnsEntry.signatureV2` against the concatenated result from the previous step. +7. If `IpnsEntry.signatureV1` or `IpnsEntry.value` is present, confirm the values in `IpnsEntry` protobuf match deserialized ones from `IpnsEntry.data`: - `IpnsEntry.value` must match `IpnsEntry.data[Value]` - `IpnsEntry.validity` must match `IpnsEntry.data[Validity]` - `IpnsEntry.validityType` must match `IpnsEntry.data[ValidityType]` - `IpnsEntry.sequence` must match `IpnsEntry.data[Sequence]` - `IpnsEntry.ttl` must match `IpnsEntry.data[TTL]` -6. Create bytes for signature verification by concatenating `ipns-signature:` prefix (bytes in hex: `69706e732d7369676e61747572653a`) with raw CBOR bytes from `IpnsEntry.data` -7. Verify signature in `IpnsEntry.signatureV2` against concatenation result from the previous step. +8. Check `Validity` + - If `ValidityType` is `0` (EOL) parse the `Validity` as an ASCII string + that follows notation from :cite[rfc3339] + (`1970-01-01T00:00:00.000000001Z`) and confirm it is bigger than the + current time. + +Value from `IpnsEntry.signatureV1` MUST never be used for signature verification. +Implementations MUST ensure `IpnsEntry.signatureV2` is used instead. -Value in `IpnsEntry.signatureV1` MUST be ignored. +Value from `IpnsEntry.value` MUST never be used unless it is the same as signed `IpnsEntry.data[Value]`. ## Integration with IPFS @@ -331,5 +447,5 @@ As the `pubsub` topics must be `utf-8` for interoperability among different impl ### Implementations -- +- - From 47bbbfb8d93f7837607c6a8dffc1b0b2855d9ff0 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Thu, 27 Jul 2023 00:33:21 +0200 Subject: [PATCH 2/7] ipip-428: apply editorial suggestions Co-authored-by: Henrique Dias --- src/ipips/ipip-0428.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/src/ipips/ipip-0428.md b/src/ipips/ipip-0428.md index ac5beedad..f33cd1890 100644 --- a/src/ipips/ipip-0428.md +++ b/src/ipips/ipip-0428.md @@ -56,10 +56,10 @@ V2-only IPNS records and ensure clients will interpret them as valid IPNS. Finish V2 story by making V2-Only records possible, namely: -- creation: document and default to lean V2-Only records, keep V1+V2 as legacy - backward-compatible variant +- Record Creation: document and default to lean V2-Only records, keep V1+V2 as legacy + backward-compatible variant. -- validation: adjust the algorithm to stop requiring V1 fields when there is no +- Record Validation: adjust the algorithm to stop requiring V1 fields when there is no `signatureV1`. For details, see the updated :cite[ipns-record] specification. @@ -67,7 +67,7 @@ For details, see the updated :cite[ipns-record] specification. ## Design rationale For modern IPNS, the outer `IpnsEntry` protobuf should effectively only have -two required fields: `data` and its `signatureV2`, and such record, as long +two required fields: `data` and its `signatureV2`, and such record, as long signature is valid, should be accepted as valid. At the same time, we don't want to break existig software, especially software @@ -90,17 +90,17 @@ We can get to that future in two steps: swarm supports V2-Only records, libraries like go-ipns/js-ipfs and implementations like Kubo stop producing V1+V2 and switch to publishing V2-only records that are protobuf with only two fields: Data - CBOR+signatureV2 + CBOR+signatureV2. ### User benefit -- End users: the main benefit for end user is smaller size of IPNS record and - less complex creation/validation of modern V2-only records. +- End users: the main benefit for end user is the smaller size of IPNS Records and + less complexity during creation/validation of modern V2-only records. - Developers interested in IPNS: by making IPNS Record creation as simple as "create DAG-CBOR with these fields, and sign it", and validation to - "signatureV2 should match the DAG-CBOR value and key". we've removed surface + "signatureV2 should match the DAG-CBOR value and key". We've removed surface for bugs, making it easier to reason about for use in greenfield projects. - IPFS ecosystem: lowering the complexity related to IPNS record creation and From 9bc0e383b14b095cbb96cdfd053b965277c7002f Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Thu, 17 Aug 2023 15:48:37 +0200 Subject: [PATCH 3/7] ipip-428: editorials --- src/ipips/ipip-0428.md | 31 ++++++++++++++++--------------- src/ipns/ipns-record.md | 27 ++++++++++++++++----------- 2 files changed, 32 insertions(+), 26 deletions(-) diff --git a/src/ipips/ipip-0428.md b/src/ipips/ipip-0428.md index f33cd1890..3a6dea323 100644 --- a/src/ipips/ipip-0428.md +++ b/src/ipips/ipip-0428.md @@ -17,6 +17,9 @@ editors: url: https://protocol.ai/ relatedIssues: - https://github.com/ipfs/specs/issues/376 + - https://github.com/ipfs/boxo/pull/339 + - https://github.com/ipfs/kubo/pull/9932 + - https://github.com/ipfs/js-ipns/pull/234 order: 428 tags: ['ipips'] --- @@ -92,7 +95,6 @@ We can get to that future in two steps: V2-only records that are protobuf with only two fields: Data CBOR+signatureV2. - ### User benefit - End users: the main benefit for end user is the smaller size of IPNS Records and @@ -133,19 +135,18 @@ included at the end of this IPIP. Describe alternate designs that were considered and related work. 1. Just switch to V2-only as the new default! - - - No, this would be a breaking change. We have to do this in two steps, - because we've rushed the way V2 was introduced in 2021, and STILL require - field copying, even when `signatureV1` is missing. So technically there was - never "V2", it was more like "V1.5". Only with this IPIP, we finally - adjust validation to only care about CBOR values when there is no `signatureV1`. + - No, this would be a breaking change. We have to do this in two steps, + because we've rushed the way V2 was introduced in 2021, and STILL require + field copying, even when `signatureV1` is missing. So technically there + was never "V2", it was more like "V1.5". Only with this IPIP, we finally + adjust validation to only care about CBOR values when there is no + `signatureV1`. 2. Why keeping the outer protobuf envelope? Could we make IPNS DAG-CBOR-only? - - Due to how long it takes for decentralized network nodes to upgrade, we prefer evolution rather than revolution. - - Protobuf is an useful envelope for two reasons: - 1. ensures the opaque V2-only record can be passed and stored in existing infrastructure - 2. allows us to evolve IPNS record ("V3") in the future without impacting existing infrastructure + - Protobuf is a useful envelope for two reasons: + 1. Ensures the opaque V2-only record can be passed and stored in existing infrastructure. + 2. Allows us to evolve IPNS record ("V3") in the future without impacting existing infrastructure. ## Test fixtures @@ -153,10 +154,10 @@ TODO: IPNS record that is valid for 100 years and 1. V1-only → record invalid 1. V1+V2 (both signatures valid) → record valid -1. V1+V2 (both signatures valid, but 'value' is different in V1) → record invalid -1. V1+V2 (only V1 valid) → record invalid -1. V1+V2 (only V2 valid, V1 missing signature) → record valid (because we do V1 validation only when signatureV1 is present in protobuf) -1. V2-only (only V2 valid) → record valid +1. V1+V2 (both signatures valid, but 'value' is different in V1 pb) → record invalid +1. V1+V2 (only signatureV1 valid) → record invalid +1. V1+V2 (only signatureV2 valid) → record valid +1. V2-only (no V1 fields) → record valid ### Copyright diff --git a/src/ipns/ipns-record.md b/src/ipns/ipns-record.md index 5be866659..2b77bca91 100644 --- a/src/ipns/ipns-record.md +++ b/src/ipns/ipns-record.md @@ -239,7 +239,7 @@ IPNS implementations MUST support sending and receiving a serialized `IpnsEntry` less than or equal to **10 KiB** in size. Records over the limit MAY be ignored. Handling records larger than the -limit is not recommended so as to keep compatibility with implementations and +limit is not recommended to keep compatibility with implementations and transports that follow this specification. ### Backward Compatibility @@ -316,7 +316,7 @@ Creating a new IPNS record MUST follow the below steps: serialized copy in `IpnsEntry.pubKey` - This step SHOULD be skipped for Ed25519, and any other key types that are - small enough (32 bytes) to be inlined inside of [IPNS Name](#ipns-name) itself. + small enough (32 bytes) to be inlined inside [IPNS Name](#ipns-name) itself. 5. Create `IpnsEntry.signatureV2` @@ -327,10 +327,9 @@ Creating a new IPNS record MUST follow the below steps: - Sign concatenated bytes from the previous step using the private key, and store the signature in `IpnsEntry.signatureV2` -7. Confirm that the serialized `IpnsEntry` bytes sum to less than or equal to +6. Confirm that the serialized `IpnsEntry` bytes sum to less than or equal to [the size limit](#record-size-limit). - Created `IpnsEntry` protobuf includes signed `data` CBOR and optional public key: ```protobuf @@ -357,14 +356,14 @@ the [DAG-CBOR specification](https://ipld.io/specs/codecs/dag-cbor/spec/): } ``` -#### Record Creation: V1+V2 with Legacy V1 Signature +#### Record Creation with Legacy SignatureV1 :::warning Fields related to `signatureV1` has been deprecated since 2021. V1 signatures are no longer used during record validation. -However it may be necessary to create a V1+V2 record that allows legacy +However, it may be necessary to create a V2+V1 record that allows legacy software to use IPNS to upgrade itself to the latest version which supports V2 signatures. In such case, follow the steps below. @@ -378,7 +377,7 @@ signatures. In such case, follow the steps below. - If you want to store additional metadata in the record, add it under unique keys at `IpnsEntry.data`. - The order of fields impacts signature verification. If you are using an alternative CBOR implementation, make sure the CBOR field order follows :cite[rfc7049] sorting rules: length and then bytewise. The order of fields impacts signature verification. 4. If your public key can't be inlined inside the IPNS Name, include a serialized copy in `IpnsEntry.pubKey` - - This step SHOULD be skipped for Ed25519, and any other key types that are inlined inside of [IPNS Name](#ipns-name) itself. + - This step SHOULD be skipped for Ed25519, and any other key types that are inlined inside [IPNS Name](#ipns-name) itself. 5. Create `IpnsEntry.signatureV2` - Create bytes for signing by concatenating `ipns-signature:` prefix (bytes in hex: `69706e732d7369676e61747572653a`) with raw CBOR bytes from `IpnsEntry.data` - Sign concatenated bytes from the previous step using the private key, and store the signature in `IpnsEntry.signatureV2` @@ -420,11 +419,13 @@ Implementations MUST ensure `IpnsEntry.signatureV2` is used instead. Value from `IpnsEntry.value` MUST never be used unless it is the same as signed `IpnsEntry.data[Value]`. -## Integration with IPFS +## Appendix: Notes for Implementers + +### Integration with IPFS Below are additional notes for implementers, documenting how IPNS is integrated within IPFS ecosystem. -### Local Record +#### Local Record This record is stored in the peer's repo datastore and contains the **latest** version of the IPNS record published by the provided key. This record is useful for republishing, as well as tracking the sequence number. A legacy convention that implementers MAY want to follow is to store serialized `IpnsEntry` under: @@ -433,7 +434,7 @@ A legacy convention that implementers MAY want to follow is to store serialized Note: Base32 according to the :cite[rfc4648]. -### Routing Record +#### Routing Record The routing record is spread across the network according to the available routing systems. The two routing systems currently available in IPFS are the [libp2p Kademlia DHT](https://github.com/libp2p/specs/tree/master/kad-dht) and :cite[ipns-pubsub-router]. @@ -445,7 +446,11 @@ The two routing systems currently available in IPFS are the [libp2p Kademlia DHT As the `pubsub` topics must be `utf-8` for interoperability among different implementations, IPNS over PubSub topics use additional wrapping `/record/base64url-unpadded(key)` -### Implementations +#### Reference Implementations + +When language-specific nuances are not covered by this specification, consider +below reference implementations as the baseline for making decisions around +interoperability. - - From cea7a1ecab0f9c2f951889818379c60013cc52e0 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Thu, 17 Aug 2023 16:18:58 +0200 Subject: [PATCH 4/7] chore: reference wip fixture work --- src/ipips/ipip-0428.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/ipips/ipip-0428.md b/src/ipips/ipip-0428.md index 3a6dea323..463d43f62 100644 --- a/src/ipips/ipip-0428.md +++ b/src/ipips/ipip-0428.md @@ -159,6 +159,8 @@ TODO: IPNS record that is valid for 100 years and 1. V1+V2 (only signatureV2 valid) → record valid 1. V2-only (no V1 fields) → record valid +Tracked in: + ### Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). From f4a2eb7c3fa97f510ae17e0855b130b65fa98c4f Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Tue, 12 Sep 2023 22:27:20 +0200 Subject: [PATCH 5/7] ipip-428: test vectors + final editorials --- src/ipips/ipip-0428.md | 42 +++++++++++++++++++++++++---------------- src/ipns/ipns-record.md | 2 +- 2 files changed, 27 insertions(+), 17 deletions(-) diff --git a/src/ipips/ipip-0428.md b/src/ipips/ipip-0428.md index 463d43f62..37488bd0b 100644 --- a/src/ipips/ipip-0428.md +++ b/src/ipips/ipip-0428.md @@ -1,7 +1,7 @@ --- title: "IPIP-0428: Allowing V2-Only Records in IPNS" date: 2023-07-24 -ipip: proposal +ipip: ratified editors: - name: Marcin Rataj github: lidel @@ -35,14 +35,14 @@ decisions made in 2021. The "V1+V2" record creation and validation was reverse-engineered and documented the current state in [ipfs/specs#319](https://github.com/ipfs/specs/pull/319), -and created a base for specifications to improve upon. +which created a base for specifications to improve upon. A quick refresher on how IPNS Record lifecycle works today (2023 Q2): -- Record Creation produces both V1 and V2 signatures, and the record has +- _Record Creation_ produces both V1 and V2 signatures, and the record has duplicated values in both top level protobuf AND `data` CBOR field. -- Record Validation only cares about V2 signature, but still requires fields +- _Record Validation_ only cares about V2 signature, but still requires fields related to V1 to be always present in a record and match values from CBOR in `data` field, for the record to be considered valid. @@ -78,9 +78,10 @@ and hardware devices which use IPNS for pulling updates. We can get to that future in two steps: -1. Reference implementations (go-ipfs, js-ipfs) will keep producing V1+V2 +1. Reference implementations (boxo/ipns, js-ipns) will keep producing V1+V2 records as backward-compatible default, but we adjust validation algorithm - to allow V2-only records, and support creation of such records as opt-in. + to allow V2-only records, and support creation of such records as opt-in in + modern implementations of IPFS+IPNS, like Kubo (GO) and Helia (JS). - Namely, only check/require fields to be duplicated in top level protobuf IF `signatureV1` is present in the `IpnsEntry` protobuf. - IF there is no `signatureV1`, the V1 record would be invalid anyway. @@ -90,8 +91,8 @@ We can get to that future in two steps: are considered valid. 2. At some point in the future, e.g. when we see the majority of the public - swarm supports V2-Only records, libraries like go-ipns/js-ipfs and - implementations like Kubo stop producing V1+V2 and switch to publishing + swarm supports V2-Only records, libraries like boxo/ipns, js-ipns and + implementations like Kubo will stop producing V1+V2 and switch to publishing V2-only records that are protobuf with only two fields: Data CBOR+signatureV2. @@ -150,16 +151,25 @@ Describe alternate designs that were considered and related work. ## Test fixtures -TODO: IPNS record that is valid for 100 years and +To make testing easier below are test vectors in form of IPNS records along +with the expected verification results. These test records are valid for 100 +years, making them safe for use in CI tests. -1. V1-only → record invalid -1. V1+V2 (both signatures valid) → record valid -1. V1+V2 (both signatures valid, but 'value' is different in V1 pb) → record invalid -1. V1+V2 (only signatureV1 valid) → record invalid -1. V1+V2 (only signatureV2 valid) → record valid -1. V2-only (no V1 fields) → record valid +1. [V1-only](https://github.com/ipfs/gateway-conformance/raw/9444cca76ef362904f1ce7a9c6d553c53da5feb3/fixtures/ipns_records/k51qzi5uqu5dm4tm0wt8srkg9h9suud4wuiwjimndrkydqm81cqtlb5ak6p7ku_v1.ipns-record) → record invalid +1. [V1+V2](https://github.com/ipfs/gateway-conformance/raw/9444cca76ef362904f1ce7a9c6d553c53da5feb3/fixtures/ipns_records/k51qzi5uqu5dlkw8pxuw9qmqayfdeh4kfebhmreauqdc6a7c3y7d5i9fi8mk9w_v1-v2.ipns-record) (both signatures valid) → record valid, value points at `/ipfs/bafkqaddwgevxmmraojswg33smq` +1. [V1+V2](https://github.com/ipfs/gateway-conformance/raw/9444cca76ef362904f1ce7a9c6d553c53da5feb3/fixtures/ipns_records/k51qzi5uqu5dlmit2tuwdvnx4sbnyqgmvbxftl0eo3f33wwtb9gr7yozae9kpw_v1-v2-broken-v1-value.ipns-record) (both signatures valid, but 'value' is different in V1 pb vs V2 CBOR) → record invalid +1. [V1+V2](https://github.com/ipfs/gateway-conformance/raw/9444cca76ef362904f1ce7a9c6d553c53da5feb3/fixtures/ipns_records/k51qzi5uqu5diamp7qnnvs1p1gzmku3eijkeijs3418j23j077zrkok63xdm8c_v1-v2-broken-signature-v2.ipns-record) (only signatureV1 valid) → record invalid +1. [V1+V2](https://github.com/ipfs/gateway-conformance/raw/9444cca76ef362904f1ce7a9c6d553c53da5feb3/fixtures/ipns_records/k51qzi5uqu5dilgf7gorsh9vcqqq4myo6jd4zmqkuy9pxyxi5fua3uf7axph4y_v1-v2-broken-signature-v1.ipns-record) (only signatureV2 valid) → record valid, value points at `/ipfs/bafkqahtwgevxmmrao5uxi2bamjzg623fnyqhg2lhnzqxi5lsmuqhmmi` +1. [V2-only](https://github.com/ipfs/gateway-conformance/raw/9444cca76ef362904f1ce7a9c6d553c53da5feb3/fixtures/ipns_records/k51qzi5uqu5dit2ku9mutlfgwyz8u730on38kd10m97m36bjt66my99hb6103f_v2.ipns-record) (no V1 fields) → record valid -Tracked in: +:::note + +Implementers can either write own tests against the above test vectors, or run +[gateway-conformance](https://github.com/ipfs/gateway-conformance/) test suite, +which includes tests for these vectors since +[gateway-conformance/pull/157](https://github.com/ipfs/gateway-conformance/pull/157). + +::: ### Copyright diff --git a/src/ipns/ipns-record.md b/src/ipns/ipns-record.md index 2b77bca91..d3617e6d9 100644 --- a/src/ipns/ipns-record.md +++ b/src/ipns/ipns-record.md @@ -3,7 +3,7 @@ title: IPNS Record and Protocol description: > Specifies the IPNS protocol in a language-agnostic manner, allowing everyone to create a compatible IPNS Record Publisher or Resolver. -date: 2023-02-13 +date: 2023-07-24 maturity: reliable editors: - name: Vasco Santos From 80c24c6e85e957e6825198f2e91b616dd96d84c5 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Tue, 12 Sep 2023 22:54:52 +0200 Subject: [PATCH 6/7] chore: switch to ipfs links --- src/ipips/ipip-0428.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/src/ipips/ipip-0428.md b/src/ipips/ipip-0428.md index 37488bd0b..7c9aba1b7 100644 --- a/src/ipips/ipip-0428.md +++ b/src/ipips/ipip-0428.md @@ -155,12 +155,12 @@ To make testing easier below are test vectors in form of IPNS records along with the expected verification results. These test records are valid for 100 years, making them safe for use in CI tests. -1. [V1-only](https://github.com/ipfs/gateway-conformance/raw/9444cca76ef362904f1ce7a9c6d553c53da5feb3/fixtures/ipns_records/k51qzi5uqu5dm4tm0wt8srkg9h9suud4wuiwjimndrkydqm81cqtlb5ak6p7ku_v1.ipns-record) → record invalid -1. [V1+V2](https://github.com/ipfs/gateway-conformance/raw/9444cca76ef362904f1ce7a9c6d553c53da5feb3/fixtures/ipns_records/k51qzi5uqu5dlkw8pxuw9qmqayfdeh4kfebhmreauqdc6a7c3y7d5i9fi8mk9w_v1-v2.ipns-record) (both signatures valid) → record valid, value points at `/ipfs/bafkqaddwgevxmmraojswg33smq` -1. [V1+V2](https://github.com/ipfs/gateway-conformance/raw/9444cca76ef362904f1ce7a9c6d553c53da5feb3/fixtures/ipns_records/k51qzi5uqu5dlmit2tuwdvnx4sbnyqgmvbxftl0eo3f33wwtb9gr7yozae9kpw_v1-v2-broken-v1-value.ipns-record) (both signatures valid, but 'value' is different in V1 pb vs V2 CBOR) → record invalid -1. [V1+V2](https://github.com/ipfs/gateway-conformance/raw/9444cca76ef362904f1ce7a9c6d553c53da5feb3/fixtures/ipns_records/k51qzi5uqu5diamp7qnnvs1p1gzmku3eijkeijs3418j23j077zrkok63xdm8c_v1-v2-broken-signature-v2.ipns-record) (only signatureV1 valid) → record invalid -1. [V1+V2](https://github.com/ipfs/gateway-conformance/raw/9444cca76ef362904f1ce7a9c6d553c53da5feb3/fixtures/ipns_records/k51qzi5uqu5dilgf7gorsh9vcqqq4myo6jd4zmqkuy9pxyxi5fua3uf7axph4y_v1-v2-broken-signature-v1.ipns-record) (only signatureV2 valid) → record valid, value points at `/ipfs/bafkqahtwgevxmmrao5uxi2bamjzg623fnyqhg2lhnzqxi5lsmuqhmmi` -1. [V2-only](https://github.com/ipfs/gateway-conformance/raw/9444cca76ef362904f1ce7a9c6d553c53da5feb3/fixtures/ipns_records/k51qzi5uqu5dit2ku9mutlfgwyz8u730on38kd10m97m36bjt66my99hb6103f_v2.ipns-record) (no V1 fields) → record valid +1. [V1-only](https://dweb.link/ipfs/bafybeifkipmlz2fehxda6y7x752uolfed7bdd46jzdammpfga5zrnkq33u/k51qzi5uqu5dm4tm0wt8srkg9h9suud4wuiwjimndrkydqm81cqtlb5ak6p7ku_v1.ipns-record) → record invalid +2. [V1+V2](https://dweb.link/ipfs/bafybeifkipmlz2fehxda6y7x752uolfed7bdd46jzdammpfga5zrnkq33u/k51qzi5uqu5dlkw8pxuw9qmqayfdeh4kfebhmreauqdc6a7c3y7d5i9fi8mk9w_v1-v2.ipns-record) (both signatures valid) → record valid, value points at `/ipfs/bafkqaddwgevxmmraojswg33smq` +3. [V1+V2](https://dweb.link/ipfs/bafybeifkipmlz2fehxda6y7x752uolfed7bdd46jzdammpfga5zrnkq33u/k51qzi5uqu5dlmit2tuwdvnx4sbnyqgmvbxftl0eo3f33wwtb9gr7yozae9kpw_v1-v2-broken-v1-value.ipns-record) (both signatures valid, but 'value' is different in V1 pb vs V2 CBOR) → record invalid +4. [V1+V2](https://dweb.link/ipfs/bafybeifkipmlz2fehxda6y7x752uolfed7bdd46jzdammpfga5zrnkq33u/k51qzi5uqu5diamp7qnnvs1p1gzmku3eijkeijs3418j23j077zrkok63xdm8c_v1-v2-broken-signature-v2.ipns-record) (only signatureV1 valid) → record invalid +5. [V1+V2](https://dweb.link/ipfs/bafybeifkipmlz2fehxda6y7x752uolfed7bdd46jzdammpfga5zrnkq33u/k51qzi5uqu5dilgf7gorsh9vcqqq4myo6jd4zmqkuy9pxyxi5fua3uf7axph4y_v1-v2-broken-signature-v1.ipns-record) (only signatureV2 valid) → record valid, value points at `/ipfs/bafkqahtwgevxmmrao5uxi2bamjzg623fnyqhg2lhnzqxi5lsmuqhmmi` +6. [V2-only](https://dweb.link/ipfs/bafybeifkipmlz2fehxda6y7x752uolfed7bdd46jzdammpfga5zrnkq33u/k51qzi5uqu5dit2ku9mutlfgwyz8u730on38kd10m97m36bjt66my99hb6103f_v2.ipns-record) (no V1 fields) → record valid :::note From 05b90835235128708b24226340b16d5073dd0389 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Tue, 12 Sep 2023 23:09:21 +0200 Subject: [PATCH 7/7] ipns: include test vectors section --- src/ipns/ipns-record.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/src/ipns/ipns-record.md b/src/ipns/ipns-record.md index d3617e6d9..d5b279362 100644 --- a/src/ipns/ipns-record.md +++ b/src/ipns/ipns-record.md @@ -421,6 +421,28 @@ Value from `IpnsEntry.value` MUST never be used unless it is the same as signed ## Appendix: Notes for Implementers +### Test Vectors + +Below are test vectors in [Record Serialization Format](#record-serialization-format), +along with the expected verification results. The records are valid for 100 +years, making them safe for use in CI tests. + +1. [V1-only](https://dweb.link/ipfs/bafybeifkipmlz2fehxda6y7x752uolfed7bdd46jzdammpfga5zrnkq33u/k51qzi5uqu5dm4tm0wt8srkg9h9suud4wuiwjimndrkydqm81cqtlb5ak6p7ku_v1.ipns-record) → record invalid +2. [V1+V2](https://dweb.link/ipfs/bafybeifkipmlz2fehxda6y7x752uolfed7bdd46jzdammpfga5zrnkq33u/k51qzi5uqu5dlkw8pxuw9qmqayfdeh4kfebhmreauqdc6a7c3y7d5i9fi8mk9w_v1-v2.ipns-record) (both signatures valid) → record valid, value points at `/ipfs/bafkqaddwgevxmmraojswg33smq` +3. [V1+V2](https://dweb.link/ipfs/bafybeifkipmlz2fehxda6y7x752uolfed7bdd46jzdammpfga5zrnkq33u/k51qzi5uqu5dlmit2tuwdvnx4sbnyqgmvbxftl0eo3f33wwtb9gr7yozae9kpw_v1-v2-broken-v1-value.ipns-record) (both signatures valid, but 'value' is different in V1 pb vs V2 CBOR) → record invalid +4. [V1+V2](https://dweb.link/ipfs/bafybeifkipmlz2fehxda6y7x752uolfed7bdd46jzdammpfga5zrnkq33u/k51qzi5uqu5diamp7qnnvs1p1gzmku3eijkeijs3418j23j077zrkok63xdm8c_v1-v2-broken-signature-v2.ipns-record) (only signatureV1 valid) → record invalid +5. [V1+V2](https://dweb.link/ipfs/bafybeifkipmlz2fehxda6y7x752uolfed7bdd46jzdammpfga5zrnkq33u/k51qzi5uqu5dilgf7gorsh9vcqqq4myo6jd4zmqkuy9pxyxi5fua3uf7axph4y_v1-v2-broken-signature-v1.ipns-record) (only signatureV2 valid) → record valid, value points at `/ipfs/bafkqahtwgevxmmrao5uxi2bamjzg623fnyqhg2lhnzqxi5lsmuqhmmi` +6. [V2-only](https://dweb.link/ipfs/bafybeifkipmlz2fehxda6y7x752uolfed7bdd46jzdammpfga5zrnkq33u/k51qzi5uqu5dit2ku9mutlfgwyz8u730on38kd10m97m36bjt66my99hb6103f_v2.ipns-record) (no V1 fields) → record valid + +:::note + +Implementers can either write own tests against the above test vectors, or run +[gateway-conformance](https://github.com/ipfs/gateway-conformance/) test suite, +which includes tests for these vectors since +[gateway-conformance/pull/157](https://github.com/ipfs/gateway-conformance/pull/157). + +::: + ### Integration with IPFS Below are additional notes for implementers, documenting how IPNS is integrated within IPFS ecosystem.