Hash raw content directly for CIP-100, CIP-108, CIP-119 (#835)

CIP-0100/README.md

@@ -41,15 +41,15 @@ This section outlines the high level format and requirements of this standard.
       NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED",  "MAY", and
       "OPTIONAL" in this document are to be interpreted as described in
       [RFC 2119](https://www.rfc-editor.org/rfc/rfc2119).
-      
+
 - On chain metadata actions in CIP-1694 (and likely any alternative proposals) have the notion of an "Anchor"; this is the URL and a hash for additional, non-ledger metadata about the action.
 - Tools which publish governance related transactions SHOULD publish metadata via these fields.
 - While that content MAY be in any format, following any standard or non-standard, for the remainder of this document SHOULD/MUST will refer to documents that are following this specification.
 - The content hosted at the anchor URL MUST be a JSON-LD document according to the rest of this specification.
 - This document SHOULD include `@context` and `@type` fields below to aid in interpretation of the document.  
 - The JSON document SHOULD be formatted for human readability, for the sake of anyone who is manually perusing the metadata.
 - That content SHOULD be hosted on a content addressable storage medium, such as IPFS or Arweave, to ensure immutability and long term archival.
-- The hash in the anchor MUST be the hash of the canonicalized form of the metadata document, using the hashing algorithm specified in the `hashAlgorithm` field. Currently only blake2b-256 is supported.
+- The hash in the anchor MUST be the hash of the the raw bytes of the content, using the hashing algorithm specified in the `hashAlgorithm` field. Currently only blake2b-256 is supported.
 - For the purposes of hashing and signature validation, we should use the [canonical RDF triplet representation](https://www.w3.org/TR/rdf-canon/), as outlined in the JSON-LD specification.
 
 ### Versioning
@@ -148,7 +148,8 @@ Additionally, we highlight the following concepts native to json-ld that are use
 
 When publishing a governance action, the certificate has an "anchor", defined as a URI and a hash of the content at the URI.
 
-For CIP-0100 compliant metadata, the hash in the anchor should be the blake2b-256 hash of the "Canonized" form of the full document. This canonicalization is to remove ambiguities in serialization format and ensure that all consumers arrive at the same hash. The canonicalization algorithm for JSON-LD is specified [here](https://w3c-ccg.github.io/rdf-dataset-canonicalization/spec/).
+For CIP-0100 compliant metadata, the hash in the anchor should be the blake2b-256 hash of the raw bytes received from the wire.
+Hashing directly the original bytes ensures that there are no ambiguities, since the process doesn't depend on parsing the metadata, which can be the source of conflicts in different implementations.
 
 A metadata has a number of authors, each of which MUST authenticate their endorsement of the document in some way.
 
@@ -204,6 +205,8 @@ Here are the goals this CIP seeks to achieve, and the rationale for how this spe
 The following alternatives were considered, and rejected:
  - Plain JSON documents
    - While ultimately flexible and simple, there is a risk that with no way to structure what is *officially* supported, and the interpretation of each field, tooling authors would have one hand tied behind their back, and would be limited to a minimum common denominator.
+ - Canonicalising the whole document before hashing it
+   - Canonicalising requires initially parsing the file as a json, which can by itself cause ambiguities
  - A custom JSON format, with reference to CIPs
   - An initial draft of this proposal had an `extensions` field that operated very similar to `@context`
   - Instead, this CIP chose to go with an industry standard format to leverage the existing tooling and thought that went into JSON-LD

CIP-0100/test-vector.md

@@ -18,7 +18,7 @@ See [cip-0100.common.schema.json](./cip-0100.common.schema.json).
 
 CIP-100 off-chain metadata json example: [example.json](./example.json)
 
-Blake2b-256 hash of the canonicalize example (to go on-chain): `cf704961a066ba770afdd810cc98e0834206df7c0febe70405e0a5f79dece3ec`
+Blake2b-256 hash of the file (to go on-chain): `04af1b48bccbf7cf9b3e3b7952dfbdde0cc851ccb87ae6643521672cc381b00d`
 
 ### Intermediate files
 
@@ -81,21 +81,14 @@ We can go back to our [example.body.json](./example.body.json) and now add in pr
 
 By adding this information we create our [example.json](example.json).
 
-#### 6. Canonicalize example.json
+#### 6. Hash example.json
 
-To be able to create a final metadata hash which can be attached on-chain we must first canonicalize the [example.json](example.json).
-Ensure the results ends in a newline.
+To be able to create a final metadata hash which can be attached on-chain we simply hash the content of the file [example.json](example.json) as is
 
-This creates [example.nq](./example.nq).
+This results in: `04af1b48bccbf7cf9b3e3b7952dfbdde0cc851ccb87ae6643521672cc381b00d`.
 
-#### 7. Hash the canonicalized example.json
-
-We then use the specified `hashAlgorithm` on [example.nq](./example.nq).
-
-This results in: `cf704961a066ba770afdd810cc98e0834206df7c0febe70405e0a5f79dece3ec`.
-
-#### 8. Submit to chain
+#### 7. Submit to chain
 
 We can then host [example.json](./example.json) somewhere easily accessible following [Best Practices](./README.md#best-practices).
 
-Then at submission time of the governance metadata anchor we can provide the on-chain transaction both the URI to the hosted [example.json](./example.json) but also the hash generated via [7.](#7-hash-the-canonicalized-examplejson).
+Then at submission time of the governance metadata anchor we can provide the on-chain transaction both the URI to the hosted [example.json](./example.json) but also the hash generated via [6.](#6-hash-examplejson).

CIP-0108/test-vector.md

@@ -19,7 +19,7 @@ See [cip-0108.common.schema.json](./cip-0108.common.schema.json).
 ### Treasury Withdrawal
 
 Example metadata document file: [treasury-withdrawal.jsonld](./examples/treasury-withdrawal.jsonld).
-Blake2b-256 of the canonicalized document (to go onchain): `6685960f5884922e135a951f8acb581ff7202fc111ec405a7e014ae725927aae`
+Blake2b-256 of the file content (to go onchain): `311b148ca792007a3b1fee75a8698165911e306c3bc2afef6cf0145ecc7d03d4`
 
 #### Intermediate files
 
@@ -30,14 +30,11 @@ Body files, used to correctly generate author's witness:
 - [treasury-withdrawal.body.nq](./examples/treasury-withdrawal.body.nq)
 
 Blake2b-256 hash digest of canonicalized body: `68d6fe27087457acf0164e65414238c43573192c99f30341926d1524924d71ca`
-
-Whole document canonical representation, used to generate final hash:
-- [treasury-withdrawal.nq](./examples/treasury-withdrawal.nq)
 
 ### Motion of No-Confidence
 
 Example metadata document file: [no-confidence.jsonld](./examples/no-confidence.jsonld).
-Blake2b-256 of the canonicalized document (to go onchain): `c382be96dd98c4bbd6d3afe81f0c8143e70fbb6b8855303e69a6606991a4e909`
+Blake2b-256 of the file content (to go onchain): `6c27e5bd0d7cdec7ddb30956be0b5eac892a8330e00689692d18f3815a71bf9f`
 
 #### Intermediate files
 
@@ -49,9 +46,6 @@ Body files, used to correctly generate author's witness:
 
 Blake2b-256 hash digest of canonicalized body: `4a7ecc544559df67ece3f7f90f76c4e3e7e329a274c79a06dcfbf28351db600e`
 
-Whole document canonical representation, used to generate final hash:
-- [no-confidence.nq](./examples/no-confidence.nq)
-
 ## How-to Recreate Examples
 
 This tutorial creates additional intermediate files, these are not required in implementations but are shown here to articulate the process.
@@ -111,23 +105,14 @@ By adding this information we create our `example.jsonld`.
 
 For [Treasury Withdrawal](#treasury-withdrawal), this will result in [treasury-withdrawal.jsonld](./examples/treasury-withdrawal.jsonld).
 
-### 6. Canonicalize example.jsonld
-
-To be able to create a final metadata hash which can be attached on-chain we must first canonicalize the `example.jsonld`.
-Ensure the results ends in a newline.
-
-This creates `example.nq`.
-
-For [Treasury Withdrawal](#treasury-withdrawal), this will result in the intermediate file of [treasury-withdrawal.nq](./examples/treasury-withdrawal.nq).
-
-### 7. Hash the canonicalized example.jsonld
+### 6. Hash example.jsonld
 
-We then use the specified `hashAlgorithm` on `example.nq`.
+To be able to create a final metadata hash which can be attached on-chain we  we simply hash the content of the file [Treasury Withdrawal](#treasury-withdrawal.jsonld) as is.
 
-For [Treasury Withdrawal](#treasury-withdrawal), this will result in `6685960f5884922e135a951f8acb581ff7202fc111ec405a7e014ae725927aae`.
+This results is: `311b148ca792007a3b1fee75a8698165911e306c3bc2afef6cf0145ecc7d03d4`.
 
-### 8. Submit to chain
+### 7. Submit to chain
 
 We can then host `example.jsonld` somewhere easily accessible following [CIP-100 Best Practices](https://github.com/cardano-foundation/CIPs/blob/master/CIP-0100/README.md#best-practices).
 
-Then at submission time of the governance metadata anchor we can provide the on-chain transaction both the URI to the hosted `example.jsonld` but also the hash generated via [7.](#7-hash-the-canonicalized-examplejsonld).
+Then at submission time of the governance metadata anchor we can provide the on-chain transaction both the URI to the hosted `example.jsonld` but also the hash generated via [6.](#6-Hash-examplejsonld).

CIP-0119/test-vector.md

@@ -18,12 +18,4 @@ See [cip-0119.common.schema.json](./cip-0119.common.schema.json).
 
 CIP-119 off-chain metadata json example: [drep.jsonld](./examples/drep.jsonld)
 
-Blake2b-256 hash of the canonicalize example (to go on-chain): `a7a5cf6bfcad40e7b43653dae26344d6fbb5d7a2ef8de56a5e43703b298b94b5`
-
-### Intermediate files
-
-File produced to articulate process, these are not necessary in implementations.
-
-Whole document canonical representation, used to generate final hash:
-
-- [drep.nq](./examples/drep.nq)
+Blake2b-256 hash of content of the file (to go on-chain): `a14a5ad4f36bddc00f92ddb39fd9ac633c0fd43f8bfa57758f9163d10ef916de`