From f0a48691c88207a88a17a3371d47f0d71cb27424 Mon Sep 17 00:00:00 2001 From: Paul Frazee Date: Tue, 26 Jun 2018 12:40:08 -0500 Subject: [PATCH 1/7] Add 0000-hypercore-header.md --- proposals/0000-hypercore-header.md | 109 +++++++++++++++++++++++++++++ 1 file changed, 109 insertions(+) create mode 100644 proposals/0000-hypercore-header.md diff --git a/proposals/0000-hypercore-header.md b/proposals/0000-hypercore-header.md new file mode 100644 index 0000000..ea918e0 --- /dev/null +++ b/proposals/0000-hypercore-header.md @@ -0,0 +1,109 @@ + +Title: **DEP-0000: Hypercore Header** + +Short Name: `0000-hypercore-header` + +Type: Standard + +Status: Undefined (as of 2018-06-26) + +Github PR: (add HTTPS link here after PR is opened) + +Authors: [Paul Frazee](https://github.com/pfrazee) + + +# Summary +[summary]: #summary + +The Dat protocol supports a variety of data structures on top of the hypercore append-only logs, including key-value stores (hyperdb) and file archives (hyperdrive). In order for a program to read the data structure within a hypercore, it must know which structure it is reading. This DEP specifies a "header" entry which can be placed at the first entry of a hypercore in order to provide identifying information. This header also supports custom data for the data-structure, and may be expanded in future DEPs to provide additional standard data. + + +# Motivation +[motivation]: #motivation + +At time of writing, Dat's standard data structures are being expanded upon (hyperdb) while existing data structures are being updated with breaking changes (hyperdrive). In order for these changes to be deployed smoothly to the network, it's important that the following needs be met: + + 1. Programs should correctly identify the data structure type and version within a hypercore. This will enable the program to use the correct handling code (eg hyperdb, hyperdrive-v1, hyperdrive-v2). + 2. Programs should abort processing a hypercore if they lack support for the type or version of the data structure. + + +# Usage Documentation +[usage-documentation]: #usage-documentation + +The "header" is the first entry in a hypercore. It includes a `type` and an optional `extension` which can contain data-structure-specific data. + +A program that is reading a hypercore will examine the `type` to determine how to process the hypercore. + +```js +function loadCorrectStructure (hypercore, cb) { + readHeader(hypercore, (err, header) => { + if (err) return cb(err) + + if (!header) { + // no header present - treat as a hypercore + return cb(null, hypercore) + } + + switch (header.type) { + case 'hyperdrive': + return cb(null, createHyperdriveV1(hypercore, header)) + case 'hyperdrive-v2': + return cb(null, createHyperdriveV2(hypercore, header)) + case 'hyperdb': + return cb(null, createHyperdbV1(hypercore, header)) + case '': + return cb(null, hypercore) // no structure, treat as a hypercore + // ... + default: + return cb(new Error('Unsupported: ' + header.type)) + } + }) +} +``` + +The `type` string should be unique to the data structure (refer to existing DEPs to avoid conflicts). When a breaking change is made to the data-structure, it should be given a new `type` string. For example, the `"hyperdrive"` type string will be updated to `"hyperdrive-v2"` for its second version. + +The `header.extension` is an optional blob of bytes. It can be used by the data structure to store custom data. In `"hyperdrive"` (v1) it is used to specify the key of the content hypercore. It is also valid to encode a protobuf message into the extension field, and therefore it's simple to add additional fields. + + +# Reference Documentation +[reference-documentation]: #reference-documentation + +The header message should be written as the first entry of the hypercore. It is not required that all hypercores possess a header. + +The header message has the following protobuf schema: + +```protobuf +message HypercoreHeader { + required string type = 1; + optional bytes extension = 2; +} +``` + +Data structures should not add additional fields to the `HypercoreHeader`. They should put any additional data into the `extension`. This is easy to do with protobuf's nested messages. For example: + +```protobuf +message MyCustomHeader { + message AdditionalData { + required string foo = 1; + required uint64 bar = 2; + } + required string type = 1; + optional AdditionalData extension = 2; +} +``` + + +# Rationale and alternatives +[alternatives]: #alternatives + +This standard is backwards compatible with v1 of hyperdrive, which used an `Index` message as the first entry in the metadata hypercore. + +An alternative approach is to simply attempt to process a given hypercore with all known message encodings. However, this is time-consuming and error-prone, and may not accurately capture the *version* of a data-structure. + + +# Changelog +[changelog]: #changelog + +- 2018-06-26: First complete draft submitted for review + From 96db1ff0778ea8880bfc6e1981e5c18a0cf2f495 Mon Sep 17 00:00:00 2001 From: Paul Frazee Date: Tue, 26 Jun 2018 12:42:49 -0500 Subject: [PATCH 2/7] Add maf as an author to hypercore-header --- proposals/0000-hypercore-header.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0000-hypercore-header.md b/proposals/0000-hypercore-header.md index ea918e0..6718c46 100644 --- a/proposals/0000-hypercore-header.md +++ b/proposals/0000-hypercore-header.md @@ -9,7 +9,7 @@ Status: Undefined (as of 2018-06-26) Github PR: (add HTTPS link here after PR is opened) -Authors: [Paul Frazee](https://github.com/pfrazee) +Authors: [Paul Frazee](https://github.com/pfrazee), [Mathias Buus](https://github.com/mafintosh) # Summary From 913a0b6663902e6c92bb82ae968040f11bcf5b22 Mon Sep 17 00:00:00 2001 From: Paul Frazee Date: Wed, 27 Jun 2018 11:30:34 -0500 Subject: [PATCH 3/7] Use more generic pseudo-code --- proposals/0000-hypercore-header.md | 47 +++++++++++++++--------------- 1 file changed, 23 insertions(+), 24 deletions(-) diff --git a/proposals/0000-hypercore-header.md b/proposals/0000-hypercore-header.md index 6718c46..3b44f65 100644 --- a/proposals/0000-hypercore-header.md +++ b/proposals/0000-hypercore-header.md @@ -34,30 +34,29 @@ The "header" is the first entry in a hypercore. It includes a `type` and an opti A program that is reading a hypercore will examine the `type` to determine how to process the hypercore. -```js -function loadCorrectStructure (hypercore, cb) { - readHeader(hypercore, (err, header) => { - if (err) return cb(err) - - if (!header) { - // no header present - treat as a hypercore - return cb(null, hypercore) - } - - switch (header.type) { - case 'hyperdrive': - return cb(null, createHyperdriveV1(hypercore, header)) - case 'hyperdrive-v2': - return cb(null, createHyperdriveV2(hypercore, header)) - case 'hyperdb': - return cb(null, createHyperdbV1(hypercore, header)) - case '': - return cb(null, hypercore) // no structure, treat as a hypercore - // ... - default: - return cb(new Error('Unsupported: ' + header.type)) - } - }) +``` +function loadCorrectStructure (hypercore) { + + var header = parseHypercoreHeader(hypercore.readEntry(0)) + + if (!header) { + // no header present - treat as a hypercore + return hypercore + } + + switch (header.type) { + case 'hyperdrive': + return new HyperdriveV1(hypercore, header) + case 'hyperdrive-v2': + return new HyperdriveV2(hypercore, header) + case 'hyperdb': + return new HyperdbV1(hypercore, header) + // ... + default: + // unknown type - treat as a hypercore + return hypercore + } + } ``` From 1a22f614a22c1aaf04f1601390f54b8d1e214456 Mon Sep 17 00:00:00 2001 From: Paul Frazee Date: Wed, 27 Jun 2018 11:32:01 -0500 Subject: [PATCH 4/7] Ignore additional fields --- proposals/0000-hypercore-header.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0000-hypercore-header.md b/proposals/0000-hypercore-header.md index 3b44f65..7a65b3c 100644 --- a/proposals/0000-hypercore-header.md +++ b/proposals/0000-hypercore-header.md @@ -79,7 +79,7 @@ message HypercoreHeader { } ``` -Data structures should not add additional fields to the `HypercoreHeader`. They should put any additional data into the `extension`. This is easy to do with protobuf's nested messages. For example: +Data structures should not add additional fields to the `HypercoreHeader`. Any additional fields should be ignored. Data-structure-specific data can be stored in the `extension`. This is easy to do with protobuf's nested messages. For example: ```protobuf message MyCustomHeader { From 5569167587e95bd98f4045471dcb4594c5d2c3e6 Mon Sep 17 00:00:00 2001 From: Paul Frazee Date: Fri, 6 Jul 2018 10:41:31 -0500 Subject: [PATCH 5/7] hypercore-v1 -> hypercore --- proposals/0000-hypercore-header.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0000-hypercore-header.md b/proposals/0000-hypercore-header.md index 7a65b3c..dffa8e9 100644 --- a/proposals/0000-hypercore-header.md +++ b/proposals/0000-hypercore-header.md @@ -23,7 +23,7 @@ The Dat protocol supports a variety of data structures on top of the hypercore a At time of writing, Dat's standard data structures are being expanded upon (hyperdb) while existing data structures are being updated with breaking changes (hyperdrive). In order for these changes to be deployed smoothly to the network, it's important that the following needs be met: - 1. Programs should correctly identify the data structure type and version within a hypercore. This will enable the program to use the correct handling code (eg hyperdb, hyperdrive-v1, hyperdrive-v2). + 1. Programs should correctly identify the data structure type and version within a hypercore. This will enable the program to use the correct handling code (eg hyperdb, hyperdrive, hyperdrive-v2). 2. Programs should abort processing a hypercore if they lack support for the type or version of the data structure. From a412d3e974ac9968fc6977bd13abdbc8b10b320e Mon Sep 17 00:00:00 2001 From: Paul Frazee Date: Fri, 6 Jul 2018 10:49:00 -0500 Subject: [PATCH 6/7] header.type -> header.dataStructureType --- proposals/0000-hypercore-header.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/proposals/0000-hypercore-header.md b/proposals/0000-hypercore-header.md index dffa8e9..c6f210e 100644 --- a/proposals/0000-hypercore-header.md +++ b/proposals/0000-hypercore-header.md @@ -30,9 +30,9 @@ At time of writing, Dat's standard data structures are being expanded upon (hype # Usage Documentation [usage-documentation]: #usage-documentation -The "header" is the first entry in a hypercore. It includes a `type` and an optional `extension` which can contain data-structure-specific data. +The "header" is the first entry in a hypercore. It includes a `dataStructureType` and an optional `extension` which can contain data-structure-specific data. -A program that is reading a hypercore will examine the `type` to determine how to process the hypercore. +A program that is reading a hypercore will examine the `dataStructureType` to determine how to process the hypercore. ``` function loadCorrectStructure (hypercore) { @@ -44,7 +44,7 @@ function loadCorrectStructure (hypercore) { return hypercore } - switch (header.type) { + switch (header.dataStructureType) { case 'hyperdrive': return new HyperdriveV1(hypercore, header) case 'hyperdrive-v2': @@ -60,7 +60,7 @@ function loadCorrectStructure (hypercore) { } ``` -The `type` string should be unique to the data structure (refer to existing DEPs to avoid conflicts). When a breaking change is made to the data-structure, it should be given a new `type` string. For example, the `"hyperdrive"` type string will be updated to `"hyperdrive-v2"` for its second version. +The `dataStructureType` string should be unique to the data structure (refer to existing DEPs to avoid conflicts). When a breaking change is made to the data-structure, it should be given a new `dataStructureType` string. For example, the `"hyperdrive"` type string might be updated to `"hyperdrive-v2"` for its second version. (Note: the actual type strings will be specified in the DEPs for the individual data structures.) The `header.extension` is an optional blob of bytes. It can be used by the data structure to store custom data. In `"hyperdrive"` (v1) it is used to specify the key of the content hypercore. It is also valid to encode a protobuf message into the extension field, and therefore it's simple to add additional fields. @@ -74,7 +74,7 @@ The header message has the following protobuf schema: ```protobuf message HypercoreHeader { - required string type = 1; + required string dataStructureType = 1; optional bytes extension = 2; } ``` @@ -87,7 +87,7 @@ message MyCustomHeader { required string foo = 1; required uint64 bar = 2; } - required string type = 1; + required string dataStructureType = 1; optional AdditionalData extension = 2; } ``` From a100c71418a736bcec73e1df09b2b9d6ec8e8aaf Mon Sep 17 00:00:00 2001 From: Paul Frazee Date: Fri, 6 Jul 2018 13:12:42 -0500 Subject: [PATCH 7/7] Publish 0007-hypercore-header --- ...{0000-hypercore-header.md => 0007-hypercore-header.md} | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) rename proposals/{0000-hypercore-header.md => 0007-hypercore-header.md} (96%) diff --git a/proposals/0000-hypercore-header.md b/proposals/0007-hypercore-header.md similarity index 96% rename from proposals/0000-hypercore-header.md rename to proposals/0007-hypercore-header.md index c6f210e..a75bdde 100644 --- a/proposals/0000-hypercore-header.md +++ b/proposals/0007-hypercore-header.md @@ -1,13 +1,13 @@ -Title: **DEP-0000: Hypercore Header** +Title: **DEP-0007: Hypercore Header** -Short Name: `0000-hypercore-header` +Short Name: `0007-hypercore-header` Type: Standard -Status: Undefined (as of 2018-06-26) +Status: Draft (as of 2018-07-06) -Github PR: (add HTTPS link here after PR is opened) +Github PR: [#34](https://github.com/datprotocol/DEPs/pull/34) Authors: [Paul Frazee](https://github.com/pfrazee), [Mathias Buus](https://github.com/mafintosh)