Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DEP: Ephemeral message extension #28

Closed
wants to merge 9 commits into from
Closed

DEP: Ephemeral message extension #28

Changes from 4 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
fb8c7d2
Add proposals/0000-ephemeral-message.md
pfrazee Jun 6, 2018
9d092b4
Set PR link on proposals/0000-ephemeral-message.md
pfrazee Jun 6, 2018
e683b60
Fix copy-pasta typos in proposals/0000-ephemeral-message.md
pfrazee Jun 6, 2018
c2bdace
Added a header with 'encoding' values and increased the max payload s…
pfrazee Jun 6, 2018
7fc180a
Change the message identifier to 'ephemeral'
pfrazee Jun 10, 2018
5299afb
Add more to the motivation of proposals/0000-ephemeral-message.md
pfrazee Jun 10, 2018
630a691
Change the payload encoding to protobuf and provide a more flexible c…
pfrazee Jun 10, 2018
c7e10ac
Add size-limit suggestion to ephemeral-message DEP
pfrazee Jun 20, 2018
b115233
Add "Privacy, security, and reliability" section
pfrazee Jul 2, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 67 additions & 0 deletions proposals/0000-ephemeral-message.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@

Title: **DEP-0000: Ephemeral Message (Extension Message)**

Short Name: `0000-ephemeral-message-extension`

Type: Informative

Status: Undefined (as of 2018-06-05)

Github PR: https://github.com/datprotocol/DEPs/pull/28

Authors: [Paul Frazee](https://github.com/pfrazee)


# Summary
[summary]: #summary

This DEP defines the non-standard `em` extension message used in the Dat replication protocol. This message provides a way to send arbitrary application data to a peer through an existing connection.


# Motivation
[motivation]: #motivation

While Dat is effective at sharing persistent datasets, applications frequently need to transmit extra information which does not need to persist. This kind of information is known as "ephemeral." Examples include: sending chat messages, proposing changes to a dat, alerting peers to events, broadcasting identity information, and sharing the URLs of related datasets.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of adding general purpose capabilities to the Dat protocol to support all these use cases, why not embed Dat withing application-specific protocols for each of these use cases? The protocol stack is pretty modular as-is, so it should be possible to borrow "just" features like discovery, stream encryption, etc. Hypercore is transport agnostic, so it should be possible to embed in each of these.

This is sort of a rhetorical/hypothetical question; what i'm really getting at is, are we trying to turn the hypercore protocol a general purpose distributed networking framework? If so, maybe we should draw up a roadmap of what that would entail.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beaker needs a quick solution to exchanging ephemeral messages between users that reside on a dat:// site. Hypercore has an extensions mechanism for sending arbitrary message types, and so this DEP is taking advantage of that. It's a low-effort, high-return feature; just a stepping stone to the broader roadmap.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's good context! I'm for pushing this through as a Draft so you can move fast but still have documentation you can point at for interop.


This extension message will establish a common mechanism for sending ephemeral messages of an arbitrary composition.


# Reference Documentation
[reference-documentation]: #reference-documentation

This DEP is implemented using the Dat replication protocol's "extension messages." In order to broadcast support for this DEP, a client should declare the `'em'` extension in the replication handshake.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does em stand for "extension message" or "ephemeral message"? I would expand it to ephemeral-message, or ephemeral if we're trying to save bytes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like ephemeral


Ephemeral messages can be sent at any time after the connection is established by sending an extension message of type `'em'`. The message includes 1 byte of header information followed by a payload no longer than 10 kilobytes. Any additional bytes should be truncated by the receiving client. Therefore the message is encoded as follows:

```
header (1 byte)
payload (0-10000 bytes)
```

The first 6 header bits are reserved. The payload's encoding may be specified in the last two header bits. Possible values are:

- `00` - binary
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hrm, this is pretty limited. I think it might be better to just not attempt to do content-type declaration at all rather than do it partially. An HTTP-style Content-Type header feels like the minimum viable here. How will the client know how to "decode the payload according to this encoding-value" at all if it doesn't know what the message type is (only the encoding)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm just looking for a way to send buffers, strings, or objects. Maf had some observations about using a protocol buffer so that we can extend it more in the future, which I agree with.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having this be a protobuf message with two fields:

optional string contentType = 1;
required bytes payload = 2;

feels more hyper-world-idiomatic to me. For the browser use-case, I think you could also just try decoding the raw bytes as JSON, and throw a warning if it fails to parse. UTF-8 strings can be encoded as a JSON string.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me. Should we use the full mime-type? "application/json" ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup; and applications can support later variants if they want (eg, for GeoJSON, application/geo+json)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like the idea of using mime types! It'll make packets a bit bigger, but I think it's worth the flexability it will bring.

- `01` - utf8
- `10` - json

The receiving client should attempt to decode the payload according to this encoding-value. The client may respond to the message by emitting an event, so that it may be handled by the client's application logic.

No acknowledgment of receipt will be provided (no "ACK").

After publishing this DEP, the "Beaker Browser" will implement a Web API for exposing the `'em'` protocol to applications. It will restrict access so that the application code of a `dat://` site will only be able to send ephemeral messages on connections related to its own content.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd leave forward-looking statements ("this client will support XYZ") out of the DEP, even at draft stage. The note about expected security policy might be worth keeping though.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's a better way to phrase it, do you think? Maybe as a recommendation to browser clients which implement the DEP?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would move it to "Motivation" and rephrase:

"A specific use case for this extension is to enable a new Web API which will expose peer message-passing channels to in-browser applications. Such an API would restrict access so that the application code of a dat:// site will only be able to send ephemeral messages on connections related to its own content."

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that's good



# Drawbacks
[drawbacks]: #drawbacks

- This DEP may present privacy concerns, as it may be used to track users in a similar fashion to HTTP Cookies, or be used to exfiltrate data.
- The payload of the `'em'` message is not authenticated in any way. The lack of trust must be considered by applications which leverage the data.
- If the recipient of the `'em'` message is not authenticated (as is currently the case in all Dat replication connections) the client will not know who is receiving the payload and may broadcast sensitive information.


# Changelog
[changelog]: #changelog

- 2018-06-05: Added a header with 'encoding' values and increased the max payload size from 256b to 10kb.
- 2018-06-05: First complete draft submitted for review