Use ephemeral key for private messages (e.g Discovery) #1686

sdbondi · 2020-04-08T16:20:39Z

Description

Added ephemeral key ECDH encryption for private messages. An "origin Message Authentication Code (MAC)" has been introduced to the message envelope header.

Given,

k_e - Ephemeral secret key
G - Ristretto generator point
k_r - Receiver secret key
K_s - Sender public key
SK - symmetric shared encryption key

Sender:

Generate an ephemeral secret key k_e and ephemeral public key k_eG
Create a Diffie-hellman encryption key using the recipient public key SK = k_e * k_rG
Use SK to encrypt the message body (Chacha20)
Generate an origin MAC containing the sender public key and signature (Schnorr) that signs the encrypted message body
Encode (protobuf) the OriginMac structure and encrypt it using SK
Set the origin_mac field in the envelope header to the resulting
cipher text from step 5.
Set the ephemeral_public_key in the envelope header

Receiver:

Generate SK = k_r * k_eG
Attempt to decrypt and decode the origin_mac field using SK
If successful, validate the message body using the signature
component
The receiver has verified the sender authenticity and message
integrity

Removed non-private options for discovery

Motivation and Context

NOTE: this change will cause a network fork

Previously, messages were encrypted using a similar process that exposed the sender public key. With this new scheme, no static public keys are leaked while being propagated on the network. As with the previous scheme, this relies on a network that participates in message propagation without a targeted destination, although a "regional" node id can be provided to allow more targeted private propagation.

How Has This Been Tested?

Existing tests updated. memorynet discoveries and store and forward discovery works.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Feature refactor (No new feature or functional changes, but performance or technical debt improvements)
New Tests
Documentation

Checklist:

I'm merging against the development branch.
I ran cargo-fmt --all before pushing.
I have squashed my commits into a single commit.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have added tests to cover my changes.

sdbondi · 2020-04-09T07:56:45Z

@philipr-za @neonknight64 extra 👁 on inbound/decryption.rs and outbound/encryption.rs please

neonknight64 · 2020-04-09T10:00:23Z

comms/dht/src/discovery/requester.rs

+    ///      node (not private)
+    ///    - `PublicKey` if any node on the network knows this public key, the message will be directed to that node.
+    ///      This sacrifices privacy for more efficient discovery in terms of network bandwidth and may result in
+    ///      quicker discovery times.


These doc strings need to be updated or moved to the correct part of the code.

Perhaps I should add the node destination back in - it's a quick change

...In a subsequent PR :)

neonknight64 · 2020-04-09T10:39:28Z

comms/dht/src/discovery/service.rs

    ) -> Result<(), DhtDiscoveryError>
    {
-        let DiscoverPeerRequest {


I'm not sure I agree with the removal of the DiscoverPeerRequest, except if it will be brought back at a later stage when discovery becomes more stable. These changes will disable the more efficient discovery using the node_id and will only allow discovery using flooding on the network. This can become expensive when the network is large.

I agree we need to find a good way to propagate encrypted single-destination messages across the network without propagating to every node while still preserving privacy. There was no way to do this before so I removed the extra cruft in the discover_peer call for now. I do plan to readd it, however when it gets added back in, it needs to make sure that "closer" nodes are online and possibly only choose out of it's currently connected neighbours to propagate the message. TLDR it's quite tricky to ensure that private targeted discovery is reliable on our current testnet network without flooding so I wanted to take a step back and add a working solution later - as you suggested (elsewhere) perhaps using the hamming distance for these kind of messages would allow privacy to be preserved while still moving the message in the right direction.

neonknight64 · 2020-04-09T10:44:22Z

comms/dht/src/outbound/encryption.rs

+                        crypt::encrypt(&shared_ephemeral_secret, &message.body).map_err(PipelineError::from_debug)?;
+
+                    // Sign the encrypted message
+                    let origin_mac = create_origin_mac(&node_identity, &encrypted_body)?;


I'm not sure there is a solution, but the detection and banning of a spamming node will be difficult with the encryption of the source identity using the shared ephemeral secret.

The general idea is that peers will reject messages over a certain size and temporarily reject/rate limit these messages if too many are received limiting the impact of spam messages. However, the heuristics for this are not completely clear.

It's worth noting that even with the source public key in the header a node could still spam by using an ephemeral public key when sending the message - the only real (authenticated) preventative measure you have is to prevent a connected peer from spamming you by banning them (so that they cannot connect to you again using the same network id).

We may also want to establish bonded relationships with peers before allowing them to send/propagate messages of this type.

neonknight64

LGTM

Storage and retreival of messages without an origin (PR #1686) All private/anonymous messages are stored as low priority (short TTL) messages. - New response type `Anonymous` for store and forward query responses - Added some more logging - DHT store and forward integration test - SAF messages can never be stored in store and forward

Added ephemeral key ECDH encryption for private messages. An "origin Message Authentication Code (MAC)" has been introduced to the message envelope header. Given, `k_e` - Ephemeral secret key `G` - Ristretto generator point `k_r` - Receiver secret key `K_s` - Sender public key `SK` - symmetric shared encryption key Sender: 1. Generate an ephemeral secret key `k_e` and ephemeral public key `k_eG` 1. Create a Diffie-hellman encryption key using the recipient public key `SK = k_e * k_rG` 1. Use `SK` to encrypt the message body (Chacha20) 1. Generate an origin MAC containing the sender public key and signature (Schnorr) that signs the encrypted message body 1. Encode (protobuf) the `OriginMac` structure and encrypt it using `SK` 1. Set the `origin_mac` field in the envelope header to the resulting cipher text from step 5. 1. Set the `ephemeral_public_key` in the envelope header Receiver: 1. Generate `SK = k_r * k_eG` 1. Attempt to decrypt and decode the `origin_mac` field using `SK` 1. If successful, validate the message body using the signature component 1. The receiver has verified the sender authenticity and message integrity - Removed non-private options for discovery

Storage and retreival of messages without an origin (PR #1686) All private/anonymous messages are stored as low priority (short TTL) messages. - New response type `Anonymous` for store and forward query responses - Added some more logging - DHT store and forward integration test - SAF messages can never be stored in store and forward

neonknight64

👍

Storage and retreival of messages without an origin (PR #1686) All private/anonymous messages are stored as low priority (short TTL) messages. - New response type `Anonymous` for store and forward query responses - Added some more logging - DHT store and forward integration test - SAF messages can never be stored in store and forward

This fixes a nasty regression caused by #1686 that caused messages to propagate between nodes endlessly. When a node broadcast encrypted messages, it encrypted each message it broadcast with a different ephemeral key and different message body. The fix is to encrypt once and shallow copy that body for each destination peer, dedup hashes are now deterministic across the network. Added a propagation test that checks that every message sent across a network of 4 nodes is in it's place.

** Major Changes from 0.0.9 **** Store and forward Peers will hold onto message for recipients that are not online and deliver the messages to them when they appear again. **** OsX package installer **** Many documentation improvements **** Ephemeral keys for private messages This is a big change that preserves privacy on the network but dramatically reduces the amount of traffic peers have to deal with. **** Emoji Ids *** Other changes - The target difficulty for a specified PoW algorithm is included in the block header. This allows the target difficulty of any block height to be calculated by only processing the last set of target difficulty samples up to that height. - Don't mark peers as offline if there are no existing connections (#1763) - Add UTXO selection strategy for large txs - Base node: Dynamically determine build version (#1760) - Include random peers for liveness ping (#1753) - RandomX - Version Update (#1754) - Add generic debug log function to FFI (#1752) - Lots of logging improvements - Added list-transactions and cancel-transaction commands (#1746) - ASCII table output for list-peers and list-connections (#1709) - Improve Difficulty adjustment manager - Modular configuration via ConfigLoader and ConfigPath traits - Fix chain monitoring bug in Transaction Service (#1739) - Empty Emoji String Bug Fix (#1736) - Coin-split base node cli command - Complete the basic OSX pkg build - Perform reorgs only on stronger tip accumulated difficulties - Use filesystem storage for dht.db on libwallet (#1735) - Fix duplicate message propagation (#1730) - Introduced accumulated difficulty validators to allow different rules for testing and running running a base node. - - Changes to peer offline handling (#1716) - Update Transaction cancellation to work for Inbound and Outbound Txs - Added oneshot reply to outbound messaging (#1703) - Add transaction stress test command to CLI - Implemented basic `make-it-rain` command - Fix MmrCache rewind issue - Use ephemeral key for private messages (e.g Discovery) (#1686) - Limit orphan pool size - Added a function to list UTXOs in the console (#1678) - Prevent adding yourself as a peer (#1665) - Update transaction weights (#1661) - Fix block period calculation - Validators will now check the weight of a block when doing validation (#1648) - Cleaned up duplicate code from the Blockchain db - The ban peer log will now supply n reason why the peer was banned (#1638)

sdbondi force-pushed the sb-dht-ephemeral-key branch 3 times, most recently from 3c62e0d to 10b5345 Compare April 8, 2020 16:32

sdbondi changed the title ~~Use ephemeral key for anonymous messages (e.g Discovery)~~ Use ephemeral key for private messages (e.g Discovery) Apr 8, 2020

sdbondi force-pushed the sb-dht-ephemeral-key branch 3 times, most recently from 82034c3 to 33c8099 Compare April 8, 2020 16:35

stringhandler previously approved these changes Apr 8, 2020

View reviewed changes

neonknight64 reviewed Apr 9, 2020

View reviewed changes

neonknight64 previously approved these changes Apr 9, 2020

View reviewed changes

sdbondi mentioned this pull request Apr 11, 2020

Removed redundant messaging authentication #1691

Merged

12 tasks

sdbondi dismissed stale reviews from neonknight64 and stringhandler via af585c0 April 14, 2020 08:31

sdbondi force-pushed the sb-dht-ephemeral-key branch from 33c8099 to af585c0 Compare April 14, 2020 08:31

neonknight64 approved these changes Apr 14, 2020

View reviewed changes

CjS77 merged commit a45872d into development Apr 14, 2020

sdbondi deleted the sb-dht-ephemeral-key branch April 14, 2020 10:12

sdbondi mentioned this pull request Apr 14, 2020

[Store and forward] Private message storage and retrieval #1697

Merged

12 tasks

sdbondi mentioned this pull request Apr 18, 2020

Fix duplicate message propagation #1730

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use ephemeral key for private messages (e.g Discovery) #1686

Use ephemeral key for private messages (e.g Discovery) #1686

sdbondi commented Apr 8, 2020 •

edited

Loading

sdbondi commented Apr 9, 2020

neonknight64 Apr 9, 2020 •

edited

Loading

sdbondi Apr 9, 2020

sdbondi Apr 9, 2020

neonknight64 Apr 9, 2020

sdbondi Apr 9, 2020

neonknight64 Apr 9, 2020

sdbondi Apr 9, 2020 •

edited

Loading

sdbondi Apr 9, 2020

neonknight64 left a comment

neonknight64 left a comment

Use ephemeral key for private messages (e.g Discovery) #1686

Use ephemeral key for private messages (e.g Discovery) #1686

Conversation

sdbondi commented Apr 8, 2020 • edited Loading

Description

Motivation and Context

How Has This Been Tested?

Types of changes

Checklist:

sdbondi commented Apr 9, 2020

neonknight64 Apr 9, 2020 • edited Loading

Choose a reason for hiding this comment

sdbondi Apr 9, 2020

Choose a reason for hiding this comment

sdbondi Apr 9, 2020

Choose a reason for hiding this comment

neonknight64 Apr 9, 2020

Choose a reason for hiding this comment

sdbondi Apr 9, 2020

Choose a reason for hiding this comment

neonknight64 Apr 9, 2020

Choose a reason for hiding this comment

sdbondi Apr 9, 2020 • edited Loading

Choose a reason for hiding this comment

sdbondi Apr 9, 2020

Choose a reason for hiding this comment

neonknight64 left a comment

Choose a reason for hiding this comment

neonknight64 left a comment

Choose a reason for hiding this comment

sdbondi commented Apr 8, 2020 •

edited

Loading

neonknight64 Apr 9, 2020 •

edited

Loading

sdbondi Apr 9, 2020 •

edited

Loading