Peer storage feature #5361

adi2011 · 2022-06-30T08:13:41Z

PEER STORAGE BACKUP

This PR implements peer storage backup which will enable nodes to exchange their respective SCBs (Static channel backup), This will be useful in case of complete data loss.

WHY

One of the major features of lightning is that the transactions are off-chain and are stored in a local database, which makes this DB highly dynamic and complex to backup.
Peer storage backup will allow users to send their encrypted backup to their peers, which can be used to recover their funds in case of complete data loss.

HOW

The strategy is that we'll store the data received in the datastore.
So to implement this I've introduced 2 new messages i.e. PEER_STORAGE and YOUR_PEER_STORAGE, these will be used to exchange the data stored between the nodes.
-PEER_STORAGE will be used to send our own encrypted backup to the peer.
-YOUR_PEER_STORAGE will be used to send the most recent backup of the peer.

The general flow of messages every time we connect is:

Alice ----------------------- ♥️ ------------------------------ Bob

PEER_STORAGE 📤-------------📨-------------> 📥
YOUR_PEER_STORAGE 📤---------📨------------> 📥

📥 <------------📨--------------PEER_STORAGE 📤
📥 <-----------📨------------YOUR_PEER_STORAGE 📤

On receiving YOUR_PEER_STORAGE bob will verify if it's correct and Alice has not changed anything in his last sent backup.
On receiving PEER_STORAGE bob will update the backup he has stored for Alice in his own datastore.

Every time we open a new channel or close an old one we will send PEER_STORAGE to every peer we're connected to (Haven't figured out yet, maybe a new RPC (sendcustommsgmulti) which will enable plugins to send a single message to multiple users at once, because the interaction between lightningd and plugins is single-threaded)

They can choose to ignore the messages since they are odd (It's okay to be odd)

In case of complete data loss, the user will reconnect to their peers and hope that they get a YOUR_PEER_STORAGE message. Then they can directly use the RPC specified in the plugin to recover the channels.

vincenzopalazzo

Concept ack I just read the code, it requires more love in the review from my side!

tests/test_misc.py

lightningd/peer_control.c

plugins/chanbackup.c

adi2011 · 2022-07-06T07:45:26Z

Thanks for the review @vincenzopalazzo, I have mostly fixed all the mentioned changes in the SCB PR, This one will get in after that :)

vincenzopalazzo · 2022-07-07T09:35:45Z

Ops! my bad I did not see that this was built on top of another PR! my bad!

niftynei

Half done review, pushing up comments.

In general, smaller commits for each independent change would be great.

plugins/chanbackup.c

niftynei · 2022-07-27T14:43:30Z

plugins/chanbackup.c

-	}
+	},
+        {
+                "peerstoragebkp",


Commands should always include a verb. Maybe populate-disk-scb-from-peer?

restore-from-peer?

restorefrompeer looks good 👍, - is not allowed I think, they create problems in other languages.

niftynei · 2022-07-27T15:00:10Z

plugins/chanbackup.c

+        {
+                "peerstoragebkp",
+                "recovery",
+                "Grabs the scb from datastore (if exists)",


Could be more descriptive of what this does?

Checks if i have got a backup from a peer, and if so, will stub those channels in the database and if is successful, will return list of channels that have been successfully stubbed

This command reads the latest backup storage data from our datastore and populates our database w/ stubs from it.

A few things about this are odd to me.

How would someone know when to call this command?

This seems like it could/should happen automatically when we get a scb from a peer?

How do we know the scb we have in the datastore is valid/up to date? What's the worst case scenario if an out of date scb is applied?

What happens for channels in the datastore scb that are already in our db?

Okay, these are some really nice questions:
Ans. 1) This would be used in case of data loss. The user could connect to their peers and collect their storage.

Ans. 2) Hmm, That's an excellent suggestion, everytime we receive scb we can automatically stub the channels in the DB (which are not already into it). What's your opinion on this @rustyrussell ?

Ans. 3) This feature would be used as a last resort to recover the funds in case of severe data loss, This can only help users to recover funds in case of severe data loss, This would never harm existing channels and funds.

Ans. 4) We skip over them and do not stub them in the DB.

niftynei · 2022-07-27T15:06:24Z

plugins/chanbackup.c

+                                    after_latestbkp,
+                                    &forward_error,
+                                    NULL);
+        json_add_string(req->js, "key", "latestbkp");


nit: call it scb not bkp

niftynei · 2022-07-27T15:06:37Z

plugins/chanbackup.c

+
+}
+
+static struct command_result *json_peerstoragebkp(struct command *cmd,


nit: rename bkp to scb

niftynei · 2022-07-27T15:29:31Z

plugins/chanbackup.c

+        return send_outreq(cmd->plugin, req);
+}
+
+static struct command_result *after_datastore(struct command *cmd,


maybe there's a generic option for htis? if not you can add one.

niftynei · 2022-07-27T15:31:57Z

plugins/chanbackup.c

+	struct stat st;
+	struct node_id *node_id = tal(cmd, struct node_id);
+
+	int fd = open("emergency.recover", O_RDONLY);


this really could be a separate function? don't you do this other places?

filenames in line is not great either fwiw.

filenames in line is not great either fwiw.

Defined a global var for it.

this really could be a separate function? don't you do this other places?

Hmm, Done!

niftynei · 2022-07-27T15:32:50Z

plugins/chanbackup.c

+	        plugin_err(cmd->plugin, "closing: %s", strerror(errno));
+	}
+
+	peers = json_get_member(buf, params, "peers");


look at json_scan?

plugins/chanbackup.c

niftynei · 2022-07-27T15:37:22Z

plugins/chanbackup.c

+		        json_to_node_id(buf, nodeid, node_id);
+
+			req = jsonrpc_request_start(cmd->plugin,
+                                    cmd,


I think you're gonna want to make this NULL.

niftynei

some comments!

niftynei · 2022-07-27T21:56:17Z

plugins/chanbackup.c

+        {
+                "peerstoragebkp",
+                "recovery",
+                "Grabs the scb from datastore (if exists)",


This command reads the latest backup storage data from our datastore and populates our database w/ stubs from it.

A few things about this are odd to me.

How would someone know when to call this command?

This seems like it could/should happen automatically when we get a scb from a peer?

How do we know the scb we have in the datastore is valid/up to date? What's the worst case scenario if an out of date scb is applied?

What happens for channels in the datastore scb that are already in our db?

plugins/chanbackup.c

niftynei · 2022-08-11T14:39:24Z

plugins/chanbackup.c

 	json_to_scb_chan(buf, scbs, &scb_chan);
 	plugin_log(cmd->plugin, LOG_INFORM, "Updating the SCB");

 	update_scb(cmd->plugin, scb_chan);
-	return notification_handled(cmd);
+	plugin_log(cmd->plugin, LOG_INFORM, "Updating the SCB2");


what's an "SCB2"? maybe make LOG_DBG

Removed, Was using it for debugging

niftynei · 2022-08-11T14:45:42Z

plugins/chanbackup.c

+			tal_bytelen(serialise_scb));
+
+		send_outreq(cmd->plugin, req);
+	}


what happens if not connected?

niftynei · 2022-08-11T14:47:36Z

plugins/chanbackup.c

@@ -388,18 +389,113 @@ static struct command_result *after_send_scb(struct command *cmd,
        return send_outreq(cmd->plugin, req);
 }

+struct info {
+	size_t idx;
+	const char *buf;


struct node_id *peers;

niftynei · 2022-08-11T14:49:44Z

plugins/chanbackup.c

+{
+        plugin_log(cmd->plugin, LOG_INFORM, "Peer storage sent to");
+	info->idx += 1;
+	return after_listpeers(cmd, info->buf, json_parse_simple(cmd, info->buf, tal_bytelen(info->buf)), info);


i would build a list of connected peers and iterate thru that, rather than holding onto the buffer w/ all the data in memory for the duration of this.

Done, we just need the index. Libplugin automatically handles multiple requests one by one

cdecker · 2022-11-01T13:34:32Z

@adi2011: this is marked for release 22.11 for which we're trying to publish a first RC this week, but it is also marked as a draft. What's the current status of this? Happy to review and merge it if it is ready, otherwise we can also postpone it to the next release, which'll happen in ~2 months again.

cdecker · 2022-11-02T12:07:33Z

After talking to @adi2011 we decided that we'll be pushing this PR to the next release.

adi2011 · 2022-11-03T09:14:03Z

Thanks! Will get onto it once my semester exams are over.

…peer_connected. This is needed for the next patch, which does this from the peer_connected hook! Signed-off-by: Rusty Russell <[email protected]> Changelog-Changed: JSON-RPC: `sendcustommsg` can now be called by a plugin from within the `peer_connected` hook.

Add msg type peer_storage and your_peer_storage

…ternal message.

We are now going to have messages which we know about, but yet we don't handle ourselves. [ I reversed this from Adi's, as that was clearer! --RR ]

…d from peers.

Signed-off-by: Rusty Russell <[email protected]>

And we should always represent them as is, not as optional: it's possible in future we could *require* "WANT_PEER_BACKUP_STORAGE". Signed-off-by: Rusty Russell <[email protected]>

node_id can be on the stack, avoiding a tal call. Signed-off-by: Rusty Russell <[email protected]>

rustyrussell · 2023-02-08T03:33:57Z

Dumb typo completely broke peer_connected hook!

Ack 5805664

When you return an allocated pointer, you should always hand in the context you want it allocated from. This is more explicit, because it may really matter to the caller! This also folds some simple operations, and avoids doing too much variable assignment in the declarations themselves: some coding styles prohibit such initializers, but that's a bit exteme. Signed-off-by: Rusty Russell <[email protected]>

Since it's not spec-final yet (hell, it's not even properly specified yet!) we need to put it behind an experimental flag. Unfortunately, we don't have support for doing this in a plugin; a plugin must present features before parsing options. So we need to do it in core. Signed-off-by: Rusty Russell <[email protected]>

rustyrussell · 2023-02-08T07:16:41Z

Ack 081a1d8

endothermicdev · 2023-02-08T14:37:53Z

Congrats @adi2011! It was a long road, but this is a very cool feature!

adi2011 · 2023-02-09T12:46:27Z

Thanks and congratulations to you too! :)

instagibbs · 2023-02-09T20:08:44Z

Test seems a bit flakey: https://pipelines.actions.githubusercontent.com/serviceHosts/e5392bd7-7462-4d15-b5db-89c31513dbe4/_apis/pipelines/1/runs/13853/signedlogcontent/81?urlExpires=2023-02-09T19%3A58%3A59.4670493Z&urlSigningMethod=HMACV1&urlSignature=foSYv9C4W2Xftr1Mop2sBqk9lnt%2B0hVTsxthcUGd62Y%3D

2023-02-09T19:56:59.1042925Z E           pyln.client.lightning.RpcError: RPC call failed: method: connect, payload: {'id': '022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59', 'host': 'localhost', 'port': 39159}, error: {'code': 402, 'message': 'disconnected during connection'}

Sjors · 2023-03-06T15:23:13Z

Is there any documentation for this, other than the name of the option?

In particular it would be useful to illustrate how recovery works, e.g. "Start a new node with the same secret, find one of your old peers e.g. using a public archive like mempool . space, call lightning-cli restore-from-peer, etc".

man lightning-restore-from-peer is not working for me either.

adi2011 force-pushed the peer-storage-feature branch 2 times, most recently from 3627845 to 1589e98 Compare July 2, 2022 01:00

adi2011 marked this pull request as ready for review July 2, 2022 01:01

adi2011 requested a review from cdecker as a code owner July 2, 2022 01:01

adi2011 force-pushed the peer-storage-feature branch 2 times, most recently from b4ca14f to cc0eb71 Compare July 4, 2022 03:40

vincenzopalazzo reviewed Jul 6, 2022

View reviewed changes

tests/test_misc.py Outdated Show resolved Hide resolved

tests/test_misc.py Outdated Show resolved Hide resolved

lightningd/peer_control.c Outdated Show resolved Hide resolved

lightningd/peer_control.c Outdated Show resolved Hide resolved

plugins/chanbackup.c Outdated Show resolved Hide resolved

adi2011 marked this pull request as draft July 8, 2022 13:05

niftynei added this to the v22.10 milestone Jul 11, 2022

openoms mentioned this pull request Jul 21, 2022

[CL]: The best way to save backups raspiblitz/raspiblitz#2983

Closed

adi2011 mentioned this pull request Jul 25, 2022

Static channel backup: What does it do? #5439

Closed

adi2011 force-pushed the peer-storage-feature branch 3 times, most recently from d790c0a to 9779edc Compare July 27, 2022 11:54

adi2011 requested a review from vincenzopalazzo July 27, 2022 12:04

niftynei reviewed Jul 27, 2022

View reviewed changes

adi2011 force-pushed the peer-storage-feature branch 3 times, most recently from 0e11176 to cc6b450 Compare August 11, 2022 10:41

adi2011 force-pushed the peer-storage-feature branch from cc6b450 to 6f32d6c Compare August 11, 2022 14:14

adi2011 force-pushed the peer-storage-feature branch from 6f32d6c to fb73a97 Compare September 1, 2022 13:16

niftynei reviewed Sep 1, 2022

View reviewed changes

adi2011 force-pushed the peer-storage-feature branch from fb73a97 to 4121d56 Compare September 6, 2022 22:01

cdecker removed this from the v22.11 milestone Nov 2, 2022

adi2011 force-pushed the peer-storage-feature branch from 4121d56 to b0d028a Compare December 8, 2022 13:15

rustyrussell and others added 17 commits February 8, 2023 14:02

wire: Add patch file for peer storage bkp

7908e51

Add msg type peer_storage and your_peer_storage

feature(PEER_STORAGE and YOUR_PEER_STORAGE) added in feature.c and in…

156bdc4

…ternal message.

peer_wire_is_internal helper.

3126cca

We are now going to have messages which we know about, but yet we don't handle ourselves. [ I reversed this from Adi's, as that was clearer! --RR ]

connectd: make exception for peer storage msgs.

234c0a3

plugins/chanbackup: PLUGIN_RESTARTABLE to PLUGIN_STATIC...

228f547

Plugins/chanbackup: Add featurebit Peerstrg and YourPeerStrg.

cb5309c

plugins/chanbackup: Define FILENAME globally (Good Manners)

3a391d6

plugins/chanbackup: use grab_file.

65496fd

Plugins/chanbackup: Add SCB on CHANNELD_AWAITING_LOCKING stage

53b427c

Plugins/chanbackup: Add hook for receiving custommsg

6f8a800

Plugins/chanbackup: Add hook for exchanging msgs on connect with a peer

837e9e1

Plugins/chanbackup: Add RPC for recovering from the latestscb receive…

8140b4a

…d from peers.

tests/test_misc.py: Add test_restorefrompeer.

3be74fd

plugins/chanbackup: switch to normal indentation.

b372392

Signed-off-by: Rusty Russell <[email protected]>

features: make name of peer storage features match spec.

21351e2

And we should always represent them as is, not as optional: it's possible in future we could *require* "WANT_PEER_BACKUP_STORAGE". Signed-off-by: Rusty Russell <[email protected]>

plugins/chanbackup: neaten a little.

349828d

node_id can be on the stack, avoiding a tal call. Signed-off-by: Rusty Russell <[email protected]>

rustyrussell force-pushed the peer-storage-feature branch from ab73723 to 5805664 Compare February 8, 2023 03:33

rustyrussell added 2 commits February 8, 2023 12:13

adi2011 force-pushed the peer-storage-feature branch from 5805664 to 081a1d8 Compare February 8, 2023 06:45

endothermicdev merged commit a71bd3e into ElementsProject:master Feb 8, 2023

TonyGiorgio mentioned this pull request Jun 23, 2023

Static Channel Backups MutinyWallet/mutiny-node#607

Merged

adi2011 mentioned this pull request Sep 25, 2023

Peer storage for nodes to distribute small encrypted blobs. lightning/bolts#1110

Open


		}

		static struct command_result json_peerstoragebkp(struct command cmd,

Peer storage feature #5361

Peer storage feature #5361

Conversation

adi2011 commented Jun 30, 2022 • edited Loading

PEER STORAGE BACKUP

WHY

HOW

Alice ----------------------- ♥️ ------------------------------ Bob

vincenzopalazzo left a comment

Choose a reason for hiding this comment

adi2011 commented Jul 6, 2022

vincenzopalazzo commented Jul 7, 2022

niftynei left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adi2011 Sep 6, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

niftynei left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cdecker commented Nov 1, 2022

cdecker commented Nov 2, 2022

adi2011 commented Nov 3, 2022 • edited Loading

rustyrussell commented Feb 8, 2023

rustyrussell commented Feb 8, 2023

endothermicdev commented Feb 8, 2023

adi2011 commented Feb 9, 2023

instagibbs commented Feb 9, 2023

Sjors commented Mar 6, 2023 • edited Loading

adi2011 commented Jun 30, 2022 •

edited

Loading

adi2011 Sep 6, 2022 •

edited

Loading

adi2011 commented Nov 3, 2022 •

edited

Loading

Sjors commented Mar 6, 2023 •

edited

Loading