Peer storage for nodes to distribute small encrypted blobs.

[adi2011]

This pull request proposes peer storage, inspired by @t-bast's peer backup storage. We create two separate messages, which users can use to distribute encrypted 'blobs' of up to 64 KB to their peers.

Users can distribute a 'blob' and update it whenever their configuration changes by sending a 'peer_storage' message. In return, they will always receive a 'your_peer_storage' message, which will serve as an acknowledgment.

Nodes can also decide whether they want to provide this service in exchange for some sats.

You can check out the implementation in CLN here

Thank you so much, @rustyrussell, for all the help and guidance!

[saubyk]

This part is unclear to me. Based on my reading, there are two instances when a node (running option_provide_storage) will send your_peer_storage message:

• When it gets a peer_storage message and has stored the data (like an ack)
• When it reconnects with the peer for which it has stored data (to ensure that peer has a chance to update stale data)

The point about "regularly" sending your_peer_storage seem to be specified loosely.

• What's the right frequency to send this to your peer?
• Does it create a spam vector for your peer?

[rustyrussell]

Agreed. You don't need to specify this, since you already said in requirements that "MAY send peer_storage whenever necessary (usually, everytime their configuration changes)".

And I made a suggestion for spam above...

[adi2011]

Yes, the two instances that you mentioned are the only time when a peer needs to send your_peer_storage.

[rustyrussell]

Agreed. You don't need to specify this, since you already said in requirements that "MAY send peer_storage whenever necessary (usually, everytime their configuration changes)".

And I made a suggestion for spam above...

[saubyk]

Ack

[t-bast]

As stated during the spec meeting, I think this should be moved to the bLIP repository: https://github.com/lightning/blips
This feels like an extension/add-on, not something that really needs to be in the BOLTs (especially if we're trying to somewhat reduce complexity here).
I wanted to move my encrypted storage proposal to a bLIP, but didn't do it because I was rather waiting for someone else to work on it first, which is happening now.

[adi2011]

I think in order to make peer storage reliable, it should have maximum coverage across the network. This approach attempts to address the backup issue, which is essential for enhancing the usability of the Lightning Network. Many small node operators may be reluctant to invest in RAID systems to protect their nodes; thus, this could serve as one of the safety nets for the network. That is why we should consider adding it to the main spec imo.

[GeorgeTsagk]

(e.g. might get paid for the service)

Could provide a bit more context here w.r.t being paid for the purpose of storing blobs

Is this why if it has an open channel with the sender is a requirement?

[adi2011]

Sure, this needs more explanation, I'll add to it when I get back to my laptop.

[ProofOfKeags]

Is this why if it has an open channel with the sender is a requirement?

I'd imagine this also helps with spam prevention. Since in order to do a drive-by to store data you'd have to commit capital.

[adi2011]

Open channel is not a requirement. Some nodes might provide this service in exchange for some sats that is what might get paid for the service means.

[GeorgeTsagk]

could also consider padding so that it is always fixed to 65531 bytes

[ProofOfKeags]

Why would you want to pad?

[ellemouton]

I think the reasoning would be privacy related: ie so that receiver cant deduce anything about the peer sending the data based on size changes.

That being said: I think all of that detail doesnt need to be specced out on this layer. How the blob is structured is the responsibility of the layer above.

Especially if the layer above is a "storage for payment" layer where they are charging per byte (or byte bucket) then we really dont want to force them to use the full 65k here.

[ProofOfKeags]

We already handle this with the randomization of ping/pong responses. The actual messages themselves are encrypted as well. Padding the message out to 65531 actually negatively impacts privacy as best I can tell since now we can do partial inference and filter out all non-max-length messages from the anonymity set.

[ProofOfKeags]

oops I misread this. Indeed the receiver may come to learn about the contents of the payload through size analysis. My commentary on ping/pong length randomization is about 3p privacy, not 2p privacy.

[GeorgeTsagk]

yeah my initial thinking was that certain data sizes can be associated with certain types of backups or data in general

That being said: I think all of that detail doesnt need to be specced out on this layer. How the blob is structured is the responsibility of the layer above.

Yes I see what you're saying here. In this case I assume the "layer above" lies within the node implementation itself? It's not like external software would have direct access to this field

[GeorgeTsagk]

Regardless, is it the BOLT's responsibility to document the need for padding?

[t-bast]

I agree, a SHOULD requirement to pad could make sense here.

[vincenzopalazzo]

I concur with @t-bast that this feature could be a BLIP.

Since this is not a one-size-fits-all solution, we can create different BLIPs for various backup use cases (if any others are proposed in the future).

If we want to add this to the BOLTs, we should address some concerns like this one ElementsProject/lightning#6652

[ellemouton]

dont think i fully understand why we need the option_want_storage feature bit..?

isnt it enough that im connecting to a node with option_provide_storage and sending peer_storage to it? Or is there a case where someone would want to seek out a node advertising option_want_storage?

[ProofOfKeags]

I'm inclined to agree. I think in practice it is very likely that people will implement this protocol wholesale or not at all (it's pretty simple). That said, I can imagine scenarios where you may specifically not want to provide this feature but still would like to consume it. At 65k per peer though, idk what the big deal would be with providing it anyway.

[adi2011]

Big nodes with large number of peers might not be interested in giving this service to all of their peers.

[ProofOfKeags]

Are you suggesting they will only want to offer it to some of their peers or rather in that situation they wouldn't want to offer it to any of them?

[ellemouton]

Big nodes with large number of peers might not be interested in giving this service to all of their peers.

I dont think having/not having option_want_storage changes this though?

[ellemouton]

the only reason you would advertise option_want_storage is so that nodes offering the service could seek you out. but that doesnt really make sense to me. Makes more sense that if you want storage - you would seek out nodes that can offer it to you.

[ellemouton]

should we not use even types for these messages? since we are only sending these if the peer advertises option_provide_storage

[ProofOfKeags]

does IOKTBO apply to message types?? I thought it was just TLVs

[ellemouton]

yeah it applies to messages too afaiu: https://github.com/lightning/bolts/blob/master/01-messaging.md#lightning-message-format

[ProofOfKeags]

I don't think this applies to the low message type numbers though afaict. If it did then things like accept_channel would be treated the same right? I would support using even messages, no issue with that at all, but I think I'm not quite understanding the intricacies of IOKTBO

[ellemouton]

yeah i think for some of the very first messages this was sort of overlooked but it wasnt a problem since nothing would work if a node didnt understand something fundamental like accept_channel

[TheBlueMatt]

IOKTBO definitely applies to all messages, its just in the very initial protocol there's no distinction cause everyone supports everything :). Its definitely nice for these to be odd because then nodes can send peer_storage without bothering to check if their peer supports the feature at all.

[TheBlueMatt]

Lets not respond with the full 64KiB blob every time data gets stored. That seems incredibly wasteful. If you want an ACK, lets include an ID.

[ProofOfKeags]

we can also ACK with a sha256 of the data provided.

[TheBlueMatt]

Don't make me hash the data, either. Honestly, I'm not really sure if we need to have an ACK at all....its useful if you trust the counterparty and want to use it as a robust data store (but, even then, I think only LDK has support for fully async persistence?, and I'm not convinced our users would prefer this over a traditional storage layer) but I'm not sure that's worth doing here when the whole design space is around "you may persist, if you want, but peers shouldn't rely on it". Maybe @t-bast has other thoughts?

[t-bast]

I agree with @TheBlueMatt here, our proposal for this feature (#881) does not have an ACK because I didn't find that useful.

[adi2011]

It serves as assurance to the sender, although not foolproof, that it won't be excessively wasteful. Additionally, updating peer_storage won't be a frequent occurrence.

[adi2011]

The sole reason for ACK was to assure the sender, is their any other efficient method to do this?

[t-bast]

But does it really assure the sender of anything meaningful? If used to store every channel state change (which is what we've been doing for years), we don't want to pause the channel's operation waiting for an ACK, it adds unnecessary complexity and risk of force-closing. This is mostly being used by nodes who will regularly disconnect (e.g. mobile phones), who have an opportunity of checking the back-up returned on every connection reestablishment, which is IMO enough assurance that the storing node behaves correctly?

[ProofOfKeags]

I'd like to remind us that we didn't need ACKs for every channel update either, and yet we are nudging ourselves towards a simplified update protocol. Granted, the purpose of this is far less consequential than ACK'ing the channel updates themselves, so we can afford to be like "🤷🏻‍♂️ who knows if they got it, we'll just assume so".

My take here is that at minimum it helps with debugging, and the cost of the ACK seems rather trivial so avoiding it seems silly. Yes, it doesn't provide much, we can always send back the packet, or the hash, or some ID, and then just yeet the data into oblivion, but does it really cost us much to just ACK it?

[TheBlueMatt]

The sole reason for ACK was to assure the sender, is their any other efficient method to do this?

To echo @t-bast's comment - to assure the sender of what? And what should the sender do with that assurance. If a given message is not actually handled by a peer/it doesn't change their behavior in some meaningful way, we should drop it.

I'd like to remind us that we didn't need ACKs for every channel update either,

Not sure what you're referring to - do you mean we don't need revoke_and_ack? We absolutely do, the protocol doesn't work without it.

My take here is that at minimum it helps with debugging, and the cost of the ACK seems rather trivial so avoiding it seems silly. Yes, it doesn't provide much, we can always send back the packet, or the hash, or some ID, and then just yeet the data into oblivion, but does it really cost us much to just ACK it?

Yes, more protocol. We have a lot of protocol, anywhere we can have just a bit less protocol is great. Its not clear to me why an ACK will help with debugging - a peer can always give us an old blob, whether they ACK'ed it or not, and if you get an old state you need to go try to figure out if the state is old and which one it is. You can always check if there was a ping/pong after the sent storage message and use that as an ack/they received it indication.

[adi2011]

Removed ACK as a requirement. Still mentioned in the rationale so that implementations can decide for themselves.

[GeorgeTsagk]

yeah my initial thinking was that certain data sizes can be associated with certain types of backups or data in general

That being said: I think all of that detail doesnt need to be specced out on this layer. How the blob is structured is the responsibility of the layer above.

Yes I see what you're saying here. In this case I assume the "layer above" lies within the node implementation itself? It's not like external software would have direct access to this field

[GeorgeTsagk]

This is a lot of data storage. Even with no overhead 64KiB/second will destroy almost any consumer SSD in a year. ISTM one update per second is much too much if you're not charging for it.

Agree and I believe directly charging is too involved for a low level feature like this. Could indirectly relate this rate to a cost, something like rate limiting to 1 update per htlc that has been routed over the channel.

[GeorgeTsagk]

what's the thinking here?

[adi2011]

To make it a bit more reliable. So that user might have a chance to get the PeerStorage back from recent old peers as well.

[GeorgeTsagk]

Regardless, is it the BOLT's responsibility to document the need for padding?

[GeorgeTsagk]

wouldn't have an option to recover the node and update its state in case they lost data.

can you elaborate a bit more here? don't really get why sending peer_storage_retrieval hurts here

[adi2011]

If data is lost, a node would need to see their storage before they can process the reestablish message.

[thomash-acinq]

In that case, we want "peer_storage_retrieval MUST sent before channel_reestablish", but there is nothing problematic about sending it again after channel_reestablish, it's just unnecessary.

[TheBlueMatt]

If nodes can't rely on their peer having the latest data, presumably they shouldn't be checking against the latest :)

Suggested change

	their data at each reconnection, by comparing the contents of the
	retrieved data with the last one they sent. However, nodes should not expect
	their data at each reconnection, by comparing the contents of the
	retrieved data with one they sent. However, nodes should not expect

[thomash-acinq]

I think you should expect your peer to send you the latest data and check against the latest. However you shouldn't rely on it because your peer may be trying to cheat you. If the data your peer sent you back is not the latest you should mark them as unreliable.

[TheBlueMatt]

IOKTBO definitely applies to all messages, its just in the very initial protocol there's no distinction cause everyone supports everything :). Its definitely nice for these to be odd because then nodes can send peer_storage without bothering to check if their peer supports the feature at all.

[TheBlueMatt]

Is it the IOPS themselves that will destroy the SSD

Yes, over one year 64KiB/second is 2 TB, which is more than most consumer SSD's write lifetimes.

rate limiting to 1 update per htlc

that's even worse :)

ISTM a rate-limit of one/minute makes a lot more sense.

[thomash-acinq]

There is no word here about what this will be used for. If we intend to use it to store critical data that will allow us to recover from a loss of channel state on our side, then we should make that explicit.
For that use-case, it's important that we get the latest blob and we should ensure that an honest peer always gives us the latest blob.

[thomash-acinq]

This seem to imply that peer_storage_retrieval is a reply to peer_storage but the requirements below only say that peer_storage_retrieval must be sent after a reconnection, not as a reply to a peer_storage message.

[thomash-acinq]

I think you should expect your peer to send you the latest data and check against the latest. However you shouldn't rely on it because your peer may be trying to cheat you. If the data your peer sent you back is not the latest you should mark them as unreliable.

[thomash-acinq]

The problem with rate limiting like this combined with no ACK is that you can't guarantee that an honest peer will store the latest data which I think is a very desirable property.

[thomash-acinq]

In that case, we want "peer_storage_retrieval MUST sent before channel_reestablish", but there is nothing problematic about sending it again after channel_reestablish, it's just unnecessary.

[Roasbeef]

There is no word here about what this will be used for. If we intend to use it to store critical data that will allow us to recover from a loss of channel state on our side, then we should make that explicit.

I think that's by design, at the surface level, it's just a blob that you send to the other party and can maybe retrieve in the future. Any higher layer that uses this feature should make minimal assumptions.

For that use-case, it's important that we get the latest blob and we should ensure that an honest peer always gives us the latest blob.

There's no way to ensure that. All we can rely on is a best effort attempt by the peer to maybe retrieve the blob, or not.

[t-bast]

The link to Bolt 1 doesn't work, I think it should rather be (untested):

Suggested change

	| 42/43 | `option_provide_storage` | Can store other nodes' encrypted backup data | IN | | [BOLT #1][bolt01-Peer-storage] |
	| 42/43 | `option_provide_storage` | Can store other nodes' encrypted backup data | IN | | [BOLT #1](01-messaging.md#peer-storage) |

[t-bast]

nit:

Suggested change

	Nodes ask their peers to store data using the `peer_storage` message & expect
	Nodes ask their peers to store data using the `peer_storage` message and expect

[t-bast]

I agree, a SHOULD requirement to pad could make sense here.