multi: implement new safe static channel backup and recovery scheme, RPCs, and cli commands

[Roasbeef]

Overview

In this PR, we implement a new safe scheme for static channel backups (SCB's) for lnd. We say safe, as care has been taken to ensure that there are no foot guns in this method of backing up channels, vs doing things like rsyncing or copying the channel.db file periodically. Those methods can be dangerous as one never knows if they have the latest state of a channel or not. Instead, we aim to provide a simple safe instead to allow users to recover the settled funds in their channels in the case of partial or complete data loss. The backups themselves are encrypted using the a key derived from the user's seed, this way we protect privacy of the users channels in the back up state, and ensure that a random node can't attempt to import another user's channels.

Once this PR is merged, given their seed and the latest back up file, the user will be able to recover both their on-chain funds, and also funds that are fully settled within their channels. By "fully settled" we mean funds that are in the base commitment outputs, and not HTLCs. We can only restore these funds as right after the channel is created, we have all the data required to make a backup. In contrast, in order to resolve HTLCs, we would also need to update the backup state with each new channel update, which is tricky to do without additional infrastructure. This infrastructure will be built out in the near future, but until then we have this scheme which will also be a fall back in the scenario that any higher level mechanisms fail.

At a later point, we also plan to propose this backup scheme as an addition to the spec, as even with the change to make the "to self" outputs static, we still need this SCB information in order to restore user funds. Additionally, the current serialization format is a bit up in the air. Atm, we use the same "codec" as we do within the wire protocol for the BOLT specs. However, we'll likely move to a TLV (type-length-value) format as it's extremely flexible and allows us to add/remove fields in the future once we gain new channel types, or modifications are made in the protocol that warrant a change to the backup format. Most importantly, if aezeed and this chanbackup scheme are added to the spec, then it will be possible to write a simple program, that given a seed+backup from any of the implementations, will be able to recover all funds (sweep to an address) the shutdown.

Recovery Flow

Skipping the backup flow for a second, given their 24-word aezeed seed, and a special channels.backup file, the recovery flow would be something like the following

1. The user uses lncli create or the gRPC WalletUnlocker.Init call to input their seed and fully serialized backups.

   1. Alternatively, if they already have a new node set up, they can use the cli and RPC commands to import channels one at a time, or the entire file.

2. lnd boots up and the wallet performs a rescan from the wallet's birthday (encoded in their aezeed) to restore all on-chain funds. Once this process is complete, the main lnd server will start up.

3. Given the set of channels to recover, the server will then (using the new chanbackup) package, will insert a series of "channel shells" into the database. These contains only the information required to initiate the DLP (data loss protection) protocol and nothing more. As a result, they're makred as "recovered" channels in the database, and we'll disallow trying to use then for any other process.

4. Once the channel is recovered, the chanbackup package will attempt to insert a LinkNode that contains all prior addresses that we were able to reach the peer at. During the process, we'll also insert the edge for that channel (only out outgoing direction) into the database as well.

5. lnd will then start up, and as usual attempt to establish connections to all peers that we have channels open with.

6. Once we connect with a peer, we'll then initiate the DLP protocol. The remote peer will discover that we've lost data, and then immediately force close their channel. Before they do though, they'll send over their latest unrevoked commitment point which we need to derive keys (will be fixed in BOLT 1.1 by making the key static) to sweep our funds.

7. Once the commitment transaction confirms, given information within the SCB we'll re-derive all keys we need, and then sweep the funds.

Backup + Recovery Methods

This PR exposes multiple safe ways to backup and recover a channel. We expect only one of them to be used primarily by unsophisticated end users, but have provided other mechanisms for more advanced users and business that already script lnd via the gRPC system.

First, the easiest method for backup+recovery. After this PR, lnd will maintain a channels.backup file in the same location that we store all the other files. Users will at any time be able to safely copy and backup this file. Each time a channel is opened or closed, lnd will update this file with the latest channel state. Users can use scripts to detect changes to the file, and upload them to their backup location. Something like fsnotify can notify a script each time the file changes to be backed up once again. The file is encrypted using an AEAD scheme, so it can safely be stored plainly in cloud storage, your SD card, etc. The file uses a special format and can be used to import via any of the recovery methods described below.

The second mechanism is via the new SubscribeChanBackups steaming gRPC method. Each time an channel is opened or closed, you'll get a new notification with all the chanbackup.Single files (described below), and a single chanbackup.Multi that contains all the information for all channels.

Finally, users are able to request a backup of a single channel, or all the channels via the cli and RPC methods. Here's an example, of a few ways users can obtain backups, see the PR for full details:

⛰ lncli --network=simnet exportchanbackup --chan_point=29be6d259dc71ebdf0a3a0e83b240eda78f9023d8aeaae13c89250c7e59467d5:0
{
    "chan_point": "29be6d259dc71ebdf0a3a0e83b240eda78f9023d8aeaae13c89250c7e59467d5:0",
    "chan_backup": "02e7b423c8cf11038354732e9696caff9d5ac9720440f70a50ca2b9fcef5d873c8e64d53bdadfe208a86c96c7f31dc4eb370a02631bb02dce6611c435753a0c1f86c9f5b99006457f0dc7ee4a1c19e0d31a1036941d65717a50136c877d66ec80bb8f3e67cee8d9a5cb3f4081c3817cd830a8d0cf851c1f1e03fee35d790e42d98df5b24e07e6d9d9a46a16352e9b44ad412571c903a532017a5bc1ffe1369c123e1e17e1e4d52cc32329aa205d73d57f846389a6e446f612eeb2dcc346e4590f59a4c533f216ee44f09c1d2298b7d6c"
}

⛰ lncli --network=simnet exportchanbackup --all
{
    "chan_points": [
        "29be6d259dc71ebdf0a3a0e83b240eda78f9023d8aeaae13c89250c7e59467d5:0"
    ],
    "multi_chan_backup": "fd73e992e5133aa085c8e45548e0189c411c8cfe42e902b0ee2dec528a18fb472c3375447868ffced0d4812125e4361d667b7e6a18b2357643e09bbe7e9110c6b28d74f4f55e7c29e92419b52509e5c367cf2d977b670a2ff7560f5fe24021d246abe30542e6c6e3aa52f903453c3a2389af918249dbdb5f1199aaecf4931c0366592165b10bdd58eaf706d6df02a39d9323a0c65260ffcc84776f2705e4942d89e4dbefa11c693027002c35582d56e295dcf74d27e90873699657337696b32c05c8014911a7ec8eb03bdbe526fe658be8abdf50ab12c4fec9ddeefc489cf817721c8e541d28fbe71e32137b5ea066a9f4e19814deedeb360def90eff2965570aab5fedd0ebfcd783ce3289360953680ac084b2e988c9cbd0912da400861467d7bb5ad4b42a95c2d541653e805cbfc84da401baf096fba43300358421ae1b43fd25f3289c8c73489977592f75bc9f73781f41718a752ab325b70c8eb2011c5d979f6efc7a76e16492566e43d94dbd42698eb06ff8ad4fd3f2baabafded"
}

⛰ lncli --network=simnet exportchanbackup --all --output_file=channels.backup

⛰ ll channels.backup
-rw-r--r--  1 roasbeef  staff   381B Dec  9 18:16 channels.backup

Static Channel Backup Scheme

Crypto

For encryption, we utilize chacha20poly1305 with a random 24 byte nonce. We use a larger nonce size as this can be safely generated via a CSPRNG without fear of frequency collisions between nonces generated. To encrypt a blob, we then use this nonce as the AD (associated data) and prepend the nonce to the front of the ciphertext package.

For key generation, in order to ensure the user only needs their passphrase and the backup file, we utilize the existing keychain to derive a private key. In order to ensure that at we don't force any hardware signer to be aware of our crypto operations, we instead opt to utilize a public key that will be hashed to derive our private key. The assumption here is that this key will only be exposed to this software, and never derived as a public facing address.

chanbackup.Single

The SCB contains all information required to initiate the data loss protection protocol once we restore the channel and connect to the remote channel peer.

The primary way outside callers will interact with this package are via the Pack and Unpack methods. Packing means writing a serialized+encrypted version of the SCB to an io.Writer. Unpacking does the opposite.

The encoding format itself uses the same encoding as we do on the wire within Lightning. Each encoded backup begins with a version so we can easily add or modify the serialization format in the future, if new channel types appear, or we need to add/remove fields. The backup contains:

• The chain a channel belongs to.
• The chanPoint of the channel.
• The shortChanID of the channel.
• The public key of the remote node.
• The series of addresses that we can use to reach the node.
• The CSV delay of the channel (required to later reconstruct our output script after BOLT 1.1)
• A keychain.KeyLocator that allows us to re-derive the payment bas epoint we need to sweep our funds .
• A keychain.KeyDescriptor that we need in order to re-derive our shachain root to validate the information the remote party gives us during the DLP protocol. (see the next section for the complications that arose here)

chanbackup.Multi

Multi is a series of static channel backups. This type of backup can contains ALL the channel
backup state in a single packed blob. This is suitable for storing on your file system, cloud storage, etc. Systems will be in place within lnd to ensure that one can easily obtain the latest version of the Multi for the node, and also that it will be kept up to date if channel state changes.

Implementation Complications and Open Questions

The main complication that arose during the implementation was that I realized late in development, that we also need to backup the details w.r.t how we derive out shachain root. We got a bit lucky here as we store the private key we use as the root, and not the public key itself. In order to derive the shachain roots, we use a special keychain.KeyFamily. However, we don't store the keychain.KeyLocator information which is a two-tuple that allow us to derive a key w/o knowing the public key or having any state in the wallet. Instead, within the backup, we're forced to store the entire public key and not just the key locator information. As a result, I needed to modify keychain.SecretKeyRing.DerivePrivKey to support a brute force scan to allow us to derive the key. In the future, we'll want to do a migration to also store the key locator information so we don't need to always do this brute force. In order to ensure we don't scan to infinity if we don't actually know the public key, I've added a cap on the max number of iterations.

As a result of the case above, it's now the case that any future hardware signers need to be aware of the shachain protocol, in order to generate and validate any points we receive.

The one other section that we maybe want to modify is the way we derive the key we use for encryption. We made an attempt to ensure that any future hardware signers don't actually need to understand our encryption protocol. So instead what we do is use a public point with the assumption that it will never be used for an address and be unveiled to the outside world. One alternative that I had (but scrapped, idk why TBH) is use a point, but then have the hardware signer provide us with an ECDH of that point and another. This would ensure that the key is derived from secret data, but allow us to not store any private data in the backup.

TODO's

• write integration tests

• write additional unit tests in channeldb

• real world recovery attempts

• update docs on how to use the recovery tools

• after rpc: Add SubscribeChannels RPC. #1988 is in, finish hooking up the chanbackup.SubSwapper so we can auto update the backup file on disk

Fixes #175

[alexbosworth]

Suggested change

	// UpdateAndSwap will attempt write a new temporary backup file to disk with
	// UpdateAndSwap will attempt to write a new temporary backup file to disk with

[lsching17]

"First, the easiest method for backup+recovery. After this PR, lnd will maintain a channels.backup file in the same location that we store all the other files. .."

Can a dedicated folder be used? If it is mounted with sshfs or nfs, the channels.backup and channel.db files can be separated into different machine.

[Roasbeef]

Can a dedicated folder be used?

I don't see why not. We can add a config flag for the backup file location.

[Roasbeef]

Alrighty, I've broken this PR up into 5 distinct PR's. Each new PR depends on the prior PR. As a result, they can go in one by one and be reviewed in smaller units, rather than waiting for the final dependents of this larger PR to be finalized. I'll keep this one as is though as it has the full description, and also builds allowing users to experiment with the set of commands. Once the final PR is ready for review (as all the prior PRs have been merged), I'll rebase this on on top of that, so everyone can use this as a central point of end to end testing.

[Roasbeef]

Pushed up a rebased version as all the dependent PRs have been merged. Once in #1988 is in, then I'll start the final push to getting this merged!

[Roasbeef]

Pushed up a new version that maintains the backup file on disk and modifies it based on new/closed channels. Will push up the integration tests next, and after that it's ready for review.

[molxyz]

Tested on a testnet node that has been running with noseedbackup, SCB still let me do exportchanbackup. Shouldn't this result in an error message instead?
https://hastebin.com/raw/urohupeful

[Roasbeef]

@molxyz at runtime, lnd doesn't know if you actually got a seed or not.

[Roasbeef]

In that case, you wouldn't actually be able to decrypt the SCB unless you read out the private data of the database.

[cfromknecht]

Awesome work on this feature @Roasbeef!

It is time to start securing our bags.

LGTM 💰

[ZapUser77]

Any chance you can include what the commands are to restore (exact syntax), and what the expected outputs would be (just and example)? Considering how important this is, just guess and 'tying to figure it out' may not be the best idea.

From my understanding, this isn't actually a "back up" of the channels, and is instead a "channel funds recovery mechanism". Correct? If you restored using this, you'd have a node with zero channels, and would have to start open channels from scratch. Correct?

[Roasbeef]

Check out the PR description, more docs will be provided later.

…

On Mon, Apr 1, 2019, 5:23 PM ZapUser77 ***@***.***> wrote: Any chance you can include what the commands are to restore (exact syntax), and what the expected outputs would be (just and example)? Considering how important this is, just guess and 'tying to figure it out' may not be the best idea. From my understanding, this isn't actually a "back up" of the channels, and is instead a "channel funds recovery mechanism". Correct? If you restored using this, you'd have a node with zero channels, and would have to start open channels from scratch. Correct? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#2313 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA87Lk9FdliyMFr1ImFE7wRLrAHHFsSZks5vcqL7gaJpZM4ZMeKM> .

[ZapUser77]

"Check out the PR description"
I did, read the entire thing, multiple times. I wouldn't have asked before reading.

"more docs will be provided later."
Thanks as always for your diligent hard work. It really is appreciated.

[gijswijs]

Would it be possible to create a "backup" manually, in the scenario where a node is lost and accessing the original channels.backup file isn't accessible anymore?

Put in another way, given that you know the chan_point, could you insert the "channel shell" into the database, so that the DLP protocol can be initiated?