-
Notifications
You must be signed in to change notification settings - Fork 320
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactor(wallet)!: rework persistence, changeset, and construction #1514
refactor(wallet)!: rework persistence, changeset, and construction #1514
Conversation
47fb0ac
to
5146a3b
Compare
Does the |
e66f755
to
1d7ddfe
Compare
Rework sqlite: Instead of only supported one schema (defined in `bdk_sqlite`), we have a schema per changeset type for more flexiblity. * rm `bdk_sqlite` crate (as we don't need `bdk_sqlite::Store` anymore). * add `sqlite` feature on `bdk_chain` which adds methods on each changeset type for initializing tables, loading the changeset and writing. Rework changesets: Some callers may want to use `KeychainTxOutIndex` where `K` may change per descriptor on every run. So we only want to persist the last revealed indices by `DescriptorId` (which uniquely-ish identifies the descriptor). * rm `keychain_added` field from `keychain_txout`'s changeset. * Add `keychain_added` to `CombinedChangeSet` (which is renamed to `WalletChangeSet`). Rework persistence: add back some safety and convenience when persisting our types. Working with changeset directly (as we were doing before) can be cumbersome. * Intoduce `struct Persisted<T>` which wraps a type `T` which stores staged changes to it. This adds safety when creating and or loading `T` from db. * `struct Persisted<T>` methods, `create`, `load` and `persist`, are avaliable if `trait PersistWith<Db>` is implemented for `T`. `Db` represents the database connection and `PersistWith` should be implemented per database-type. * For async, we have `trait PersistedAsyncWith<Db>`. * `Wallet` has impls of `PersistedWith<rusqlite::Connection>`, `PersistedWith<rusqlite::Transaction>` and `PersistedWith<bdk_file_store::Store>` by default. Rework wallet-construction: Before, we had multiple methods for loading and creating with different input-counts so it would be unwieldly to add more parameters in the future. This also makes it difficult to impl `PersistWith` (which has a single method for `load` that takes in `PersistWith::LoadParams` and a single method for `create` that takes in `PersistWith::CreateParams`). * Introduce a builder pattern when constructing a `Wallet`. For loading from persistence or `ChangeSet`, we have `LoadParams`. For creating a new wallet, we have `CreateParams`.
Remove returning `Result` for builder methods on `CreateParams` and `LoadParams`.
1d7ddfe
to
22d02ed
Compare
Here are some perspectives from @LLFourn. The From the other angle, lets see what will be required to have sqlite features in a separate crate...
Is there a drawback to have a feature flag for |
We already know that there will be more than one SQL based store for We don't want to make every data store a feature flagged options in the chain crate. Likely other data store crates won't even live in the Same goes for the We already have to update downstream blockchain client crates when the |
@notmandatory you can still implement custom persistence externally. It just makes sense from a maintenance point of view to update the encoding/decoding together with the types for the persistence backends that BDK plans to maintain. Edit: My current stance is I'm okay with both. I'll see what @LLFourn 's arguments are tomorrow. If my stance doesn't change, I'll create a commit to split the sqlite stuff out and have everyone else decide which one to go forward with. Edit: if we have feature flags for different DBs on |
I do think we should be able to update the persistence related dependencies without having to bump the chain and wallet crates. I feel pretty strongly that sqlite dependencies, even optional, don't belong in the chain (or wallet) crate. |
These kinds of choices as much of an art as they are a science so intuition is important. I think it's important from the outset to understand that it's easy to go to an external crate from feature flags in a major release if we belive it's going to be stable for a while. Also it's not an either/or decision. You can have feature flags on In so far as there is a science to making these decisions here are the considerations I see:
Taking these into account, I think For
Just noting, this works without too much friction: For each of these general databases you make a feature flag. For custom project specific databases it's easy since you can implement |
Ok let's go ahead and start as @evanlinjin has it and we can re-evaluate for the next major release if we like it or want to move the rusqlite store into it's own crate. Post beta release I'll help @matthiasdebernardini adapt his sqlx+postgres db store and see how much of a hassle wrapping the sqlx db connection pool is. |
Previously, `Persist{Async}With::persist` can be directly called as a method on the type (i.e. `Wallet`). However, the `db: Db` (which is an input) may not be initialized. We want a design which makes it harder for the caller to make this mistake. We change `Persist{Async}With::persist` to be an "associated function" which takes two inputs: `db: &mut Db` and `changeset`. However, the implementer cannot take directly from `Self` (as it's no longer an input). So we introduce a new trait `Staged` which defines the staged changeset type and a method that gives us a `&mut` of the staged changes.
|
||
/// A wrapper that we use to impl remote traits for types in our crate or dependency crates. | ||
pub struct Impl<T>(pub T); | ||
|
||
impl<T> From<T> for Impl<T> { | ||
fn from(value: T) -> Self { | ||
Self(value) | ||
} | ||
} | ||
|
||
impl<T> core::ops::Deref for Impl<T> { | ||
type Target = T; | ||
|
||
fn deref(&self) -> &Self::Target { | ||
&self.0 | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@LLFourn would you be okay with this?
Also fix imports and rename `sqlite` module to `rusqlite_impl`.
2b474ce
to
2cf07d6
Compare
Things to do later: We need docs for how to impl Basically, the following two constraints need to be met for the associated types: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ACK 2cf07d6
I primarily focused on the Wallet and persistence API changes and overall this is a big improvement. I added a few small nits, feel free to ignore them, they can be fixed after the beta release.
The comments from @ValuedMammal should be addressed before merging.
self.index.index_txout(outpoint, txout); | ||
} | ||
|
||
self.graph.apply_changeset(changeset.graph); | ||
self.graph.apply_changeset(changeset.tx_graph); | ||
} | ||
|
||
/// Determines the [`ChangeSet`] between `self` and an empty [`IndexedTxGraph`]. | ||
pub fn initial_changeset(&self) -> ChangeSet<A, I::ChangeSet> { | ||
let graph = self.graph.initial_changeset(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
⛏️ small nit.
let graph = self.graph.initial_changeset(); | |
let tx_graph = self.graph.initial_changeset(); |
@@ -89,21 +94,30 @@ where | |||
pub fn apply_update(&mut self, update: TxGraph<A>) -> ChangeSet<A, I::ChangeSet> { | |||
let graph = self.graph.apply_update(update); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
⛏️ same small nit.
let graph = self.graph.apply_update(update); | |
let tx_graph = self.graph.apply_update(update); |
ChangeSet { | ||
tx_graph: graph, | ||
indexer, | ||
} | ||
} | ||
|
||
/// Insert a floating `txout` of given `outpoint`. | ||
pub fn insert_txout(&mut self, outpoint: OutPoint, txout: TxOut) -> ChangeSet<A, I::ChangeSet> { | ||
let graph = self.graph.insert_txout(outpoint, txout); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
⛏️ same nit.
let graph = self.graph.insert_txout(outpoint, txout); | |
let tx_graph = self.graph.insert_txout(outpoint, txout); |
ChangeSet { | ||
tx_graph: graph, | ||
indexer, | ||
} | ||
} | ||
|
||
/// Insert and index a transaction into the graph. | ||
pub fn insert_tx(&mut self, tx: Transaction) -> ChangeSet<A, I::ChangeSet> { | ||
let graph = self.graph.insert_tx(tx); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
⛏️ nit and same for rest of file.
let graph = self.graph.insert_tx(tx); | |
let tx_graph = self.graph.insert_tx(tx); |
pub struct ChangeSet<K> { | ||
/// Contains the keychains that have been added and their respective descriptor | ||
pub keychains_added: BTreeMap<K, Descriptor<DescriptorPublicKey>>, | ||
pub struct ChangeSet { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
📖 docs need to be updated since K was removed no more K to descriptor mapping.
|
||
use super::{ChangeSet, LoadError, PersistedWallet}; | ||
|
||
/// This atrocity is to avoid having type parameters on [`CreateParams`] and [`LoadParams`]. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
⛏️ I'd avoid using the word "atrocity" you're only doing what has to be done. 🙂
I am working on cleaning up a few things on this PR. I am not changing anything about the sqlite queries that are actually run. My main goal is to have This is mostly done but I need another 24h to modify the API builder things that evan created to work with this new approach. |
After quite some work I've decided to abandon the work I was doing for now. Although I made a lot of progress on making things nicer internally I don't think it's worth investing the time to finish the job -- in particular the tests are going to take me another few hours that I don't have right now. In any case, this PR can be merged as there are no severe issues and it's a big step forward and only a few smalls steps back which can be addressed soonish. I fixed the bugs discovered by @ValuedMammal above. I will leave testing them for now. The merge bug is no longer really expressible in the PR I was working on due to some improvements I made FWIW. ACK: 64eb576 other comments: While developing I very much turned against the As I discussed with @evanlinjin I think the long term solution here is for every method that wants to persist changes to take a E.g.: let db = ...;
let mut wallet = db.persist(|changeset| {
Wallet::new(params, changeset)
});
// now i want to get an address
let address = db.persist(|changeset| {
wallet.reveal_next_address(KeychainKind::External, changeset)
}); So the idea would be that |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Re ACK 64eb576
@evanlinjin and @LLFourn thanks for moving the rusqlite impl code into it's own module and final cleaning for the issues @ValuedMammal found.
Closes #1496
Closes #1498
Closes #1500
Description
Rework sqlite: Instead of only supported one schema (defined in
bdk_sqlite
), we have a schema per changeset type for more flexiblity.bdk_sqlite
crate (as we don't needbdk_sqlite::Store
anymore).sqlite
feature onbdk_chain
which adds methods on each changeset type for initializing tables, loading the changeset and writing.Rework changesets: Some callers may want to use
KeychainTxOutIndex
whereK
may change per descriptor on every run. So we only want to persist the last revealed indices byDescriptorId
(which uniquely-ish identifies the descriptor).keychain_added
field fromkeychain_txout
's changeset.keychain_added
toCombinedChangeSet
(which is renamed toWalletChangeSet
).Rework persistence: add back some safety and convenience when persisting our types. Working with changeset directly (as we were doing before) can be cumbersome.
struct Persisted<T>
which wraps a typeT
which stores staged changes to it. This adds safety when creating and or loadingT
from db.struct Persisted<T>
methods,create
,load
andpersist
, are available iftrait PersistWith<Db>
is implemented forT
.Db
represents the database connection andPersistWith
should be implemented per database-type.trait PersistedAsyncWith<Db>
.Wallet
has impls ofPersistedWith<rusqlite::Connection>
,PersistedWith<rusqlite::Transaction>
andPersistedWith<bdk_file_store::Store>
by default.Rework wallet-construction: Before, we had multiple methods for loading and creating with different input-counts so it would be unwieldly to add more parameters in the future. This also makes it difficult to impl
PersistWith
(which has a single method forload
that takes inPersistWith::LoadParams
and a single method forcreate
that takes inPersistWith::CreateParams
).Wallet
. For loading from persistence orChangeSet
, we haveLoadParams
. For creating a new wallet, we haveCreateParams
.Notes to the reviewers
TODO
Changelog notice
Checklists
All Submissions:
cargo fmt
andcargo clippy
before committingNew Features: