Simplify wallet persistence

[evanlinjin]

Description

Removed the persistence module in bdk_chain (which contained the PersistWith and PersistAsyncWith traits and a Persisted<T> wrapper). Replaced it with simplified versions that are Wallet-specific.

The new traits (WalletPersister and AsyncWalletPersister) are simpler since they have less type-parameters (being wallet-specific). The old versions were more complex as they were intended to be used with any type. However, we need more time to finalize a works-for-all-bdk-types persistence API.

Additionally, WalletPersister and AsyncWalletPersister also introduce the initialize method. It is handy to contain db-initialization (i.e. create tables for SQL) here. We can call initialize in PersistedWallet constructors. So the PersistedWallet::persist method does not need to ensure the database is initialized. To accommodate this, I made the .init_sqlite_table methods on changeset-types public, and removed the call of this in from_sqlite and .persist_to_sqlite. The initialize method now loads from the persister, so the load associated function is removed.

There was a bug in the old PersistAsyncWith trait where the lifetime bounds were not strict enough (refer to the conversation in #1552). This is now fixed in the new AsyncWalletPersister trait.

Docs for the new types are clearer (hopefully). I mentioned implementation details about the new traits and what safety checks were guaranteed in PersistedWallet.

I think it makes sense just to have a wallet-specific persistence API which we will maintain alongside wallet for v1.0. This is less baggage for users and maintainers alike.

Notes to the reviewers

How breaking are these changes?

Unless if you were implementing your own persistence, then not breaking at all! If you were implementing your own persistence for BDK, the breakages are minimal. The main change is introducing the initialize method to persistence traits (which replaces load).

Acknowledgements

I came up with the idea of this while responding to @tnull's suggestions on Discord. Unfortunately we cannot fulfill all of his requests. However, I do believe that this will make things easier for our users.

Changelog notice

• Removed the persist module in bdk_chain.
• Added WalletPersister/AsyncWalletPersister traits and PersistedWallet struct to bdk_wallet. These are simplified and safer versions of old structs provided by the persist module in bdk_chain.
• Change .init_sqlite_tables method to be public on changeset types. from_sqlite and persist_into_sqlite methods no longer call .init_sqlite_tables internally.

Checklists

All Submissions:

• I've signed all my commits
• I followed the contribution guidelines
• I ran cargo fmt and cargo clippy before committing

What is left to do:

• Better docs.

[notmandatory]

Concept ACK

Even though we need to stop making breaking changes, this looks like like a reasonable isolated change that will simplify the persistence API for Wallet users.

[notmandatory]

LGTM, needs fmt, rebase on master, and commits cleanup. I tested with wallet and cli examples and worked fine, including loading existing db created with prior version.

[evanlinjin]

An idea would be to introduce some sort of test framework for WalletPersister and AsyncWalletPersister implementations.

Checks include:

• Calling initialize multiple times should not error, or make database inconsistent.
• Calling initialize after persist should not error, or make database inconsistent.
• What previously went in should come out.

Maybe a more elaborate version of

bdk/crates/wallet/tests/wallet.rs

Line 105 in e0822d7

fn wallet_is_persisted() -> anyhow::Result<()> {

?

[notmandatory]

I did a quick review and found a few doc fixes but otherwise looks good. I'll do a deeper look this afternoon but so far looks ready to go.

[ValuedMammal]

I did a first pass. Also spent some time implementing AsyncWalletPersister for a locally defined type and so far results are positive.

[ValuedMammal]

Suggested change

	pub fn staged_mut(&mut self) -> Option<&mut ChangeSet> {
	pub(crate) fn staged_mut(&mut self) -> Option<&mut ChangeSet> {

[evanlinjin]

I was thinking that this method may be useful for some users. I.e. they don't need to query the staged area twice if they wanted to do something like we do in PersistedWallet::persist.

[ValuedMammal]

For the sake of discussion I think you can do it without ever getting the stage mutably such as having a method on Wallet that resets the stage but doesn't return it, and this would prevent users from freely modifying their changesets.

  /// Clears the stage.
  pub fn clear_stage(&mut self) { self.stage = ChangeSet::default(); }

Then persist would look like

  /// Persist staged changes.
  pub fn persist(&mut self, persister: &mut P) -> Result<bool, P::Error> {
      if let Some(change) = self.inner.staged() {
          P::persist(persister, change)?;
          self.inner.clear_stage();
          return Ok(true);
      }
      Ok(false)
  }

[oleonardolima]

Agree, I think it's a better approach to have/use a clear_stage method, having explicit and clear meaning.

[ValuedMammal]

wondering if this would achieve the same thing

Suggested change

	fn persist<'a>(
	persister: &'a mut Self,
	changeset: &'a ChangeSet,
	) -> FutureResult<'a, (), Self::Error>
	where
	Self: 'a;
	fn persist<'a>(&'a mut self, changeset: &'a ChangeSet) -> FutureResult<'a, (), Self::Error>;

[ValuedMammal]

oh, I think we still need Self: 'a.

[evanlinjin]

Also, I made purposely made persist a non-member so that it's harder to call haha

[evanlinjin]

@ValuedMammal @notmandatory I updated the docs! Let me know if everything makes sense.

I'll be disappearing this weekend and be back Tuesday night. Feel free to push on commits for more doc fixes and merge (if appropriate).

[oleonardolima]

Concept ACK
It's looking good to me. I guess I found only two documentation fixes.

[ValuedMammal]

ACK f434c93 (pending comments by @oleonardolima )

[ValuedMammal]

small nit:

It might be more convenient for the implementer if the return type is Result<Option<ChangeSet>, Self::Error>.

[notmandatory]

ACK 340808e