Bitcoin wallet tracker (bwt) client

[notmandatory]

bwt is a lightweight wallet xpub tracker and query engine for Bitcoin, implemented in Rust.

Implement block data source trait for bwt server REST API.

depends on #84

[justinmoon]

We'd probably want to directly connect to Bitcoin RPC. Just implement whatever bwt functionality that we need. I have a WIP PR that implements some of this: #22. So I think this could be merged with #79.

[notmandatory]

I agree, it looks like bwt wouldn't be a good fit since you need to start it up with the xpub(s) you want to track and then it only offers a REST API with similar functionality to the RPC api.

I'm closing this issue and if BWT evolves into something less statefull we can reconsider reopening it.

[shesek]

you need to start it up with the xpub(s) you want to track

Does having to configure this when starting up complicate things? I could make it possible to track additional ones in runtime if that helps.

offers a REST API

In addition to the HTTP REST API, bwt also offers an Electrum RPC server, which bdk could connect to using the existing Electrum backend implementation.

Alternatively, it can also be initialized and accessed directly from Rust. You can see an example of that here. The Query API is documented here. One thing that's still not easily doable with the Rust API is subscribing to real-time events, but implementing that would be pretty straightforward. (real-time updates are currently only available via the Electrum RPC subscription mechanism, via HTTP Server-Sent-Events, or using WebHooks.)

if BWT evolves into something less statefull

I'm not sure what you mean; bwt is entirely stateless and has no persistent storage of its own. All the state is managed within Bitcoin Core.

--

One thing that's missing in bwt is support for tracking script descriptors. There's an open PR for this by @justinmoon that I've been meaning to push forward but still haven't (sorry that its been taking so long justin!). Knowing that bwt might find use in bdk if it supported descriptors would definitely give me some motivation though. :-)

[notmandatory]

hi @shesek, thanks for the feed back, bwt is certainly a cool project so I'm happy to reopen this issue to continue the discussion on how we can work together.

What I meant by bwt being stateful is that, as I understand it, when you startup a bwt server you need to tell it which xpubs to track, and then it keeps indexes to track utxos for addresses generated from only those xpubs, and returns corresponding data stored in your full node. But with an Electrum server, or calling core RPC directly, there's no server setup, the wallet client just needs the server's URL, and you can ask it for whatever info those APIs can provide. But I see the advantages with the bwt approach, that your indexing can be much smaller and faster since your node is only tracking what your wallet cares about. It's just a different model then how existing bdk blockchain modules work.

Probably the best first step is, as you mentioned, with bwt-dev/bwt#37. With this PR it sounds like a basic bdk + bwt wallet would work something like this:

1. user generates a new bdk based wallet xprv descriptor
2. bdk wallet also generates and displays the corresponding xpub descriptor
3. user configures the bdk wallet with their bwt server URL and protocol (electrum RPC or [future] bwt REST)
4. user configures their bwt server with the xpub descriptor displayed by bdk wallet

Having a runtime bwt API to add new descriptors to track would make the wallet/server setup experience easier, but it probably more of a nice to have feature right now.

We can use this bdk issue to track testing a bdk wallet with bwt via electrum RPC and possibly also implementing the bwt REST client api if there's interest in that. How does that sound?

[shesek]

as I understand it, when you startup a bwt server you need to tell it which xpubs to track, and then it keeps indexes to track utxos for addresses generated from only those xpubs

Yes, you understand correctly (though the indexes also track transactions, not just utxos). But this is inherent if you're not keeping a full history index for all addresses, not something unique to bwt.

or calling core RPC directly, there's no server setup, the wallet client just needs the server's URL, and you can ask it for whatever info those APIs can provide

The core RPC can't give you the transaction history of arbitrary addresses, you'll have to first import the addresses and do a rescan, which is what bwt does internally. The only way that there'll be no server setup is with a full address index (which core doesn't have).

We can use this bdk issue to track testing a bdk wallet with bwt via electrum RPC and possibly also implementing the bwt REST client api if there's interest in that. How does that sound?

Sounds good! Though I probably wouldn't bother with the REST API for now if you can already get everything you need via the Electrum RPC.

[shesek]

I tried running bdk against bwt, and found an issue that is now fixed (bwt was returning an error from blockchain.scripthash.listunspent if the scripthash has no history, instead of returning an empty set).

With this fix in place, I was able to sync bdk as follows:

$ bitcoind -regtest -datadir=/tmp/bd
$ bwt --network regtest --bitcoind-dir /tmp/bd --xpub vpub5SLqN2bLY4Wea2jFkKNCZUzwRAsAGixRfD4PFq5NJ4zUaQBejH13Km2wQynyL7DXyuVU37cj2qb6XpBtSGsQ9tGPv6z34MZf4MXULEqbPRM
$ repl --descriptor "wpkh(tpubD6NzVbkrYhZ4YRJLsDvYBaXK6EFi9MSV34h8BAvHPzW8RtUpJpBFiL23hRnvrRUcb6Fz9eKiVG8EzZudGXYfdo5tiP8BuhrsBmFAsREPZG4/0/*)" --network regtest --server tcp://127.0.0.1:60401 sync

Note that I had to convert the xpub into its zpub equivalent to get bwt to use p2wpkh. This won't be necessary once descriptor support lands in bwt (it'll accept a --descriptor param similar to bdk's).

--

As an aside, why does bdk use blockchain.scripthash.listunspent? The electrum wallet itself never calls it (which is why I didn't catch that bug earlier), instead it determines the wallet's utxo set based on the transaction history retrieved with get_history, which seems like a more reliable way to do this. (otherwise, the wallet might get into a state where the utxo set and the tx history are out-of-sync and disagree with each other -- so for exmaple you could have a utxo without having the tx funding it in the history, or have a tx in your history that creates a utxo which isn't in your utxo set)