BDK only finding a single transaction for the mining rewards wallet on Regtest

[mightyguava]

Describe the bug
While setting up regtest with BDK, bitcoin mining rewards generated to the wallet specified in generatetowallet is getting funds. No matter how many blocks I generate, BDK seems to only record, arbitrarily but deterministically, only a single transaction.

To Reproduce

The example below uses bdk-cli to make the issue easier to reproduce, but I initially encountered this directly using bdk 0.28.0 as a library in my tests.

I'm running a bitcoind in regtest mode, and Blockstream's Electrs fork (which runs the Esplora REST API on :50000 and the Electrum JSONRPC API on :50001)

docker-compose.yaml

services:
  bitcoind:
    image: lncm/bitcoind:v25.0
    command:
      - -regtest
      - -rpcbind=0.0.0.0:18443
      - -rpcuser=test
      - -rpcpassword=test
      - -rpcallowip=0.0.0.0/0
      - -server=1
      - -txindex=1
      - -prune=0
    volumes:
      - bitcoin-data:/data/.bitcoin
    ports:
      - 18443:18443

  esplora:
    image: blockstream/esplora:electrs-latest
    # Esplora has no arm image. Setting it explicitly to x86 gets rid of a noisy warning.
    platform: linux/amd64
    command:
      - ./electrs_bitcoin/bin/electrs
      - -vvvv
      - --timestamp
      - --daemon-dir=/app/.bitcoin
      - --db-dir=/app/db
      - --network
      - regtest
      - --http-addr=0.0.0.0:50000
      - --electrum-rpc-addr=0.0.0.0:50001
      - --daemon-rpc-addr=bitcoind:18443
      - --cookie=test:test
      - --electrum-txs-limit=1000000
      - --utxos-limit=1000000
    volumes:
      - bitcoin-data:/app/.bitcoin
    ports:
      - 50000:50000
      - 50001:50001

volumes:
  bitcoin-data:

1. Mine a few blocks

❯ bdk-cli wallet -d "wpkh(tprv8ggNwbZwS4zbfiCncSLPediwpth1JKC4HS5XDxpMBCmirfN2sos9sAQUKUpFDr1DmaV3r8RLBbMAQ5qevGN2Jbxjx4tTf4QAFx71sc84APA/84'/1'/0'/0/*)#nuddvulp" get_new_address

bcrt1qrt37mr0kf2th5dgsqq6k87tl8k220e7nj4ts5u

❯ bitcoin-cli -regtest -rpcuser=test -rpcpassword=test generatetoaddress 5 bcrt1qrt37mr0kf2th5dgsqq6k87tl8k220e7nj4ts5u
[
  "10c1d2db9573c017bcd516ddd8e53793b1c67a162a9d52e0ed498eecd20473ab",
  "69c51b6f756d809e3ce86f98f17d68e522f18125ebd20f8840033dd5852cb182",
  "19e5a6d9d8aa0e3afce541a17e63eb971ab050b1ed7dce291b2c87fe15e27a02",
  "6c6e66ec9dca49e7d29ce12c354a5c23808430c5d532a0e852071eec24a8374a",
  "0256b425e7d265d1f6bc80f63d36cd0d865c6d13d2c5e1fab2a132dff2f8c0e6"
]

2. Wait for electrum to index (< 10 seconds)
3. Sync & check balances

❯ RUST_LOG=debug bdk-cli -n regtest wallet -s tcp://localhost:50001 --descriptor "wpkh(tprv8ggNwbZwS4zbfiCncSLPediwpth1JKC4HS5XDxpMBCmirfN2sos9sAQUKUpFDr1DmaV3r8RLBbMAQ5qevGN2Jbxjx4tTf4QAFx71sc84APA/84'/1'/0'/0/*)#nuddvulp" sync
<noisy logs omitted>
[2023-10-05T14:27:49Z DEBUG bdk::blockchain::script_sync] found 5 transactions for script pubkey Script(OP_0 OP_PUSHBYTES_20 1ae3ed8df64a977a3510003563f97f3d94a7e7d3)
<noisy logs omitted>

❯ bdk-cli -n regtest wallet -s tcp://localhost:50001 --descriptor "wpkh(tprv8ggNwbZwS4zbfiCncSLPediwpth1JKC4HS5XDxpMBCmirfN2sos9sAQUKUpFDr1DmaV3r8RLBbMAQ5qevGN2Jbxjx4tTf4QAFx71sc84APA/84'/1'/0'/0/*)#nuddvulp" get_balance
{
  "satoshi": {
    "confirmed": 0,
    "immature": 5000000000,
    "trusted_pending": 0,
    "untrusted_pending": 0
  }
}

❯ bdk-cli -n regtest wallet -s tcp://localhost:50001 --descriptor "wpkh(tprv8ggNwbZwS4zbfiCncSLPediwpth1JKC4HS5XDxpMBCmirfN2sos9sAQUKUpFDr1DmaV3r8RLBbMAQ5qevGN2Jbxjx4tTf4QAFx71sc84APA/84'/1'/0'/0/*)#nuddvulp" list_transactions
[
  {
    "confirmation_time": {
      "height": 3,
      "timestamp": 1696516052
    },
    "fee": 0,
    "received": 5000000000,
    "sent": 0,
    "transaction": null,
    "txid": "410ae4028cd7f073ca7937bfc570c1ff41673f92156a0c3d6fffaded4f992e05"
  }
]

Notice above that BDK only found a single transaction.

4. Check Electrum via the REST API

❯ curl --silent http://localhost:50000/address/bcrt1qrt37mr0kf2th5dgsqq6k87tl8k220e7nj4ts5u | jq
{
  "address": "bcrt1qrt37mr0kf2th5dgsqq6k87tl8k220e7nj4ts5u",
  "chain_stats": {
    "funded_txo_count": 5,
    "funded_txo_sum": 25000000000,
    "spent_txo_count": 0,
    "spent_txo_sum": 0,
    "tx_count": 5
  },
  "mempool_stats": {
    "funded_txo_count": 0,
    "funded_txo_sum": 0,
    "spent_txo_count": 0,
    "spent_txo_sum": 0,
    "tx_count": 0
  }
}

Notice that Electrum indexed all 5 transactions and reports a much larger sum than BDK does.

5. Try this again with another 100 transactions. BDK will still pick up only a single transaction. Sometimes it may pick up one whose block rewards have matured and so there's a usable balance, but usually it'll report 0 spendable balance.

Expected behavior
BDK should see a transaction for every block mined and report the correct balance.

Build environment

• BDK tag/commit: bdk-cli 0.27.1
• OS+version: Mac OSX 13.6
• Rust/Cargo version: cargo 1.72.1 (103a7ff2e 2023-08-15)
• Rust/Cargo target: x86_64-apple-darwin

Additional context

[notmandatory]

As we discussed on discord I think this is related to all the tx being coinbase.

I'll create a new integration test to repro it as you have above and then see if I can figure out what's going on.

nevermind, @mightyguava found #1090 so I think we just need to update bdk-cli to use bdk 0.29 which has the fix.

[danielabrozzoni]

I'm closing this one as it's been fixed on bdk's side, I opened a bug report in bdk-cli instead: bitcoindevkit/bdk-cli#163