Fix sqlite database set_utxo to insert or update utxos#605
Fix sqlite database set_utxo to insert or update utxos#605notmandatory merged 4 commits intobitcoindevkit:masterfrom
Conversation
b41f24e to
b8954a2
Compare
b8954a2 to
1281ffd
Compare
| "INSERT INTO transaction_details SELECT txid, timestamp, received, sent, fee, height FROM transaction_details_old;", | ||
| "DROP TABLE transaction_details_old;", | ||
| "ALTER TABLE utxos ADD COLUMN is_spent;", | ||
| "DELETE FROM utxos WHERE rowid > (SELECT MIN(rowid) FROM utxos u2 WHERE utxos.txid = u2.txid AND utxos.vout = u2.vout);", |
There was a problem hiding this comment.
I discussed a bit with @afilini about this, while theoretically the duplicated lines should contain the same info, we can't be so sure about that...
If the duplicated lines contain different info (for example: same outpoint, different is_spent), we might end up in an inconsistent state.
Alekos suggested wiping the entire db.
I suggested wiping it only if we find some duplicated outpoints with the different info - if this can be done in a single SQL query (should be doable, but I'm not entirely sure). What do you think?
There was a problem hiding this comment.
But I'm so proud of my little SQL de-duplicating! haha, anyway, rather than risk an inconsistent state I'll update this to always wipe the database, it's only a cache after all. And I have to agree with @afilini to wipe the data even if there are no dups since either way we will need to warn users that it MIGHT be wiped on this release and they'll need to be prepared to handle it with a sync() (and any reasonable app will do a sync at startup).
1281ffd to
154b10b
Compare
|
I force-pushed 154b10b to DELETE all data from tables (except for script version table). Tested locally with bdk-cli and verified MIGRATIONS scripts all run and able to re-sync after. I also added a warning in the CHANGELOG that anyone using the |
47936d4 to
4023a7d
Compare
4023a7d to
2471908
Compare
|
This is now ready to go, I also added a fix for our blockchain integration tests in CI that haven't been running. Once this PR is merge I'm going to cherry pick it into a |
danielabrozzoni
left a comment
There was a problem hiding this comment.
utACK 35feb10 - Code looks good, but I didn't do any local test to see if the db gets wiped
|
Thanks for the utACK @danielabrozzoni, I'm going to merge now, should be good based on my manually testing, as discussed on the call this is what I did with the unreleased master branch version of
cargo run --no-default-features --features cli,electrum,sqlite-db -- wallet --descriptor "wpkh(tpubEBr4i6yk5nf5DAaJpsi9N2pPYBeJ7fZ5Z9rmN4977iYLCGco1VyjB9tvvuvYtfZzjD5A8igzgw3HeWeeKFmanHYqksqZXYXGsw5zjnj7KM9/*)" sync
|
|
Post-merge ACK 35feb10 |
Description
This PR fixes #591 by:
MIGRATIONSstatements to remove duplicate utxos and add unique utxos index on txid and vout.set_utxo().set_utxo()doesn't insert duplicate utxos.Notes to the reviewers
I verified the updated
test_utxofails as expected before my fix and passes after the fix. I tested the new migrations using the belowbdk-clicommand and a manually updated sqlite db with duplicate utxos.cargo run --no-default-features --features cli,sqlite-db,esplora-ureq -- wallet -w test1 --descriptor "wpkh(tpubEBr4i6yk5nf5DAaJpsi9N2pPYBeJ7fZ5Z9rmN4977iYLCGco1VyjB9tvvuvYtfZzjD5A8igzgw3HeWeeKFmanHYqksqZXYXGsw5zjnj7KM9/*)" syncChecklists
All Submissions:
cargo fmtandcargo clippybefore committingNew Features:
CHANGELOG.mdBugfixes: