Design assumeUTXO user flow

[GBKS]

AssumeUTXO is a bootstrapping technique that will allow a user to see some data about their wallet, and possible make transactions, before having gone through the full initial block download. Let's figure out how this works and what the best user experience for this feature is.

There seem to be a few steps:

• User opts-in to this feature, or it is enabled by default
• The application downloads the startup data (just going to call it this for sake of a better term) automatically, or the user uploads it from their hard drive
• Initial block download starts for full validation
  • Do we communicate this?
• The wallet can show some data
  • Wallet balance
  • Can it also show the payment history?
  • Can the user spend?
• Initial block download finds a discrepancy to the startup data
  • How to resolve this?
• Initial block download finishes
  • Do we communicate this? What changes for the user?
• The user can create and download their own startup data to use elsewhere

Some detail to sort through here. Let's figure it out.

--

List of related issues and PRs (update as we go):

• AssumeUTXO: Block clock changes #91
• UI Only AU Snapshot Loading Flow bitcoin-core/gui-qml#421

[yashrajd]

Based on what I read, it seems the feature is opt-in and disabled by default. I've created a doc to track user flow/spec.

Also a few questions from me too, hope they're relevant:

• Can assumeutxo be enabled anytime during IBD (presumably before IBD reaches the snapshot block height) or only before onboarding starts?
• Is downloading UTXO snapshot from peers possible in 26.0?
• What’s the block height at which the snapshot hash is hardcoded in 26.0? Would it be updated in every Core version? Would it be able to partially validate snapshots taken at higher block heights?
• Do we allow snapshots that fail hash validation to be loaded?
• Should the user have to wait for the UTXO set to be updated till network tip before they can use wallet functions? (Does code prioritize processing blocks above the snapshot height to make things faster?)
• Should the block clock (IBD status metrics/display) change in any way (if yes, what) once a snapshot is successfully loaded?
• Disk space requirement: It seems 2 UTXO sets will be maintained until a certain point in IBD. What point is that? How much extra in disk space requirements would be required & needed to be communicated? Does this look different on whether the snapshot is downloaded from peers or loaded from a file?
• Can snapshot validation be skipped if the user themselves generated the snapshot (or any other scenarios)?

[yashrajd]

Answers to questions above would be very helpful to get more clarity, but following might be one way assumeutxo could potentially be accommodated within current user flows. Looking for feedback and thoughts on this.

(edit: added following headings)
During onboarding:

Post-onboarding:

[D33r-Gee]

Hello @GBKS and @yashrajd

It’s great to see that you are both figuring out the best UX to make use of this new powerful feature!

I have been working on an early implementation of AssumeUTXO (AU) into qml for the last few months.

The current wip branch can be found here

Also here are some screenshots of the progress so far (Built on WSL Ubuntu 22.04 on signet)

Please keep in mind that this only place holder stuff to test the functionality and should not be considered anywhere near deployment ready UX ;)

And here are some pictures of the mainnet snapshot loaded onto a Samsung Galaxy S8+ with 64GB of storage:

This is the total storage use after full snapshot validation:

[D33r-Gee]

@GBKS

There seem to be a few steps:

• User opts-in to this feature, or it is enabled by default

This could be OS dependent i.e. on Android AU is the default and on others it’s opt-in and IBD is the default

• The application downloads the startup data (just going to call it this for sake of a better term) automatically, or the user uploads it from their hard drive

Currently the user has to have a snapshot already generated or downloaded separatly

• Initial block download starts for full validation

  • Do we communicate this?

Yes that a great idea

• The wallet can show some data

  • Wallet balance
  • Can it also show the payment history?
  • Can the user spend?

Haven’t tested with wallet yet will do soon…

• Initial block download finds a discrepancy to the startup data

  • How to resolve this?

Currently it stops the loading of the snapshot and keeps going with regular IBD

• Initial block download finishes

  • Do we communicate this? What changes for the user?

Here I’d say that when the snapshot is fully validated perhaps a simple notification would be a nice to have.
Currently one has to check the debug log to see if the background validation has completed

• The user can create and download their own startup data to use elsewhere
  Yes

[D33r-Gee]

@yashrajd

Thanks for your questions, hope my answers help…

Based on what I read, it seems the feature is opt-in and disabled by default. I've created a doc to track user flow/spec.

Also a few questions from me too, hope they're relevant:

• Can assumeutxo be enabled anytime during IBD (presumably before IBD reaches the snapshot block height) or only before onboarding starts?

AU can be initiated anytime before full IDB has finished synchronizing.
Once launched, it will check and wait till IBD headers have hit the snapshot block height then it activates the snapshot (see ActivateSnapshot()), for more info please see the mark down doc here

• Is downloading UTXO snapshot from peers possible in 26.0?

Not that I know of in the doc it says: “There is currently no canonical source for snapshots,...”

• What’s the block height at which the snapshot hash is hardcoded in 26.0?

The block heights that were used to test and validate the AU feature were on:

mainnet 780,000 and 800,000

signet 160,000

testnet 2,500,000

*Would it be updated in every Core version?

Anyone with a full IDB can generate their own snapshot at whatever height they prefer using the following dev tool: ./contrib/devtools/utxo_snapshot.sh
More information can be found in the original PR

Would it be able to partially validate snapshots taken at higher block heights?

I haven’t tested this but my guess is that no it would return an error.

• Do we allow snapshots that fail hash validation to be loaded?

No if the hash validation fails the snapshot loading process stops.

• Should the user have to wait for the UTXO set to be updated till network tip before they can use wallet functions? (Does code prioritize processing blocks above the snapshot height to make things faster?)

Great question, I currently don’t know haven’t tested with a wallet yet (Definitely a TODO)

• Should the block clock (IBD status metrics/display) change in any way (if yes, what) once a snapshot is successfully loaded?

With the test I’ve been doing as soon as the snapshot is loading, the block clock displays 56% completion

• Disk space requirement: It seems 2 UTXO sets will be maintained until a certain point in IBD. What point is that?

Until the back background validation of the snapshot has completed on signet it remains around 1 GB

On mainnet it can get up 30 GB while both chainstates are present then it drops to 21 GB after full validation (These number may differ depending on the OS)

• How much extra in disk space requirements would be required & needed to be communicated?

Well it doesn’t require more space than the raw IBD however it is more than just the prune state, so something to play with most definitely

• Does this look different on whether the snapshot is downloaded from peers or loaded from a file?

At this moment there is no peers download option, so no

• Can snapshot validation be skipped if the user themselves generated the snapshot (or any other scenarios)?

No, it still has to go through the full validation to ensure chainstate fidelity

[yashrajd]

edit: deleting some irrelevant quoted text, plus minor edit in storage question

Thanks for getting the ball rolling on the dev side and even the UI @D33r-Gee great stuff! Your answers were very helpful, my designs (not included here) hopefully reflect that. I still have a few questions:

• Can assumeutxo be enabled anytime during IBD (presumably before IBD reaches the snapshot block height) or only before onboarding starts?

AU can be initiated anytime before full IDB has finished synchronizing. Once launched, it will check and wait till IBD headers have hit the snapshot block height then it activates the snapshot (see ActivateSnapshot()), for more info please see the mark down doc here

My understanding from the doc is that "activate" basically happens automatically after loading (which is the user-initiated activity) and validation of the snapshot. Is that correct? I ask coz I noticed that one of your screen mockups has an activate toggle in addition to the Load snapshot toggle. Is it required and the user expected to decide to toggle it at some point?

Another question here: what's the time scale of loading & validating processes expected to take?
If it's several seconds or longer, we might need a progress/transition screen etc.

• What’s the block height at which the snapshot hash is hardcoded in 26.0?

The block heights that were used to test and validate the AU feature were on:

mainnet 780,000 and 800,000

signet 160,000

testnet 2,500,000

*Would it be updated in every Core version?

Anyone with a full IDB can generate their own snapshot at whatever height they prefer using the following dev tool: ./contrib/devtools/utxo_snapshot.sh More information can be found in the original PR

Would it be able to partially validate snapshots taken at higher block heights?

I haven’t tested this but my guess is that no it would return an error.

Assuming the snapshot generation takes place at the current block height but the hardcoded hashes are at the block heights you mentioned, wouldn't it always result in a mismatch during validation?

• Disk space requirement: It seems 2 UTXO sets will be maintained until a certain point in IBD. What point is that?

Until the back background validation of the snapshot has completed on signet it remains around 1 GB

On mainnet it can get up 30 GB while both chainstates are present then it drops to 21 GB after full validation (These number may differ depending on the OS)

• How much extra in disk space requirements would be required & needed to be communicated?

Well it doesn’t require more space than the raw IBD however it is more than just the prune state, so something to play with most definitely

I take this to mean, yes, if a snapshot needs to be loaded the space requirement would increase but I'm unsure if this is right and by how much?
If it's right we should specify and communicate the same on the Storage screen during onboarding.

Also, if it's a lot, and the user tries to load a snapshot AFTER the onboarding UX is done and IBD in progress, do we just disallow loading a screenshot?

• Does this look different on whether the snapshot is downloaded from peers or loaded from a file?

At this moment there is no peers download option, so no

• Can snapshot validation be skipped if the user themselves generated the snapshot (or any other scenarios)?

No, it still has to go through the full validation to ensure chainstate fidelity

[yashrajd]

Adding a few mockups based on the scheme, would love to know what others think.

** Loading a snapshot (before IBD) **

                                                         |
                                                         |
                                                         |
                                                         v

                                                         |
                                                         |
                                                         |
                                                         v     

The above flow could sit in the onboarding below as detailed below:

Post-onboarding:

Once onboarding user flow is complete, the user can go to Settings > Wallet > AssumeUTXO to load (and later, generate) the snapshot. The load snapshot user-flow should look pretty much the same from here as well.

The Generate snapshot user-flow might look like below:

                                                         |
                                                         |
                                                         |
                                                         v

                                                         |
                                                         |
                                                         |
                                                         v

The above flow could sit in the application interface as detailed below:

[yashrajd]

• [Alternative] *

There's also a much simpler UI that can essentially be captured on a single screen (shown below), but might not be intuitive enough or allow for good feedback or information. What do folks think?

Send the user to this screen (see below) whether before (i.e- during onboarding screens, detailed in previous comment) or after IBD has started, and require one-step click generate/load buttons to get process started.

[D33r-Gee]

edit: deleting some irrelevant quoted text, plus minor edit in storage question

Thanks for getting the ball rolling on the dev side and even the UI @D33r-Gee great stuff! Your answers were very helpful, my designs (not included here) hopefully reflect that. I still have a few questions:

with pleasure @yashrajd , thanks for the mock ups and questions

• Can assumeutxo be enabled anytime during IBD (presumably before IBD reaches the snapshot block height) or only before onboarding starts?

AU can be initiated anytime before full IDB has finished synchronizing. Once launched, it will check and wait till IBD headers have hit the snapshot block height then it activates the snapshot (see ActivateSnapshot()), for more info please see the mark down doc here

My understanding from the doc is that "activate" basically happens automatically after loading (which is the user-initiated activity) and validation of the snapshot. Is that correct? I ask coz I noticed that one of your screen mockups has an activate toggle in addition to the Load snapshot toggle. Is it required and the user expected to decide to toggle it at some point?

Your understanding of loading/activating the snapshot is correct, the reason I added an activate toggle in the Load UTXO options is because I have yet to figure out how to load/activate the snapshot during node init.

Another question here: what's the time scale of loading & validating processes expected to take?

That’s device dependent, i.e. on the WSL device it took a few hours (way less than IBD) to complete the background validation.
On the Galaxy S8+ it took a couple of weeks to fully validate the snapshot.

If it's several seconds or longer, we might need a progress/transition screen etc.

yes that would be great to have since I had to manually look into the debug log, definitely not the most user friendly way.

• What’s the block height at which the snapshot hash is hardcoded in 26.0?

The block heights that were used to test and validate the AU feature were on:
mainnet 780,000 and 800,000
signet 160,000
testnet 2,500,000

*Would it be updated in every Core version?

Anyone with a full IDB can generate their own snapshot at whatever height they prefer using the following dev tool: ./contrib/devtools/utxo_snapshot.sh More information can be found in the original PR
Would it be able to partially validate snapshots taken at higher block heights?
I haven’t tested this but my guess is that no it would return an error.

Assuming the snapshot generation takes place at the current block height but the hardcoded hashes are at the block heights you mentioned, wouldn't it always result in a mismatch during validation?

If the snapshot generation is at the current block (i.e. 824,361) then it needs to be updated in the chainparams from 800,000 to 824,361 like it was done here

• Disk space requirement: It seems 2 UTXO sets will be maintained until a certain point in IBD. What point is that?

Until the back background validation of the snapshot has completed on signet it remains around 1 GB
On mainnet it can get up 30 GB while both chainstates are present then it drops to 21 GB after full validation (These number may differ depending on the OS)

• How much extra in disk space requirements would be required & needed to be communicated?

Well it doesn’t require more space than the raw IBD however it is more than just the prune state, so something to play with most definitely

I take this to mean, yes, if a snapshot needs to be loaded the space requirement would increase but I'm unsure if this is right and by how much? If it's right we should specify and communicate the same on the Storage screen during onboarding.

To communicate this there’s definitely some math that can be extracted depending which chain is used (mainnet, signet and/or testnet). Will dig around and see how that could be done…

Also, if it's a lot, and the user tries to load a snapshot AFTER the onboarding UX is done and IBD in progress, do we just disallow loading a screenshot?

Well it’s definitely much less than IBD so allowing loading a snapshot after onboarding can still be an option as long as IBD hasn’t finished and blocks haven’t been pruned.

• Does this look different on whether the snapshot is downloaded from peers or loaded from a file?

At this moment there is no peers download option, so no

• Can snapshot validation be skipped if the user themselves generated the snapshot (or any other scenarios)?

No, it still has to go through the full validation to ensure chainstate fidelity

[D33r-Gee]

@yashrajd The links to the following images seems to be broken:

The above flow could sit in the onboarding below as detailed below:

assumeutxo setup

Post-onboarding:

The above flow could sit in the application interface as detailed below:

manage utxo setup