remote-externalities: 'instant' snapshots, threading refactor, better progress logs by liamaharon · Pull Request #14057 · paritytech/substrate

liamaharon · 2023-05-01T12:19:54Z

Instant(ish) snapshot loading

The existing snapshots contain key / values without the trie. Loading the key/values can take a while even when batch inserting because the trie needs to be re-computed.
In this PR, the snapshot was refactored to contain the raw trie itself, and be loaded directly into the underlying TrieBackend storage that backs TestExternalties

Snapshot load performance (Macbook Pro M2 Max)

Chain	Build	Before	After	Improvement
Kusama	debug	12.13s	0.88s	92.75%
Kusama	release	1.93s	0.18s	90.67%
Polkadot	debug	59.08s	2.66s	95.50%
Polkadot	release	8.25s	0.44s	94.67%

Removes threading

Uses async/await futures for parallel requests instead of threads
Closes try-runtime-cli panics on Polkadot #14006
It makes loading state from a live node a little bit slower because it waits until all data has been downloaded before loading it into the trie. However, the degradation is small, and it should be encouraged to use snapshots instead when performance is required especially now that we have much faster snapshot loading.
Bonus: simplifies a lot of complex code

Better progress logs

Spinner for operations where the eta cannot be estimated
Progress bar for operations where the eta can be estimated
Partial try-runtime-cli: polish output #13975

Kusama `on_runtime_upgrade` `live` logs

Screen.Recording.2023-05-03.at.12.11.20.mp4

TODO Check pulling data from these remotes works:

wss://polkadot-try-runtime-node.parity-chains.parity.io:443
wss://kusama-try-runtime-node.parity-chains.parity.io:443
wss://westend-try-runtime-node.parity-chains.parity.io:443

utils/frame/remote-externalities/src/lib.rs

niklasad1

nice, LGTM

Can you add an example of the changed CLI output in the PR description so the reviewers can a feeling how output will look like after this change?

…mote-externalities-refactor

liamaharon · 2023-05-02T08:59:12Z

Converting this to a draft, I'd like to include a switch to inserting keys directly into OverlayedChanges into this PR.

This reverts commit c0ddb87.

This reverts commit 1d49362.

This reverts commit 06df786.

ggwpez · 2023-05-02T17:31:32Z

Tested it for all four parity nodes and it works with this patch, thanks!

kianenigma

It looks very promising!

I think now's a good time to revisit https://github.com/paritytech/devops/issues/2083 and its importance as well.

With the end goal of making this usable (in a stable and fast way 🙈) in our CI, it would make a lot of sense to have snapshots ready in our infra, and have the CI use those. With this fast loading time, it would be a smooth process.

ggwpez · 2023-05-04T08:53:35Z

With the end goal of making this usable (in a stable and fast way ) in our CI, it would make a lot of sense to have snapshots ready in our infra, and have the CI use those. With this fast loading time, it would be a smooth process.

It is pretty fast now. Takes about 3 minutes locally and 15 in CI.
I think we should rather focus on stability and add tests instead of optimizing more.

ggwpez

Looks pretty good, thanks!

What is this file utils/frame/remote-externalities/test_data/proxy_test?

primitives/state-machine/src/testing.rs

utils/frame/remote-externalities/src/lib.rs

…mote-externalities-refactor

liamaharon · 2023-05-04T15:10:25Z

@ggwpez utils/frame/remote-externalities/test_data/proxy_test is a snapshot file used in a test. I needed to update it to use the new snapshot format.

…ress logs (#14057) * remote externalities refactor * remove redundant logs * use const for parallel requests * prefer functional * improve variable naming * handle requests error * use overlayedchanges * Revert "use overlayedchanges" This reverts commit c0ddb87. * Revert "Revert "use overlayedchanges"" This reverts commit 1d49362. * Revert "Revert "Revert "use overlayedchanges""" This reverts commit 06df786. * backup/load raw storage values * test raw storage drain and restore * update snapshot tests * improve logs * clippy suggestions * address comments * fix example * fix test * clippy

…ress logs (paritytech#14057) * remote externalities refactor * remove redundant logs * use const for parallel requests * prefer functional * improve variable naming * handle requests error * use overlayedchanges * Revert "use overlayedchanges" This reverts commit c0ddb87. * Revert "Revert "use overlayedchanges"" This reverts commit 1d49362. * Revert "Revert "Revert "use overlayedchanges""" This reverts commit 06df786. * backup/load raw storage values * test raw storage drain and restore * update snapshot tests * improve logs * clippy suggestions * address comments * fix example * fix test * clippy

liamaharon added 5 commits May 1, 2023 15:21

remote externalities refactor

9d8d1c6

remove redundant logs

9f97308

use const for parallel requests

c9f8b3e

prefer functional

266de13

improve variable naming

5ec0155

liamaharon requested review from ggwpez, kianenigma and niklasad1 May 1, 2023 12:19

liamaharon added A0-please_review Pull request needs code review. B0-silent Changes should not be mentioned in any release notes C1-low PR touches the given topic and has a low impact on builders. labels May 1, 2023

niklasad1 reviewed May 1, 2023

View reviewed changes

utils/frame/remote-externalities/src/lib.rs Show resolved Hide resolved

niklasad1 reviewed May 1, 2023

View reviewed changes

utils/frame/remote-externalities/src/lib.rs Outdated Show resolved Hide resolved

niklasad1 approved these changes May 1, 2023

View reviewed changes

liamaharon added 2 commits May 2, 2023 10:24

handle requests error

7026d00

Merge branch 'master' of github.com:paritytech/substrate into liam-re…

094bdc1

…mote-externalities-refactor

liamaharon marked this pull request as draft May 2, 2023 08:56

liamaharon changed the title ~~remote-externalities: threading and progress indicator refactor~~ remote-externalities: threading refactor, logs overhaul and performance improvement May 2, 2023

liamaharon added A3-in_progress Pull request is in progress. No review needed at this stage. and removed A0-please_review Pull request needs code review. labels May 2, 2023

liamaharon mentioned this pull request May 2, 2023

try-runtime-cli: Lazy Download #13562

Closed

liamaharon added 4 commits May 2, 2023 13:40

use overlayedchanges

c0ddb87

Revert "use overlayedchanges"

1d49362

This reverts commit c0ddb87.

Revert "Revert "use overlayedchanges""

06df786

This reverts commit 1d49362.

Revert "Revert "Revert "use overlayedchanges"""

c0e5fba

This reverts commit 06df786.

liamaharon added 3 commits May 3, 2023 10:43

backup/load raw storage values

476e727

test raw storage drain and restore

c4ba497

update snapshot tests

9bd60e5

liamaharon added the D3-trivial 🧸 PR contains trivial changes in a runtime directory that do not require an audit label May 3, 2023

liamaharon changed the title ~~remote-externalities: threading refactor, logs overhaul and 'instant' loading snapshots~~ remote-externalities: threading refactor, logs overhaul and 'instant' snapshots May 3, 2023

liamaharon mentioned this pull request May 3, 2023

try-runtime-cli: remove DB operations in async code #14066

Closed

liamaharon changed the title ~~remote-externalities: threading refactor, logs overhaul and 'instant' snapshots~~ remote-externalities: threading refactor, better progress logs, and 'instant' snapshots May 3, 2023

liamaharon added A0-please_review Pull request needs code review. and removed A3-in_progress Pull request is in progress. No review needed at this stage. labels May 3, 2023

clippy suggestions

736363f

liamaharon changed the title ~~remote-externalities: threading refactor, better progress logs, and 'instant' snapshots~~ remote-externalities: threading refactor, better progress logs, 'instant' snapshots May 3, 2023

liamaharon changed the title ~~remote-externalities: threading refactor, better progress logs, 'instant' snapshots~~ remote-externalities: 'instant' snapshots, threading refactor, better progress logs May 3, 2023

kianenigma reviewed May 3, 2023

View reviewed changes

ggwpez reviewed May 4, 2023

View reviewed changes

liamaharon added 2 commits May 4, 2023 18:25

Merge branch 'master' of github.com:paritytech/substrate into liam-re…

b7fd6b8

…mote-externalities-refactor

address comments

27b5028

liamaharon added 2 commits May 4, 2023 19:26

fix example

aefb9d2

fix test

121de0a

liamaharon requested a review from ggwpez May 4, 2023 16:24

ggwpez approved these changes May 4, 2023

View reviewed changes

clippy

caa8d87

liamaharon merged commit fb98ee1 into master May 5, 2023

liamaharon deleted the liam-remote-externalities-refactor branch May 5, 2023 07:15

kiltbot mentioned this pull request Jun 21, 2023

1️⃣ [AUTOMATIC] Update Polkadot dependencies from 0.9.42 to 0.9.43 KILTprotocol/kilt-node#531

Closed

kacperzuk-neti mentioned this pull request Jun 23, 2023

Polkadot v0.9.43 liberland/liberland_substrate#295

Merged

15 tasks

liamaharon mentioned this pull request Aug 14, 2023

Remove DB operations in async code paritytech/try-runtime-cli#14

Open

dmitrylavrenov mentioned this pull request Oct 29, 2024

Tracking issue to bump substrate related deps to polkadot-v0.9.43 humanode-network/humanode#923

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

remote-externalities: 'instant' snapshots, threading refactor, better progress logs#14057

remote-externalities: 'instant' snapshots, threading refactor, better progress logs#14057
liamaharon merged 21 commits intomasterfrom
liam-remote-externalities-refactor

liamaharon commented May 1, 2023 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

niklasad1 left a comment •

edited

Loading

Uh oh!

liamaharon commented May 2, 2023

Uh oh!

ggwpez commented May 2, 2023

Uh oh!

kianenigma left a comment

Uh oh!

ggwpez commented May 4, 2023

Uh oh!

ggwpez left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

liamaharon commented May 4, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

liamaharon commented May 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Instant(ish) snapshot loading

Snapshot load performance (Macbook Pro M2 Max)

Removes threading

Better progress logs

Kusama on_runtime_upgrade live logs

TODO Check pulling data from these remotes works:

Uh oh!

Uh oh!

Uh oh!

niklasad1 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liamaharon commented May 2, 2023

Uh oh!

ggwpez commented May 2, 2023

Uh oh!

kianenigma left a comment

Choose a reason for hiding this comment

Uh oh!

ggwpez commented May 4, 2023

Uh oh!

ggwpez left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

liamaharon commented May 4, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

liamaharon commented May 1, 2023 •

edited

Loading

Kusama `on_runtime_upgrade` `live` logs

niklasad1 left a comment •

edited

Loading