Sidechain peer fetch blocks - RPC client/server by murerfel · Pull Request #580 · integritee-network/worker

murerfel · 2021-12-16T11:13:31Z

Preparatory step for #567. Implements the RPC client and server side to fetch sidechain blocks from a peer validateer.

Added a new 'peer-fetch' crate that implements both server and client logic
Extended the sidechain storage to retrieve blocks with a certain ancestry
Added a rpc call on the untrusted server side to retrieve sidechain blocks
Use temp dir in sidechain storage unit test (closes [sidechain_storage] use TmpDir in tests #401)

The new rpc call is not yet used by the worker. This will only be added in a later step.

gaudenzkessler

I have only cosmetic comments.

sidechain/storage/Cargo.toml

core-primitives/api-client-extensions/Cargo.toml

sidechain/storage/src/storage.rs

sidechain/storage/src/storage_tests_get_blocks_following.rs

core/rpc-client/src/url_utils.rs

sidechain/storage/src/storage.rs

clangenb

Looks quite good, not many comments.

sidechain/peer-fetch/Cargo.toml

sidechain/storage/src/error.rs

sidechain/storage/src/storage.rs

clangenb · 2021-12-29T10:07:12Z

sidechain/storage/src/test_db_fixture.rs

+/// Test fixture for a DB, cleans itself up when it goes out of scope.
+///
+/// Use with `_dbFixture`, not `_` - because the latter immediately drops!
+pub struct TestDbFixture {


This setup is not very stable. I once had a panic in a test, and it corrupted the state because such that I had to manually delete the directories to get the tests running again.

Now that we update the tests anyhow, could you solve: [sidechain_storage] use TmpDir in tests #401

Added, please take a look :)

sidechain/peer-fetch/src/untrusted_peer_fetch.rs

murerfel

Just some minor comments - thanks for continuing my work on this PR!

sidechain/storage/src/storage.rs

sidechain/peer-fetch/src/untrusted_peer_fetch.rs

clangenb

Some minors, but it looks good in general 👍

sidechain/peer-fetch/src/block_fetch_server.rs

sidechain/peer-fetch/src/lib.rs

sidechain/peer-fetch/src/untrusted_peer_fetch.rs

sidechain/storage/src/storage.rs

murerfel

Just a minor re-naming issue - otherwise it's looking good

sidechain/storage/src/storage_tests_get_blocks_following.rs

clangenb

Only a minor, good otherwise. I will pre-emptively approve and trust you to fix the rest. ;)

sidechain/peer-fetch/src/untrusted_peer_fetch.rs

sidechain/storage/src/storage_tests_get_blocks_following.rs

* Make mu-ra and untrusted worker url queriable (integritee-network#595) * extract request_keys() to separate file * remove providr input, add dummy getter function * add node_api worker_for_shard call * fix error message * add primitives cache and rpc call * fix tests * add primitives-cache to workspace * fix unit tests * remove obsolete .yml provider from request-keys cmd * remove provider_addr from CI py scripts * fix reported worker address * improve usability of rpc-client * make it work * fix rebase error * add some delay * update local setup script * remove ugly async worker url, replace with enclave getter function * some steps towards a working exmaple.. * add peer_updater * fix unit test * fix some test clippy warnings * fix function name * fix client mu ra url * fix comment * fix comment * rename state_sync to appropriate request keys * fix comments and add missing _size to untrusted_worker_addr * update cargo.lock after rebase * fix typos * rename store_peers to set_peers * fix comment * move set_primitives to primitves cache repository * return read guard instead of primittves clone * rename config worker_rpc_port to trusted_worker_port * remove obsolete Error enum from request_keys.rs * fix unit tests * move thread spawning back into watch fn * rename worker-rpc-port to trusted-worker-port * readd external worker address * fix unit tests * fix unit test * add external addresses, optional port input and unit tests * update test names * [cli.yml] update shorts * fix local setup configs * change untrusted worker port to w * [sidechain] detect out of sync error (integritee-network#606) * inital commit * remove unwrap_err from assert_matches * Update substrate sp-core to version 4.1.0-dev (integritee-network#612) Co-authored-by: Gaudenz Kessler <gaudenz.kessler@scs.ch> * Renaming of unspecific SB and PB variable names (integritee-network#605) * some clean up & add hanlde import error * remove logic changes * fix unit tests * [aura block importer] rename SidechainBlock to SignedSidechainBlock * fix rebase errors * [aura mock] rename xxT import to xxTrait * [aura verifier] rename SidechainBlock to SignedSidechainblock where appropriate * fix rebase errors * [aura] rename Sidechainblock to SignedSidechainBlock * rename SB & PB to full written version and adapt to SignedSidechainBlock Where necessary * [sidechain block imported] remove extra generic from SignedParentchainBlock * some further SB & PB clean up * rename B & SB to SidechainBlock & SignedSidechainBlock * some further renaming * completely remove SB * rename all left over PBs * remove rebase error & rename to SignedSidechainBlock * rename to SignedSidechainBlock * Sidechain peer fetch blocks - RPC client/server (integritee-network#580) * WIP: RPC call to fetch sidechain blocks * WIP: sidechain peer fetch crate with RPC server and client impl * WIP: test for RPC peer sidechain block fetching * fix unit test * remove obsolete comment * fix rebase error * cargo fmt * fix tests * fix Fixme + add som Send+Sync to errors * update add_block_to_batch to return error. Otherwise silent fail * small comment fixes * make some comments better understandable * remove FIXME comment * remove new lines * fix trailing comments * [peer-fetch] change order or crates in .toml * [sidechain storage] fix error message of HeaderAncestryMismatch * [sidechain storage] exchange match statement with ok_or * [sidechain storage] use temp-dir in tests * [sidechain storage] remove extra genesis block check * fix rebase errors * remove untrstued url, replace with untrusted peer fetcher * [sidechain storage] fix comment * update delete_last_block description comment * [sidechain storage] fix comment grammer * move FetchUntrustedPeers trait to the top * [FetchBlocksFromPeer] extend description comment * update cargo.lock * rename get_blocks_following to get_blocks_after * rename get_blocks_following to get_blocks_after * rename all leftover "blocks_following" to "block_after" Co-authored-by: Bigna Härdi <bhaerdi@devsgx02.scs-ad.scs.ch> * [Sidechain] Peer block fetching o-call implementation (integritee-network#619) * introduce o-call for fetching sidechain blocks from peer * re-name api-client-extensions to node-api-extensions Sub-task of integritee-network#567 * add direct call rpc doc (integritee-network#620) * add some doc * add some structure to the links * restructure rpc interface * Update docs/README.md Co-authored-by: gaudenzkessler <92718752+gaudenzkessler@users.noreply.github.com> * adapt readmes according to review comments Co-authored-by: gaudenzkessler <92718752+gaudenzkessler@users.noreply.github.com> * [Sidechain] Peer sync and block production suspension (integritee-network#618) Peer syncer implementation (not in use yet) and block production suspender (also not in use yet) * update to most recent teaclave commit (integritee-network#624) * Sync state from fellow validateer (integritee-network#615) * rename request_keys to sync_state * rename request_key_prov to request_state_prov * rename request_keys.rs to sync_state.rs * restructure key and state provisioning server * some refactoring * add TlsServer struct * add test file * rename key_provision_server to state_provisioning_server * add unit test * update unit test * introduce mockable key handler struct * shielding key success * remove clippy warnings * fix test * add unit tests for KeyHandler * rename to prepare for state inclusion * rename seal_handler * add shard as argument to sync state * some more renaming * add shard read & write process * [SealHandler] add unit tests & fix state * update networking test to include state * add default shard * add some documentation * remove ugly for loop * move authentications to separate file * update comment * remove obsolete, never ending loop * add error logs * remove extra phantom field * add sgx feature flag * remove global variables from test * add join handle to test * add some more logging info * Change tokio runtime to use 2 worker threads. Gossiping spawns new tokio tasks. (integritee-network#626) * Add state update sequence (integritee-network#632) * add bock_import_sequence.svg * move block_import.svg to docs/diagramms * update diagramm * add block import sequence * RPC call to get metadata from sgx-runtime (integritee-network#642) * RPC call to get metadata from sgx-runtime - rcp call - print sgx metadata cli * Change from review: Metadata is already encoded * Change from review Co-authored-by: echevrier <edith.chevrier@scs.ch> * bump substrate to commit 59649dd (integritee-network#645) * update .tomls to new substrate versions * cargo update * RawEvent -> Event * remove default from Accountid * RawEvent -> Event * cargo update enclave-runtime * fix bump errors * remove unused patch * finaly compiling * update sgx-runtime and substrate-api-client to github * remove integritee-node-runtime patch * cargo update -p std-std --precise 59649dd * update Github Actions integritee node * remove bh-config * fix clippy * fix cargo test * update spec version * update substrate-api-client * update sgx-runtime source * update substrate * adjust node version values * detect new game * solve merge conflicts * update sgx-runtime * fix some things * cargo format Co-authored-by: gaudenzkessler <92718752+gaudenzkessler@users.noreply.github.com> Co-authored-by: Gaudenz Kessler <gaudenz.kessler@scs.ch> Co-authored-by: Felix Müller <mullefel@users.noreply.github.com> Co-authored-by: echevrier <84318241+echevrier@users.noreply.github.com> Co-authored-by: echevrier <edith.chevrier@scs.ch>

* Make mu-ra and untrusted worker url queriable (integritee-network#595) * extract request_keys() to separate file * remove providr input, add dummy getter function * add node_api worker_for_shard call * fix error message * add primitives cache and rpc call * fix tests * add primitives-cache to workspace * fix unit tests * remove obsolete .yml provider from request-keys cmd * remove provider_addr from CI py scripts * fix reported worker address * improve usability of rpc-client * make it work * fix rebase error * add some delay * update local setup script * remove ugly async worker url, replace with enclave getter function * some steps towards a working exmaple.. * add peer_updater * fix unit test * fix some test clippy warnings * fix function name * fix client mu ra url * fix comment * fix comment * rename state_sync to appropriate request keys * fix comments and add missing _size to untrusted_worker_addr * update cargo.lock after rebase * fix typos * rename store_peers to set_peers * fix comment * move set_primitives to primitves cache repository * return read guard instead of primittves clone * rename config worker_rpc_port to trusted_worker_port * remove obsolete Error enum from request_keys.rs * fix unit tests * move thread spawning back into watch fn * rename worker-rpc-port to trusted-worker-port * readd external worker address * fix unit tests * fix unit test * add external addresses, optional port input and unit tests * update test names * [cli.yml] update shorts * fix local setup configs * change untrusted worker port to w * [sidechain] detect out of sync error (integritee-network#606) * inital commit * remove unwrap_err from assert_matches * Update substrate sp-core to version 4.1.0-dev (integritee-network#612) Co-authored-by: Gaudenz Kessler <gaudenz.kessler@scs.ch> * Renaming of unspecific SB and PB variable names (integritee-network#605) * some clean up & add hanlde import error * remove logic changes * fix unit tests * [aura block importer] rename SidechainBlock to SignedSidechainBlock * fix rebase errors * [aura mock] rename xxT import to xxTrait * [aura verifier] rename SidechainBlock to SignedSidechainblock where appropriate * fix rebase errors * [aura] rename Sidechainblock to SignedSidechainBlock * rename SB & PB to full written version and adapt to SignedSidechainBlock Where necessary * [sidechain block imported] remove extra generic from SignedParentchainBlock * some further SB & PB clean up * rename B & SB to SidechainBlock & SignedSidechainBlock * some further renaming * completely remove SB * rename all left over PBs * remove rebase error & rename to SignedSidechainBlock * rename to SignedSidechainBlock * Sidechain peer fetch blocks - RPC client/server (integritee-network#580) * WIP: RPC call to fetch sidechain blocks * WIP: sidechain peer fetch crate with RPC server and client impl * WIP: test for RPC peer sidechain block fetching * fix unit test * remove obsolete comment * fix rebase error * cargo fmt * fix tests * fix Fixme + add som Send+Sync to errors * update add_block_to_batch to return error. Otherwise silent fail * small comment fixes * make some comments better understandable * remove FIXME comment * remove new lines * fix trailing comments * [peer-fetch] change order or crates in .toml * [sidechain storage] fix error message of HeaderAncestryMismatch * [sidechain storage] exchange match statement with ok_or * [sidechain storage] use temp-dir in tests * [sidechain storage] remove extra genesis block check * fix rebase errors * remove untrstued url, replace with untrusted peer fetcher * [sidechain storage] fix comment * update delete_last_block description comment * [sidechain storage] fix comment grammer * move FetchUntrustedPeers trait to the top * [FetchBlocksFromPeer] extend description comment * update cargo.lock * rename get_blocks_following to get_blocks_after * rename get_blocks_following to get_blocks_after * rename all leftover "blocks_following" to "block_after" Co-authored-by: Bigna Härdi <bhaerdi@devsgx02.scs-ad.scs.ch> * [Sidechain] Peer block fetching o-call implementation (integritee-network#619) * introduce o-call for fetching sidechain blocks from peer * re-name api-client-extensions to node-api-extensions Sub-task of integritee-network#567 * add direct call rpc doc (integritee-network#620) * add some doc * add some structure to the links * restructure rpc interface * Update docs/README.md Co-authored-by: gaudenzkessler <92718752+gaudenzkessler@users.noreply.github.com> * adapt readmes according to review comments Co-authored-by: gaudenzkessler <92718752+gaudenzkessler@users.noreply.github.com> * [Sidechain] Peer sync and block production suspension (integritee-network#618) Peer syncer implementation (not in use yet) and block production suspender (also not in use yet) * update to most recent teaclave commit (integritee-network#624) * Sync state from fellow validateer (integritee-network#615) * rename request_keys to sync_state * rename request_key_prov to request_state_prov * rename request_keys.rs to sync_state.rs * restructure key and state provisioning server * some refactoring * add TlsServer struct * add test file * rename key_provision_server to state_provisioning_server * add unit test * update unit test * introduce mockable key handler struct * shielding key success * remove clippy warnings * fix test * add unit tests for KeyHandler * rename to prepare for state inclusion * rename seal_handler * add shard as argument to sync state * some more renaming * add shard read & write process * [SealHandler] add unit tests & fix state * update networking test to include state * add default shard * add some documentation * remove ugly for loop * move authentications to separate file * update comment * remove obsolete, never ending loop * add error logs * remove extra phantom field * add sgx feature flag * remove global variables from test * add join handle to test * add some more logging info * Change tokio runtime to use 2 worker threads. Gossiping spawns new tokio tasks. (integritee-network#626) * Add state update sequence (integritee-network#632) * add bock_import_sequence.svg * move block_import.svg to docs/diagramms * update diagramm * add block import sequence * RPC call to get metadata from sgx-runtime (integritee-network#642) * RPC call to get metadata from sgx-runtime - rcp call - print sgx metadata cli * Change from review: Metadata is already encoded * Change from review Co-authored-by: echevrier <edith.chevrier@scs.ch> * bump substrate to commit 59649dd (integritee-network#645) * update .tomls to new substrate versions * cargo update * RawEvent -> Event * remove default from Accountid * RawEvent -> Event * cargo update enclave-runtime * fix bump errors * remove unused patch * finaly compiling * update sgx-runtime and substrate-api-client to github * remove integritee-node-runtime patch * cargo update -p std-std --precise 59649dd * update Github Actions integritee node * remove bh-config * fix clippy * fix cargo test * update spec version * update substrate-api-client * update sgx-runtime source * update substrate * adjust node version values * detect new game * solve merge conflicts * update sgx-runtime * fix some things * cargo format Co-authored-by: haerdib <73821294+haerdib@users.noreply.github.com> Co-authored-by: Gaudenz Kessler <gaudenz.kessler@scs.ch> Co-authored-by: Felix Müller <mullefel@users.noreply.github.com> Co-authored-by: Bigna Härdi <bhaerdi@devsgx02.scs-ad.scs.ch> Co-authored-by: echevrier <84318241+echevrier@users.noreply.github.com> Co-authored-by: echevrier <edith.chevrier@scs.ch>

haerdib self-assigned this Dec 22, 2021

haerdib force-pushed the feature/fm-sidechain-peer-fetch-client branch from e13e48a to df3d811 Compare December 28, 2021 08:56

haerdib marked this pull request as ready for review December 28, 2021 08:56

haerdib requested review from clangenb and gaudenzkessler December 28, 2021 11:01

haerdib changed the title ~~WIP: Sidechain peer fetch blocks - RPC client/server~~ Sidechain peer fetch blocks - RPC client/server Dec 28, 2021

This was referenced Dec 28, 2021

Sync state by retrieving sidechain blocks from storage of a different, given worker #567

Closed

Make mu-ra and untrusted worker url queriable #595

Merged

gaudenzkessler approved these changes Dec 28, 2021

View reviewed changes

haerdib reviewed Dec 28, 2021

View reviewed changes

core/rpc-client/src/url_utils.rs Outdated Show resolved Hide resolved

sidechain/storage/src/storage.rs Show resolved Hide resolved

clangenb reviewed Dec 29, 2021

View reviewed changes

haerdib force-pushed the feature/fm-sidechain-peer-fetch-client branch from 40def1f to 33ec0eb Compare December 30, 2021 08:53

haerdib reviewed Dec 30, 2021

View reviewed changes

sidechain/peer-fetch/src/untrusted_peer_fetch.rs Show resolved Hide resolved

haerdib requested a review from clangenb December 30, 2021 09:00

murerfel commented Jan 10, 2022

View reviewed changes

sidechain/storage/src/storage.rs Outdated Show resolved Hide resolved

sidechain/storage/src/storage.rs Outdated Show resolved Hide resolved

sidechain/storage/src/storage.rs Show resolved Hide resolved

sidechain/peer-fetch/src/untrusted_peer_fetch.rs Outdated Show resolved Hide resolved

clangenb reviewed Jan 10, 2022

View reviewed changes

haerdib force-pushed the feature/fm-sidechain-peer-fetch-client branch from fac56e9 to 7f4de4a Compare January 10, 2022 14:41

Felix Müller and others added 13 commits January 10, 2022 15:58

WIP: RPC call to fetch sidechain blocks

cdf888d

WIP: sidechain peer fetch crate with RPC server and client impl

0c6c021

WIP: test for RPC peer sidechain block fetching

b25daae

fix unit test

37c9dbe

remove obsolete comment

8ee038c

fix rebase error

cddfd65

cargo fmt

386612e

fix tests

4025a5f

fix Fixme + add som Send+Sync to errors

df105eb

update add_block_to_batch to return error. Otherwise silent fail

2a11833

small comment fixes

6bfadfa

make some comments better understandable

de789c8

remove FIXME comment

233826a

haerdib added 17 commits January 10, 2022 15:58

remove new lines

671f5d6

fix trailing comments

ca67bda

[peer-fetch] change order or crates in .toml

cdb57d5

[sidechain storage] fix error message of HeaderAncestryMismatch

693326d

[sidechain storage] exchange match statement with ok_or

02da5e7

[sidechain storage] use temp-dir in tests

e2d0b6a

[sidechain storage] remove extra genesis block check

0484572

fix rebase errors

fc64268

remove untrstued url, replace with untrusted peer fetcher

e38df1e

[sidechain storage] fix comment

a360d06

update delete_last_block description comment

3252214

[sidechain storage] fix comment grammer

2650e86

move FetchUntrustedPeers trait to the top

1efc8b8

[FetchBlocksFromPeer] extend description comment

fcbac71

update cargo.lock

c47be5e

rename get_blocks_following to get_blocks_after

311238b

rename get_blocks_following to get_blocks_after

8e29607

haerdib force-pushed the feature/fm-sidechain-peer-fetch-client branch from dc7caea to 8e29607 Compare January 10, 2022 14:59

murerfel commented Jan 11, 2022

View reviewed changes

sidechain/storage/src/storage_tests_get_blocks_following.rs Show resolved Hide resolved

clangenb reviewed Jan 11, 2022

View reviewed changes

sidechain/peer-fetch/src/untrusted_peer_fetch.rs Show resolved Hide resolved

sidechain/storage/src/storage_tests_get_blocks_following.rs Show resolved Hide resolved

clangenb approved these changes Jan 11, 2022

View reviewed changes

rename all leftover "blocks_following" to "block_after"

2461f6b

haerdib merged commit 17e9776 into master Jan 12, 2022

haerdib deleted the feature/fm-sidechain-peer-fetch-client branch January 12, 2022 06:52

Conversation

murerfel commented Dec 16, 2021 • edited by haerdib Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gaudenzkessler left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

clangenb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

clangenb Dec 29, 2021

Choose a reason for hiding this comment

Uh oh!

haerdib Dec 29, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

murerfel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

clangenb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

murerfel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

clangenb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

murerfel commented Dec 16, 2021 •

edited by haerdib

Loading