[Sidechain] Peer block fetching o-call implementation by murerfel · Pull Request #619 · integritee-network/worker

murerfel · 2022-01-13T10:28:04Z

Implements the o-call structures needed for fetching sidechain blocks from a peer from inside the enclave (sub-task of #567 )

collisions with other services running on those ports. (cherry picked from commit edd8f37)

…from peer

murerfel

Comments to document this PR

murerfel · 2022-01-13T10:36:17Z

core-primitives/ocall-api/src/lib.rs

+	fn fetch_sidechain_blocks_from_peer<SignedSidechainBlock: Decode>(
+		&self,
+		last_known_block_hash: BlockHash,
+		shard_identifier: ShardIdentifier,
+	) -> SgxResult<Vec<SignedSidechainBlock>>;


Extend the o-call API on the enclave side

murerfel · 2022-01-13T10:38:54Z

enclave-runtime/src/ocall/sidechain_ocall.rs

+		// We have to pre-allocate the vector and hope it's large enough
+		let mut signed_blocks_encoded: Vec<u8> = vec![0; 4096 * 16];


This is what worries me a bit: In order to get the sidechain blocks over the o-call interface, we have to pre-allocate the vector into which the block data is filled. I think there's no other elegant solution to this? An alternative would be to have an additional o-call to query the size first, so we can then allocate exactly the right amount. But it would basically double the number of calls into our block storage.

Ouf, you're right. That's quite ugly. I do see a different solution, but that might require some redesigning:

Instead of fetching the blocks directly with this ocall, you could start a fetcher function on the untrusted side with it. This function then retrieves the sidechain blocks and buffers them on the untrusted side (not sure if buffering is needed, there might be a simpler solution). For each retrieved sidechain block (or a chunk of sidechain blocks), it would then do an ecall to import the block (I guess that could be the same ecall as for the gossiped sidechain blocks).

The block production suspension would then also need to be lifted via ecall.

Hmm I think that would be more complex to do, because we would go from a pull model to a push model (from the enclave perspective), which is not easy to do in the current PeerBlockSyncer.
An alternative would be to extend the o-call to have an additional output parameter, which is the total buffer size required. And this will be set in case the default one is not enough. We can then detect this in our o-call and adjust our buffer size accordingly and make the call again.

I do agree that this is more complex. But simply adjusting the buffer to an arbitrary amount, I also don't think is sensible - except if we limit our sidechain buffering to a reasonable size which we can import in one go.
Either way - I think this issue is worth its own PR and discussion, so I suggest opening a new issue.

Issue created -> #621

murerfel · 2022-01-13T10:40:37Z

service/src/globals/tokio_handle.rs

+/// Implementation for a scoped Tokio handle
+///
+///
+pub struct ScopedTokioHandle {
+	tokio_runtime: tokio::runtime::Runtime,
+}


Added a new scoped tokio handle for use in tests. As opposed to the global tokio handle, which is initialized once at startup of the application.

murerfel · 2022-01-13T10:46:45Z

service/src/ocall_bridge/sidechain_ocall.rs

+	fn fetch_sidechain_blocks_from_peer(
+		&self,
+		last_known_block_hash_encoded: Vec<u8>,
+		shard_identifier_encoded: Vec<u8>,
+	) -> OCallBridgeResult<Vec<u8>> {
+		let last_known_block_hash: BlockHash =
+			Decode::decode(&mut last_known_block_hash_encoded.as_slice()).map_err(|_| {
+				OCallBridgeError::FetchSidechainBlocksFromPeer(
+					"Failed to decode last known block hash".to_string(),
+				)
+			})?;
+
+		let shard_identifier: ShardIdentifier =
+			Decode::decode(&mut shard_identifier_encoded.as_slice()).map_err(|_| {
+				OCallBridgeError::FetchSidechainBlocksFromPeer(
+					"Failed to decode shard identifier".to_string(),
+				)
+			})?;
+
+		let tokio_handle = self.tokio_handle.get_handle();
+
+		let signed_sidechain_blocks = tokio_handle
+			.block_on(
+				self.peer_block_fetcher
+					.fetch_blocks_from_peer(last_known_block_hash, shard_identifier),
+			)
+			.map_err(|e| {
+				OCallBridgeError::FetchSidechainBlocksFromPeer(format!(
+					"Failed to execute block fetching from peer: {:?}",
+					e
+				))
+			})?;
+
+		Ok(signed_sidechain_blocks.encode())
+	}


This is the implementation that calls into the peer block fetcher from the sidechain crate. Because it's an asynchronous call that we run synchronously, we need a tokio handle.

murerfel · 2022-01-13T10:47:40Z

service/src/tests/mock.rs

-pub const W1_URL: &str = "127.0.0.1:2222";
-pub const W2_URL: &str = "127.0.0.1:3333";
+pub const W1_URL: &str = "127.0.0.1:22222";
+pub const W2_URL: &str = "127.0.0.1:33333";


This is just cherry-picked from the other PR #618

murerfel · 2022-01-13T10:49:35Z

service/src/utils.rs

+pub fn write_slice_and_whitespace_pad(writable: &mut [u8], data: Vec<u8>) -> Result<(), Error> {
+	ensure!(data.len() <= writable.len(), Error::InsufficientBufferSize);
 	let (left, right) = writable.split_at_mut(data.len());
 	left.clone_from_slice(&data);
 	// fill the right side with whitespace
 	right.iter_mut().for_each(|x| *x = 0x20);
+	Ok(())


Changed the signature of this method to return a result in case of failure (instead of panicking). Otherwise this could lead to unintentional shut downs of a worker.

murerfel · 2022-01-13T10:50:53Z

service/src/worker_peers_updater.rs

 }

-pub struct GlobalPeerUpdater<Worker> {
+pub struct WorkerPeersUpdater<Worker> {


Re-named from GlobalPeerUpdater to WorkerPeersUpdater, because this updater itself is not global or has any global component per se.

murerfel · 2022-01-13T10:52:09Z

sidechain/peer-fetch/src/mocks/fetch_blocks_from_peer_mock.rs

+pub struct FetchBlocksFromPeerMock<SignedBlock> {
+	signed_blocks_map: HashMap<ShardIdentifier, Vec<SignedBlock>>,
+}


Added a new mock for the FetchBlocksFromPeer trait, to be used in unit tests in other crates (and hence also added a new mocks feature to this crate).

…actory

* move the node API factory to that crate * move the untrusted peer fetch back to the sidechain crate * remove the new 'parentchain node api' crate

murerfel

More comments

murerfel · 2022-01-13T14:51:39Z

Cargo.lock

 "ita-stf",
 "itc-rpc-client",
- "itp-api-client-extensions",
+ "itp-node-api-extensions",


After discussing with @haerdib, I decided to rename the crate and move the node api factory into this crate (so it can be re-used)

murerfel · 2022-01-13T14:52:49Z

core-primitives/node-api-extensions/src/node_api_factory.rs

+	limitations under the License.
+
+*/
+


Move the NodeApiFactory here, so it can be re-used from the sidechain crate (was in the service crate before)

murerfel · 2022-01-13T14:55:58Z

service/src/main.rs

+	let node_api_factory =
+		Arc::new(NodeApiFactory::new(config.node_url(), AccountKeyring::Alice.pair()));


The signer is now passed into the node API factory and doesn't have to be set anymore as an additional step (is included in the factory now)

haerdib

Apart from the pre-allocation, only minor comments. Thanks for all the unit tests!

I think the pre-allocation is fine for now, but only if you file an appropriate issue. In my eyes, the current design is not future-proof. At least chunk sized sidechain fetching should be made possible I guess.

service/src/globals/tokio_handle.rs

service/src/ocall_bridge/ffi/fetch_sidechain_blocks_from_peer.rs

clangenb

Looks good! I value that you are so consistently improving the test-infrastructure. :)

Add period at the end of comment Co-authored-by: haerdib <73821294+haerdib@users.noreply.github.com>

* Make mu-ra and untrusted worker url queriable (integritee-network#595) * extract request_keys() to separate file * remove providr input, add dummy getter function * add node_api worker_for_shard call * fix error message * add primitives cache and rpc call * fix tests * add primitives-cache to workspace * fix unit tests * remove obsolete .yml provider from request-keys cmd * remove provider_addr from CI py scripts * fix reported worker address * improve usability of rpc-client * make it work * fix rebase error * add some delay * update local setup script * remove ugly async worker url, replace with enclave getter function * some steps towards a working exmaple.. * add peer_updater * fix unit test * fix some test clippy warnings * fix function name * fix client mu ra url * fix comment * fix comment * rename state_sync to appropriate request keys * fix comments and add missing _size to untrusted_worker_addr * update cargo.lock after rebase * fix typos * rename store_peers to set_peers * fix comment * move set_primitives to primitves cache repository * return read guard instead of primittves clone * rename config worker_rpc_port to trusted_worker_port * remove obsolete Error enum from request_keys.rs * fix unit tests * move thread spawning back into watch fn * rename worker-rpc-port to trusted-worker-port * readd external worker address * fix unit tests * fix unit test * add external addresses, optional port input and unit tests * update test names * [cli.yml] update shorts * fix local setup configs * change untrusted worker port to w * [sidechain] detect out of sync error (integritee-network#606) * inital commit * remove unwrap_err from assert_matches * Update substrate sp-core to version 4.1.0-dev (integritee-network#612) Co-authored-by: Gaudenz Kessler <gaudenz.kessler@scs.ch> * Renaming of unspecific SB and PB variable names (integritee-network#605) * some clean up & add hanlde import error * remove logic changes * fix unit tests * [aura block importer] rename SidechainBlock to SignedSidechainBlock * fix rebase errors * [aura mock] rename xxT import to xxTrait * [aura verifier] rename SidechainBlock to SignedSidechainblock where appropriate * fix rebase errors * [aura] rename Sidechainblock to SignedSidechainBlock * rename SB & PB to full written version and adapt to SignedSidechainBlock Where necessary * [sidechain block imported] remove extra generic from SignedParentchainBlock * some further SB & PB clean up * rename B & SB to SidechainBlock & SignedSidechainBlock * some further renaming * completely remove SB * rename all left over PBs * remove rebase error & rename to SignedSidechainBlock * rename to SignedSidechainBlock * Sidechain peer fetch blocks - RPC client/server (integritee-network#580) * WIP: RPC call to fetch sidechain blocks * WIP: sidechain peer fetch crate with RPC server and client impl * WIP: test for RPC peer sidechain block fetching * fix unit test * remove obsolete comment * fix rebase error * cargo fmt * fix tests * fix Fixme + add som Send+Sync to errors * update add_block_to_batch to return error. Otherwise silent fail * small comment fixes * make some comments better understandable * remove FIXME comment * remove new lines * fix trailing comments * [peer-fetch] change order or crates in .toml * [sidechain storage] fix error message of HeaderAncestryMismatch * [sidechain storage] exchange match statement with ok_or * [sidechain storage] use temp-dir in tests * [sidechain storage] remove extra genesis block check * fix rebase errors * remove untrstued url, replace with untrusted peer fetcher * [sidechain storage] fix comment * update delete_last_block description comment * [sidechain storage] fix comment grammer * move FetchUntrustedPeers trait to the top * [FetchBlocksFromPeer] extend description comment * update cargo.lock * rename get_blocks_following to get_blocks_after * rename get_blocks_following to get_blocks_after * rename all leftover "blocks_following" to "block_after" Co-authored-by: Bigna Härdi <bhaerdi@devsgx02.scs-ad.scs.ch> * [Sidechain] Peer block fetching o-call implementation (integritee-network#619) * introduce o-call for fetching sidechain blocks from peer * re-name api-client-extensions to node-api-extensions Sub-task of integritee-network#567 * add direct call rpc doc (integritee-network#620) * add some doc * add some structure to the links * restructure rpc interface * Update docs/README.md Co-authored-by: gaudenzkessler <92718752+gaudenzkessler@users.noreply.github.com> * adapt readmes according to review comments Co-authored-by: gaudenzkessler <92718752+gaudenzkessler@users.noreply.github.com> * [Sidechain] Peer sync and block production suspension (integritee-network#618) Peer syncer implementation (not in use yet) and block production suspender (also not in use yet) * update to most recent teaclave commit (integritee-network#624) * Sync state from fellow validateer (integritee-network#615) * rename request_keys to sync_state * rename request_key_prov to request_state_prov * rename request_keys.rs to sync_state.rs * restructure key and state provisioning server * some refactoring * add TlsServer struct * add test file * rename key_provision_server to state_provisioning_server * add unit test * update unit test * introduce mockable key handler struct * shielding key success * remove clippy warnings * fix test * add unit tests for KeyHandler * rename to prepare for state inclusion * rename seal_handler * add shard as argument to sync state * some more renaming * add shard read & write process * [SealHandler] add unit tests & fix state * update networking test to include state * add default shard * add some documentation * remove ugly for loop * move authentications to separate file * update comment * remove obsolete, never ending loop * add error logs * remove extra phantom field * add sgx feature flag * remove global variables from test * add join handle to test * add some more logging info * Change tokio runtime to use 2 worker threads. Gossiping spawns new tokio tasks. (integritee-network#626) * Add state update sequence (integritee-network#632) * add bock_import_sequence.svg * move block_import.svg to docs/diagramms * update diagramm * add block import sequence * RPC call to get metadata from sgx-runtime (integritee-network#642) * RPC call to get metadata from sgx-runtime - rcp call - print sgx metadata cli * Change from review: Metadata is already encoded * Change from review Co-authored-by: echevrier <edith.chevrier@scs.ch> * bump substrate to commit 59649dd (integritee-network#645) * update .tomls to new substrate versions * cargo update * RawEvent -> Event * remove default from Accountid * RawEvent -> Event * cargo update enclave-runtime * fix bump errors * remove unused patch * finaly compiling * update sgx-runtime and substrate-api-client to github * remove integritee-node-runtime patch * cargo update -p std-std --precise 59649dd * update Github Actions integritee node * remove bh-config * fix clippy * fix cargo test * update spec version * update substrate-api-client * update sgx-runtime source * update substrate * adjust node version values * detect new game * solve merge conflicts * update sgx-runtime * fix some things * cargo format Co-authored-by: gaudenzkessler <92718752+gaudenzkessler@users.noreply.github.com> Co-authored-by: Gaudenz Kessler <gaudenz.kessler@scs.ch> Co-authored-by: Felix Müller <mullefel@users.noreply.github.com> Co-authored-by: echevrier <84318241+echevrier@users.noreply.github.com> Co-authored-by: echevrier <edith.chevrier@scs.ch>

* Make mu-ra and untrusted worker url queriable (integritee-network#595) * extract request_keys() to separate file * remove providr input, add dummy getter function * add node_api worker_for_shard call * fix error message * add primitives cache and rpc call * fix tests * add primitives-cache to workspace * fix unit tests * remove obsolete .yml provider from request-keys cmd * remove provider_addr from CI py scripts * fix reported worker address * improve usability of rpc-client * make it work * fix rebase error * add some delay * update local setup script * remove ugly async worker url, replace with enclave getter function * some steps towards a working exmaple.. * add peer_updater * fix unit test * fix some test clippy warnings * fix function name * fix client mu ra url * fix comment * fix comment * rename state_sync to appropriate request keys * fix comments and add missing _size to untrusted_worker_addr * update cargo.lock after rebase * fix typos * rename store_peers to set_peers * fix comment * move set_primitives to primitves cache repository * return read guard instead of primittves clone * rename config worker_rpc_port to trusted_worker_port * remove obsolete Error enum from request_keys.rs * fix unit tests * move thread spawning back into watch fn * rename worker-rpc-port to trusted-worker-port * readd external worker address * fix unit tests * fix unit test * add external addresses, optional port input and unit tests * update test names * [cli.yml] update shorts * fix local setup configs * change untrusted worker port to w * [sidechain] detect out of sync error (integritee-network#606) * inital commit * remove unwrap_err from assert_matches * Update substrate sp-core to version 4.1.0-dev (integritee-network#612) Co-authored-by: Gaudenz Kessler <gaudenz.kessler@scs.ch> * Renaming of unspecific SB and PB variable names (integritee-network#605) * some clean up & add hanlde import error * remove logic changes * fix unit tests * [aura block importer] rename SidechainBlock to SignedSidechainBlock * fix rebase errors * [aura mock] rename xxT import to xxTrait * [aura verifier] rename SidechainBlock to SignedSidechainblock where appropriate * fix rebase errors * [aura] rename Sidechainblock to SignedSidechainBlock * rename SB & PB to full written version and adapt to SignedSidechainBlock Where necessary * [sidechain block imported] remove extra generic from SignedParentchainBlock * some further SB & PB clean up * rename B & SB to SidechainBlock & SignedSidechainBlock * some further renaming * completely remove SB * rename all left over PBs * remove rebase error & rename to SignedSidechainBlock * rename to SignedSidechainBlock * Sidechain peer fetch blocks - RPC client/server (integritee-network#580) * WIP: RPC call to fetch sidechain blocks * WIP: sidechain peer fetch crate with RPC server and client impl * WIP: test for RPC peer sidechain block fetching * fix unit test * remove obsolete comment * fix rebase error * cargo fmt * fix tests * fix Fixme + add som Send+Sync to errors * update add_block_to_batch to return error. Otherwise silent fail * small comment fixes * make some comments better understandable * remove FIXME comment * remove new lines * fix trailing comments * [peer-fetch] change order or crates in .toml * [sidechain storage] fix error message of HeaderAncestryMismatch * [sidechain storage] exchange match statement with ok_or * [sidechain storage] use temp-dir in tests * [sidechain storage] remove extra genesis block check * fix rebase errors * remove untrstued url, replace with untrusted peer fetcher * [sidechain storage] fix comment * update delete_last_block description comment * [sidechain storage] fix comment grammer * move FetchUntrustedPeers trait to the top * [FetchBlocksFromPeer] extend description comment * update cargo.lock * rename get_blocks_following to get_blocks_after * rename get_blocks_following to get_blocks_after * rename all leftover "blocks_following" to "block_after" Co-authored-by: Bigna Härdi <bhaerdi@devsgx02.scs-ad.scs.ch> * [Sidechain] Peer block fetching o-call implementation (integritee-network#619) * introduce o-call for fetching sidechain blocks from peer * re-name api-client-extensions to node-api-extensions Sub-task of integritee-network#567 * add direct call rpc doc (integritee-network#620) * add some doc * add some structure to the links * restructure rpc interface * Update docs/README.md Co-authored-by: gaudenzkessler <92718752+gaudenzkessler@users.noreply.github.com> * adapt readmes according to review comments Co-authored-by: gaudenzkessler <92718752+gaudenzkessler@users.noreply.github.com> * [Sidechain] Peer sync and block production suspension (integritee-network#618) Peer syncer implementation (not in use yet) and block production suspender (also not in use yet) * update to most recent teaclave commit (integritee-network#624) * Sync state from fellow validateer (integritee-network#615) * rename request_keys to sync_state * rename request_key_prov to request_state_prov * rename request_keys.rs to sync_state.rs * restructure key and state provisioning server * some refactoring * add TlsServer struct * add test file * rename key_provision_server to state_provisioning_server * add unit test * update unit test * introduce mockable key handler struct * shielding key success * remove clippy warnings * fix test * add unit tests for KeyHandler * rename to prepare for state inclusion * rename seal_handler * add shard as argument to sync state * some more renaming * add shard read & write process * [SealHandler] add unit tests & fix state * update networking test to include state * add default shard * add some documentation * remove ugly for loop * move authentications to separate file * update comment * remove obsolete, never ending loop * add error logs * remove extra phantom field * add sgx feature flag * remove global variables from test * add join handle to test * add some more logging info * Change tokio runtime to use 2 worker threads. Gossiping spawns new tokio tasks. (integritee-network#626) * Add state update sequence (integritee-network#632) * add bock_import_sequence.svg * move block_import.svg to docs/diagramms * update diagramm * add block import sequence * RPC call to get metadata from sgx-runtime (integritee-network#642) * RPC call to get metadata from sgx-runtime - rcp call - print sgx metadata cli * Change from review: Metadata is already encoded * Change from review Co-authored-by: echevrier <edith.chevrier@scs.ch> * bump substrate to commit 59649dd (integritee-network#645) * update .tomls to new substrate versions * cargo update * RawEvent -> Event * remove default from Accountid * RawEvent -> Event * cargo update enclave-runtime * fix bump errors * remove unused patch * finaly compiling * update sgx-runtime and substrate-api-client to github * remove integritee-node-runtime patch * cargo update -p std-std --precise 59649dd * update Github Actions integritee node * remove bh-config * fix clippy * fix cargo test * update spec version * update substrate-api-client * update sgx-runtime source * update substrate * adjust node version values * detect new game * solve merge conflicts * update sgx-runtime * fix some things * cargo format Co-authored-by: haerdib <73821294+haerdib@users.noreply.github.com> Co-authored-by: Gaudenz Kessler <gaudenz.kessler@scs.ch> Co-authored-by: Felix Müller <mullefel@users.noreply.github.com> Co-authored-by: Bigna Härdi <bhaerdi@devsgx02.scs-ad.scs.ch> Co-authored-by: echevrier <84318241+echevrier@users.noreply.github.com> Co-authored-by: echevrier <edith.chevrier@scs.ch>

Felix Müller added 5 commits January 13, 2022 08:12

introduce o-call for fetching sidechain blocks from peer

2e8c57d

WIP: unit tests for the sidechain block fetching ocalls

4fffffe

change test WS ports 2222 -> 22222 and 3333 -> 33333 for less

8d30d58

collisions with other services running on those ports. (cherry picked from commit edd8f37)

unit test for the sidechain_ocall implementation for fetching blocks …

c295a2b

…from peer

Fix file comment in ocall bridge ffi module

2c15003

murerfel commented Jan 13, 2022

View reviewed changes

extract new node-api crate for the peer worker fetcher and node api f…

fd4fa45

…actory

murerfel changed the title ~~Sidechain peer fetch o-call implementation~~ [Sidechain] Peer block fetching o-call implementation Jan 13, 2022

murerfel self-assigned this Jan 13, 2022

Felix Müller added 2 commits January 13, 2022 14:30

add minor comment for extracting ocall bridge to separate crate

87de715

re-name api-client-extensions to node-api-extensions

62a4ea4

* move the node API factory to that crate * move the untrusted peer fetch back to the sidechain crate * remove the new 'parentchain node api' crate

murerfel commented Jan 13, 2022

View reviewed changes

murerfel requested review from clangenb and haerdib January 13, 2022 15:13

murerfel marked this pull request as ready for review January 13, 2022 15:14

haerdib approved these changes Jan 14, 2022

View reviewed changes

service/src/globals/tokio_handle.rs Outdated Show resolved Hide resolved

service/src/ocall_bridge/ffi/fetch_sidechain_blocks_from_peer.rs Outdated Show resolved Hide resolved

clangenb approved these changes Jan 16, 2022

View reviewed changes

Update service/src/globals/tokio_handle.rs

45b3cd4

Add period at the end of comment Co-authored-by: haerdib <73821294+haerdib@users.noreply.github.com>

murerfel mentioned this pull request Jan 17, 2022

[Sidechain] O-call to fetch sidechain blocks from peer has to pre-allocate buffer and fails if buffer size is not enough #621

Open

fix test utility function name

da2f482

murerfel merged commit 88a075a into master Jan 17, 2022

murerfel deleted the feature/fm-sidechain-peer-fetch-ocall branch January 17, 2022 11:10

		// We have to pre-allocate the vector and hope it's large enough
		let mut signed_blocks_encoded: Vec<u8> = vec![0; 4096 * 16];

		let node_api_factory =
		Arc::new(NodeApiFactory::new(config.node_url(), AccountKeyring::Alice.pair()));

Conversation

murerfel commented Jan 13, 2022

Uh oh!

murerfel left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

haerdib Jan 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

haerdib Jan 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

murerfel left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

haerdib left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

clangenb left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

haerdib Jan 14, 2022 •

edited

Loading

haerdib Jan 17, 2022 •

edited

Loading