Make mu-ra and untrusted worker url queriable by haerdib · Pull Request #595 · integritee-network/worker

haerdib · 2021-12-22T18:55:30Z

Untrusted worker port is now settable at start-up, not hardcoded anymore.
Stores the mu-ra and untrusted worker url inside the enclave
Introduces rpc calls to retrieve mu-ra und untrusted url via trusted enclave server for fellow validateers (which only know the trusted ws address via the teerex pallet)
Introduces a "peers" field inside the Worker struct, accompanied by a "PeerUpdater". A first step to separate block gossiping from peer querying.
Allows setting of external urls via cli (trusted, untrusted an mu-ra) in case the worker is behind an nginx oder docker envinroment and the external url differs from the local. (including port)

closes #566

cli/src/main.rs

core-primitives/enclave-api/ffi/src/lib.rs

core/rpc-client/src/direct_client.rs

local-setup/launch.py

local-setup/github-action-config.json

service/src/main.rs

service/src/sync_state.rs

service/src/worker.rs

service/src/request_keys.rs

core-primitives/enclave-api/ffi/src/lib.rs

service/src/sync_state.rs

core-primitives/primitives-cache/src/error.rs

core-primitives/primitives-cache/src/primitives_cache.rs

service/src/globals/worker.rs

gaudenzkessler

A few small things I have seen.

service/src/config.rs

core-primitives/enclave-api/src/enclave_base.rs

clangenb

Looks good, was much more work than I thought. :)

But I have some comments.

core-primitives/primitives-cache/src/primitives_cache.rs

clangenb · 2021-12-27T14:47:13Z

service/src/cli.yml

+    - untrusted-worker-port:
+        short: U
+        long: untrusted-worker-port
+        help: Set the untrsuted websocket port of the worker


We definitely need to document, which parts are running trustedly and which parts are not.

Agreed. Do you want me to do this in this PR?

Did some updates, though no renaming of files / folders. I'm afraid of rebase conflicts with PR #580

clangenb · 2021-12-27T14:49:38Z

service/src/cli.yml

                long: skip-ra
                short: s
                help: skip remote attestation. Set this flag if running enclave in SW mode
-            - w-server:


No, this is not obsolete. This is relevant for all workers running behind an nginx instance for wss setup because internally, they will be started with a local IP address, but they need to register with on the parentchain with the public address of the nginx instance

enclave-runtime/src/lib.rs

service/src/global_peer_updater.rs

service/src/main.rs

service/src/request_keys.rs

service/src/tests/mock.rs

service/src/worker.rs

clangenb

Except for the thread thingy, you could convince me everywhere. :)

cli/src/main.rs

clangenb · 2021-12-28T07:16:47Z

service/src/cli.yml

                long: skip-ra
                short: s
                help: skip remote attestation. Set this flag if running enclave in SW mode
-            - w-server:


external-address seems fine to me. I'd use address here because it is generic and implies that we can use both: ip-addresses, and urls.

However, there is documentation that needs to be adjusted. So I wonder if you want to do this now.

service/src/config.rs

service/src/global_peer_updater.rs

service/src/main.rs

service/src/tests/mock.rs

service/src/main.rs

haerdib · 2021-12-28T14:11:58Z

service/src/config.rs

+	///
+	/// In case external_address is set, it should be considered that the internal trusted server is a tls
+	/// websocket and must have have a wss:// primary to the internal worker ip.
+	pub fn trusted_worker_url_external(&self) -> String {


Is this an acceptable approach @clangenb ?

As I advised above, I think this is not acceptable as it will break reverse-proxied setups.

I suggest parsing the external address and do the following:

if the external address contains a port, use it as is.

if not, default to the trusted-worker-port

All right all right ,I give up, will do 🥲

Added this option with the function add_port_if_necessary. Added some unit tests as well, because config got relatively complex.

haerdib · 2021-12-28T14:12:35Z

service/src/cli.yml

                long: skip-ra
                short: s
                help: skip remote attestation. Set this flag if running enclave in SW mode
-            - w-server:


Readded it. But I didn't see any documentation.. not in the book and not in the worker.

cli/demo_direct_call.sh

clangenb

We need to discuss the port thing again, but otherwise it looks good.

cli/src/main.rs

service/src/cli.yml

clangenb · 2021-12-28T14:36:41Z

service/src/cli.yml

                long: skip-ra
                short: s
                help: skip remote attestation. Set this flag if running enclave in SW mode
-            - w-server:


Ok, but @gaudenzkessler needs to know that he needs to update the flags in the docker demo

clangenb · 2021-12-28T14:39:10Z

service/src/config.rs

+	///
+	/// In case external_address is set, it should be considered that the internal trusted server is a tls
+	/// websocket and must have have a wss:// primary to the internal worker ip.
+	pub fn trusted_worker_url_external(&self) -> String {


As I advised above, I think this is not acceptable as it will break reverse-proxied setups.

I suggest parsing the external address and do the following:

if the external address contains a port, use it as is.

if not, default to the trusted-worker-port

clangenb · 2021-12-28T14:42:22Z

service/src/config.rs

+	/// Returns the untrusted worker url that should be addressed by external clients.
+	///
+	/// In case external_address is set, it should be considered that the internal untrusted worker url
+	/// must have a ws:// primary to the internal worker ip.
+	pub fn untrusted_worker_url_external(&self) -> String {
+		match &self.external_worker_address {
+			Some(external_address) =>
+				format!("{}:{}", external_address, self.untrusted_worker_port),
+			None => format!("ws://{}:{}", self.worker_ip, self.untrusted_worker_port),
+		}
 	}


Hmm, this one is very tricky because we will always have wss in the published external address, but it will not always be wss for the untrusted part.

Spontaneously, I have no idea how to solve that cleanly such that it always works.

Me neither. Added all external urls as an input option. I think we should have covered all cases like that. Maybe not in an elegant way, but it should work for all cases.

service/src/global_peer_updater.rs

clangenb

Some last questions about the url, but then we are good. :)

service/src/cli.yml

service/src/config.rs

clangenb · 2021-12-29T07:24:35Z

service/src/config.rs

+				// url is of format 127.0.0.1:4000, port was added
+				url.to_string()


This one should be invalid, shouldn't it?

mu-ra server actually expects one without. When I tried to add ws or wss the client failed. But I haven't looked into details there.

Oh my gosh, we need to unify the setup here, this is not bearable.

clangenb · 2021-12-29T07:25:33Z

service/src/config.rs

+				url.to_string()
+			}
+		},
+		1 => format!("{}:{}", url, port),


This one should also be invalid, don't we always need to define the protocol?

clangenb · 2021-12-29T07:27:39Z

service/src/config.rs

+		let resulting_url = add_port_if_necessary(url, port);
+
+		assert_eq!(resulting_url, format!("{}:{}", url, port));
+	}


Wow, nice testing.

Just a comment, your test-names are sometimes too verbose that I don't really get what they are testing.

True enough, I also wasn't happy. Let me think about some better names.

Updated - but making them "short" was not really possible. Hope it's better understandable now, either way.

gaudenzkessler · 2021-12-29T07:59:35Z

service/src/cli.yml

+        takes_value: true
+        default_value: "2001"
+    - trusted-external-address:
+        short: trusted-ext


shouldn't we still offer a one letter short option (like U, p and P) which does not have to be verbose?

In case you have to set external addresses, you will probably be setting all three some them. I imagine it quite messy with one letter shortcuts, but I'm open for suggestions.

I did not see that before, but AFAIK, must not be more than one letter. This will not work.

the idea is that you can still use the long or the short option, as you prefer.

I see. I'll adapt then, thanks for pointing that out.

clangenb

Looks great! 🦖

gaudenzkessler · 2021-12-29T14:34:46Z

service/src/cli.yml

                long: skip-ra
                short: s
                help: skip remote attestation. Set this flag if running enclave in SW mode
-            - w-server:


* Make mu-ra and untrusted worker url queriable (integritee-network#595) * extract request_keys() to separate file * remove providr input, add dummy getter function * add node_api worker_for_shard call * fix error message * add primitives cache and rpc call * fix tests * add primitives-cache to workspace * fix unit tests * remove obsolete .yml provider from request-keys cmd * remove provider_addr from CI py scripts * fix reported worker address * improve usability of rpc-client * make it work * fix rebase error * add some delay * update local setup script * remove ugly async worker url, replace with enclave getter function * some steps towards a working exmaple.. * add peer_updater * fix unit test * fix some test clippy warnings * fix function name * fix client mu ra url * fix comment * fix comment * rename state_sync to appropriate request keys * fix comments and add missing _size to untrusted_worker_addr * update cargo.lock after rebase * fix typos * rename store_peers to set_peers * fix comment * move set_primitives to primitves cache repository * return read guard instead of primittves clone * rename config worker_rpc_port to trusted_worker_port * remove obsolete Error enum from request_keys.rs * fix unit tests * move thread spawning back into watch fn * rename worker-rpc-port to trusted-worker-port * readd external worker address * fix unit tests * fix unit test * add external addresses, optional port input and unit tests * update test names * [cli.yml] update shorts * fix local setup configs * change untrusted worker port to w * [sidechain] detect out of sync error (integritee-network#606) * inital commit * remove unwrap_err from assert_matches * Update substrate sp-core to version 4.1.0-dev (integritee-network#612) Co-authored-by: Gaudenz Kessler <gaudenz.kessler@scs.ch> * Renaming of unspecific SB and PB variable names (integritee-network#605) * some clean up & add hanlde import error * remove logic changes * fix unit tests * [aura block importer] rename SidechainBlock to SignedSidechainBlock * fix rebase errors * [aura mock] rename xxT import to xxTrait * [aura verifier] rename SidechainBlock to SignedSidechainblock where appropriate * fix rebase errors * [aura] rename Sidechainblock to SignedSidechainBlock * rename SB & PB to full written version and adapt to SignedSidechainBlock Where necessary * [sidechain block imported] remove extra generic from SignedParentchainBlock * some further SB & PB clean up * rename B & SB to SidechainBlock & SignedSidechainBlock * some further renaming * completely remove SB * rename all left over PBs * remove rebase error & rename to SignedSidechainBlock * rename to SignedSidechainBlock * Sidechain peer fetch blocks - RPC client/server (integritee-network#580) * WIP: RPC call to fetch sidechain blocks * WIP: sidechain peer fetch crate with RPC server and client impl * WIP: test for RPC peer sidechain block fetching * fix unit test * remove obsolete comment * fix rebase error * cargo fmt * fix tests * fix Fixme + add som Send+Sync to errors * update add_block_to_batch to return error. Otherwise silent fail * small comment fixes * make some comments better understandable * remove FIXME comment * remove new lines * fix trailing comments * [peer-fetch] change order or crates in .toml * [sidechain storage] fix error message of HeaderAncestryMismatch * [sidechain storage] exchange match statement with ok_or * [sidechain storage] use temp-dir in tests * [sidechain storage] remove extra genesis block check * fix rebase errors * remove untrstued url, replace with untrusted peer fetcher * [sidechain storage] fix comment * update delete_last_block description comment * [sidechain storage] fix comment grammer * move FetchUntrustedPeers trait to the top * [FetchBlocksFromPeer] extend description comment * update cargo.lock * rename get_blocks_following to get_blocks_after * rename get_blocks_following to get_blocks_after * rename all leftover "blocks_following" to "block_after" Co-authored-by: Bigna Härdi <bhaerdi@devsgx02.scs-ad.scs.ch> * [Sidechain] Peer block fetching o-call implementation (integritee-network#619) * introduce o-call for fetching sidechain blocks from peer * re-name api-client-extensions to node-api-extensions Sub-task of integritee-network#567 * add direct call rpc doc (integritee-network#620) * add some doc * add some structure to the links * restructure rpc interface * Update docs/README.md Co-authored-by: gaudenzkessler <92718752+gaudenzkessler@users.noreply.github.com> * adapt readmes according to review comments Co-authored-by: gaudenzkessler <92718752+gaudenzkessler@users.noreply.github.com> * [Sidechain] Peer sync and block production suspension (integritee-network#618) Peer syncer implementation (not in use yet) and block production suspender (also not in use yet) * update to most recent teaclave commit (integritee-network#624) * Sync state from fellow validateer (integritee-network#615) * rename request_keys to sync_state * rename request_key_prov to request_state_prov * rename request_keys.rs to sync_state.rs * restructure key and state provisioning server * some refactoring * add TlsServer struct * add test file * rename key_provision_server to state_provisioning_server * add unit test * update unit test * introduce mockable key handler struct * shielding key success * remove clippy warnings * fix test * add unit tests for KeyHandler * rename to prepare for state inclusion * rename seal_handler * add shard as argument to sync state * some more renaming * add shard read & write process * [SealHandler] add unit tests & fix state * update networking test to include state * add default shard * add some documentation * remove ugly for loop * move authentications to separate file * update comment * remove obsolete, never ending loop * add error logs * remove extra phantom field * add sgx feature flag * remove global variables from test * add join handle to test * add some more logging info * Change tokio runtime to use 2 worker threads. Gossiping spawns new tokio tasks. (integritee-network#626) * Add state update sequence (integritee-network#632) * add bock_import_sequence.svg * move block_import.svg to docs/diagramms * update diagramm * add block import sequence * RPC call to get metadata from sgx-runtime (integritee-network#642) * RPC call to get metadata from sgx-runtime - rcp call - print sgx metadata cli * Change from review: Metadata is already encoded * Change from review Co-authored-by: echevrier <edith.chevrier@scs.ch> * bump substrate to commit 59649dd (integritee-network#645) * update .tomls to new substrate versions * cargo update * RawEvent -> Event * remove default from Accountid * RawEvent -> Event * cargo update enclave-runtime * fix bump errors * remove unused patch * finaly compiling * update sgx-runtime and substrate-api-client to github * remove integritee-node-runtime patch * cargo update -p std-std --precise 59649dd * update Github Actions integritee node * remove bh-config * fix clippy * fix cargo test * update spec version * update substrate-api-client * update sgx-runtime source * update substrate * adjust node version values * detect new game * solve merge conflicts * update sgx-runtime * fix some things * cargo format Co-authored-by: gaudenzkessler <92718752+gaudenzkessler@users.noreply.github.com> Co-authored-by: Gaudenz Kessler <gaudenz.kessler@scs.ch> Co-authored-by: Felix Müller <mullefel@users.noreply.github.com> Co-authored-by: echevrier <84318241+echevrier@users.noreply.github.com> Co-authored-by: echevrier <edith.chevrier@scs.ch>

* Make mu-ra and untrusted worker url queriable (integritee-network#595) * extract request_keys() to separate file * remove providr input, add dummy getter function * add node_api worker_for_shard call * fix error message * add primitives cache and rpc call * fix tests * add primitives-cache to workspace * fix unit tests * remove obsolete .yml provider from request-keys cmd * remove provider_addr from CI py scripts * fix reported worker address * improve usability of rpc-client * make it work * fix rebase error * add some delay * update local setup script * remove ugly async worker url, replace with enclave getter function * some steps towards a working exmaple.. * add peer_updater * fix unit test * fix some test clippy warnings * fix function name * fix client mu ra url * fix comment * fix comment * rename state_sync to appropriate request keys * fix comments and add missing _size to untrusted_worker_addr * update cargo.lock after rebase * fix typos * rename store_peers to set_peers * fix comment * move set_primitives to primitves cache repository * return read guard instead of primittves clone * rename config worker_rpc_port to trusted_worker_port * remove obsolete Error enum from request_keys.rs * fix unit tests * move thread spawning back into watch fn * rename worker-rpc-port to trusted-worker-port * readd external worker address * fix unit tests * fix unit test * add external addresses, optional port input and unit tests * update test names * [cli.yml] update shorts * fix local setup configs * change untrusted worker port to w * [sidechain] detect out of sync error (integritee-network#606) * inital commit * remove unwrap_err from assert_matches * Update substrate sp-core to version 4.1.0-dev (integritee-network#612) Co-authored-by: Gaudenz Kessler <gaudenz.kessler@scs.ch> * Renaming of unspecific SB and PB variable names (integritee-network#605) * some clean up & add hanlde import error * remove logic changes * fix unit tests * [aura block importer] rename SidechainBlock to SignedSidechainBlock * fix rebase errors * [aura mock] rename xxT import to xxTrait * [aura verifier] rename SidechainBlock to SignedSidechainblock where appropriate * fix rebase errors * [aura] rename Sidechainblock to SignedSidechainBlock * rename SB & PB to full written version and adapt to SignedSidechainBlock Where necessary * [sidechain block imported] remove extra generic from SignedParentchainBlock * some further SB & PB clean up * rename B & SB to SidechainBlock & SignedSidechainBlock * some further renaming * completely remove SB * rename all left over PBs * remove rebase error & rename to SignedSidechainBlock * rename to SignedSidechainBlock * Sidechain peer fetch blocks - RPC client/server (integritee-network#580) * WIP: RPC call to fetch sidechain blocks * WIP: sidechain peer fetch crate with RPC server and client impl * WIP: test for RPC peer sidechain block fetching * fix unit test * remove obsolete comment * fix rebase error * cargo fmt * fix tests * fix Fixme + add som Send+Sync to errors * update add_block_to_batch to return error. Otherwise silent fail * small comment fixes * make some comments better understandable * remove FIXME comment * remove new lines * fix trailing comments * [peer-fetch] change order or crates in .toml * [sidechain storage] fix error message of HeaderAncestryMismatch * [sidechain storage] exchange match statement with ok_or * [sidechain storage] use temp-dir in tests * [sidechain storage] remove extra genesis block check * fix rebase errors * remove untrstued url, replace with untrusted peer fetcher * [sidechain storage] fix comment * update delete_last_block description comment * [sidechain storage] fix comment grammer * move FetchUntrustedPeers trait to the top * [FetchBlocksFromPeer] extend description comment * update cargo.lock * rename get_blocks_following to get_blocks_after * rename get_blocks_following to get_blocks_after * rename all leftover "blocks_following" to "block_after" Co-authored-by: Bigna Härdi <bhaerdi@devsgx02.scs-ad.scs.ch> * [Sidechain] Peer block fetching o-call implementation (integritee-network#619) * introduce o-call for fetching sidechain blocks from peer * re-name api-client-extensions to node-api-extensions Sub-task of integritee-network#567 * add direct call rpc doc (integritee-network#620) * add some doc * add some structure to the links * restructure rpc interface * Update docs/README.md Co-authored-by: gaudenzkessler <92718752+gaudenzkessler@users.noreply.github.com> * adapt readmes according to review comments Co-authored-by: gaudenzkessler <92718752+gaudenzkessler@users.noreply.github.com> * [Sidechain] Peer sync and block production suspension (integritee-network#618) Peer syncer implementation (not in use yet) and block production suspender (also not in use yet) * update to most recent teaclave commit (integritee-network#624) * Sync state from fellow validateer (integritee-network#615) * rename request_keys to sync_state * rename request_key_prov to request_state_prov * rename request_keys.rs to sync_state.rs * restructure key and state provisioning server * some refactoring * add TlsServer struct * add test file * rename key_provision_server to state_provisioning_server * add unit test * update unit test * introduce mockable key handler struct * shielding key success * remove clippy warnings * fix test * add unit tests for KeyHandler * rename to prepare for state inclusion * rename seal_handler * add shard as argument to sync state * some more renaming * add shard read & write process * [SealHandler] add unit tests & fix state * update networking test to include state * add default shard * add some documentation * remove ugly for loop * move authentications to separate file * update comment * remove obsolete, never ending loop * add error logs * remove extra phantom field * add sgx feature flag * remove global variables from test * add join handle to test * add some more logging info * Change tokio runtime to use 2 worker threads. Gossiping spawns new tokio tasks. (integritee-network#626) * Add state update sequence (integritee-network#632) * add bock_import_sequence.svg * move block_import.svg to docs/diagramms * update diagramm * add block import sequence * RPC call to get metadata from sgx-runtime (integritee-network#642) * RPC call to get metadata from sgx-runtime - rcp call - print sgx metadata cli * Change from review: Metadata is already encoded * Change from review Co-authored-by: echevrier <edith.chevrier@scs.ch> * bump substrate to commit 59649dd (integritee-network#645) * update .tomls to new substrate versions * cargo update * RawEvent -> Event * remove default from Accountid * RawEvent -> Event * cargo update enclave-runtime * fix bump errors * remove unused patch * finaly compiling * update sgx-runtime and substrate-api-client to github * remove integritee-node-runtime patch * cargo update -p std-std --precise 59649dd * update Github Actions integritee node * remove bh-config * fix clippy * fix cargo test * update spec version * update substrate-api-client * update sgx-runtime source * update substrate * adjust node version values * detect new game * solve merge conflicts * update sgx-runtime * fix some things * cargo format Co-authored-by: haerdib <73821294+haerdib@users.noreply.github.com> Co-authored-by: Gaudenz Kessler <gaudenz.kessler@scs.ch> Co-authored-by: Felix Müller <mullefel@users.noreply.github.com> Co-authored-by: Bigna Härdi <bhaerdi@devsgx02.scs-ad.scs.ch> Co-authored-by: echevrier <84318241+echevrier@users.noreply.github.com> Co-authored-by: echevrier <edith.chevrier@scs.ch>

haerdib self-assigned this Dec 22, 2021

haerdib force-pushed the bh-remove-async-url branch from 389296a to 2464e50 Compare December 23, 2021 16:13

haerdib commented Dec 27, 2021

View reviewed changes

haerdib changed the title ~~Remove async worker url~~ Make mu-ra and untrusted worker url queriable Dec 27, 2021

haerdib requested review from clangenb and gaudenzkessler December 27, 2021 06:51

haerdib marked this pull request as ready for review December 27, 2021 09:30

haerdib mentioned this pull request Dec 27, 2021

replace own worker address by localhost and do not end gossip loop when peer connections fails #601

Closed

gaudenzkessler requested changes Dec 27, 2021

View reviewed changes

service/src/config.rs Outdated Show resolved Hide resolved

service/src/config.rs Outdated Show resolved Hide resolved

core-primitives/enclave-api/src/enclave_base.rs Outdated Show resolved Hide resolved

clangenb reviewed Dec 27, 2021

View reviewed changes

haerdib added 19 commits December 27, 2021 18:06

extract request_keys() to separate file

c132ad7

remove providr input, add dummy getter function

9583224

add node_api worker_for_shard call

fbc5b36

fix error message

b96a829

add primitives cache and rpc call

52b7094

fix tests

0c87e08

add primitives-cache to workspace

a5c043d

fix unit tests

62937ce

remove obsolete .yml provider from request-keys cmd

0d5f61d

remove provider_addr from CI py scripts

ddf2868

fix reported worker address

8ce4a90

improve usability of rpc-client

b0f0798

make it work

c1b8307

fix rebase error

189bf5f

add some delay

84dc22b

update local setup script

b6b3990

remove ugly async worker url, replace with enclave getter function

8096a96

some steps towards a working exmaple..

7f40e56

add peer_updater

349f1fb

haerdib added 3 commits December 27, 2021 20:10

rename config worker_rpc_port to trusted_worker_port

0966f1e

remove obsolete Error enum from request_keys.rs

8afa53f

fix unit tests

547638a

clangenb reviewed Dec 28, 2021

View reviewed changes

haerdib added 3 commits December 28, 2021 14:10

move thread spawning back into watch fn

8ac6b2e

rename worker-rpc-port to trusted-worker-port

cd2e7a9

readd external worker address

44082be

haerdib commented Dec 28, 2021

View reviewed changes

haerdib requested a review from gaudenzkessler December 28, 2021 14:30

clangenb reviewed Dec 28, 2021

View reviewed changes

haerdib added 3 commits December 28, 2021 15:54

fix unit tests

396d768

fix unit test

0a0d501

add external addresses, optional port input and unit tests

09dc4ed

haerdib requested a review from clangenb December 28, 2021 18:35

haerdib mentioned this pull request Dec 28, 2021

Sidechain peer fetch blocks - RPC client/server #580

Merged

clangenb reviewed Dec 29, 2021

View reviewed changes

gaudenzkessler reviewed Dec 29, 2021

View reviewed changes

haerdib added 2 commits December 29, 2021 12:53

update test names

7200956

[cli.yml] update shorts

89ba374

haerdib requested review from clangenb and gaudenzkessler December 29, 2021 11:55

clangenb approved these changes Dec 29, 2021

View reviewed changes

gaudenzkessler approved these changes Dec 29, 2021

View reviewed changes

haerdib added 2 commits December 29, 2021 17:03

fix local setup configs

16c5f21

change untrusted worker port to w

8844c29

haerdib merged commit 412fc8f into master Dec 30, 2021

haerdib deleted the bh-remove-async-url branch December 30, 2021 06:24

haerdib mentioned this pull request Dec 30, 2021

Discover from which validateer to sync the state #443

Closed

		// url is of format 127.0.0.1:4000, port was added
		url.to_string()

Conversation

haerdib commented Dec 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gaudenzkessler left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

clangenb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

clangenb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

clangenb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

haerdib commented Dec 22, 2021 •

edited

Loading

clangenb Dec 29, 2021 •

edited

Loading