NDRS-43: add initial storage components by Fraser999 · Pull Request #4 · casper-network/casper-node

Fraser999 · 2020-05-28T08:53:16Z

No description provided.

marc-casperlabs · 2020-05-28T08:54:40Z

src/app/config.rs

Derive Debug and add docs maybe?

This is still missing Debug =)

Yes - there's no need for a Debug impl here - there'd be no info shown other than the struct's type and we'd never need to print that anyway, since this only used while the tracing subscriber is handling a tracing event.

I eagerly put the Debug trait on anything where it is cheap as a very weak nod to https://rust-lang.github.io/api-guidelines/interoperability.html .

Debug still shows up in panic handling, for example.

goral09 · 2020-05-28T09:56:35Z

src/app/config.rs

I thought that the decision to go with 2-tier network hasn't been made yet.

This repo is still an early WIP effectively. For now, Marc and I are partially pair-programming and partially reviewing via Slack/zoom convos while we build out the basic structure of this repo.

I agree. This isn't final and not meant to decide this, we can easily pull it out and change it, for example. We just want to put an impl in now.

src/app/config.rs

goral09 · 2020-05-28T10:01:04Z

src/components/storage.rs

I've raised my concern about this couple of times (including questions in https://casperlabs.atlassian.net/wiki/spaces/EN/pages/477659259/Node-rs+Consensus+components+design document) but I really would like to first hear how is reactor going to ensure that no events are lost. We cannot afford to lose PutBlock event when another process is waiting for it. It might result in equivocations that will trigger slashing and validator losing all of his money.

I'm not sure if there are plans to have a run through of the finalised reactor structure once we finish implementing it. There were some design changes made last night during a prolonged zoom meeting, so these need to be implemented first I'd suggest.

how is reactor going to ensure that no events are lost

The queue for events is unbounded and infallible. Losing an event thus should not be possible except due to memory exhaustion, which should crash the whole program.

Or some panic that crashes the program, or a bug, or node operator turning it off, or a power outage, or machine crash etc etc.

Durability for events is not something we aim for currently, as many depend on volatile state (e.g. retrying a TCP connection would not be something that carries over smoothly to a restarted node).

Panics should be avoided by not calling functions that can panic (or checking preconditions first), they are going to leave your program in an undefined state (from the "business logic" perspective) anyway, so all bets are off. In general, components are expected to never crash and always gracefully handle even "invalid" events, which given the state machine nature, is not hard to do.

Power outage or operator turning it off - persistence would be handled by each component in a meaningful way, outside of the event handling. If it it is really important that an event go through at least once, you'll have to employ transactional techniques inside said component (e.g. similar to a WAL). When recovering from a crash, simply re-schedule these events.

Transactionality is paramount for the correctness of this software and the success of the project. Any failure to provide exactly-once-delivery and transactionality will result in validators losing money. Previous communication model was giving this implicitly out of the box.

src/components/storage.rs

goral09 · 2020-05-28T10:09:21Z

src/components/small_network/endpoint.rs

What is endpoint?

It's the endpoint of a remote peer in the small network. We'll build up the docs as required once the design stabilises.

Why do you need an Ord, PartialEq etc for it?

To be able to tell when an endpoint supercedes another. e.g if you have two endpoints received, take the one with the newer timestamp, if that is a toss-up, go by IP. This is required for conflict resolution in cases where multiple conflicting endpoints might be received.

Sounds like you're reinventing Kademlia.

We're not quite there yet! :)

The small network is pretty basic, in that it aims to be fully connected - every node connected to all peers in the network. There's no notion of a routing table structured by k-buckets, and no notion of "XOR-distance" between peers which are absolutely fundamental to Kademlia.

so if we will have 50 thousands validators everyone will see everyone else?

so if we will have 50 thousands validators everyone will see everyone else

If we have 50 thousand validators, we will have swapped out this networking component with a better one, possibly based on one of the available Kademlia crates.

This networking module is intended to be a concrete version of a small networking component that can be swapped out later, similar to the Pothole consensus.

marc-casperlabs

Are we maybe going a little overboard with the visilibity restrictions? I usually use pub a little more liberally, since it is a lot less noisy to read.

In the TLS module, many things are no longer exposed at all. Do we pull those back in when we need to?

src/app/config.rs

marc-casperlabs · 2020-05-28T09:38:59Z

src/app/config.rs

Newline would be nice (sorry).

You mean before this line?

src/app/config.rs

src/components/storage.rs

src/effect.rs

src/lib.rs

src/reactor.rs

marc-casperlabs · 2020-05-28T10:12:08Z

src/tls.rs

Hmm, why do we need to hide this behind another module?

It's purely to keep it out of the crate's public API and hence out of the generated docs since the macro unconditionally makes the struct pub.

I'll add a comment to this effect.

I'll add a comment to this effect.

👍 👍 👍

Fraser999 · 2020-05-28T11:50:43Z

Are we maybe going a little overboard with the visilibity restrictions? I usually use pub a little more liberally, since it is a lot less noisy to read.

I'm less concerned with the noise than that the public API truly reflects what we want it to reflect. I think we'll need to change the visibility a fair bit if/when we decide e.g. components should be separate crates.

In the TLS module, many things are no longer exposed at all. Do we pull those back in when we need to?

That's generally my preferred way of working. Keep the public API as trimmed down as possible while keeping it usable. That can be pretty subjective of course - I'm never sure about how many traits to derive for new structs, since it's not always easy to envisage all the use cases.

I think this approach also helps with the docs too - avoiding users of a crate/module getting bogged down reading docs/comments of what should be private implementation details.

marc-casperlabs · 2020-05-29T09:44:51Z

I'm kinda with you on the keeping public API surface small - hopefully people will remember where we put the code before they go and reimplement it.

marc-casperlabs

Good enough for me at this point in time. We will decide the precise shade of red for the shed some time in the future.

Impl JsonSchema for PublicKey and Signature

NDRS-807: Separate block body and header in the database

change client complex args to use checksummed hex decoding

…a_peer-selection

Merge dev into backpressure poc

Fixed size sender

Update default chainspec values

Binary port tests

Transaction validity is critical to the chain consistency. The specification of the routines is defined in https://github.com/FuelLabs/fuel-specs/blob/master/specs/protocol/tx_validity.md Some of these specs may intersect with VM logic; these intersections are outside the scope of this lib and should be implemented directly in the VM. Resolves #3

The traits were moved to `fuel-storage` so the map requirements won't overlap with the infrastructure types for the VM and client.

# This is the 1st commit message: rebasing # The commit message #2 will be skipped: # WIP # The commit message #3 will be skipped: # WIP # The commit message #4 will be skipped: # WIP # The commit message #5 will be skipped: # WIP # The commit message #6 will be skipped: # Removing notion of install upgrade wasms # The commit message #7 will be skipped: # Removing notion of install upgrade wasms # The commit message #8 will be skipped: # reinstantiated install/upgrade # The commit message #9 will be skipped: # mid additional_computation_factor refactor # The commit message casper-network#10 will be skipped: # mid removing TransactionCategory

merge dev

Fraser999 requested a review from marc-casperlabs May 28, 2020 08:54

marc-casperlabs reviewed May 28, 2020

View reviewed changes

Fraser999 force-pushed the NDRS-38-async-storage branch from 2ddcd44 to eb29dab Compare May 28, 2020 08:58

goral09 reviewed May 28, 2020

View reviewed changes

src/app/config.rs Outdated Show resolved Hide resolved

goral09 reviewed May 28, 2020

View reviewed changes

src/components/storage.rs Outdated Show resolved Hide resolved

goral09 reviewed May 28, 2020

View reviewed changes

marc-casperlabs suggested changes May 28, 2020

View reviewed changes

Fraser999 force-pushed the NDRS-38-async-storage branch from b08a61e to 517e258 Compare May 28, 2020 11:53

EdHastingsCasperAssociation self-requested a review May 28, 2020 13:52

EdHastingsCasperAssociation approved these changes May 28, 2020

View reviewed changes

marc-casperlabs approved these changes May 29, 2020

View reviewed changes

Fraser999 added 6 commits May 29, 2020 11:14

EE-1020: restructure project in preparation for new crates being added

fc95a00

NDRS-38: added initial cut of storage component

fbc847e

NDRS-38: put storage calls behind async interface

70a01ba

NDRS-38: move RwLock into InMemBlockStore

d3fce2a

NDRS-43: improve logging

78806d6

NDRS-42: address review comments

f2fc5d7

Fraser999 force-pushed the NDRS-38-async-storage branch from 517e258 to f2fc5d7 Compare May 29, 2020 10:14

Fraser999 merged commit 4a8431a into casper-network:master May 29, 2020

Fraser999 deleted the NDRS-38-async-storage branch May 29, 2020 10:23

darthsiroftardis referenced this pull request in darthsiroftardis/casper-node Nov 25, 2020

Merge pull request #4 from darthsiroftardis/NDRS-591-rpc-docs-karan

627791a

Impl JsonSchema for PublicKey and Signature

bors bot pushed a commit that referenced this pull request Feb 11, 2021

Merge pull request #4 from xcthulhu/mpwd/NDRS-807

88db27a

NDRS-807: Separate block body and header in the database

casperlabs-bors-ng bot pushed a commit that referenced this pull request Oct 15, 2021

Merge pull request #4 from Fraser999/checksummed-hex

c424a25

change client complex args to use checksummed hex decoding

goral09 added a commit to goral09/casper-node-1 that referenced this pull request Apr 15, 2022

Merge pull request casper-network#4 from marc-casperlabs/gov-285-extr…

b6fbb5e

…a_peer-selection

casperlabs-bors-ng bot pushed a commit that referenced this pull request May 12, 2022

Merge pull request #4 from rafal-ch/merge_dev_into_backpressure_poc

233a1a4

Merge dev into backpressure poc

marc-casperlabs added a commit that referenced this pull request Jul 5, 2022

Merge pull request #4 from marc-casperlabs/fixed_size_sender

e17ed9f

Fixed size sender

rafal-ch mentioned this pull request Apr 27, 2023

Fix the flakiness of the nightly tests #3862

Closed

sacherjj added a commit that referenced this pull request Apr 27, 2023

Merge pull request #4 from Fraser999/chainspec_updates_for_1.4.15

a553638

Update default chainspec values

casperlabs-bors-ng bot pushed a commit that referenced this pull request Feb 29, 2024

Merge pull request #4 from rafal-ch/binary_port_tests

fc5b7b0

Binary port tests

rafal-ch pushed a commit that referenced this pull request Sep 11, 2024

Add test and format CI. (#4)

e54078a

rafal-ch pushed a commit that referenced this pull request Sep 11, 2024

Remove license. (#4)

be23bd0

rafal-ch pushed a commit that referenced this pull request Sep 11, 2024

Remove data traits in favor of fuel-storage (#4)

7b5e6a1

The traits were moved to `fuel-storage` so the map requirements won't overlap with the infrastructure types for the VM and client.

igor-casper added a commit that referenced this pull request Mar 27, 2025

Merge pull request #4 from igor-casper/dev

3c60273

merge dev

Conversation

Fraser999 commented May 28, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marc-casperlabs May 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

goral09 May 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

goral09 May 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marc-casperlabs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Fraser999 commented May 28, 2020

Uh oh!

marc-casperlabs commented May 29, 2020

Uh oh!

marc-casperlabs left a comment

Choose a reason for hiding this comment

marc-casperlabs May 28, 2020 •

edited

Loading

goral09 May 28, 2020 •

edited

Loading

goral09 May 29, 2020 •

edited

Loading