Skip to content

NDRS-76: add LMDB storage implementation#21

Merged
Fraser999 merged 5 commits intocasper-network:masterfrom
Fraser999:NDRS-76-storage
Jun 15, 2020
Merged

NDRS-76: add LMDB storage implementation#21
Fraser999 merged 5 commits intocasper-network:masterfrom
Fraser999:NDRS-76-storage

Conversation

@Fraser999
Copy link
Contributor

@Fraser999 Fraser999 commented Jun 14, 2020

Note: This PR is based on top of #20, so that should ideally be reviewed and merged first. Only the final four commits of this PR are not included in #20.

This PR adds the LMDB storage implementation along with corresponding config options. There is a separate LMDB environment for each of the separate types of store (Block and Deploy).

It also includes a simplification of the storage component by merging the discrete internal error types into a single storage Error enum, and by changing the StorageType trait's associated types from Store<T> to T.

Finally, it also adds a new HTTP endpoint to allow the client to receive a list of hashes of all stored deploys.

goral09
goral09 reviewed Jun 15, 2020
View reviewed changes
src/components/storage.rs
Comment on lines +41 to +42
+ Send
+ Sync;
Copy link
Contributor

@goral09 goral09 Jun 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just an idea: if something is Clone and Copy why does it have to be also Send and Sync? Wouldn't it be simpler to just clone it?

Copy link
Contributor Author

@Fraser999 Fraser999 Jun 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Clone and Copy traits are somewhat orthogonal to Send and Sync. The latter are just "marker traits" - they don't really do anything except indicate the implementing type can be safely sent or shared between threads (see https://doc.rust-lang.org/nomicon/send-and-sync.html for a better explanation than my effort!)

Copy link
Contributor

@goral09 goral09 Jun 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I am aware of that. I just thought that as a caller you could decide how to do it. My reasoning was that if you copy the value every time it's sent between threads then you don't need Send but maybe that's precisely what Send would do/decide. I will re-read the nomicon. Thans.

goral09
goral09 reviewed Jun 15, 2020
View reviewed changes
src/components/storage.rs
Comment on lines +55 to +56
type Block: Value;
type Deploy: Value;
Copy link
Contributor

@goral09 goral09 Jun 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the reason for making a single point of access to deploy and block storage?

Copy link
Contributor Author

@Fraser999 Fraser999 Jun 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's really to have a single point of access to all storage pieces. The storage component will manage all aspects of storage, which is important mostly in the case of on-disk storage.

We made the design decision a while back since it seems like the best way handle things like managing disk usage, and having fewer IO components (where an IO component has a need to break from the normal component paradigm since its operations can be blocking for an unacceptably long period)

goral09
goral09 reviewed Jun 15, 2020
View reviewed changes
README.md

#### List stored `Deploy`s

To get a list of all stored `Deploy`s' hashes:
Copy link
Contributor

@goral09 goral09 Jun 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just for PoC right?

Copy link
Contributor Author

@Fraser999 Fraser999 Jun 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would think so, I certainly put it in for that reason. If it were to remain, it'd need some work (e.g. to handle larger responses)

goral09
goral09 reviewed Jun 15, 2020
View reviewed changes
src/components/api_server.rs Outdated
hex_hashes.push(',');
}
}
hex_hashes.push(']');
Copy link
Contributor

@goral09 goral09 Jun 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't serde(-json) do this for you?

Copy link
Contributor Author

@Fraser999 Fraser999 Jun 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, it won't encode binary data as hex, so if I were to use serde_json, I'd need to first convert the Vec<DeployHash> to Vec<String>, and use serde_json on that vec. I didn't think it was worth the effort to use serde_json for that, but looking at it with fresh eyes, it might be worthwhile. I'll try it.

@Fraser999 Fraser999 merged commit 23b051f into casper-network:master Jun 15, 2020
@Fraser999 Fraser999 deleted the NDRS-76-storage branch June 15, 2020 15:50
mpapierski pushed a commit to mpapierski/casper-node that referenced this pull request Nov 30, 2021
@darthsiroftardis
Merge pull request casper-network#21 from casper-network/Distribute-p…
e5f996c 
…urses

Do not create temporary purses during distribute
casperlabs-bors-ng bot pushed a commit that referenced this pull request Apr 28, 2022
Merge pull request #21 from TomVasile/rel-146-ci-fix
148b092 
CI: hardcode protocol 1 to 1.4.5
jacek-casper pushed a commit to jacek-casper/casper-node that referenced this pull request Apr 3, 2024
@EdHastingsCasperAssociation
Merge pull request casper-network#21 from alsrdn/burn_fee_tests
4ff4e37 
reactor/tests: add tests for burning fees
rafal-ch pushed a commit that referenced this pull request Sep 11, 2024
@vlopes11
Add serde-types feature (#21)
0186cd0 
* Add serde-types feature

* Update format conventions for Cargo.toml
rafal-ch pushed a commit that referenced this pull request Sep 11, 2024
@vlopes11
Add Receipt and receipt root calculation (#21)
4f9d0dc
rafal-ch pushed a commit that referenced this pull request Sep 11, 2024
@Voxelot
make docs feature aware on docs.rs (#21)
829e82f
darthsiroftardis added a commit that referenced this pull request Mar 24, 2025
@darthsiroftardis
Merge pull request #21 from zajko/update_wasm_lane_sorting
9425c87 
changing ordering logic
zajko added a commit to zajko/casper-node that referenced this pull request Jun 16, 2025
@zajko
# This is a combination of 22 commits.
17af5bc 
# This is the 1st commit message:

Initial wip

# The commit message #2 will be skipped:

# Basic case working. Pre-fixture tests

# The commit message #3 will be skipped:

# Initial fixture tests passing

# The commit message #4 will be skipped:

# Test case passing for disabled versions

# The commit message #5 will be skipped:

# EE tests passing

# The commit message #6 will be skipped:

# Additional fixture tests and PR prep

# The commit message #7 will be skipped:

# Run make format

# The commit message #8 will be skipped:

# X

# The commit message #9 will be skipped:

# X

# The commit message casper-network#10 will be skipped:

# X

# The commit message casper-network#11 will be skipped:

# X

# The commit message casper-network#12 will be skipped:

# Add flag to engine config to return error

# The commit message casper-network#13 will be skipped:

# Add flag to chainspec.toml

# The commit message casper-network#14 will be skipped:

# Address CI issues

# The commit message casper-network#15 will be skipped:

# Add contract runtime tests

# The commit message casper-network#16 will be skipped:

# rebasing with darthsiroftardis/change-package-version-calls

# The commit message casper-network#17 will be skipped:

# fmt fix

# The commit message casper-network#18 will be skipped:

# fmt fix

# The commit message casper-network#19 will be skipped:

# fmt fix

# The commit message casper-network#20 will be skipped:

# refactoring put_key contract

# The commit message casper-network#21 will be skipped:

# refactoring put_key contract

# The commit message casper-network#22 will be skipped:

# X
zajko added a commit to zajko/casper-node that referenced this pull request Oct 13, 2025
@zajko
# This is a combination of 23 commits.
c145bb3 
# This is the 1st commit message:

rebasing

# The commit message #2 will be skipped:

# WIP

# The commit message #3 will be skipped:

# WIP

# The commit message #4 will be skipped:

# wip

# The commit message #5 will be skipped:

# wip

# The commit message #6 will be skipped:

# wip

# The commit message #7 will be skipped:

# wip

# The commit message #8 will be skipped:

# wip

# The commit message #9 will be skipped:

# wip

# The commit message casper-network#10 will be skipped:

# wip

# The commit message casper-network#11 will be skipped:

# wip

# The commit message casper-network#12 will be skipped:

# wip

# The commit message casper-network#13 will be skipped:

# rebasing

# The commit message casper-network#14 will be skipped:

# wip

# The commit message casper-network#15 will be skipped:

# wip

# The commit message casper-network#16 will be skipped:

# wip

# The commit message casper-network#17 will be skipped:

# wip

# The commit message casper-network#18 will be skipped:

# wip

# The commit message casper-network#19 will be skipped:

# wip

# The commit message casper-network#20 will be skipped:

# rebasing

# The commit message casper-network#21 will be skipped:

# wip

# The commit message casper-network#22 will be skipped:

# wip

# The commit message casper-network#23 will be skipped:

# wip
zajko added a commit to zajko/casper-node that referenced this pull request Oct 13, 2025
@zajko
# This is a combination of 29 commits.
d0da6e3 
# This is the 1st commit message:

rebasing

# The commit message #2 will be skipped:

# WIP

# The commit message #3 will be skipped:

# WIP

# The commit message #4 will be skipped:

# wip

# The commit message #5 will be skipped:

# wip

# The commit message #6 will be skipped:

# wip

# The commit message #7 will be skipped:

# wip

# The commit message #8 will be skipped:

# wip

# The commit message #9 will be skipped:

# wip

# The commit message casper-network#10 will be skipped:

# wip

# The commit message casper-network#11 will be skipped:

# wip

# The commit message casper-network#12 will be skipped:

# wip

# The commit message casper-network#13 will be skipped:

# rebasing

# The commit message casper-network#14 will be skipped:

# wip

# The commit message casper-network#15 will be skipped:

# wip

# The commit message casper-network#16 will be skipped:

# wip

# The commit message casper-network#17 will be skipped:

# wip

# The commit message casper-network#18 will be skipped:

# wip

# The commit message casper-network#19 will be skipped:

# wip

# The commit message casper-network#20 will be skipped:

# rebasing

# The commit message casper-network#21 will be skipped:

# wip

# The commit message casper-network#22 will be skipped:

# wip

# The commit message casper-network#23 will be skipped:

# wip

# The commit message casper-network#24 will be skipped:

# wip

# The commit message casper-network#25 will be skipped:

# wip

# The commit message casper-network#26 will be skipped:

# wip

# The commit message casper-network#27 will be skipped:

# wip

# The commit message casper-network#28 will be skipped:

# wip

# The commit message casper-network#29 will be skipped:

# wip
zajko added a commit to zajko/casper-node that referenced this pull request Oct 13, 2025
@zajko
# This is a combination of 32 commits.
599c5c7 
# This is the 1st commit message:

rebasing

# The commit message #2 will be skipped:

# WIP

# The commit message #3 will be skipped:

# WIP

# The commit message #4 will be skipped:

# wip

# The commit message #5 will be skipped:

# wip

# The commit message #6 will be skipped:

# wip

# The commit message #7 will be skipped:

# wip

# The commit message #8 will be skipped:

# wip

# The commit message #9 will be skipped:

# wip

# The commit message casper-network#10 will be skipped:

# wip

# The commit message casper-network#11 will be skipped:

# wip

# The commit message casper-network#12 will be skipped:

# wip

# The commit message casper-network#13 will be skipped:

# rebasing

# The commit message casper-network#14 will be skipped:

# wip

# The commit message casper-network#15 will be skipped:

# wip

# The commit message casper-network#16 will be skipped:

# wip

# The commit message casper-network#17 will be skipped:

# wip

# The commit message casper-network#18 will be skipped:

# wip

# The commit message casper-network#19 will be skipped:

# wip

# The commit message casper-network#20 will be skipped:

# rebasing

# The commit message casper-network#21 will be skipped:

# wip

# The commit message casper-network#22 will be skipped:

# wip

# The commit message casper-network#23 will be skipped:

# wip

# The commit message casper-network#24 will be skipped:

# wip

# The commit message casper-network#25 will be skipped:

# wip

# The commit message casper-network#26 will be skipped:

# wip

# The commit message casper-network#27 will be skipped:

# wip

# The commit message casper-network#28 will be skipped:

# wip

# The commit message casper-network#29 will be skipped:

# wip

# The commit message casper-network#30 will be skipped:

# wip

# The commit message casper-network#31 will be skipped:

# wip

# The commit message casper-network#32 will be skipped:

# wip
zajko added a commit to zajko/casper-node that referenced this pull request Oct 13, 2025
@zajko
# This is a combination of 40 commits.
1759a90 
# This is the 1st commit message:

rebasing

# The commit message #2 will be skipped:

# WIP

# The commit message #3 will be skipped:

# WIP

# The commit message #4 will be skipped:

# wip

# The commit message #5 will be skipped:

# wip

# The commit message #6 will be skipped:

# wip

# The commit message #7 will be skipped:

# wip

# The commit message #8 will be skipped:

# wip

# The commit message #9 will be skipped:

# wip

# The commit message casper-network#10 will be skipped:

# wip

# The commit message casper-network#11 will be skipped:

# wip

# The commit message casper-network#12 will be skipped:

# wip

# The commit message casper-network#13 will be skipped:

# rebasing

# The commit message casper-network#14 will be skipped:

# wip

# The commit message casper-network#15 will be skipped:

# wip

# The commit message casper-network#16 will be skipped:

# wip

# The commit message casper-network#17 will be skipped:

# wip

# The commit message casper-network#18 will be skipped:

# wip

# The commit message casper-network#19 will be skipped:

# wip

# The commit message casper-network#20 will be skipped:

# rebasing

# The commit message casper-network#21 will be skipped:

# wip

# The commit message casper-network#22 will be skipped:

# wip

# The commit message casper-network#23 will be skipped:

# wip

# The commit message casper-network#24 will be skipped:

# wip

# The commit message casper-network#25 will be skipped:

# wip

# The commit message casper-network#26 will be skipped:

# wip

# The commit message casper-network#27 will be skipped:

# wip

# The commit message casper-network#28 will be skipped:

# wip

# The commit message casper-network#29 will be skipped:

# wip

# The commit message casper-network#30 will be skipped:

# wip

# The commit message casper-network#31 will be skipped:

# wip

# The commit message casper-network#32 will be skipped:

# wip

# The commit message casper-network#33 will be skipped:

# wip

# The commit message casper-network#34 will be skipped:

# wip

# The commit message casper-network#35 will be skipped:

# wip

# The commit message casper-network#36 will be skipped:

# wip

# The commit message casper-network#37 will be skipped:

# wip

# The commit message casper-network#38 will be skipped:

# wip

# The commit message casper-network#39 will be skipped:

# wip

# The commit message casper-network#40 will be skipped:

# rebasing
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@EdHastingsCasperAssociation EdHastingsCasperAssociation EdHastingsCasperAssociation approved these changes

@marc-casperlabs marc-casperlabs Awaiting requested review from marc-casperlabs

@fizyk20 fizyk20 Awaiting requested review from fizyk20

+1 more reviewer

@goral09 goral09 goral09 left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Fraser999 @goral09 @EdHastingsCasperAssociation