From e9057bf1f613b8fa8562072c69c28a9cf08c2f84 Mon Sep 17 00:00:00 2001 From: Islam El-Ashi Date: Mon, 28 Apr 2025 16:27:20 +0300 Subject: [PATCH 1/4] docs: add intro documentation to be used in mdbook. There are a number of concepts in stable structures that need more thorough documentation, and I think it warrants some kind of "book" or reference guide that can be used by developers. This commit adds the skeleton of what would become this book, and in the future we'll use mdbook to serve this documentation. An example of what this book will look like: https://proptest-rs.github.io/proptest/ --- docs/src/SUMMARY.md | 7 +++ docs/src/available-data-structures.md | 9 ++++ docs/src/design-principles.md | 21 +++++++++ docs/src/introduction.md | 34 ++++++++++++++ docs/src/schema-upgrades.md | 66 +++++++++++++++++++++++++++ 5 files changed, 137 insertions(+) create mode 100644 docs/src/SUMMARY.md create mode 100644 docs/src/available-data-structures.md create mode 100644 docs/src/design-principles.md create mode 100644 docs/src/introduction.md create mode 100644 docs/src/schema-upgrades.md diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md new file mode 100644 index 00000000..b5e0bd95 --- /dev/null +++ b/docs/src/SUMMARY.md @@ -0,0 +1,7 @@ +# Summary + +- [Introduction](./introduction.md) + - [Design Principles](./design-principles.md) + - [Available Data Structures](./available-data-structures.md) + +- [Schema Upgrades](./schema-upgrades.md) diff --git a/docs/src/available-data-structures.md b/docs/src/available-data-structures.md new file mode 100644 index 00000000..64d95b58 --- /dev/null +++ b/docs/src/available-data-structures.md @@ -0,0 +1,9 @@ +# Available Data Structures + +The library provides several stable data structures: + +* **BTreeMap**: A key-value store that maintains keys in sorted order +* **Vec**: A growable array +* **Log**: An append-only list of variable-size entries +* **Cell**: A serializable value +* **MinHeap**: A priority queue diff --git a/docs/src/design-principles.md b/docs/src/design-principles.md new file mode 100644 index 00000000..6d2e7fdf --- /dev/null +++ b/docs/src/design-principles.md @@ -0,0 +1,21 @@ +# Design Principles + +The library is built on several key principles: + +* **Radical simplicity**: Each data structure follows the most straightforward design that solves the problem at hand. +This makes the code easier to understand, debug, and maintain. + +* **Backward compatibility**: Upgrading the library version must preserve the data. +All data structures have a metadata section with the layout version, ensuring that new versions can read data written by old versions. + +* **No `pre_upgrade` hooks**: A bug in the `pre_upgrade` hook can make your canister non-upgradable. +The best way to avoid this issue is not to have a `pre_upgrade` hook at all. + +* **Limited blast radius**: If a single data structure has a bug, it should not corrupt the contents of other data structures. +This isolation helps prevent cascading failures. + +* **No reallocation**: Moving large amounts of data is expensive and can lead to prohibitively high cycle consumption. +All data structures must manage their memory without costly moves. + +* **Compatibility with multi-memory WebAssembly**: The design should work when canisters have multiple stable memories. +This ensures future compatibility with upcoming IC features. diff --git a/docs/src/introduction.md b/docs/src/introduction.md new file mode 100644 index 00000000..3b3a2c49 --- /dev/null +++ b/docs/src/introduction.md @@ -0,0 +1,34 @@ +# Introduction + +## Background + +Smart contracts on the [Internet Computer](https://internetcomputer.org) are referred to as [canisters](https://learn.internetcomputer.org/hc/en-us/articles/34210839162004-Canister-Smart-Contracts). + +Canisters, compared to traditional smart contracts, have some unique properties including: + +**Mutability**: A canister can have a set of controllers, and controllers are able to upgrade the code of the canister (e.g., to add new features, fix bugs, etc.) + +**Scale**: Canisters have access to hundreds of gigabytes of memory and ample amounts of compute, allowing developers to build fully functioning dapps without relying on external cloud providers. + +### The Challenge of Upgrades + +When upgrading a canister, the canister's code is replaced with the new code. +In Rust, the new version of the code is not guaranteed to understand the memory layout established by the previous version. +This is because Rust's memory layout can change between different versions of the code, making it unsafe to directly access the old memory layout. +Therefore, by default, when a canister is upgraded and a new module is installed, the canister's main memory is wiped. + +To persist state, the Internet Computer provides canisters with an additional memory called _stable memory_. +The conventional approach to canister state persistence follows these steps: + +1. Serialize and store the state of the canister just before the upgrade using the `pre_upgrade` hook. +2. Install the new Wasm module of the canister (and wipe out the canister's main memory). +3. Deserialize the data that was stored in stable memory in step 1 using the `post_upgrade` hook. + +This approach is easy to implement and works well for relatively small datasets. +Unfortunately, it does not scale well and can render a canister non-upgradable. + +### The Solution: Stable Structures + +Rather than using standard Rust data structures, which store their data in the canister's main memory, you can use stable structures. +Stable structures are designed to use stable memory as the primary storage, allowing them to grow to gigabytes in size without the need for `pre_upgrade`/`post_upgrade` hooks. +This is the key characteristic that distinguishes stable structures from Rust's standard data structures. diff --git a/docs/src/schema-upgrades.md b/docs/src/schema-upgrades.md new file mode 100644 index 00000000..12758e50 --- /dev/null +++ b/docs/src/schema-upgrades.md @@ -0,0 +1,66 @@ +# Schema Upgrades + +Stable structures store data directly in stable memory and do not require upgrade hooks. +Since these structures are designed to persist throughout the lifetime of the canister, it's nearly inevitable that developers would want to make modifications to the data's schema as the canister evolves. + +Let's say you are storing assets in your canister. The declaration of it can look something like this: + +```rust +#[derive(Serialize, Deserialize, CandidType)] +struct Asset { + // The contents of the asset. + contents: Vec, +} + +impl Storable for Asset { + fn to_bytes(&self) -> std::borrow::Cow<[u8]> { + let mut bytes = vec![]; + ciborium::ser::into_writer(&self, &mut bytes).unwrap(); + Cow::Owned(bytes) + } + + fn from_bytes(bytes: std::borrow::Cow<[u8]>) -> Self { + ciborium::de::from_reader(&*bytes).expect("deserialization must succeed.") + } + + const BOUND: Bound = Bound::Unbounded; +} +``` + +> **Note:** Stables structures do not enforce a specific data format. +It's up to the developer to use the data format that fits their use-case. +In this example, CBOR is used for encoding `Asset`. + + +## Adding an attribute + +Adding a new field can be as simple as adding the field, like this: + +```rust +#[derive(Serialize, Deserialize)] +struct Asset { + // The contents of the asset. + contents: Vec, + + // The timestamp the asset was created at. + #[serde(default)] + created_at: u64, +} +``` + +If the new attribute being added doesn't have a sensible default value, consider wrapping it in an `Option`: + +```rust +#[derive(Serialize, Deserialize, CandidType)] +struct Asset { + // The contents of the asset. + contents: Vec, + + // The timestamp the asset was created at. + #[serde(default)] + created_at: u64, + + // The username of the uploader. + uploaded_by: Option, +} +``` From 00ec807a610a2777bc92482dd56f737143b64127 Mon Sep 17 00:00:00 2001 From: Islam El-Ashi Date: Mon, 28 Apr 2025 16:28:29 +0300 Subject: [PATCH 2/4] . --- docs/schema-upgrades.md | 66 ----------------------------------------- 1 file changed, 66 deletions(-) delete mode 100644 docs/schema-upgrades.md diff --git a/docs/schema-upgrades.md b/docs/schema-upgrades.md deleted file mode 100644 index 12758e50..00000000 --- a/docs/schema-upgrades.md +++ /dev/null @@ -1,66 +0,0 @@ -# Schema Upgrades - -Stable structures store data directly in stable memory and do not require upgrade hooks. -Since these structures are designed to persist throughout the lifetime of the canister, it's nearly inevitable that developers would want to make modifications to the data's schema as the canister evolves. - -Let's say you are storing assets in your canister. The declaration of it can look something like this: - -```rust -#[derive(Serialize, Deserialize, CandidType)] -struct Asset { - // The contents of the asset. - contents: Vec, -} - -impl Storable for Asset { - fn to_bytes(&self) -> std::borrow::Cow<[u8]> { - let mut bytes = vec![]; - ciborium::ser::into_writer(&self, &mut bytes).unwrap(); - Cow::Owned(bytes) - } - - fn from_bytes(bytes: std::borrow::Cow<[u8]>) -> Self { - ciborium::de::from_reader(&*bytes).expect("deserialization must succeed.") - } - - const BOUND: Bound = Bound::Unbounded; -} -``` - -> **Note:** Stables structures do not enforce a specific data format. -It's up to the developer to use the data format that fits their use-case. -In this example, CBOR is used for encoding `Asset`. - - -## Adding an attribute - -Adding a new field can be as simple as adding the field, like this: - -```rust -#[derive(Serialize, Deserialize)] -struct Asset { - // The contents of the asset. - contents: Vec, - - // The timestamp the asset was created at. - #[serde(default)] - created_at: u64, -} -``` - -If the new attribute being added doesn't have a sensible default value, consider wrapping it in an `Option`: - -```rust -#[derive(Serialize, Deserialize, CandidType)] -struct Asset { - // The contents of the asset. - contents: Vec, - - // The timestamp the asset was created at. - #[serde(default)] - created_at: u64, - - // The username of the uploader. - uploaded_by: Option, -} -``` From f7e26e2a2e093afe23e1ee86868aa26f2dd51ce8 Mon Sep 17 00:00:00 2001 From: Islam El-Ashi Date: Mon, 28 Apr 2025 16:28:58 +0300 Subject: [PATCH 3/4] . --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index db1bbba3..499c9729 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,7 @@ For more information about the philosophy behind the library, see [Roman's tutor ## Tutorials -[Schema Upgrades](./docs/schema-upgrades.md) +[Schema Upgrades](./docs/src/schema-upgrades.md) ## How it Works @@ -164,4 +164,4 @@ cargo +nightly fuzz list # To run a target cargo +nightly fuzz run -``` \ No newline at end of file +``` From 3db3a0010032405b46b3134bd4dd6a0252c9a857 Mon Sep 17 00:00:00 2001 From: Islam El-Ashi Date: Mon, 28 Apr 2025 17:51:47 +0300 Subject: [PATCH 4/4] . --- docs/src/design-principles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/design-principles.md b/docs/src/design-principles.md index 6d2e7fdf..2a1f29b6 100644 --- a/docs/src/design-principles.md +++ b/docs/src/design-principles.md @@ -8,7 +8,7 @@ This makes the code easier to understand, debug, and maintain. * **Backward compatibility**: Upgrading the library version must preserve the data. All data structures have a metadata section with the layout version, ensuring that new versions can read data written by old versions. -* **No `pre_upgrade` hooks**: A bug in the `pre_upgrade` hook can make your canister non-upgradable. +* **No [`pre_upgrade` hooks](https://internetcomputer.org/docs/references/ic-interface-spec#system-api-upgrades)**: A bug in the `pre_upgrade` hook can make your canister non-upgradable. The best way to avoid this issue is not to have a `pre_upgrade` hook at all. * **Limited blast radius**: If a single data structure has a bug, it should not corrupt the contents of other data structures.