bug: recovery from ic_stable_structures::Vec may fail for complex data structures

[linnefromice]

• Different data contents, some data recoverable and some unrecoverable
  • Data structure is the same
• Data store where the event occurs
  • Only for Vec of stable memory (Perhaps even Map)
  • No problem in following cases
    • heap, Cell of stable memory
• Guess
  • Data is not being acquired correctly when retrieving from Vec.
    • The bytes of input and the contents are different when encode.
  • There is no problem if you do a command encode -> decode -> encode for struce.
    • So I guess this is the part that handles stable memory.

All checked codes can be found below
https://github.com/linnefromice/stable_memory_complex_struct
The README contains all the instructions.

The reference log output during encode and decode before saving/retrieving is also attached.
by linnefromice/stable_memory_complex_struct#1

% dfx canister call backend add_from_default
2024-01-10 12:03:40.792010 UTC: [Canister bkyz2-fmaaa-aaaaa-qaaaq-cai] to_bytes
2024-01-10 12:03:40.792010 UTC: [Canister bkyz2-fmaaa-aaaaa-qaaaq-cai] [44, 49, 44, 4c, 6, 6c, 2, f1, fe, e1, 8d, 3, 1, d6, a9, bb, ae, a, 78, 6c, 3, db, b7, 1, 71, dd, d1, 91, 44, 2, bd, d3, d1, ab, 6, 71, 6c, 7, f6, dc, b8, 81, 1, 3, 97, bc, 9b, 9a, 3, 75, f2, 9d, 80, ee, 4, 71, 97, ae, 9c, 8f, 9, 71, b4, e3, ad, e8, 9, 71, 82, b7, bd, e7, d, 71, e1, ea, f9, c8, f, 75, 6d, 4, 6c, 2, 0, 71, 1, 5, 6c, 5, b6, af, f5, dc, 2, 71, b7, 92, 93, fb, 2, 71, 94, ea, bf, b8, 3, 7e, d0, 8a, da, ff, 5, 71, b7, d8, 8f, f8, b, 71, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
()
% dfx canister call backend get_last_data --query
2024-01-10 12:03:43.985242 UTC: [Canister bkyz2-fmaaa-aaaaa-qaaaq-cai] from_bytes
2024-01-10 12:03:43.985242 UTC: [Canister bkyz2-fmaaa-aaaaa-qaaaq-cai] [44, 49, 44, 4c, 6, 6c, 2, f1, fe, e1, 8d, 3, 1, d6, a9, bb, ae, a, 78, 6c, 3, db, b7, 1, 71, dd, d1, 91, 44, 2, bd, d3, d1, ab, 6, 71, 6c, 7, f6, dc, b8, 81, 1, 3, 97, bc, 9b, 9a, 3, 75, f2, 9d, 80, ee, 4, 71, 97, ae, 9c, 8f, 9, 71, b4, e3, ad, e8, 9, 71, 82, b7, bd, e7, d, 71, e1, ea, f9, c8, f, 75, 6d, 4, 6c, 2, 0, 71, 1, 5, 6c, 5, b6, af, f5, dc, 2, 71, b7, 92, 93, fb, 2, 71, 94, ea, bf, b8, 3, 7e, d0, 8a, da, ff, 5, 71, b7, d8, 8f, f8, b, 71, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
(
  record {
    value = record {
      id = "";
      result = record {
        ticks = vec {};
        tick_current = 0 : int32;
        liquidity = "";
        token0 = "";
        address = "";
        sqrt_ratio_x96 = "";
        tick_spacing = 0 : int32;
      };
      jsonrpc = "";
    };
    timestamp = 0 : nat64;
  },
)
% dfx canister call backend add_from_dummy
2024-01-10 12:03:49.675502 UTC: [Canister bkyz2-fmaaa-aaaaa-qaaaq-cai] to_bytes
2024-01-10 12:03:49.675502 UTC: [Canister bkyz2-fmaaa-aaaaa-qaaaq-cai] [44, 49, 44, 4c, 6, 6c, 2, f1, fe, e1, 8d, 3, 1, d6, a9, bb, ae, a, 78, 6c, 3, db, b7, 1, 71, dd, d1, 91, 44, 2, bd, d3, d1, ab, 6, 71, 6c, 7, f6, dc, b8, 81, 1, 3, 97, bc, 9b, 9a, 3, 75, f2, 9d, 80, ee, 4, 71, 97, ae, 9c, 8f, 9, 71, b4, e3, ad, e8, 9, 71, 82, b7, bd, e7, d, 71, e1, ea, f9, c8, f, 75, 6d, 4, 6c, 2, 0, 71, 1, 5, 6c, 5, b6, af, f5, dc, 2, 71, b7, 92, 93, fb, 2, 71, 94, ea, bf, b8, 3, 7e, d0, 8a, da, ff, 5, 71, b7, d8, 8f, f8, b, 71, 1, 0, 1, 31, 0, 70, d1, fe, ff, 17, 30, 78, 31, 64, 32, 66, 30, 39, 31, 66, 66, 30, 39, 66, 62, 36, 37, 31, 37, 34, 37, 33, 38, 2a, 30, 78, 36, 62, 31, 37, 35, 34, 37, 34, 65, 38, 39, 30, 39, 34, 63, 34, 34, 64, 61, 39, 38, 62, 39, 35, 34, 65, 65, 64, 65, 61, 63, 34, 39, 35, 32, 37, 31, 64, 30, 66, 2a, 30, 78, 63, 32, 65, 39, 66, 32, 35, 62, 65, 36, 32, 35, 37, 63, 32, 31, 30, 64, 37, 61, 64, 66, 30, 64, 34, 63, 64, 36, 65, 33, 65, 38, 38, 31, 62, 61, 32, 35, 66, 38, 19, 30, 78, 35, 35, 33, 37, 36, 63, 64, 32, 61, 64, 30, 35, 62, 38, 31, 35, 37, 38, 30, 65, 63, 66, 62, 3c, 0, 0, 0, 3, 32, 2e, 30, a5, 87, 9e, 65, 0, 0, 0, 0]
()
% dfx canister call backend get_last_data --query
2024-01-10 12:03:51.564396 UTC: [Canister bkyz2-fmaaa-aaaaa-qaaaq-cai] from_bytes
2024-01-10 12:03:51.564396 UTC: [Canister bkyz2-fmaaa-aaaaa-qaaaq-cai] [44, 49, 44, 4c, 6, 6c, 2, f1, fe, e1, 8d, 3, 1, d6, a9, bb, ae, a, 78, 6c, 3, db, b7, 1, 71]
2024-01-10 12:03:51.564396 UTC: [Canister bkyz2-fmaaa-aaaaa-qaaaq-cai] Panicked at 'called `Result::unwrap()` on an `Err` value: Custom(Cannot parse header 4449444c066c02f1fee18d0301d6a9bbae0a786c03dbb70171

Caused by:
    binary parser error: id at byte offset 25)', src/backend/src/types.rs:14:39
Error: Failed query call.
Caused by: Failed query call.
  The replica returned a replica error: reject code CanisterError, reject message IC0503: Canister bkyz2-fmaaa-aaaaa-qaaaq-cai trapped explicitly: Panicked at 'called `Result::unwrap()` on an `Err` value: Custom(Cannot parse header 4449444c066c02f1fee18d0301d6a9bbae0a786c03dbb70171

Caused by:
    binary parser error: id at byte offset 25)', src/backend/src/types.rs:14:39, error code Some("IC0503")

[0xhawk]

Hi @ielashi, could you please check this?

[ielashi]

Sorry folks, I didn't notice this issue until today. I took a look, and the reason you're seeing this failure is because your dummy() snapshot, when serialized, is 281 bytes in length. However, the MAX_SIZE you specified for Snapshot is 100. The serialized size of an element must always be <= MAX_SIZE. Setting a larger MAX_SIZE will fix the problem.

This should've been handled better by this library, though. Specifically, Vec should've panicked telling you that you were trying to store an element that's too big. I just opened PR #180 that addresses this.

[ielashi]

Version 0.6.2 of stable structures has been released. I updated your example with the new version and the problem immediately becomes apparent:

$ dfx canister call backend add_from_dummy
2024-01-23 10:59:54.322902 UTC: [Canister bkyz2-fmaaa-aaaaa-qaaaq-cai] Panicked at 'expected an element with length <= 100 bytes, but found 281 bytes', /Users/ielashi/.cargo/registry/src/index.crates.io-6f17d22bba15001f/ic-stable-structures-0.6.2/src/storable.rs:41:17

I'll now close ticket. Feel free to let us know if you're still encountering issues.

And here's the diff if you'd like to try it out yourself:

diff --git a/Cargo.toml b/Cargo.toml
index 54b31e9..e15e770 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -8,4 +8,4 @@ candid = "0.9.6"
 ic-cdk = "0.11.3"
 ic-cdk-macros = "0.8.1"
 ic-cdk-timers = "0.5.0"
-ic-stable-structures = "0.5.5"
+ic-stable-structures = "0.6.2"
diff --git a/src/backend/src/types.rs b/src/backend/src/types.rs
index b414757..d52aade 100644
--- a/src/backend/src/types.rs
+++ b/src/backend/src/types.rs
@@ -1,5 +1,6 @@
 use std::{borrow::Cow, collections::BTreeMap};

+use ic_stable_structures::storable::Bound;
 use candid::{Decode, Encode};

 #[derive(Debug, Clone, Default, candid::CandidType, candid::Deserialize, serde::Serialize)]
@@ -14,10 +15,11 @@ impl ic_stable_structures::Storable for Snapshot {
     fn to_bytes(&self) -> std::borrow::Cow<[u8]> {
         Cow::Owned(Encode!(self).unwrap())
     }
-}
-impl ic_stable_structures::BoundedStorable for Snapshot {
-    const MAX_SIZE: u32 = 100;
-    const IS_FIXED_SIZE: bool = false;
+
+    const BOUND: Bound = Bound::Bounded {
+        max_size: 100,
+        is_fixed_size: false,
+    };
 }

 #[derive(Debug, Clone, Default, candid::CandidType, candid::Deserialize, serde::Serialize)]

[0xhawk]

@ielashi Thank you so much for your updates!