storage: enable WAL, schema versioning, and batched transactions#140
Merged
Conversation
Durability pragmas, tracked schema migrations, and an atomic store_events batch API replace the old open path that ran every insert in its own implicit transaction with no recovery settings. - Set journal_mode=WAL (skipped for :memory:), synchronous=FULL, and foreign_keys=ON on every open. - Introduce a MIGRATIONS slice and a schema_version table so existing databases can evolve without hand-written ALTERs. - Add store_events for batched inserts inside a single transaction; store_event stays backwards-compatible via a shared INSERT OR IGNORE helper. - Cover the new behavior with file-backed tempdir tests for WAL / synchronous / foreign_keys pragmas, migration idempotency, batch insert success, batch atomicity on duplicate primary keys, and post-failure connection recovery. Closes #116 Progresses #108
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #116
Progresses #108
Summary
StorageEventStore::open:journal_mode = WAL(skipped for:memory:databases, where it is a no-op),synchronous = FULL, andforeign_keys = ON.MIGRATIONSslice and aschema_versiontable so the existingCREATE TABLE/CREATE INDEXblock becomes migration 1, and any future schema changes can be appended without hand-writtenALTERs. Each pending migration is applied inside its own transaction together with its version-row insert so crashes mid-migration leave the database either fully migrated or untouched.store_events(&[(String, Event)]), which wraps the whole batch in a single transaction so onefsyncflushes every event;store_eventstays backwards-compatible and now routes through a sharedINSERT OR IGNOREhelper. Intra-batch duplicate primary keys raise a UNIQUE constraint error and roll the batch back, while cross-call duplicates keep their old silent-dedup semantics.Test Plan
Added tests (
crates/storage/src/store.rstest module, file-backed viatempfile::tempdir()):wal_mode_is_enabledassertsPRAGMA journal_modereturnswal.synchronous_full_is_enabledassertsPRAGMA synchronousreturns2(FULL).foreign_keys_enabledassertsPRAGMA foreign_keysreturns1.schema_version_table_exists_after_openasserts version1is recorded.migrations_are_idempotentopens the same database twice and asserts theschema_versionrow count matchesMIGRATIONS.len().batched_store_inserts_all_rowsis the happy-path batch case.batched_store_is_atomicpasses a batch containing a duplicate primary key, asserts the call errors, and asserts thatcount() == 0— the whole batch is rolled back.store_event_after_failed_batch_still_workscovers connection recovery after a rolled-back batch.Verification run on this branch:
cargo fmt --checkcargo clippy --workspace --all-targets -- -D warningscargo test --workspace(all 27willow-storagetests pass, including the 8 new ones; the full workspace suite is green)cargo check --target wasm32-unknown-unknown -p willow-identity -p willow-state -p willow-channel -p willow-messaging -p willow-crypto -p willow-transport -p willow-common -p willow-network -p willow-client -p willow-web(storage is native-only and not in the WASM target list, so unchanged there)