Skip to content

storage: enable WAL, schema versioning, and batched transactions#140

Merged
intendednull merged 1 commit into
mainfrom
claude/issue-116-storage-durability
Apr 11, 2026
Merged

storage: enable WAL, schema versioning, and batched transactions#140
intendednull merged 1 commit into
mainfrom
claude/issue-116-storage-durability

Conversation

@intendednull
Copy link
Copy Markdown
Owner

Closes #116
Progresses #108

Summary

  • Turns on SQLite durability pragmas on every StorageEventStore::open: journal_mode = WAL (skipped for :memory: databases, where it is a no-op), synchronous = FULL, and foreign_keys = ON.
  • Introduces a MIGRATIONS slice and a schema_version table so the existing CREATE TABLE / CREATE INDEX block becomes migration 1, and any future schema changes can be appended without hand-written ALTERs. Each pending migration is applied inside its own transaction together with its version-row insert so crashes mid-migration leave the database either fully migrated or untouched.
  • Adds store_events(&[(String, Event)]), which wraps the whole batch in a single transaction so one fsync flushes every event; store_event stays backwards-compatible and now routes through a shared INSERT OR IGNORE helper. Intra-batch duplicate primary keys raise a UNIQUE constraint error and roll the batch back, while cross-call duplicates keep their old silent-dedup semantics.

Test Plan

Added tests (crates/storage/src/store.rs test module, file-backed via tempfile::tempdir()):

  • wal_mode_is_enabled asserts PRAGMA journal_mode returns wal.
  • synchronous_full_is_enabled asserts PRAGMA synchronous returns 2 (FULL).
  • foreign_keys_enabled asserts PRAGMA foreign_keys returns 1.
  • schema_version_table_exists_after_open asserts version 1 is recorded.
  • migrations_are_idempotent opens the same database twice and asserts the schema_version row count matches MIGRATIONS.len().
  • batched_store_inserts_all_rows is the happy-path batch case.
  • batched_store_is_atomic passes a batch containing a duplicate primary key, asserts the call errors, and asserts that count() == 0 — the whole batch is rolled back.
  • store_event_after_failed_batch_still_works covers connection recovery after a rolled-back batch.

Verification run on this branch:

  • cargo fmt --check
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo test --workspace (all 27 willow-storage tests pass, including the 8 new ones; the full workspace suite is green)
  • cargo check --target wasm32-unknown-unknown -p willow-identity -p willow-state -p willow-channel -p willow-messaging -p willow-crypto -p willow-transport -p willow-common -p willow-network -p willow-client -p willow-web (storage is native-only and not in the WASM target list, so unchanged there)

Durability pragmas, tracked schema migrations, and an atomic
store_events batch API replace the old open path that ran every
insert in its own implicit transaction with no recovery settings.

- Set journal_mode=WAL (skipped for :memory:), synchronous=FULL,
  and foreign_keys=ON on every open.
- Introduce a MIGRATIONS slice and a schema_version table so
  existing databases can evolve without hand-written ALTERs.
- Add store_events for batched inserts inside a single
  transaction; store_event stays backwards-compatible via a shared
  INSERT OR IGNORE helper.
- Cover the new behavior with file-backed tempdir tests for WAL /
  synchronous / foreign_keys pragmas, migration idempotency, batch
  insert success, batch atomicity on duplicate primary keys, and
  post-failure connection recovery.

Closes #116
Progresses #108
@intendednull intendednull merged commit 8e9cb3b into main Apr 11, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[storage] Enable WAL, synchronous=FULL, transactions, and schema versioning

2 participants