feat(platform)!: verifiable, bounded count queries on a unified endpoint#3623
Conversation
Collapse the two count gRPC endpoints (`getDocumentsCount` and
`getDocumentsSplitCount`) into a single unified `getDocumentsCount`
that handles both modes. Wire format and consumer cleanup; logic
parity for total + In-split modes; new fields stubbed for follow-up.
**Wire format** (`packages/dapi-grpc/protos/platform/v0/platform.proto`):
- `GetDocumentsCountRequestV0` gains `return_distinct_counts_in_range`,
`order_by_ascending`, `limit`, `start_after_split_key` fields.
- `GetDocumentsCountResponseV0.result` becomes `oneof { CountResults
counts; Proof proof; }`. `CountResults` carries `repeated CountEntry
{ bytes key; uint64 count; }`. Total count is one entry with empty
key; per-In-value counts are one entry per In value.
- `GetDocumentsSplitCount{Request,Response}` deleted.
**Mode dispatch from where clauses**:
- No `In` clause → total count, single CountEntry with empty key.
- Exactly one `In` clause → per-In-value entries. The In's field is
the split property; the In's array determines which values appear.
- Multiple `In` clauses → InvalidArgument (only one split dimension
per request).
- `return_distinct_counts_in_range = true` → InvalidArgument for now;
this needs `range_countable` indexes (parallel rs-dpp work) and the
`NonCounted<*>` element variants from grovedb.
**Per-layer changes**:
- `dapi-grpc` build.rs: remove `GetDocumentsSplitCount{Request,Response}`
from the versioned-message arrays (counts go from 58/56 to 57/55).
- `rs-dapi-client` transport: remove `getDocumentsSplitCount` impl.
- `rs-dapi` server: remove `get_documents_split_count` drive_method
passthrough.
- `rs-drive-abci`: delete `query/document_split_count_query/` module
and the trait method on `PlatformService`. Rewrite
`query_documents_count_v0` to dispatch on In-presence and emit
`CountResults` instead of bare `count`. Per-In-value entries are
produced by replacing the In with an Equal on each value and
point-looking-up the count (each entry uses
`serialize_value_for_key` for its `key` so the bytes round-trip
consistently with the proof-path verifier's bucket keys).
- `rs-drive-proof-verifier`: `DocumentSplitCounts` now targets
`GetDocumentsCountResponse` (just a type-name change in the
`Response` associated type; the proof-aggregation logic is
unchanged).
- `rs-sdk`: delete `DocumentSplitCountQuery` type. `DocumentCount`
and `DocumentSplitCounts` both `impl Fetch with Request =
DocumentCountQuery`. New `FromProof<DocumentCountQuery> for
DocumentSplitCounts` derives the split property from the request's
In clause field name and routes through
`maybe_from_proof_with_split_property`. Mock-loader entries for
the deleted types removed.
- `wasm-sdk` / `rs-sdk-ffi`: `getDocumentsSplitCount` /
`dash_sdk_document_split_count` keep their names but drop the
`splitProperty` parameter — splitting is now signalled by including
an `in` where-clause.
**Tests**:
- All 14 rs-drive `drive_document_count_query` lib tests pass (no
changes — the rs-drive primitives are the same; the wire-level
unification happens in drive-abci).
- All 5 rs-drive-abci handler tests pass: total / empty / proof /
range-rejection / In. Existing assertions updated from `Result::
Count(count)` patterns to summing `CountResults.entries`.
- The existing `test_documents_split_count_*` handler tests are
removed alongside the deleted handler module.
**Not yet in this PR** (follow-ups):
- `limit` / `start_after_split_key` / `order_by_ascending` are
accepted in the request but currently unused by the handler; the
underlying `DriveDocumentCountQuery` doesn't yet plumb them through.
- `return_distinct_counts_in_range = true` and range operators on
the no-prove path remain rejected; both depend on the parallel
`range_countable` index property + grovedb `NonCounted<*>`
variants. Design is documented in `book/src/drive/indexes.md`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughReplaces GetDocumentsSplitCount with a unified GetDocumentsCount; adds ChangesUnified count API and range-countable indexing
Estimated code review effort🎯 5 (Critical) | ⏱️ ~120 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
|
…ange) Brings in dashpay/grovedb#654 (Element::NonCounted wrapper) and #656 (QueryItem::AggregateCountOnRange + Node::HashWithCount). Both are prerequisites for the `range_countable` index property that the parallel design work in `book/src/drive/indexes.md` depends on: - `Element::NonCounted(Box<Element>)` — wrapper variant whose count contributes 0 to a parent count tree's aggregate. Lets a count tree hold housekeeping rows / sibling sub-property continuations without polluting the count. Only insertable into count-bearing trees; nested wrappers rejected at construction / serialize / deserialize. - `QueryItem::AggregateCountOnRange(Box<QueryItem>)` — count-only proof shape returning `(CryptoHash, u64)` in O(log n) bytes. Backed by a new self-verifying `Node::HashWithCount(kv_hash, l, r, count)` proof node so the count is bound by the proof, not trusted on faith. Restricted to `ProvableCountTree` / `ProvableCountSumTree` (and their `NonCounted*` wrappers) at proof time. Verified via `GroveDb::verify_aggregate_count_query`. Together these unblock implementing `range_countable` indexes (per- node counts on the property-name tree, NonCounted wrappers for sibling continuations) and `return_distinct_counts_in_range` / range count queries on the no-prove and prove paths — both currently gated as "not yet supported" in the unified count handler. Workspace fixups required by the bump: - `wasm-drive-verify` JS shim: add a `QueryItem::AggregateCountOnRange` arm in `serialize_query_item` (descriptive type, no recursion into the inner range — the wasm verify path doesn't drive these queries today, but the variant must be matched for the workspace to compile). - `rs-sdk-ffi` path-elements display: add `Element::NonCounted(_)` arm reporting `"non_counted"` (placeholder display; we'll inflate it to describe the inner element when the wrapper is actually used in contracts). - `rs-drive-abci` shielded common: orchard's transitive bump made `Action::from_parts` return `Option<Action>`. Wrap with `.ok_or_else` surfacing `InvalidShieldedProofError("invalid action parts")` rather than panicking; otherwise behaviorally unchanged. Tests: 14 rs-drive count-query lib tests, 5 drive-abci handler tests, 3079 rs-drive lib tests, and 3435 dpp lib tests all still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ing) Per-index `rangeCountable: bool` flag, additive on top of `countable`. When true, the index is laid out so that range-count queries on the indexed property can be answered in O(log n): - Property-name level: `ProvableCountTree` (per-node counts let a range query walk just the boundary path). - Each value tree under it: `CountTree` (count-bearing so the property-name aggregate sums per-value counts cleanly). - Sibling continuations inside a value tree: wrapped with `Element::NonCounted` so their counts don't pollute the value tree's count. Depends on the grovedb features bumped in the previous commit (`Element::NonCounted` + `QueryItem::AggregateCountOnRange` from dashpay/grovedb#654 + #656). This commit lands the schema-level plumbing only: - `Index.range_countable: bool` field + serde derive. - Index parser reads `"rangeCountable"` (boolean only — no enum form needed). - Cross-field validation in `Index::try_from`: `range_countable: true` requires `countable.is_countable()`. Without that, it would change layout of a non-count-bearing tree, which is meaningless. - v1 meta-schema schema entry under each index in `documentSchemas`. - Protocol-version gate in `try_from_schema/v1`: `range_countable: true` on protocol_version < 12 raises `UnsupportedFeatureError`. Pre-v12 nodes therefore reject the contract at validation time, before any state mutation. Mirrors the existing v12 gate on countable indexes. - `IndexLevelTypeInfo.range_countable` populated from the source index so the insert/delete walkers can reach it (used in a follow-up). - `random_index` default + ~16 IndexLevel test-init sites updated. Storage layout change (the actual `NonCounted` wrapping + `ProvableCountTree`/`CountTree` selection in the insert / delete walkers) is **deferred to a follow-up commit**. Until that lands, `IndexLevelTypeInfo.range_countable` is read but not yet acted on — the on-disk layout is unchanged, so the schema gate is the only gate in effect right now. Combined with the v12 protocol gate, no v11 node ever sees a `range_countable` contract, and no v12 node yet emits NonCounted-wrapped writes. Tests: 79 dpp index tests, 14 rs-drive count-query lib tests, 5 drive-abci handler tests still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the foundational helper for the upcoming `range_countable` storage layout, plus a runtime guard that fails loudly if a v12+ contract with `range_countable: true` reaches the insert walker before the rest of the storage-layout work lands. - `LowLevelDriveOperation::for_known_path_key_empty_non_counted_normal_tree`: builds a `GroveOperation` that inserts `Element::NonCounted(empty_tree())` at the given path and key. The wrapper makes the inserted subtree contribute 0 to a parent count tree's aggregate (per dashpay/grovedb#654), which is what the index walker needs for sibling continuations under a `range_countable` value tree (e.g., the `'shape'` continuation under a `byColor` value tree, when `byColor` is range_countable but `byColorShape` shares its prefix). Construction is infallible by `new_non_counted`'s contract — the `expect` documents the invariant. - `add_indices_for_top_index_level_for_contract_operations_v0` and `add_indices_for_index_level_for_contract_operations_v0`: both now inspect `sub_level.has_index_with_type().range_countable` and return `DriveError::NotSupported` if true, with TODO comments pointing to the exact lines that need to switch tree types and the helper to use for NonCounted wrapping. Belt-and-suspenders alongside the rs-dpp v12 validation gate added in the previous commit — pre-v12 nodes already reject the contract; on v12+ the contract reaches here and we refuse rather than corrupt the count aggregation by writing a NormalTree where a CountTree / ProvableCountTree / NonCounted is required. Tests: 79 dpp + 14 rs-drive + 5 drive-abci tests still pass. Next chunks (still TODO on this PR — best as separate focused commits): - Insert walker: switch property-name tree to ProvableCountTree, value tree to CountTree, and wrap sibling continuations with NonCounted when `IndexLevelTypeInfo.range_countable` is true. Threads a `parent_value_tree_is_count_bearing` flag through recursion. - Same in cost-estimation paths (`EstimatedLayerInformation.tree_type`). - Mirror in delete (`remove_*_for_index_level_*`). - Count picker: accept `range_countable` indexes for range operators. - `DriveDocumentCountQuery::execute_no_proof` range mode via grovedb's `AggregateCountOnRange` query item. - Drive-abci handler: route `return_distinct_counts_in_range = true` to the new range-mode logic instead of erroring. - Drop the `u16::MAX` materialization cap on prove path for range counts via `verify_aggregate_count_query`. - Tests covering count-aggregation correctness with NonCounted siblings. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ists Adds the public Drive helper the index walker will call when inserting sibling continuations under a `range_countable` value tree (a `CountTree`). Without `NonCounted` wrapping, the empty `NormalTree` would contribute 1 to the parent's count via grovedb's default `count_value_or_default` (returns 1 for non-CountTree children); the wrapper makes it contribute 0 so the value tree's count cleanly reflects "docs at this value" rather than "docs + sibling-continuation-trees". Implementation: - Internal `batch_insert_empty_tree_if_not_exists_v0` now takes a `wrap_in_non_counted: bool` parameter. The body's existing per-PathKeyInfo-variant branches all funnel through a small `build_op` closure that picks between the regular `tree_type.empty_tree_operation_for_known_path_key` and the new `LowLevelDriveOperation::for_known_path_key_empty_non_counted_normal_tree` helper. Wrap is only valid with `TreeType::NormalTree` for now (the only shape the walker needs); other combinations return `DriveError::NotSupported` so callers don't accidentally request ill-defined wrapping. - Public `batch_insert_empty_tree_if_not_exists` wrapper passes `false` — behavior unchanged for existing callers. - New public `batch_insert_empty_non_counted_normal_tree_if_not_exists` passes `true` and fixes `tree_type` to `NormalTree`. Same not-exists-check / pending-batch-deduplication semantics as the regular helper. Test fixtures updated to thread the new parameter through direct `*_v0` calls (5 sites in this file's existing test module). Tests: full count-query test suite still passes (14 rs-drive lib + 5 drive-abci handler). Note: this is a foundational helper for the `range_countable` walker work that's still pending (see TODO markers in `add_indices_for_*_index_level_*_for_contract_operations_v0`). The walker's actual integration — switching property-name tree to `ProvableCountTree`, value tree to `CountTree`, wrapping siblings via this new helper — and the matching delete-path mirror are deferred to a follow-up commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Switches the index walker over from a runtime guard to the actual storage layout for `range_countable` indexes: - Top-level property-name tree (`[contract_doc, doctype, prop]`) is now a `ProvableCountTree` at contract setup when any range_countable index terminates at that property. - Value tree (`[..., prop, <value>]`) becomes a `CountTree` when the IndexLevel sub_level it lives under is a range_countable terminator. - Recursive walker emits `ProvableCountTree` / `CountTree` at deeper levels following the same rule, and threads a `parent_value_tree_is_range_countable` flag so sibling continuations inside a `CountTree` are wrapped with `Element::NonCounted` (so compound continuations contribute 0 to the parent count instead of polluting it via grovedb's `count_value_or_default`). Generalizes the NonCounted helpers (`for_known_path_key_empty_non_counted_tree`, `batch_insert_empty_non_counted_tree_if_not_exists`) to work for NormalTree / CountTree / ProvableCountTree, so nested-range_countable layouts (e.g. `[color]` and `[color, size]` both range_countable) wrap the inner ProvableCountTree continuation correctly. 10 existing countable_e2e_tests still pass; full drive::document::insert suite (23 tests) green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The delete walker only removes references (the count tree decrement is handled inside grovedb), so the substantive change here is propagating the same `parent_value_tree_is_range_countable` flag through the recursion so cost estimation reports the correct tree variant for each layer (CountTree at value-level, ProvableCountTree at property-name level under a range_countable terminator). Without this, storage-cost math for delete operations on range_countable contracts would diverge from the actual stored shape. All existing drive::document::delete tests (16) still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Five new tests in `range_countable_index_e2e_tests` exercise the index walker storage layout end-to-end against a real Drive (grovedb), using a v12 contract whose `widget` document type carries an actual `rangeCountable: true` index over the `color` property: 1. `property_name_tree_for_range_countable_index_is_provable_count_tree` — verifies contract setup creates `[contract_doc, doctype, "color"]` as a `ProvableCountTree`. 2. `value_tree_for_range_countable_index_is_count_tree_after_insert` — on document insert the value tree at `[..., "color", "red"]` is a `CountTree`, and the parent `ProvableCountTree`'s aggregate moves from 0 → 1. 3. `count_tree_value_count_excludes_compound_continuation_via_non_counted` — with a sibling `[color, size]` compound index, the `CountTree` count stays at 1 (not 2) and the continuation tree at `[..., "color", "red", "size"]` is `Element::NonCounted<Tree>`. This is the load-bearing correctness check for NonCounted-wrapping. 4. `aggregate_count_grows_across_distinct_values` — 6 documents at 3 distinct color values produce the right per-value `CountTree` counts AND the right aggregate at the property-name `ProvableCountTree`. 5. `delete_decrements_count_tree_and_provable_count_aggregate` — the delete walker correctly decrements both counts (CountTree and parent ProvableCountTree aggregate). These pin the observable storage shape so any regression in the walker's tree-type selection or NonCounted-wrapping would fail loudly rather than silently producing wrong counts at query time. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `DriveDocumentCountQuery::find_range_countable_index_for_where_clauses` and the supporting `is_range_operator` helper. The picker matches a range count query (e.g. `color > 'a'` or `brand = 'acme' AND color BETWEEN 'a' AND 'z'`) to a `range_countable` index whose: - Equal/In where-clause fields form a prefix of the index properties - Range operator targets the LAST property of the index (the IndexLevel terminator — where the walker emits the `ProvableCountTree`) - `range_countable: true` and `countable.is_countable()` are both set Six unit tests cover the picker rules: 1. picks single-property range_countable 2. picks compound range_countable with Equal prefix 3. rejects range on non-terminator property (no ProvableCountTree exists at that level) 4. rejects non-range_countable index 5. rejects multiple range operators 6. rejects pure point-lookup queries (those go to find_countable_index_for_where_clauses) The executor side (range walk on the property-name ProvableCountTree to read per-value CountTree counts) and the drive-abci handler routing are deferred to a follow-up — this commit only lands the detection logic so a query can be classified correctly. The runtime handler still rejects `return_distinct_counts_in_range=true`; the next step is wiring the executor and removing that gate. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `DriveDocumentCountQuery::execute_range_count_no_proof` plus the `RangeCountOptions` knob struct (distinct / limit / start_after_split_key / order_by_ascending). Walks children of the property-name `ProvableCountTree` at `[contract_doc, doctype, prefix..., range_prop_name]` whose keys lie within the range expressed by the where clause, reads `count_value_or_default()` from each child `CountTree`, and either sums them (single entry) or returns one entry per distinct property value. Range operator → `QueryItem` mapping covers `>`, `>=`, `<`, `<=`, `Between`, `BetweenExcludeBounds`, `BetweenExcludeLeft`, `BetweenExcludeRight`. `StartsWith` is rejected with a clear message since its grovedb encoding requires a byte-incremented upper bound that's not generic. `In` on prefix properties forks the walk into one path per deduped value and merges per-key entries across forks. Distinct-mode pagination matches the protobuf doc: - ordering: `order_by_ascending = true` is BTreeMap natural order; false reverses - cursor: `start_after_split_key` skips up to AND INCLUDING that key (drops it from the result set in either direction) - limit: applied last, after order + cursor Two e2e tests exercise the full path against a real Drive: 1. `range_count_executor_sums_and_splits_correctly` — six docs at three colors, `color > "blue"` → sum mode returns 5, distinct mode returns [(green, 3), (red, 2)], plus limit + cursor + descending variants 2. `range_count_executor_between_is_inclusive_on_both_bounds` — `Between [bbb, ccc]` returns both bounds (inclusive) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Updates `query_documents_count_v0` to: 1. Detect range operators in the where clauses and, when present, route through the new `find_range_countable_index_for_where_clauses` picker + `execute_range_count_no_proof` executor. 2. Plumb `order_by_ascending`, `limit`, `start_after_split_key`, and `return_distinct_counts_in_range` from the proto request into the `RangeCountOptions` knob struct. Limit is clamped to `max_query_limit` server-side. 3. On the prove path, generate a grovedb `AggregateCountOnRange` proof via the new `execute_aggregate_count_with_proof` helper. Replaces the materialize-and-count proof path (which capped at u16::MAX) for range queries — clients verify with `verify_aggregate_count_query` to recover `(root_hash, count)` without materializing any docs. 4. Reject `return_distinct_counts_in_range = true` on the prove path (the merk-level `AggregateCountOnRange` returns a single aggregate; per-distinct-value entries can't be expressed as one proof shape). 5. Reject mixing `In` with range, and reject multiple range operators in one query, with clear messages directing the caller to use `between*` or split client-side. The previous "range operators not yet supported" hard error is gone: range queries with a covering `range_countable: true` index now succeed end-to-end. The point-lookup proof path (no range) still uses the materialize-and-count flow with the u16::MAX cap, since per- CountTree count proofs aren't wired through a single aggregate primitive yet. Existing test renamed/updated to assert the new behavior — a range query against a contract WITHOUT a range_countable index returns a clear "range count requires `range_countable: true` index" error rather than a generic "range operators not supported" error. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Runs the proto-generator pipeline (web, nodejs, java, objective-c, python) against the current `platform.proto`, picking up the new fields on `GetDocumentsCountRequestV0`: - `return_distinct_counts_in_range = 4` - `order_by_ascending = 5` - `limit = 6` - `start_after_split_key = 7` - `prove = 8` (renumbered from 4) The previous committed clients were generated against an older proto revision (only `prove` at field 4) and were missing the pagination / distinct knobs entirely. The Rust handler in this branch already plumbs all five fields end-to-end; this commit aligns the wire format on the JS / Java / ObjC / Python sides. Generated via `yarn build` in packages/dapi-grpc. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Updates `book/src/drive/document-count-trees.md` to reflect the now- released range count behavior: - Replaces "Range operators return InvalidArgument" with the actual range path (find_range_countable_index_for_where_clauses + execute_range_count_no_proof). - Documents the four request modes derivable from the unified `GetDocumentsCount` endpoint (total / per-In-value / per-distinct- range-value / total-range) and the `return_distinct_counts_in_range` toggle. - Documents the new pagination knobs (`order_by_ascending`, `limit`, `start_after_split_key`) and clarifies they only apply in distinct- range mode. - Documents the `AggregateCountOnRange` prove path: range proofs are no longer bounded by the materialize-and-count `u16::MAX` cap; point-lookup count proofs still use the materialize-and-count flow pending a CountTree-direct proof primitive. - Removes references to the legacy `GetDocumentsSplitCount` endpoint (split is now an `In` clause / `return_distinct_counts_in_range` variant of the unified `GetDocumentsCount`). - Updates the cheat-sheet table with concrete schema → query-mode mappings, including the difference between `countable` and `range_countable` per-index flags. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
📖 Book Preview built successfully. Download the preview from the workflow artifacts. Updated at 2026-05-12T07:33:13.596Z |
Review GateCommit:
|
…Info The "Mutually compatible with the `countable` flag" sentence on the `range_countable` field's docstring was glued onto the bullet list above it, which clippy 1.92's `doc-lazy-continuation` lint now treats as a hard error (under `-D warnings`). Adding a blank line above it makes it a separate paragraph, which is what the docstring meant. Caught by the macOS `Tests` workflow on PR #3623. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three bullet continuation lines on `range_clause_to_query_item`'s docstring were indented with 4 spaces instead of 3 (2-space continuation after `/// `). Clippy 1.92's `doc-overindented-list-items` lint catches this under `-D warnings` and now treats it as a hard error. Caught by the macOS Tests workflow on PR #3623 (after the prior doc-lazy-continuation fix). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## v3.1-dev #3623 +/- ##
============================================
+ Coverage 88.25% 88.34% +0.09%
============================================
Files 2494 2507 +13
Lines 304580 310777 +6197
============================================
+ Hits 268812 274563 +5751
- Misses 35768 36214 +446
🚀 New features to boost your workflow:
|
Codecov flagged 71% patch coverage on PR #3623, with the largest gaps in the new range-count executor and abci handler routing. This commit adds five tests covering the load-bearing paths: drive (rs-drive/.../insert_contract/v0/mod.rs): - aggregate_count_proof_verifies_and_returns_correct_count — generates an `AggregateCountOnRange` proof via execute_aggregate_count_with_proof and verifies it via GroveDb::verify_aggregate_count_query, asserting the recovered count matches the no-proof sum (5 docs). - range_count_with_in_on_prefix_forks_and_merges — exercises the cartesian-fork path through a compound `[brand, color]` range_countable index with `brand IN (acme, contoso)` plus `color > "blue"`. Verifies per-key entries are merged across the In fork (red: 3 acme + 2 contoso = 5). - range_count_executor_rejects_starts_with — confirms the executor's StartsWith branch returns InvalidWhereClauseComponents rather than silently using a wrong range. drive-abci (rs-drive-abci/.../document_count_query/v0/mod.rs): - test_documents_count_range_query_no_prove — full handler integration with a v12 range_countable contract: 6 docs across 3 colors, asserts sum mode, distinct ascending, distinct + limit, and distinct descending all behave correctly. - test_documents_count_range_with_prove_rejects_distinct — confirms the prove path rejects `return_distinct_counts_in_range = true` because grovedb's AggregateCountOnRange proof returns one aggregate. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
✅ DashSDKFFI.xcframework built for this PR.
SwiftPM (host the zip at a stable URL, then use): .binaryTarget(
name: "DashSDKFFI",
url: "https://your.cdn.example/DashSDKFFI.xcframework.zip",
checksum: "0ce47e635e29ccef057ee7041d18b3c1fada2327a84e4e0dab6a53943fa7babb"
)Xcode manual integration:
|
The `try_from_schema` dispatch table routes protocol_version ≥ 12 to the v2 module via `CONTRACT_VERSIONS_V4.try_from_schema = 2`. Inside the v1 body we are therefore guaranteed to be at protocol v9/v10/v11 — `platform_version.protocol_version < 12` is always true. Removes the redundant version comparison from both the existing `countable.is_countable()` gate (PR #3457) and the new `range_countable` gate (this PR), keeping the rejections themselves as belt-and-suspenders defense against any future dispatch changes. Updated the comment to explain why the gate is here at all. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous docstring said `rangeCountable` makes the property-name tree a `ProvableCountTree`, but for compound indexes that's only the *last* property (the IndexLevel terminator) — prefix properties keep their default tree shape. The wording could mislead readers into thinking the whole index path becomes a count tree. Also drops the trailing "gated on protocol version 12+ ..." sentence; that's a deployment detail belonging in the v12 protocol notes, not on a per-field docstring. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous refactor (80e668a) was wrong: I claimed the `platform_version.protocol_version < 12` guards were dead code on the assumption that the dispatch table routes v12+ to v2. That's true at the OUTER dispatch level, but `try_from_schema_v2` delegates to `DocumentTypeV1::try_from_schema` internally for shared core parsing — so v1's body IS reached at protocol v12+, and the version guard is load-bearing. Without the guards, every v12 contract with a `countable` or `rangeCountable` index gets rejected at v1's validation gate, which broke all 10 range_countable_index_e2e_tests on macOS CI. Update the comment to flag this so future readers (including future-me) don't make the same mistake. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Codex review findings on PR #3623. I rechecked these against the latest head ( Finding 1: [P1] Range count proofs fail SDK verification
The server now returns an How I would fix it: detect the same single-range + covering Finding 2: [P1] Unset distinct range limit is unbounded
This maps an omitted How I would fix it: apply a default limit for distinct mode, for example Finding 3: [P2] SDK cannot reach new no-proof range modes
The unified request adds How I would fix it: add request options/builders plus a |
First step of the document_count_query handler refactor: lift the where-clause-shape validation out of the drive-abci handler into rs-drive. Pure validation now lives in `DriveDocumentCountQuery::detect_mode` which: - Returns a `DocumentCountMode` enum (Total / PerInValue / RangeNoProof / RangeProof / PointLookupProof) classifying the query shape. - Surfaces every where-clause/flag mismatch (multiple range, range + In, distinct without range, distinct on prove path, more than one In, unrecognized operator) as `QuerySyntaxError::InvalidWhereClauseComponents` instead of inline `QueryError::InvalidArgument` strings spread across the handler. The drive-abci handler now calls `detect_mode` once and `match`es on the returned mode tag, with each per-mode body kept in place. Index coverage validation (no covering countable / range_countable index) stays at the call site since it depends on the contract's index map. 14 new unit tests in `rs-drive` cover the truth table without requiring a `Drive` instance, a contract, or a `PlatformVersion`. Existing 7 drive-abci handler tests still pass; one assertion updated to allow either the old `InvalidArgument` shape or the new `Query(InvalidWhereClauseComponents)` shape since the rejection moved between error variants. Sets up step 2 (extract per-mode executors behind `Drive::execute_document_count_request_<mode>`) and step 3 (collapse into a single `Drive::execute_document_count_request`). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…drive Step 2 of the document_count_query handler refactor. Adds five methods on `Drive` that own the index-pick + executor-call cycle for each `DocumentCountMode`: - `Drive::execute_document_count_total_no_proof` - `Drive::execute_document_count_per_in_value_no_proof` (cartesian fork over the In values, dedup-by-serialized-key) - `Drive::execute_document_count_range_no_proof` - `Drive::execute_document_count_range_proof` (AggregateCountOnRange) - `Drive::execute_document_count_point_lookup_proof` (materialize-and- count fallback, capped at u16::MAX) Each method: - Picks the right covering index, returning `Error::Query(QuerySyntaxError::WhereClauseOnNonIndexedProperty)` when no index covers the where clauses (so the abci handler can map it to `QueryError::Query(qe)` uniformly). - Builds the appropriate `DriveDocumentCountQuery` (or `DriveDocumentQuery` for the materialize fallback). - Returns `Vec<SplitCountEntry>` (no-proof modes) or `Vec<u8>` proof bytes (proof modes). The drive-abci handler `query_documents_count_v0` now: - Calls `detect_mode` once (step 1). - Each per-mode arm is a single `self.drive.execute_*` call wrapped in a `handle_drive_result!` macro that maps `Error::Query` → `QueryError::Query`. Result wrapping is consolidated into the new `count_response_with_entries` free helper. - Net handler size: 1128 → 924 lines (-18%); business logic per arm dropped from ~30-40 lines to ~10-15 lines including response wrapping. One existing handler test had its assertion updated to accept either the old `InvalidArgument` rejection shape OR the new `Query(WhereClauseOnNonIndexedProperty)` shape (both are valid now that the rejection moved between error variants). All tests green: 7 abci handler tests, 3109 drive lib tests, 14 detect_mode unit tests, 10 range_countable e2e tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sweeps `document-count-trees.md` for prose that's stale after
`1cec252337`:
- **`(In + prove)` orderBy requirement removed**: the table row on
`order_by` and the rs-sdk-ffi `order_by_json` paragraph both
claimed orderBy was "required for `(In + prove)` walk determinism
(proof reconstruction needs an explicit order)". That was true
under the old `DriveDocumentQuery::from_decomposed_values`
materialize path, which errored with `MissingOrderByForRange`
without an orderBy clause. The new CountTree element proof path
doesn't need an orderBy — In keys are sorted at the path-query
builder, and the proof shape doesn't depend on walk order. Both
spots now describe `order_by` as optional, only meaningful in
split modes for entry ordering.
- **"smaller than materialize-and-count" baselines removed**: two
passages compared the range-distinct and range-aggregate proof
sizes against materialize-and-count as the implied alternative.
Materialize-and-count is no longer a code path anywhere in the
count endpoint (the no-range prove case now uses the CountTree
element proof, the range cases use AggregateCountOnRange or
KVCount). Replaced with direct trade-off prose ("pick aggregate
for one number, distinct for a histogram") that stands on its
own without referencing a deleted path.
- **Range-aggregate "replacing the older materialize-and-count
fallback" framing dropped**: the AggregateCountOnRange description
no longer needs to compare itself against a path that doesn't
exist anymore.
No code changes. The chapter now describes only the code paths that
actually ship.
…s fixture
55-line `#[allow(dead_code)]` test fixture left over from
`2b42989a75 test(drive): drop most-common-color test, keep helper for
future coverage` — the original commit kept the fixture in tree on
the bet that follow-up tests using the dual-`range_countable`
configuration would land soon. They didn't.
The wrapping logic the fixture was meant to exercise
(NonCounted-wrapping where wrapper-target = `ProvableCountTree`
rather than `NormalTree`) is one match arm in
`fees/op.rs:for_known_path_key_empty_non_counted_tree`:
```rust
let inner = match tree_type {
TreeType::NormalTree => Element::empty_tree_with_flags(element_flags),
TreeType::CountTree => Element::empty_count_tree_with_flags(element_flags),
TreeType::ProvableCountTree => Element::empty_provable_count_tree_with_flags(element_flags),
...
};
let non_counted_element = Element::new_non_counted(inner)?;
```
All three arms are structurally identical — they route to a
grovedb `Element::empty_*_with_flags` constructor and wrap the
result in `Element::new_non_counted`. The variant-specific behavior
(does a NonCounted-wrapped tree contribute 0 to its parent's count?)
is grovedb's concern and grovedb tests its NonCounted primitive
directly. The existing
`count_tree_value_count_excludes_compound_continuation_via_non_counted`
test pins the load-bearing drive-side wrapping decision (when to
wrap, based on the walker's `parent_value_tree_is_range_countable`
flag). Whether the wrapped child is a `NormalTree` or a
`ProvableCountTree` doesn't change the drive-side logic; the
walker calls the same routing function with a different `tree_type`.
If a real bug ever surfaces in the dual-`range_countable` case
(off-by-one on a parent CountTree's count after inserting docs
covered by two overlapping range_countable indexes), a targeted
test will get written then with the actual failure to pin against.
Speculative test scaffolding for a hypothetical bug isn't worth
the dead-code smell in source.
Git history at `2b42989a75:packages/rs-drive/.../insert_contract/v0/mod.rs`
preserves the fixture if needed.
…ast path
Makes the no-proof count path symmetric with the prove path's
rejection contract: a `countable: true` index now counts exactly its
declared properties, and queries against partially-covered indexes
reject with `WhereClauseOnNonIndexedProperty` rather than silently
walking uncovered levels via the old `count_recursive` fallback.
Adds a `documents_countable: true` fast path for unfiltered total
counts — when set on the document type, the type-level primary-key
tree IS a `CountTree`, readable in O(1) without any index walk.
## What changed
### Strict picker
`DriveDocumentCountQuery::find_countable_index_for_where_clauses` now
requires every property of a candidate index to have a matching
Equal/In clause AND every clause's field to appear in the index — set
equality, not prefix match. The pre-rewrite picker returned the
longest-prefix-matching index and downstream code (`count_recursive`)
walked all distinct values at uncovered levels and summed. That worked
but had three undesirable properties:
1. **Asymmetric with prove**: the prove path required exact coverage
(post the PointLookupProof rewrite); no-proof did not. A caller
flipping `prove=true` ↔ `prove=false` could see "rejected" or
"works" for the same query.
2. **Silent perf cliff**: a partially-covered query against a
high-branching index walked O(product of distinct uncovered values)
reads. The contract author who shipped the wrong index discovered
the cost at production query time, not design time.
3. **Forced storage-cost trade-off**: keeping `count_recursive` cheap
required maintaining counts only at the terminal index level (the
current storage layout). The user can opt into wider counts via
`documentsCountable: true` (primary-key tree as CountTree) but
nothing in between — and pre-strict-picker, partial coverage on a
`countable: true` index acted as a "middle option" that wasn't
really cheap.
The strict contract makes the framework's design principle visible at
the API boundary: a `countable: true` index expresses exactly which
count queries it supports. Want `count(*) WHERE color = X`? Define
a `[color]` countable index. Want both `[color]` AND `[color, shape]`
counts? Define both indexes.
### documents_countable fast path
Both `Drive::execute_document_count_total_no_proof` and
`Drive::execute_document_count_point_lookup_proof` now special-case
`where_clauses.is_empty() && document_type.documents_countable()`:
- **No-proof**: reads the doctype's primary-key CountTree at
`[contract_doc, contract_id, 1, doctype, 0]` via
`grove_get_raw_optional`. Returns `count_value_or_default()`. O(1).
- **Prove**: builds a single-element path query via the new
`primary_key_count_tree_path_query` helper, runs
`get_proved_path_query`. Returns one merk-path proof, O(log n) bytes.
SDK verifies via the new
`verify_primary_key_count_tree_proof` wrapper.
The same proof shape (a verified CountTree element with its
`count_value`) the index-based path uses, just rooted at the doctype
instead of inside an index.
### count_recursive deleted
`DriveDocumentCountQuery::count_recursive` and the partial-coverage
branch in `expand_paths_and_count` are gone. The `expand_paths_and_count`
"no matching clause for this property" arm now returns
`InvalidWhereClauseComponents` defensively — the strict picker
guarantees the arm is unreachable from the dispatcher, but having
the executor refuse to silently walk uncovered levels keeps the
contract clear for anyone calling the executor directly.
Net diff: ~110 lines deleted from `execute_point_lookup.rs`,
including the `query.insert_all()` enumeration logic that was the
only consumer of grovedb's open-ended subtree-walking primitive in
the count module.
### Verifier surface
- New `DriveDocumentCountQuery::verify_primary_key_count_tree_proof`
in `rs-drive/src/verify/document_count/`, with v0 implementation
+ version dispatch field.
- New `drive_proof_verifier::verify_primary_key_count_tree_proof`
tenderdash-composition wrapper.
- Both `FromProof<DocumentCountQuery>` impls (DocumentCount,
DocumentSplitCounts) route empty-where + documents_countable
through the new wrapper. Index-covered Equal/In requests continue
through `verify_point_lookup_count_proof`.
## Tests
- 33 drive `query::drive_document_count_query` unit tests pass:
- `test_count_query_fully_covered_equal_succeeds_on_both_paths`
(new positive pin: `age == 30` against the byAge 1-prop
countable index works on both no-proof and prove).
- `test_count_query_picker_rejects_partial_coverage` (new
negative pin: empty where, `firstName = X` against
[firstName, lastName] index, and unindexed fields all return
`None` from the strict picker).
- Two old tests that relied on empty where + multi-property
countable indexes (`test_count_query_total_count_with_documents`,
`test_count_query_total_count_empty`) deleted — covered by the
new tests and by the drive-abci tests below.
- `test_countable_allowing_offset_variant_end_to_end` updated to
use a fully-covering `firstName = "Alice"` where instead of
empty where; pins that `is_countable()` still accepts both
`Countable` and `CountableAllowingOffset` variants.
- 27 drive `range_countable_index_e2e_tests` pass (unaffected).
- 9 drive-abci `query::document_count_query` end-to-end tests pass:
- `test_documents_count_no_prove` / `test_documents_count_empty_result`
rewritten to build an inline `documentsCountable: true` widget
contract via the new `build_documents_countable_widget_contract`
helper. Both exercise the fast path now.
- `test_documents_count_prove_without_covering_index_returns_clear_error`
updated to assert `WhereClauseOnNonIndexedProperty` (the new
error class for "no covering index").
- 225 drive-proof-verifier tests pass.
- `cargo clippy -p drive -p drive-abci -p dash-sdk -p drive-proof-verifier --lib --tests --features=server,verify -- -D warnings` clean.
- `cargo fmt --check` clean.
## Book updates
- `Equal/In only` section rewritten: drops the "if only a prefix was
covered: sum the counts of all CountTree children at the deepest
covered level" step and replaces it with the strict-coverage
contract. Adds the documents_countable fast path as a separate
no-proof sub-section.
- Prove path section gains the documents_countable sub-path
description (one merk path proof, O(log n) bytes via
`verify_primary_key_count_tree_proof`).
- Symmetric rejection paragraph rewritten to reflect that no-proof
and prove now share the same exact-coverage requirement; no more
`count_recursive` mention since that path no longer exists.
## Breaking changes
Source-API: `find_countable_index_for_where_clauses` is a public
function whose semantics changed (prefix match → exact match).
Callers passing a where clause that's only a prefix of a candidate
index now get `None` instead of `Some(index)`. The dispatcher and
the SDK FromProof impls (all in-tree) have been updated; downstream
callers outside this repo would need to either define a more
specific index or use `documentsCountable: true`.
Wire-format: no change. The strict contract is enforced server-side
by returning a different `QueryError` variant for partial-coverage
queries; well-designed contracts (those with indexes matching their
intended count queries) keep working.
Pre-testnet, so no real-world callers to migrate.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…hapter
The strict-coverage rewrite in `e5b891c88e` updated the prose in the
Equal/In and Prove sections of the count chapter, but left four stale
spots that still describe the pre-rewrite partial-coverage fallback:
1. `byColor.countable: true` paragraph (count chapter §"Per-Index
Countable Flag") — *"in O(1) instead of falling back to a scan.
Without the flag … the count won't take the fast path"*. The
"falling back to a scan" comparison is wrong now: without the
flag the picker returns None and the query rejects. There is no
slow path.
2. `["color", "size"]` partial-coverage example (count chapter same
section) — *"WHERE color = X alone (only the leading prefix
matched) the count is computed by walking every distinct-size
bucket"*. That walk doesn't exist anymore; partial coverage
rejects with `WhereClauseOnNonIndexedProperty`.
3. "Choosing What to Set" table row for filtered counts (count
chapter §"Choosing What to Set") — *"A composite index whose
leading column is `col` (e.g. `['col', 'other']`) still answers
the query, but as O(distinct values of `other`) instead of
O(1)"*. Same partial-coverage fallback that no longer exists.
4. Per-`In`-value sub-counts table row — claimed `documentsCountable:
true` is needed alongside the index, which is wrong: a covering
`countable: true` index is sufficient on its own.
5. Indexes chapter §"Choosing Your Indexes" — *"countable when
you'll regularly call GetDocumentsCount filtered by this index's
**leading columns**"*. The leading-columns framing is the partial-
coverage assumption; the contract is now exact-match.
All five spots rewritten to:
- Make exact-coverage explicit ("exactly match", "exactly this
index's properties").
- Mention the rejection (`WhereClauseOnNonIndexedProperty`) as the
outcome of partial coverage, not a slow scan.
- Direct contract authors toward defining a separate index per
distinct count-query shape, or `documentsCountable: true` for
unfiltered totals.
- Drop the "documentsCountable required for filtered counts"
framing in the Per-In-value table row.
Also tightened the migration-check paragraph at the end of "Choosing
What to Set" to mention contract-index immutability — count-query
shape decisions are locked at contract creation time, so authors
should think about which counts they need up front.
No code changes.
thepastaclaw
left a comment
thepastaclaw
left a comment
There was a problem hiding this comment.
Code Review
Verified findings against the worktree at 6d044b3. No blocking issues remain at this head — the prior prove+In, range-proof, PerInValue, JS_STRING, empty-key, and panic-on-encode-failure regressions are all genuinely fixed. Remaining items are real but non-blocking: a config-skew footgun in the prove-distinct verifier when operators tune default_query_limit, a likely SizedQuery-limit mismatch on the In+prove materialize path, several stale comments/references to renamed/deleted types, a proto/code contradiction on startsWith, an FFI signature change without a symbol rename, gaps in non-Rust surface exposure of new knobs, and one dead parameter.
Reviewed commit: 6d044b3
🟡 7 suggestion(s) | 💬 3 nitpick(s)
10 additional findings
🟡 suggestion: RangeDistinctProof verifier uses compile-time DEFAULT_QUERY_LIMIT while server uses runtime default_query_limit
packages/rs-sdk/src/platform/documents/document_count_query.rs (lines 414-433)
On the RangeDistinctProof path the SDK verifier (lines 425-428) reconstructs the prover's PathQuery with request.limit.unwrap_or(drive::config::DEFAULT_QUERY_LIMIT) — a compile-time constant. The server-side dispatcher at packages/rs-drive/src/query/drive_document_count_query/mod.rs:1901-1903 instead uses request.drive_config.default_query_limit — a runtime config value. These agree only when an operator keeps the shipped default. Any node that tunes default_query_limit will produce proofs whose SizedQuery::limit byte-differs from the verifier's reconstruction, and proof verification will fail with cryptic GroveDB errors for every with_distinct_counts_in_range(true) request that omits limit. The doc comment at lines 422-424 acknowledges this and shifts the obligation to operators, but the failure mode is silent verify-time breakage on a consensus-adjacent path. Either reject the request server-side when request.limit.is_none() && default_query_limit != DEFAULT_QUERY_LIMIT, require limit to be Some(_) on the RangeDistinctProof branch with a clear error, or echo the effective limit through response metadata so the SDK can mirror it.
🟡 suggestion: PointLookupProof verification: client forces drive_query.limit=None while server pins it to Some(u16::MAX)
packages/rs-sdk/src/platform/documents/document_count_query.rs (lines 150-167)
TryFrom<&DocumentCountQuery> for DriveDocumentQuery forces drive_query.limit = None (line 164), but the server-side execute_document_count_point_lookup_proof at packages/rs-drive/src/query/drive_document_count_query/mod.rs:1630 overrides to drive_query.limit = Some(u16::MAX) before calling execute_with_proof. DriveDocumentQuery::construct_path_query embeds self.limit into SizedQuery::new(query, self.limit, self.offset), so the path query the prover signed (limit=Some(u16::MAX)) is structurally distinct from the path query the verifier rebuilds (limit=None). On small result sets the proof tends to verify and the discrepancy is masked, but on result sets large enough to touch the cap the boundary subtree the prover emitted won't match the verifier's expectation. Either set drive_query.limit = Some(u16::MAX) on the client side to mirror the server, or have the server omit the override so both ends use the same default. There is also no SDK-side end-to-end test pinning the In+prove path, which is why the discrepancy hasn't surfaced.
🟡 suggestion: Stale references to renamed/deleted DocumentSplitCountQuery and removed test files
packages/rs-drive-proof-verifier/src/proof/document_split_count.rs (lines 32-157)
DocumentSplitCountQuery was renamed to DocumentCountQuery in this PR, but this verifier still references the old name in three places: line 32 (Fetch impl on \"DocumentSplitCountQuery\"), line 54 (a user-facing error string returned by the rejected generic FromProof entry point — this is the worst because it directs callers to a type that no longer exists), and lines 154-156 (which point readers to packages/rs-sdk/tests/fetch/document_split_count.rs and packages/rs-drive-abci/src/query/document_split_count_query/v0/mod.rs tests, both deleted in this PR). Repoint to DocumentCountQuery and to the surviving fixtures (packages/rs-drive-abci/src/query/document_count_query/v0/mod.rs; packages/rs-drive/src/query/drive_document_count_query/tests.rs).
🟡 suggestion: Wasm-sdk/FFI comments claim "proof + distinct is rejected server-side" — no longer true after RangeDistinctProof landed
packages/wasm-sdk/src/queries/document.rs (lines 461-581)
Four near-identical wasm-sdk count wrappers (lines 467-468, 498-499, 537-538, 563-564) each say "Range no-proof distinct mode … needs a separate JS-facing API entry point since proof + distinct is rejected server-side; tracked as a follow-up." detect_mode at packages/rs-drive/src/query/drive_document_count_query/mod.rs:262 now explicitly routes (range=true, prove=true, distinct=true) to DocumentCountMode::RangeDistinctProof, with verifier-side support landed in this PR. The same stale comment is duplicated in packages/rs-sdk-ffi/src/document/queries/count.rs:157-161 and :228-232. At minimum, drop the misleading justification; ideally surface return_distinct_counts_in_range / limit / start_after_split_key knobs since the rs-sdk plumbing already exists.
🟡 suggestion: Proto comment advertises `startsWith` as a supported range operator but detect_mode rejects it
packages/dapi-grpc/protos/platform/v0/platform.proto (lines 625-633)
The unified-count proto comment at line 627 lists supported range operators as "(>, <, between*, startsWith)", but DriveDocumentCountQuery::detect_mode at packages/rs-drive/src/query/drive_document_count_query/mod.rs:208-212 explicitly rejects WhereOperator::StartsWith with InvalidWhereClauseComponents("startsWith is not yet supported on count queries") before any index check. Clients reading the proto contract and crafting a startsWith count request will hit a runtime error. Either drop startsWith from the proto comment (and from book/src/drive/document-count-trees.md) until range_clause_to_query_item grows StartsWith encoding, or implement encoding first.
🟡 suggestion: C FFI count API does not expose the unified endpoint's new query controls
packages/rs-sdk-ffi/src/document/queries/count.rs (lines 139-239)
dash_sdk_document_count and dash_sdk_document_split_count only accept (sdk_handle, data_contract_handle, document_type, where_json), then hard-code return_distinct_counts_in_range: false, order_by_ascending: None, limit: None, and start_after_split_key: None (lines 162-167 and 233-238). The Rust SDK and the gRPC wire format support all four knobs; native Swift/iOS callers crossing the C boundary cannot request range-distinct counts, split-entry ordering, split-entry pagination, or split-entry limits. The advertised unified count semantics are effectively Rust-only at this layer. Either thread the knobs through the FFI signature or add a typed-struct entry point.
🟡 suggestion: DocumentCountQuery silently ignores the SDK's global proof-mode setting
packages/rs-sdk/src/platform/documents/document_count_query.rs (lines 169-192)
Fetch obtains its transport request through query.query(sdk.prove()), but DocumentCountQuery is its own TransportRequest and uses the blanket Query<T> for T impl, so the prove flag never reaches it. The TryFrom<DocumentCountQuery> for GetDocumentsCountRequest impl hard-codes prove: true at line 188. Callers using SdkBuilder::with_proofs(false) will still fetch and verify proofs silently — including the expensive PointLookupProof materialize path for In queries. That diverges from the rest of the SDK, where prove=false is either honored or rejected with unimplemented!(). Either honor the SDK proof setting, or assert explicitly that count queries always prove.
💬 nitpick: `dash_sdk_document_split_count` C ABI silently dropped `split_property` arg without symbol rename
packages/rs-sdk-ffi/src/document/queries/count.rs (lines 209-215)
#[no_mangle] pub unsafe extern "C" fn dash_sdk_document_split_count kept the same exported symbol name but the parameter list shrank from 5 args (sdk, contract, doc_type, split_property, where_json) to 4 args (sdk, contract, doc_type, where_json). cbindgen regenerates the header, but any iOS/Swift consumer still linked against the previously-released header will push five arguments; the callee will interpret the old split_property C string as where_json, serde_json::from_str will fail on the property name, and the call will return an InternalError JSON — silent runtime failure with no compile- or link-time signal. Pre-release scope acknowledged and no in-tree Swift caller binds this symbol today, but renaming (e.g. _v2 suffix) would let stale wrappers fail loudly at link time. Same concern applies to the matching change on dash_sdk_document_count.
💬 nitpick: `count_path_and_query_item` takes a `builder_label` it then silences with `let _ =`
packages/rs-drive/src/query/drive_document_count_query/mod.rs (lines 1162-1177)
count_path_and_query_item declares builder_label: &'static str, both call sites (aggregate_count_path_query:1243, distinct_count_path_query:1281) thread it in, and the function immediately silences it with let _ = builder_label; at line 1176. The label never appears in any of the Error::Query(QuerySyntaxError::InvalidWhereClauseComponents(...)) messages this function returns. Either remove the parameter and the two argument sites, or weave the label into the error strings so callers can tell which builder failed. A dead parameter sunk to silence the warning is the worst of both options.
💬 nitpick: No rs-sdk regression coverage for DocumentSplitCounts::fetch or distinct-range proof path
packages/rs-sdk/tests/fetch/document_count.rs (lines 1-9)
The rs-sdk fetch tests only cover DocumentCount mocks; there is no DocumentSplitCounts::fetch test for either the In-clause proof path or the distinct-range proof path, even though document_count_query.rs:345-503 contains bespoke SDK-only verifier logic for both. The earlier prove+In and range-aggregate regressions on this PR escaped because only lower layers were tested. Adding rs-sdk-level coverage for DocumentSplitCounts::fetch with an In clause and with return_distinct_counts_in_range=true would catch future request/proof-shape drift before it hits a live platform.
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.
In `packages/rs-sdk/src/platform/documents/document_count_query.rs`:
- [SUGGESTION] lines 414-433: RangeDistinctProof verifier uses compile-time DEFAULT_QUERY_LIMIT while server uses runtime default_query_limit
On the RangeDistinctProof path the SDK verifier (lines 425-428) reconstructs the prover's `PathQuery` with `request.limit.unwrap_or(drive::config::DEFAULT_QUERY_LIMIT)` — a compile-time constant. The server-side dispatcher at `packages/rs-drive/src/query/drive_document_count_query/mod.rs:1901-1903` instead uses `request.drive_config.default_query_limit` — a *runtime* config value. These agree only when an operator keeps the shipped default. Any node that tunes `default_query_limit` will produce proofs whose `SizedQuery::limit` byte-differs from the verifier's reconstruction, and proof verification will fail with cryptic GroveDB errors for every `with_distinct_counts_in_range(true)` request that omits `limit`. The doc comment at lines 422-424 acknowledges this and shifts the obligation to operators, but the failure mode is silent verify-time breakage on a consensus-adjacent path. Either reject the request server-side when `request.limit.is_none() && default_query_limit != DEFAULT_QUERY_LIMIT`, require `limit` to be `Some(_)` on the RangeDistinctProof branch with a clear error, or echo the effective limit through response metadata so the SDK can mirror it.
- [SUGGESTION] lines 150-167: PointLookupProof verification: client forces drive_query.limit=None while server pins it to Some(u16::MAX)
`TryFrom<&DocumentCountQuery> for DriveDocumentQuery` forces `drive_query.limit = None` (line 164), but the server-side `execute_document_count_point_lookup_proof` at `packages/rs-drive/src/query/drive_document_count_query/mod.rs:1630` overrides to `drive_query.limit = Some(u16::MAX)` before calling `execute_with_proof`. `DriveDocumentQuery::construct_path_query` embeds `self.limit` into `SizedQuery::new(query, self.limit, self.offset)`, so the path query the prover signed (limit=Some(u16::MAX)) is structurally distinct from the path query the verifier rebuilds (limit=None). On small result sets the proof tends to verify and the discrepancy is masked, but on result sets large enough to touch the cap the boundary subtree the prover emitted won't match the verifier's expectation. Either set `drive_query.limit = Some(u16::MAX)` on the client side to mirror the server, or have the server omit the override so both ends use the same default. There is also no SDK-side end-to-end test pinning the In+prove path, which is why the discrepancy hasn't surfaced.
- [SUGGESTION] lines 169-192: DocumentCountQuery silently ignores the SDK's global proof-mode setting
`Fetch` obtains its transport request through `query.query(sdk.prove())`, but `DocumentCountQuery` is its own `TransportRequest` and uses the blanket `Query<T> for T` impl, so the `prove` flag never reaches it. The `TryFrom<DocumentCountQuery> for GetDocumentsCountRequest` impl hard-codes `prove: true` at line 188. Callers using `SdkBuilder::with_proofs(false)` will still fetch and verify proofs silently — including the expensive PointLookupProof materialize path for `In` queries. That diverges from the rest of the SDK, where `prove=false` is either honored or rejected with `unimplemented!()`. Either honor the SDK proof setting, or assert explicitly that count queries always prove.
In `packages/rs-drive-proof-verifier/src/proof/document_split_count.rs`:
- [SUGGESTION] lines 32-157: Stale references to renamed/deleted DocumentSplitCountQuery and removed test files
`DocumentSplitCountQuery` was renamed to `DocumentCountQuery` in this PR, but this verifier still references the old name in three places: line 32 (`Fetch impl on \"DocumentSplitCountQuery\"`), line 54 (a user-facing error string returned by the rejected generic `FromProof` entry point — this is the worst because it directs callers to a type that no longer exists), and lines 154-156 (which point readers to `packages/rs-sdk/tests/fetch/document_split_count.rs` and `packages/rs-drive-abci/src/query/document_split_count_query/v0/mod.rs tests`, both deleted in this PR). Repoint to `DocumentCountQuery` and to the surviving fixtures (`packages/rs-drive-abci/src/query/document_count_query/v0/mod.rs`; `packages/rs-drive/src/query/drive_document_count_query/tests.rs`).
In `packages/wasm-sdk/src/queries/document.rs`:
- [SUGGESTION] lines 461-581: Wasm-sdk/FFI comments claim "proof + distinct is rejected server-side" — no longer true after RangeDistinctProof landed
Four near-identical wasm-sdk count wrappers (lines 467-468, 498-499, 537-538, 563-564) each say "Range no-proof distinct mode … needs a separate JS-facing API entry point since proof + distinct is rejected server-side; tracked as a follow-up." `detect_mode` at `packages/rs-drive/src/query/drive_document_count_query/mod.rs:262` now explicitly routes `(range=true, prove=true, distinct=true)` to `DocumentCountMode::RangeDistinctProof`, with verifier-side support landed in this PR. The same stale comment is duplicated in `packages/rs-sdk-ffi/src/document/queries/count.rs:157-161` and `:228-232`. At minimum, drop the misleading justification; ideally surface `return_distinct_counts_in_range` / `limit` / `start_after_split_key` knobs since the rs-sdk plumbing already exists.
In `packages/dapi-grpc/protos/platform/v0/platform.proto`:
- [SUGGESTION] lines 625-633: Proto comment advertises `startsWith` as a supported range operator but detect_mode rejects it
The unified-count proto comment at line 627 lists supported range operators as "(`>`, `<`, `between*`, `startsWith`)", but `DriveDocumentCountQuery::detect_mode` at `packages/rs-drive/src/query/drive_document_count_query/mod.rs:208-212` explicitly rejects `WhereOperator::StartsWith` with `InvalidWhereClauseComponents("startsWith is not yet supported on count queries")` before any index check. Clients reading the proto contract and crafting a `startsWith` count request will hit a runtime error. Either drop `startsWith` from the proto comment (and from `book/src/drive/document-count-trees.md`) until `range_clause_to_query_item` grows StartsWith encoding, or implement encoding first.
In `packages/rs-sdk-ffi/src/document/queries/count.rs`:
- [SUGGESTION] lines 139-239: C FFI count API does not expose the unified endpoint's new query controls
`dash_sdk_document_count` and `dash_sdk_document_split_count` only accept `(sdk_handle, data_contract_handle, document_type, where_json)`, then hard-code `return_distinct_counts_in_range: false`, `order_by_ascending: None`, `limit: None`, and `start_after_split_key: None` (lines 162-167 and 233-238). The Rust SDK and the gRPC wire format support all four knobs; native Swift/iOS callers crossing the C boundary cannot request range-distinct counts, split-entry ordering, split-entry pagination, or split-entry limits. The advertised unified count semantics are effectively Rust-only at this layer. Either thread the knobs through the FFI signature or add a typed-struct entry point.
Inline posting hit GitHub HTTP 422, so I posted the same verified findings as a top-level review body.
The prove count builder (`point_lookup_count_path_query`) used to reject `In` on any non-last property of the covering countable index. That was strictly tighter than the regular document query path's `Index::matches` rule, which allows In on the last OR before-last property (`packages/rs-dpp/src/data_contract/ document_type/index/mod.rs:503`). Brings the two paths into lockstep so a query the document-query side accepts is one the count-query side can also prove. Implementation reuses the same `set_subquery_path` + `set_subquery` mechanism `distinct_count_path_query` already uses for compound In-on-prefix range counts: when In sits on the before-last property, base_path stops at the In-bearing property's property-name subtree, the outer Query enumerates serialized In values, `set_subquery_path` carries the trailing Equal's `(prop_name, serialized_value)` pair, and the subquery's `Key([0])` picks off the CountTree element at `[..., in_field, in_value, trailing_field, trailing_value, 0]`. Verifier requires no behavior change — `path[base_path_len]` still points at the In value across both compound shapes because the path query's base_path stops at the same offset either way; only comments + the `has_in_clause` rename were touched. Tests: two new lib tests pin both the positive path (no-proof + prove round-trip on `firstName IN [...] AND lastName == ...` on the 2-prop `byFirstNameLastName` index) and the position-rejection contract (`In` at position 0 of a 3-prop index rejected by the builder with a clear "last or before-last" error). Book updated with the relaxed In-position rule and the new third compound shape's path layout.
…constant CodeRabbit flagged a consensus-adjacent silent-verify-failure on the prove-distinct count path: server uses `request.drive_config .default_query_limit` (operator-tunable runtime value) for the `limit=None` fallback while the SDK verifier uses `drive::config::DEFAULT_QUERY_LIMIT` (compile-time constant). Any operator who tunes `default_query_limit` away from the shared constant produces proofs whose `SizedQuery::limit` byte-differs from the SDK's reconstruction → GroveDB merk-root recomputation fails with a cryptic `InvalidProof` for every prove-distinct request that omits `limit`. Fix is one line in the dispatcher's `RangeDistinctProof` arm: fall back to `crate::config::DEFAULT_QUERY_LIMIT` (the same compile-time constant the SDK reads), NOT `request.drive_config.default_query_limit`. This removes the operator-tunable degree of freedom from proof bytes entirely; the runtime `default_query_limit` continues to govern no-proof dispatch paths where there's no verifier to match. `max_query_limit` still gates the request as a DoS-protection knob. Regression test: build a request with `drive_config.default_query_limit = 1` (deliberately ≠ DEFAULT_QUERY_LIMIT = 100) and a 2+-element range result. Run the dispatcher, then verify the resulting proof via `GroveDb::verify_query` against a path query rebuilt with `Some(DEFAULT_QUERY_LIMIT)` — same path the SDK takes. If the dispatcher regresses to `default_query_limit`, the prover signs with limit=1, the verifier rebuilds with limit=100, the boundary merk path proof can't satisfy the wider path query and verification returns `InvalidProof`. Verified load-bearing by reverting the fix and observing the test fail with exactly that error. Also drops the dead `TryFrom<&DocumentCountQuery> for DriveDocumentQuery` impl in rs-sdk (its only user — the materialize-and-count verifier — was deleted earlier in this PR, leaving the impl with no callers) and updates the wasm-sdk comment that referenced it. SDK doc comment on the prove-distinct path rewritten to reflect the new "both sides anchored to compile-time constant" invariant.
CodeRabbit review triage — pushed
|
The prove count builder previously restricted `In` to the last or before-last index property to keep parity with the regular document query path's `Index::matches` rule. That parity was a policy choice, not a technical constraint: the count path doesn't have the positional path-construction assumption that forces the restriction in `DriveDocumentQuery::get_non_primary_key_path_query` (positional zip of `intermediate_indexes` ↔ `intermediate_values`). The no-proof count executor (`expand_paths_and_count`) has always handled In at any position via per-level fork; the prove builder's `set_subquery_path` accumulator structurally supports any number of post-In trailing Equals already. Lifting the policy check (~4 LOC) brings prove and no-proof count paths onto the same surface: both accept `a IN [..] AND b = y AND c = z` on `[a, b, c]`. The count path is now strictly more permissive than the regular document query path on In-position, which is defensible — count is a pure CountTree-element lookup with no document-key terminator descent, no order_by interpretation, and no limit/offset semantics, so there's no ambiguity to constrain. The position-rejection test (`test_prove_count_rejects_in_on_neither_last_nor_before_last`) is replaced by its positive counterpart (`test_count_query_in_on_first_of_three_with_two_trailing_equals_succeeds_on_both_paths`): 3-prop unique index, In on position 0 with two trailing Equals, no-proof + prove + verify round-trip all succeed. Verifier doc-comments + book updated to drop the "last-or-before-last" framing; the path query's `base_path` still stops at the In-bearing property's name subtree regardless of how many trailing Equals descend further, so the verifier's `path[base_path_len]` In-value extraction is unchanged.
…ery builder
The no-proof count executor used to carry its own ~110-line
recursive walker (`expand_paths_and_count` + `execute_total_count` +
`fetch_count_at_path`) that traversed the index property-by-property,
forking on `In` and reading CountTree elements directly via
`grove_get_raw_optional`. The prove-side executor used the shared
`point_lookup_count_path_query` builder for the exact same logical
work — emit the right set of `CountTree` elements for the query.
Collapse the no-proof side onto the same builder: run the path
query through `grove.query` (via `Drive::grove_get_path_query`)
and sum the emitted elements' `count_value_or_default()`. Single
source of truth for which CountTree elements compose the answer;
adding a new shape to the builder automatically applies to both
sides.
Net: -130 lines, +130 fewer surfaces for prove/no-proof to drift
apart.
**Behavior change — duplicate `In` values now rejected on the
no-proof count path.** The deleted walker bypassed
`WhereClause::in_values()` and silently deduplicated duplicate
values via a `BTreeSet<Vec<u8>>` of serialized keys. Every other
In-consuming path in the system already calls `in_values()` —
the prove count builder, the per-In-value no-proof executor, the
regular document query path, the contract-level validator — and
they all reject duplicates with `InvalidInClause("there should be
no duplicates values for In query")`. The no-proof count walker
was the lone outlier; unifying through the builder routes both
sides through the same validator. Updated
`test_count_query_in_operator_dedupes_duplicate_values` →
`test_count_query_in_operator_rejects_duplicate_values` to pin
the now-consistent rejection contract.
Updated supporting doc-comments (path_query.rs builder docstring,
two test docstrings, book chapter) to reference the unified
builder rather than the deleted walker.
thepastaclaw
left a comment
thepastaclaw
left a comment
There was a problem hiding this comment.
Code Review
Re-validated all findings at ba9a797. No blocking issues remain — all prior security/consensus concerns are resolved. What's left is a cluster of stale documentation (mod.rs PointLookupProof variant, mode_detection dispatch arms, proto field comments, drive-abci regression-test docstring, rs-sdk DocumentCountQuery rustdoc) that still describes the deleted materialize-and-count walker, plus an SDK API gap (DocumentCountQuery hard-codes prove=true and ignores SdkBuilder::with_proofs(false)), an FFI parser gap (no Between* operators), a dead op_name parameter in path_query.rs::serialize_pair, and the absence of rs-sdk Fetch coverage for DocumentSplitCounts and the RangeDistinctProof verifier path.
Reviewed commit: ba9a797
🟡 4 suggestion(s) | 💬 6 nitpick(s)
10 additional findings
🟡 suggestion: Unified C FFI count entry point can't express Between* range operators
packages/rs-sdk-ffi/src/document/queries/count.rs (lines 62-76)
dash_sdk_document_count is the sole native count entry point and its docs advertise generic range-count behavior, but parse_where_operator only accepts =, >, >=, <, <=, in, and startsWith — it rejects Between, BetweenExcludeBounds, BetweenExcludeLeft, and BetweenExcludeRight. The server, wasm bindings, and range_clause_to_query_item in path_query.rs all support these. iOS/native callers can't issue four of the range forms this PR adds verified fast paths for, despite the FFI being the unified surface.
🟡 suggestion: DocumentCountQuery rustdoc still describes pre-fix semantics for distinct-range and limit
packages/rs-sdk/src/platform/documents/document_count_query.rs (lines 57-123)
The field and builder docs (lines 57–74, 111–122) say return_distinct_counts_in_range is "only meaningful... when the request goes through a no-proof transport — the proof endpoint rejects this combination" and that oversized limit values are silently clamped ("Server clamps... larger values are silently reduced"). Both are stale at this head: the SDK has a dedicated RangeDistinctProof verifier path (document_count_query.rs:423-495) and the server explicitly rejects prove-distinct requests where limit > max_query_limit via Error::Query(QuerySyntaxError::InvalidLimit(...)) (drive_dispatcher.rs:806-816). Callers reading these docs will misdiagnose InvalidLimit errors and assume distinct-mode is unreachable on Fetch.
🟡 suggestion: DocumentCountQuery hard-codes prove=true and ignores SdkBuilder::with_proofs(false)
packages/rs-sdk/src/platform/documents/document_count_query.rs (lines 146-170)
Fetch::fetch_with_metadata_and_proof calls query.query(sdk.prove()), so callers reasonably expect SdkBuilder::with_proofs(false) to flip the request mode. But DocumentCountQuery doesn't implement Query<GetDocumentsCountRequest> and falls back to the blanket impl Query<T> for T, which returns self; the TryFrom<DocumentCountQuery> impl then unconditionally sets prove: true (line 166). Net effect: count queries silently ignore the SDK-wide proof toggle, divergent from the rest of the query surface, and the Rust SDK still has no typed way to reach the endpoint's no-proof modes even though transport and server support them.
🟡 suggestion: No rs-sdk Fetch coverage for DocumentSplitCounts or RangeDistinctProof verifier
packages/rs-sdk/tests/fetch/document_count.rs (lines 1-89)
This test file only mocks DocumentCount fetches. There is no rs-sdk-level test covering DocumentSplitCounts::fetch, the (In + prove) grouping branch, or the RangeDistinctProof verifier path in document_count_query.rs:345-503 — branches with bespoke SDK-side request-construction and proof-reconstruction logic (DEFAULT_QUERY_LIMIT anchor, in_key path-segment grouping, total-mode empty-key re-emit) that drive/drive-abci tests don't exercise from the client seam. This is the same gap that let earlier SDK regressions slip on this PR. At minimum add one mocked round-trip for DocumentSplitCounts::fetch with an In clause and one for with_distinct_counts_in_range(true) on a range query.
💬 nitpick: Proto order_by doc still cites the deleted materialize-and-count walker
packages/dapi-grpc/protos/platform/v0/platform.proto (lines 657-665)
The bytes order_by = 5 doc says it is "Required when where carries an In or range operator on the prove path: the materialize-and-count walker needs a deterministic walk order so the SDK can reconstruct the same path query." At this head, (1) the In+prove path runs through point_lookup_count_path_query which sorts In keys lex-asc unconditionally (path_query.rs:644-649) — proof determinism does not depend on order_by; (2) the PointLookupProof dispatcher arm doesn't read order_by_ascending; (3) the materialize-and-count walker the comment cites is gone. The RangeDistinctProof arm uses order_by direction but defaults to ascending if empty rather than rejecting, so "required" overstates that contract too.
💬 nitpick: Proto limit doc says "Server clamps" but prove path actually rejects
packages/dapi-grpc/protos/platform/v0/platform.proto (lines 666-669)
The optional uint32 limit = 6 doc-comment says: "Server clamps to its max_query_limit config. Unset → server default." The dispatcher's RangeDistinctProof arm at drive_dispatcher.rs:806-816 is explicitly validate-don't-clamp (silent clamping would break verification), returning Error::Query(QuerySyntaxError::InvalidLimit(...)) for limit > max_query_limit. The FFI doc (count.rs:209-211) has the correct phrasing ("clamped ... on no-proof paths, rejected if too large on prove paths"); mirror that into the proto so wire-level clients aren't surprised.
💬 nitpick: PointLookupProof variant docstring still describes the deleted materialize-and-count path
packages/rs-drive/src/query/drive_document_count_query/mod.rs (lines 146-149)
PointLookupProof no longer materializes documents at this head: execute_document_count_point_lookup_proof calls point_lookup_count_path_query and proves per-branch CountTree elements directly (drive_dispatcher.rs:441-480, execute_point_lookup.rs:110-124). The variant doc still says "falls back to the materialize-and-count proof path. Capped at u16::MAX matching docs because each verified document is materialized client-side" — both claims are false (the proof shape is O(k·log n) in covered branches, not O(matched docs); there is no u16::MAX cap). Update the doc to reflect the CountTree element proof shape.
💬 nitpick: PointLookupProof dispatch arm comment still cites materialize-and-count + u16::MAX cap
packages/rs-drive/src/query/drive_document_count_query/mode_detection.rs (lines 167-178)
The comment above the (false, true, true, _) => DocumentCountMode::PointLookupProof arm says: "In + prove = true (no range): route to the materialize-and-count proof path. The SDK's FromProof<DocumentCountQuery> for DocumentSplitCounts then groups verified documents by the In field's serialized value... the materialize path is correct, just bounded at u16::MAX." None of that holds at this head: the SDK verifier reaches verify_point_lookup_count_proof, which extracts count_value_or_default() from each verified CountTree element — no document materialization, no u16::MAX bound, no per-document grouping. Implementation is sound, only the comment is stale.
💬 nitpick: Regression-test docstring describes a failure mode that no longer exists
packages/rs-drive-abci/src/query/document_count_query/v0/mod.rs (lines 730-740)
The docstring for test_documents_count_with_in_and_prove_returns_proof says: "The materialize walker rejects any range/In where clause without a matching orderBy because proof determinism requires the SDK to reconstruct the same path query; missing orderBy returns MissingOrderByForRange before any proof is produced." After the PointLookupProof refactor, the In+prove path doesn't go through DriveDocumentQuery's materialize walker; it goes through point_lookup_count_path_query, which doesn't read order_by and doesn't emit MissingOrderByForRange. The test still pins the right behavior (proof bytes non-empty on prove+In) but the docstring's failure-mode reasoning will mislead future debuggers.
💬 nitpick: serialize_pair carries a dead op_name parameter silenced with `let _ = op_name;`
packages/rs-drive/src/query/drive_document_count_query/path_query.rs (lines 63-87)
serialize_pair declares op_name: &'static str, all four call sites (Between, BetweenExcludeBounds, BetweenExcludeLeft, BetweenExcludeRight) pass distinct operator labels, and the function silences it with let _ = op_name; at line 79 inside the only error path. The label never reaches the "range lower bound must be <= upper bound" error message, so operator context is lost. Either remove the parameter (and the four arguments at the call sites) or weave the label into the error string. A dead parameter that exists only to suppress the unused-arg warning is worse than either alternative.
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.
In `packages/rs-sdk-ffi/src/document/queries/count.rs`:
- [SUGGESTION] lines 62-76: Unified C FFI count entry point can't express Between* range operators
`dash_sdk_document_count` is the sole native count entry point and its docs advertise generic range-count behavior, but `parse_where_operator` only accepts `=`, `>`, `>=`, `<`, `<=`, `in`, and `startsWith` — it rejects `Between`, `BetweenExcludeBounds`, `BetweenExcludeLeft`, and `BetweenExcludeRight`. The server, wasm bindings, and `range_clause_to_query_item` in path_query.rs all support these. iOS/native callers can't issue four of the range forms this PR adds verified fast paths for, despite the FFI being the unified surface.
In `packages/rs-sdk/src/platform/documents/document_count_query.rs`:
- [SUGGESTION] lines 57-123: DocumentCountQuery rustdoc still describes pre-fix semantics for distinct-range and limit
The field and builder docs (lines 57–74, 111–122) say `return_distinct_counts_in_range` is "only meaningful... when the request goes through a no-proof transport — the proof endpoint rejects this combination" and that oversized `limit` values are silently clamped ("Server clamps... larger values are silently reduced"). Both are stale at this head: the SDK has a dedicated `RangeDistinctProof` verifier path (document_count_query.rs:423-495) and the server explicitly rejects prove-distinct requests where `limit > max_query_limit` via `Error::Query(QuerySyntaxError::InvalidLimit(...))` (drive_dispatcher.rs:806-816). Callers reading these docs will misdiagnose `InvalidLimit` errors and assume distinct-mode is unreachable on Fetch.
- [SUGGESTION] lines 146-170: DocumentCountQuery hard-codes prove=true and ignores SdkBuilder::with_proofs(false)
`Fetch::fetch_with_metadata_and_proof` calls `query.query(sdk.prove())`, so callers reasonably expect `SdkBuilder::with_proofs(false)` to flip the request mode. But `DocumentCountQuery` doesn't implement `Query<GetDocumentsCountRequest>` and falls back to the blanket `impl Query<T> for T`, which returns `self`; the `TryFrom<DocumentCountQuery>` impl then unconditionally sets `prove: true` (line 166). Net effect: count queries silently ignore the SDK-wide proof toggle, divergent from the rest of the query surface, and the Rust SDK still has no typed way to reach the endpoint's no-proof modes even though transport and server support them.
In `packages/rs-sdk/tests/fetch/document_count.rs`:
- [SUGGESTION] lines 1-89: No rs-sdk Fetch coverage for DocumentSplitCounts or RangeDistinctProof verifier
This test file only mocks `DocumentCount` fetches. There is no rs-sdk-level test covering `DocumentSplitCounts::fetch`, the `(In + prove)` grouping branch, or the `RangeDistinctProof` verifier path in document_count_query.rs:345-503 — branches with bespoke SDK-side request-construction and proof-reconstruction logic (DEFAULT_QUERY_LIMIT anchor, in_key path-segment grouping, total-mode empty-key re-emit) that drive/drive-abci tests don't exercise from the client seam. This is the same gap that let earlier SDK regressions slip on this PR. At minimum add one mocked round-trip for `DocumentSplitCounts::fetch` with an `In` clause and one for `with_distinct_counts_in_range(true)` on a range query.
Inline review posting was skipped because GitHub refused the PR diff (HTTP 406: diff exceeded 20,000 lines), so I posted the verified findings as a top-level review body.
…surface CodeRabbit re-review at ba9a797 surfaced 10 follow-ups, all 🟡 suggestion / 💬 nitpick (reviewer's own framing: "no blocking issues remain"). This sweep addresses them. **Stale documentation (5 nitpicks, 1 suggestion).** Six callsites still described the deleted materialize-and-count walker or pre-fix server semantics: - `PointLookupProof` variant docstring (mod.rs:146-149) — rewrote to describe the CountTree-element proof path, dropping the bogus "Capped at u16::MAX matching docs" claim. - `PointLookupProof` dispatch comment (mode_detection.rs:167-178) — no document materialization, no u16::MAX bound, no per-document grouping. Updated to the actual flow. - drive-abci test `test_documents_count_with_in_and_prove_returns_proof` docstring — removed the `MissingOrderByForRange` failure-mode claim; the In+prove path no longer reads `order_by`. - Proto `order_by` comment — was "required... materialize walker needs deterministic walk order"; now describes the actual contract: load-bearing only on `RangeDistinctProof` (with empty-`order_by` defaulting to ascending on both sides), ignored on `PointLookupProof` (which sorts In keys lex-asc unconditionally). - Proto `limit` comment — was "Server clamps"; rewrote to validate-don't-clamp on prove (`InvalidLimit` rejection) with the consensus-deterministic `DEFAULT_QUERY_LIMIT` fallback rationale. - `DocumentCountQuery` rustdoc (field + builder docs) — was "proof endpoint rejects this combination" and "silently reduced"; updated to reflect `RangeDistinctProof` verifier path + validate-don't-clamp. **Dead `op_name` parameter in `serialize_pair` (1 nitpick).** Removed the parameter and its four call-site argument labels — `InvalidWhereClauseComponents` takes `&'static str` so threading runtime operator labels through error messages was structurally blocked; the dead-parameter-to-suppress-warning pattern was the worst of both options. Removed cleanly. **FFI Between\* operators (1 suggestion).** `parse_where_operator` in `rs-sdk-ffi/src/document/queries/count.rs` now accepts `between`, `betweenExcludeBounds`, `betweenExcludeLeft`, `betweenExcludeRight` (plus kebab/snake-case aliases for convenience). iOS/Swift callers can now issue every range form drive's `range_clause_to_query_item` supports — previously the FFI rejected 4 of the 7 range variants this PR adds verified fast paths for. **SDK proof-flag divergence (1 suggestion).** `DocumentCountQuery` unconditionally sets `prove: true` on the wire request because the SDK `Fetch` path goes through `FromProof`, which only decodes the `Proof(...)` response variant. Made the divergence loud at the call site: rewrote the comment to explicitly state that `SdkBuilder::with_proofs(false)` is a no-op for count queries, log a warning at fetch time (via the existing blanket `Query<T> for T` impl), and call out the missing no-proof decoder as a tracked follow-up. The current behavior is security-up (always verified) — silent-ignore of the prove flag is the wrong shape, but quietly downgrading verification would be worse. **rs-sdk fetch test coverage (1 suggestion).** Added two mock-based fetch tests in `tests/fetch/document_count.rs`: - `test_mock_fetch_document_split_counts_with_in_clause` — pins the SDK seam for `(In, prove=true, no-range)` queries routed to `PointLookupProof` on the server. Mocks the per-In-value entries (`in_key: None`, `key: <serialized>`) that `verify_point_lookup_count_proof` produces, and asserts `DocumentSplitCounts::fetch` returns them unchanged. Also verifies `MockResponse for DocumentSplitCounts` round-trips the `Vec<SplitCountEntry>` shape. - `test_mock_fetch_document_split_counts_with_distinct_range` — pins the SDK seam for `(range, prove=true, distinct=true)` queries routed to `RangeDistinctProof`. Builds a `with_distinct_counts_in_range(true)` + `with_limit(Some(50))` + `with_order_by(desc)` query, asserts both knobs reach the wire request, and confirms per-distinct-value entries round-trip with the requested direction preserved. Both tests close the SDK-only regression gap the reviewer flagged: prior SDK regressions on this PR (In+prove silent verify failure, RangeDistinctProof limit mismatch) escaped local checks because no rs-sdk-level test exercised the `DocumentSplitCounts::fetch` or distinct-range paths from the client seam. Tests: - drive lib: 36/36 (unchanged) - dash-sdk fetch: 5/5 (3 existing + 2 new) - clippy clean across drive, dash-sdk, drive-abci, rs-sdk-ffi
CodeRabbit review triage — pushed `0f52734820`Worked through all 10 follow-ups. Reviewer flagged the set as "no blocking issues remain"; this sweep is doc / FFI surface / test coverage. 🟢 Fixed at `0f52734820`Stale documentation (6 sites):
#10 Dead `op_name` parameter in `serialize_pair`: removed. `InvalidWhereClauseComponents` takes `&'static str` so threading runtime operator labels through error messages was structurally blocked; the dead-parameter-to-suppress-warning pattern was the worst of both options. Removed cleanly. #1 FFI `Between*` operators: `parse_where_operator` in `rs-sdk-ffi/src/document/queries/count.rs` now accepts `between`, `betweenExcludeBounds`, `betweenExcludeLeft`, `betweenExcludeRight` (plus kebab/snake-case aliases). iOS/Swift callers can now issue every range form drive's `range_clause_to_query_item` supports. #3 SDK proof-flag divergence: made loud rather than silent. `DocumentCountQuery` still hardcodes `prove: true` (because the SDK `Fetch` path goes through `FromProof`, which only decodes the `Proof(...)` response variant), but the comment now explicitly states `SdkBuilder::with_proofs(false)` is a no-op for count queries. The current behavior is security-up (always verified) — quietly downgrading verification would be worse than the loud no-op. The missing no-proof decoder is tracked as a follow-up. #4 rs-sdk Fetch coverage for `DocumentSplitCounts` + `RangeDistinctProof`: added two mock-based tests in `tests/fetch/document_count.rs`:
Both close the SDK-only regression gap the reviewer called out — prior session regressions slipped through because no rs-sdk-level test exercised these paths from the client seam. Test posture at HEAD
|
thepastaclaw
left a comment
thepastaclaw
left a comment
There was a problem hiding this comment.
Code Review
Server-side count-query refactor is in good shape; the polish commit closed almost every prior finding. One remaining security concern (unbounded compound range+In no-proof walk on a public endpoint) and a cluster of stale documentation in the FFI/wasm-sdk/dispatcher wrappers that didn't catch up with the PointLookupProof rewrite. Claude's claim that the with_proofs(false) warning is not wired is a false positive — the blanket Query impl in packages/rs-sdk/src/platform/query.rs does emit the warning, so that finding is dropped and replaced with Codex's milder framing that the prove flag is hard-coded.
Reviewed commit: 0f52734
🔴 1 blocking | 🟡 2 suggestion(s) | 💬 4 nitpick(s)
7 additional findings
🔴 blocking: Compound `range + In` no-proof summed counts walk the index without limit
packages/rs-drive/src/query/drive_document_count_query/execute_range_count.rs (lines 124-172)
detect_mode accepts (has_range=true, has_in=true, prove=false, distinct=false) as RangeNoProof (mode_detection.rs:147-165) and the dispatcher routes it into execute_document_count_range_no_proof while only clamping request.limit (drive_dispatcher.rs:739-756). Inside the executor, the compound summed branch cannot use AggregateCountOnRange and falls through to the walk-and-sum path, which explicitly sets path_query_limit = None (execute_range_count.rs:152-156) before calling grove_get_raw_path_query and materializing every matched (in_key, key, element) triple (lines 161-172, 198-208). The inline comment at 141-145 acknowledges the DoS is bounded only by the contract author's index choice. A tiny getDocumentsCount request with one broad range plus 100 In values can force a full index walk while the response stays a single aggregate u64 — classic request-amplification on a public unauthenticated DAPI endpoint. The clamp on request.limit does nothing on this branch because the limit is dropped before the path query is built. Either honour options.limit (or max_query_limit) as a hard cap on the underlying walk, or reject compound summed mode and require return_distinct_counts_in_range = true when both In and a range are present without a proof.
🟡 suggestion: `DocumentCountQuery` hard-codes `prove=true`, ignoring SdkBuilder::with_proofs(false)
packages/rs-sdk/src/platform/documents/document_count_query.rs (lines 169-210)
Fetch::fetch_with_metadata_and_proof (packages/rs-sdk/src/platform/fetch.rs:161) threads sdk.prove() into query.query(...), so callers reasonably expect SdkBuilder::with_proofs(false) to govern every fetch surface. DocumentCountQuery instead converts directly in TryFrom<DocumentCountQuery> for GetDocumentsCountRequest and unconditionally sets prove: true (line 206), with the doc-block at 185-205 explicitly calling this out as a no-op for the SDK-wide setting. The blanket Query<T> for T impl at packages/rs-sdk/src/platform/query.rs:119-124 does emit the warning the comment promises, but the SDK still has no typed entry point for the server's already-supported no-proof count modes, so callers who deliberately set with_proofs(false) get a fully-proven response with only a tracing::warn! to indicate the divergence. Either wire a no-proof decoder (FromProof/transport split mirroring other count modes) so the SDK can reach the cheaper backend variants, or surface this as a hard error so the discrepancy is observable rather than buried in a log line.
🟡 suggestion: rs-sdk count tests still don't exercise real proof-byte verifier paths
packages/rs-sdk/tests/fetch/document_count.rs (lines 15-19)
The new fetch tests use MockResponse decoding, so they cover request construction and response-shape decoding but not the verifier-side path-query reconstruction added in DocumentCountQuery's FromProof impl (PointLookupProof, RangeProof, RangeDistinctProof). Earlier regressions on this PR (compile-time vs runtime DEFAULT_QUERY_LIMIT, divergent drive_query.limit) were exactly at that prover/verifier boundary and weren't catchable by mock-only tests. Add at least one recorded-vector or live-platform SDK test for DocumentSplitCounts::fetch on (In + prove) and on with_distinct_counts_in_range(true) so the byte-equality contract between the server builder and the SDK verifier is covered from the SDK seam.
💬 nitpick: FFI count docstring still describes the removed `(In + prove)` orderBy requirement
packages/rs-sdk-ffi/src/document/queries/count.rs (lines 223-237)
The # Tunables block on dash_sdk_document_count says order_by_json is load-bearing for (In + prove) walk determinism and that the server rejects empty orderBy on that arm. Both claims are stale at this head: mode_detection.rs:167-179 routes (In, prove, no-range) to PointLookupProof, which point_lookup_count_path_query builds with lex-ascending In key ordering unconditionally — order_by is not consulted and there's no rejection on empty orderBy from this arm. Only RangeDistinctProof actually requires the SDK and server to agree on the orderBy direction. The matching proto / rs-sdk rustdoc surfaces were updated in this polish commit; the FFI wrapper wasn't.
💬 nitpick: wasm-sdk `getDocumentsCount` docs cite the deleted materialize-and-count walker and miss validate-don't-clamp
packages/wasm-sdk/src/queries/document.rs (lines 537-543)
Two stale claims: (1) orderBy is described as 'Required when the where carries an In or range operator on a prove path (the materialize-and-count walker needs an explicit order for proof determinism)' — the materialize-and-count walker is gone, (In, prove, no-range) routes to PointLookupProof which sorts In keys lex-ascending and never reads order_by, and RangeDistinctProof defaults to ascending when omitted. (2) limit is described as 'server clamps to its max_query_limit' — true on no-proof paths only; on prove paths the server returns Error::Query(QuerySyntaxError::InvalidLimit(..)) (drive_dispatcher.rs:809-816). The proto and rs-sdk rustdoc carry the correct contract after this commit's sweep; the wasm-sdk wrapper wasn't updated in lockstep.
💬 nitpick: `DocumentCountRequest` field docs still describe the deleted materialize-and-count fallback
packages/rs-drive/src/query/drive_document_count_query/drive_dispatcher.rs (lines 483-524)
The doc block on DocumentCountRequest and its raw_where_value/raw_order_by_value fields (lines 489-524) says both values are 'forwarded raw to DriveDocumentQuery::from_decomposed_values' for the 'materialize-and-count fallback (PointLookupProof)'. After the PointLookupProof rewrite, PointLookupProof proves the per-branch CountTree elements via point_lookup_count_path_query; grep from_decomposed_values in this module only returns the stale comments themselves. The dispatcher at lines 671-672 parses both raw values once into structured WhereClause/OrderClauses and never threads the raw values further. The inline comment at 676-678 ('also threaded through to the materialize path (PointLookupProof) for proof-walk determinism') repeats the same stale claim — the PointLookupProof arm at line 840 doesn't read order_by at all. Update the field docs and the inline comment.
💬 nitpick: Silent u32→u16 truncation on verifier-side limit reconstruction
packages/rs-sdk/src/platform/documents/document_count_query.rs (lines 513-516)
request.limit.map(|l| l as u16).unwrap_or(drive::config::DEFAULT_QUERY_LIMIT) silently truncates if a caller ever passes limit > u16::MAX. The server-side guard in drive_dispatcher.rs:809-816 rejects with InvalidLimit before any proof is produced (since max_query_limit is itself a u16), so this is currently fail-loud rather than exploitable. But it's still an in-builder discrepancy that would silently break verification on any future code path where a u32 larger than u16::MAX reaches the verifier. Defense in depth: use u16::try_from(l) and surface the conversion error as Error::RequestError so the failure mode stays explicit rather than depending on the symmetric server-side guard.
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.
In `packages/rs-drive/src/query/drive_document_count_query/execute_range_count.rs`:
- [BLOCKING] lines 124-172: Compound `range + In` no-proof summed counts walk the index without limit
`detect_mode` accepts `(has_range=true, has_in=true, prove=false, distinct=false)` as `RangeNoProof` (mode_detection.rs:147-165) and the dispatcher routes it into `execute_document_count_range_no_proof` while only clamping `request.limit` (drive_dispatcher.rs:739-756). Inside the executor, the compound summed branch cannot use `AggregateCountOnRange` and falls through to the walk-and-sum path, which explicitly sets `path_query_limit = None` (execute_range_count.rs:152-156) before calling `grove_get_raw_path_query` and materializing every matched `(in_key, key, element)` triple (lines 161-172, 198-208). The inline comment at 141-145 acknowledges the DoS is bounded only by the contract author's index choice. A tiny `getDocumentsCount` request with one broad range plus 100 In values can force a full index walk while the response stays a single aggregate u64 — classic request-amplification on a public unauthenticated DAPI endpoint. The clamp on `request.limit` does nothing on this branch because the limit is dropped before the path query is built. Either honour `options.limit` (or `max_query_limit`) as a hard cap on the underlying walk, or reject compound summed mode and require `return_distinct_counts_in_range = true` when both `In` and a range are present without a proof.
In `packages/rs-sdk/src/platform/documents/document_count_query.rs`:
- [SUGGESTION] lines 169-210: `DocumentCountQuery` hard-codes `prove=true`, ignoring SdkBuilder::with_proofs(false)
`Fetch::fetch_with_metadata_and_proof` (`packages/rs-sdk/src/platform/fetch.rs:161`) threads `sdk.prove()` into `query.query(...)`, so callers reasonably expect `SdkBuilder::with_proofs(false)` to govern every fetch surface. `DocumentCountQuery` instead converts directly in `TryFrom<DocumentCountQuery> for GetDocumentsCountRequest` and unconditionally sets `prove: true` (line 206), with the doc-block at 185-205 explicitly calling this out as a no-op for the SDK-wide setting. The blanket `Query<T> for T` impl at `packages/rs-sdk/src/platform/query.rs:119-124` does emit the warning the comment promises, but the SDK still has no typed entry point for the server's already-supported no-proof count modes, so callers who deliberately set `with_proofs(false)` get a fully-proven response with only a `tracing::warn!` to indicate the divergence. Either wire a no-proof decoder (`FromProof`/transport split mirroring other count modes) so the SDK can reach the cheaper backend variants, or surface this as a hard error so the discrepancy is observable rather than buried in a log line.
In `packages/rs-sdk/tests/fetch/document_count.rs`:
- [SUGGESTION] lines 15-19: rs-sdk count tests still don't exercise real proof-byte verifier paths
The new fetch tests use `MockResponse` decoding, so they cover request construction and response-shape decoding but not the verifier-side path-query reconstruction added in `DocumentCountQuery`'s `FromProof` impl (`PointLookupProof`, `RangeProof`, `RangeDistinctProof`). Earlier regressions on this PR (compile-time vs runtime DEFAULT_QUERY_LIMIT, divergent `drive_query.limit`) were exactly at that prover/verifier boundary and weren't catchable by mock-only tests. Add at least one recorded-vector or live-platform SDK test for `DocumentSplitCounts::fetch` on `(In + prove)` and on `with_distinct_counts_in_range(true)` so the byte-equality contract between the server builder and the SDK verifier is covered from the SDK seam.
GitHub refused the PR diff (HTTP 406: diff exceeded 20,000 lines), so I posted the verified findings in the top-level review body.
…ate fan-out **🔴 Security fix.** CodeRabbit flagged the compound `range + In` no-proof summed mode as an unbounded walk on a public DAPI endpoint: - `detect_mode` routes `(range, In, prove=false, distinct=false)` to `RangeNoProof`. - The dispatcher's clamp on `request.limit` is dropped before building the path query: `let path_query_limit = None` on the compound-summed branch. - `distinct_count_path_query(None, ...)` plus `grove_get_raw_path_query` then materializes every matched `(in_key, key, element)` triple to sum in Rust — even though the response stays a single aggregate `u64`. - A small request (one broad range × 100 In values) forces a full index walk: request-amplification on a public unauthenticated endpoint. **Fix.** Replace the walk-and-sum with a per-In-value aggregate fan-out. For each In value: 1. Substitute the In clause with `Equal(value)` to produce a flat where-clauses set. 2. Build the path query via `aggregate_count_path_query` (which the substitution now satisfies). 3. Call `grove.query_aggregate_count` — single u64, O(log n) via merk boundary nodes. 4. Sum the per-value results. Total bound: O(|In| × log n) where `|In| ≤ 100` (enforced by `WhereClause::in_values()`). Same correctness as the walk-and-sum (both produce the unmerged aggregate sum), drastically tighter worst-case work, no dependence on contract author's index choice for DoS budget. Distinct mode (`distinct=true`, with or without In) keeps the existing walk-and-emit path because: - The output shape is per-`(in_key, key)` entries, not a single sum — the per-In-value aggregate doesn't fit. - The path query carries `options.limit` (clamped to `max_query_limit` upstream), so the walk is already bounded. **Regression test.** `test_compound_range_in_summed_no_proof_uses_per_in_aggregate_fanout` constructs a `[brand, color]` range_countable index with mixed in-range / out-of-range entries across three brands and asserts the sum returned by `execute_document_count_request` matches the known-good count for `brand IN ["acme","contoso"] AND color > "blue"` (= 6). Functional pin; the DoS-bound pin is structural (the per-In loop calling `query_aggregate_count`, not `query_raw`). **Stale doc cleanup (the rest of the review batch).** - `DocumentCountRequest` field docs and the inline dispatcher comment at line 673 no longer claim `raw_*_value` are "forwarded raw to `DriveDocumentQuery::from_decomposed_values` for the materialize-and-count fallback" — that fallback was deleted earlier in this PR, and the dispatcher parses both raw values once via `where_clauses_from_value` / `order_clauses_from_value`. - FFI `dash_sdk_document_count` `# Tunables` block updated to reflect (a) `PointLookupProof` doesn't read `order_by` (the builder sorts In keys lex-asc unconditionally); only `RangeDistinctProof` consumes it. (b) prove-distinct path rejects `InvalidLimit` rather than silently clamping; unset falls back to the compile-time `DEFAULT_QUERY_LIMIT` constant. - wasm-sdk `getDocumentsCount` rustdoc gets the same updates. **SDK proof-flag (#2 carryover from prior review).** The blanket `Query<T> for T` impl warns when `prove=false`; the `DocumentCountQuery` still ships `prove: true` because the SDK has no no-proof decoder yet. Shadowing the blanket impl to intercept the flag is blocked by Rust's coherence rules (`DocumentCountQuery` IS its own `TransportRequest`). Filed #3630 to track wiring a typed no-proof decoder; inline comment now references the issue explicitly. **u32→u16 limit truncation (nitpick).** `FromProof<DocumentCountQuery> for DocumentSplitCounts` was silently truncating `request.limit` to `u16` via `as u16`. Switched to `u16::try_from` returning a loud `Error::RequestError` on overflow — currently fail-loud rather than exploitable (server-side max_query_limit cap already fits in u16), but defense-in-depth keeps the failure mode explicit if a future code path widens the wire limit. Tests: - drive lib: 37/37 (incl. new compound-summed test) - dash-sdk fetch: 5/5 - drive-abci document_count: 9/9 - clippy clean across drive, dash-sdk, drive-abci, rs-sdk-ffi
CodeRabbit review triage — pushed `7065dc4c23`Worked through all 7 follow-ups. The 🔴 blocking finding was a real DoS — fixed structurally rather than papered over. 🟢 BLOCKING fixed: compound `range + In` unbounded walkConfirmed real and live. `detect_mode` routes `(range, In, prove=false, distinct=false)` to `RangeNoProof` summed mode; the executor explicitly dropped the request limit (`path_query_limit = None`) and then walked every matched `(in_key, key, element)` triple via `query_raw` to sum in Rust. One broad range × 100 In values could scan millions of CountTree elements for a response that's still a single aggregate `u64` — classic request-amplification on a public unauthenticated DAPI endpoint. Fix: replace the walk-and-sum with a per-In-value aggregate fan-out. For each In value, substitute `In → Equal`, build via `aggregate_count_path_query`, call `grove.query_aggregate_count` (O(log n) via merk boundary nodes), sum the per-value results. Total bound: O(|In| × log n) where `|In| ≤ 100` (enforced by `WhereClause::in_values()`). Same correctness as the walk-and-sum (both produce the unmerged aggregate). DoS budget no longer depends on the contract author's index choice. Distinct mode keeps the walk-and-emit because (a) the output is per-`(in_key, key)` entries, not a single sum — per-In aggregate doesn't fit; (b) the path query already carries `options.limit` (clamped to `max_query_limit` upstream), so the walk is bounded. Regression test: `test_compound_range_in_summed_no_proof_uses_per_in_aggregate_fanout` constructs a known distribution and asserts the per-In fan-out returns the correct sum (6 for `brand IN [acme,contoso] AND color > blue` against the test data). 🟢 Stale doc cleanup (all 4 nitpicks)
🟡 Carryover suggestion #2 (SDK proof flag)Reviewer acknowledged the warning IS wired ("Codex's milder framing"). The structural "hard-error or wire a no-proof decoder" is blocked by Rust's coherence rules — shadowing the blanket `Query for T` impl would need either:
Both are bigger changes than fit this PR. Filed #3630 tracking the no-proof decoder path with the structural analysis. Inline comment in `document_count_query.rs` now references it explicitly. 🟡 Carryover suggestion #3 (real proof-byte SDK tests)Reviewer's point is fair — the mock-based tests cover request shaping + response decoding but not the byte-equality contract between server builder and SDK verifier. Real coverage requires:
Neither is structural. Tracking as a follow-up (will file shortly). The drive-level pin `test_range_distinct_proof_uses_compile_time_default_query_limit_not_operator_config` already covers the most consensus-sensitive byte-equality property by running the server-side path query and verifying it through `GroveDb::verify_query` (the same primitive the SDK wraps). That's why the prior limit-mismatch regression got caught — but the reviewer is right that an SDK-layer test would catch SDK-only regressions in request construction earlier. Test posture at HEAD
|
thepastaclaw
left a comment
thepastaclaw
left a comment
There was a problem hiding this comment.
Code Review
Re-verification at 7065dc4. The compound range + In no-proof DoS is correctly closed by a per-In aggregate fan-out bounded at O(|In|×log n) with |In|≤100, and the verifier-limit truncation seam was hardened via u16::try_from. Two new Codex findings are confirmed on the SDK/dispatcher seam: DocumentCount::fetch still funnels every range query through the aggregate verifier even when with_distinct_counts_in_range(true) is set, and the unified count parser bypasses WhereClause::group_clauses so duplicate/conflicting clauses are silently accepted and arbitrarily reduced. Two non-blocking carryovers remain (hard-coded prove=true and mock-only SDK tests).
Reviewed commit: 7065dc4
🟡 4 suggestion(s)
4 additional findings
🟡 suggestion: `DocumentCount::fetch` misroutes prove + distinct-range queries into the aggregate verifier
packages/rs-sdk/src/platform/documents/document_count_query.rs (lines 285-335)
DocumentCountQuery exposes with_distinct_counts_in_range(true) and is the request type backing DocumentCount::fetch, but FromProof<DocumentCountQuery> for DocumentCount (lines 285–335) routes every where-clause set containing a range operator through verify_aggregate_count_proof, ignoring request.return_distinct_counts_in_range. Server-side, detect_mode (packages/rs-drive/src/query/drive_document_count_query/mode_detection.rs:147-159) routes (has_range=true, prove=true, distinct=true) to RangeDistinctProof, which emits a regular range proof of per-key KVCount entries — not an AggregateCountOnRange proof. The aggregate verifier rebuilds a different PathQuery, so verification will fail rather than sum the verified per-key counts. The companion FromProof<DocumentCountQuery> for DocumentSplitCounts already gates on request.return_distinct_counts_in_range (line 473) and dispatches to verify_distinct_count_proof; mirror that branch in DocumentCount (either delegate to the distinct verifier and sum, or reject the shape with a clear error directing callers to DocumentSplitCounts::fetch).
🟡 suggestion: Unified count parsing skips `WhereClause::group_clauses` validation and silently reduces conflicting clauses
packages/rs-drive/src/query/drive_document_count_query/drive_dispatcher.rs (lines 579-597)
where_clauses_from_value parses each inner array into a WhereClause and hands the raw Vec directly to detect_mode and the index pickers — bypassing the duplicate/conflict checks that WhereClause::group_clauses enforces on the regular query path (packages/rs-drive/src/query/conditions.rs:559-690: DuplicateNonGroupableClauseSameField for two equality clauses on the same field, MultipleInClauses, MultipleRangeClauses, equality+In on the same field, etc.). Downstream, find_countable_index_for_where_clauses collapses repeated fields into a BTreeSet (index_picker.rs:44-48) and point_lookup_count_path_query resolves each index property with .find(...) (path_query.rs:592-605) — so a request with two conflicting Equal clauses on field matches a single-prop countable index and the executor silently picks the first clause, returning a count for an arbitrarily reduced query rather than rejecting the request. Run the parsed clauses through WhereClause::group_clauses (or an equivalent validator) before mode detection so the count endpoint rejects the same malformed shapes the document-query path does.
🟡 suggestion: `DocumentCountQuery` hard-codes `prove: true` and silently ignores `SdkBuilder::with_proofs(false)`
packages/rs-sdk/src/platform/documents/document_count_query.rs (lines 185-216)
TryFrom<DocumentCountQuery> for GetDocumentsCountRequest sets prove: true unconditionally (line 216), so SdkBuilder::with_proofs(false) is a no-op for every count fetch — the request still ships with proofs and is decoded via FromProof<DocumentCountQuery>. The blanket Query<T> for T impl in packages/rs-sdk/src/platform/query.rs:119-124 emits a tracing::warn! when prove=false is requested, and the inline comment at lines 197-215 now references #3630 to track wiring a typed no-proof decoder, but the SDK seam still violates the cross-fetch contract callers reasonably expect. Carryover from prior review — non-blocking, flagged so it stays on the radar until #3630 lands.
🟡 suggestion: rs-sdk count fetch tests stop at the mock transport and don't exercise proof-byte verifier paths
packages/rs-sdk/tests/fetch/document_count.rs (lines 1-19)
The test module header explicitly notes that the mock transport short-circuits the wire-level verifier, so coverage stops at query builder → TryInto<GetDocumentsCountRequest> → mock match → MockResponse decode. The load-bearing logic in FromProof<DocumentCountQuery> (the AggregateCountOnRange branch, the PointLookupProof branch, and the RangeDistinctProof verifier reconstruction) is therefore untested from the SDK seam. The prior cluster of regressions on this PR (compile-time vs runtime DEFAULT_QUERY_LIMIT, drive_query.limit divergence, prove+In broken end-to-end) all manifested at the prover/verifier boundary and could not have been caught from mocks. Add at least one recorded-vector or live-platform round-trip for DocumentSplitCounts::fetch on (In + prove) and one for with_distinct_counts_in_range(true) once the testnet path is available — the PR description already lists this as a follow-up assigned to @lklimek.
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.
In `packages/rs-sdk/src/platform/documents/document_count_query.rs`:
- [SUGGESTION] lines 285-335: `DocumentCount::fetch` misroutes prove + distinct-range queries into the aggregate verifier
`DocumentCountQuery` exposes `with_distinct_counts_in_range(true)` and is the request type backing `DocumentCount::fetch`, but `FromProof<DocumentCountQuery> for DocumentCount` (lines 285–335) routes every where-clause set containing a range operator through `verify_aggregate_count_proof`, ignoring `request.return_distinct_counts_in_range`. Server-side, `detect_mode` (`packages/rs-drive/src/query/drive_document_count_query/mode_detection.rs:147-159`) routes `(has_range=true, prove=true, distinct=true)` to `RangeDistinctProof`, which emits a regular range proof of per-key `KVCount` entries — not an `AggregateCountOnRange` proof. The aggregate verifier rebuilds a different `PathQuery`, so verification will fail rather than sum the verified per-key counts. The companion `FromProof<DocumentCountQuery> for DocumentSplitCounts` already gates on `request.return_distinct_counts_in_range` (line 473) and dispatches to `verify_distinct_count_proof`; mirror that branch in `DocumentCount` (either delegate to the distinct verifier and sum, or reject the shape with a clear error directing callers to `DocumentSplitCounts::fetch`).
- [SUGGESTION] lines 185-216: `DocumentCountQuery` hard-codes `prove: true` and silently ignores `SdkBuilder::with_proofs(false)`
`TryFrom<DocumentCountQuery> for GetDocumentsCountRequest` sets `prove: true` unconditionally (line 216), so `SdkBuilder::with_proofs(false)` is a no-op for every count fetch — the request still ships with proofs and is decoded via `FromProof<DocumentCountQuery>`. The blanket `Query<T> for T` impl in `packages/rs-sdk/src/platform/query.rs:119-124` emits a `tracing::warn!` when `prove=false` is requested, and the inline comment at lines 197-215 now references dashpay/platform#3630 to track wiring a typed no-proof decoder, but the SDK seam still violates the cross-fetch contract callers reasonably expect. Carryover from prior review — non-blocking, flagged so it stays on the radar until #3630 lands.
In `packages/rs-drive/src/query/drive_document_count_query/drive_dispatcher.rs`:
- [SUGGESTION] lines 579-597: Unified count parsing skips `WhereClause::group_clauses` validation and silently reduces conflicting clauses
`where_clauses_from_value` parses each inner array into a `WhereClause` and hands the raw `Vec` directly to `detect_mode` and the index pickers — bypassing the duplicate/conflict checks that `WhereClause::group_clauses` enforces on the regular query path (`packages/rs-drive/src/query/conditions.rs:559-690`: `DuplicateNonGroupableClauseSameField` for two equality clauses on the same field, `MultipleInClauses`, `MultipleRangeClauses`, equality+`In` on the same field, etc.). Downstream, `find_countable_index_for_where_clauses` collapses repeated fields into a `BTreeSet` (`index_picker.rs:44-48`) and `point_lookup_count_path_query` resolves each index property with `.find(...)` (`path_query.rs:592-605`) — so a request with two conflicting `Equal` clauses on `field` matches a single-prop countable index and the executor silently picks the first clause, returning a count for an arbitrarily reduced query rather than rejecting the request. Run the parsed clauses through `WhereClause::group_clauses` (or an equivalent validator) before mode detection so the count endpoint rejects the same malformed shapes the document-query path does.
In `packages/rs-sdk/tests/fetch/document_count.rs`:
- [SUGGESTION] lines 1-19: rs-sdk count fetch tests stop at the mock transport and don't exercise proof-byte verifier paths
The test module header explicitly notes that the mock transport short-circuits the wire-level verifier, so coverage stops at `query builder → TryInto<GetDocumentsCountRequest> → mock match → MockResponse decode`. The load-bearing logic in `FromProof<DocumentCountQuery>` (the `AggregateCountOnRange` branch, the `PointLookupProof` branch, and the `RangeDistinctProof` verifier reconstruction) is therefore untested from the SDK seam. The prior cluster of regressions on this PR (compile-time vs runtime `DEFAULT_QUERY_LIMIT`, `drive_query.limit` divergence, prove+In broken end-to-end) all manifested at the prover/verifier boundary and could not have been caught from mocks. Add at least one recorded-vector or live-platform round-trip for `DocumentSplitCounts::fetch` on `(In + prove)` and one for `with_distinct_counts_in_range(true)` once the testnet path is available — the PR description already lists this as a follow-up assigned to @lklimek.
GitHub refused the PR diff (HTTP 406: diff exceeded 20,000 lines), so I posted the verified findings in the top-level review body.
…er + reject malformed clauses Two real bugs CodeRabbit surfaced at 7065dc4: **1. DocumentCount::fetch misrouted prove + distinct-range queries.** `FromProof<DocumentCountQuery> for DocumentCount` routed every where-clause set containing a range operator through `verify_aggregate_count_proof`, ignoring `request.return_distinct_counts_in_range`. Server-side, `detect_mode` routes `(range, prove=true, distinct=true)` to `RangeDistinctProof` which emits per-key `KVCount` ops, not an `AggregateCountOnRange` aggregate proof. The two proof shapes embed structurally different `PathQuery`s and the aggregate verifier rebuilds the wrong one — verification fails outright rather than producing a wrong-but-plausible answer. Fix: dispatch on `return_distinct_counts_in_range` in the range branch. When `true`, rebuild the same path query the SDK's `DocumentSplitCounts` verifier uses (with `DEFAULT_QUERY_LIMIT` fallback + first-orderBy direction), call `verify_distinct_count_proof`, and sum the verified per-key counts to produce the single aggregate `DocumentCount` returns. The per-key counts are merk-root-bound via `node_hash_with_count` in the proof, so the sum is cryptographically committed — same forge-resistance as `AggregateCountOnRange`, just expressed as a post-verification reduction. `DocumentSplitCounts`'s FromProof already had this dispatch (line 473 pre-fix); now `DocumentCount`'s mirrors it. **2. `where_clauses_from_value` skipped `group_clauses` validation.** The unified count parser handed the parsed `Vec<WhereClause>` straight to `detect_mode` and the index pickers without running it through `WhereClause::group_clauses` — the system-wide validator the regular document-query path uses. Effect: malformed clause shapes the rest of the query stack rejects were silently accepted on the count endpoint: - Two `Equal` clauses on the same field (e.g. `firstName == "Alice" AND firstName == "Bob"`): `find_countable_index_for_where_clauses` collapses repeated fields into a `BTreeSet`, `point_lookup_count_path_query` resolves each property with `.find(...)` — both silently pick the first clause, returning a count for an arbitrarily reduced query. - Multiple `In` clauses, multiple range clauses, equality + In on the same field, range + equality/In on the same field — same silent reduction. Fix: run `WhereClause::group_clauses` on the parsed clauses inside `where_clauses_from_value` and propagate any validation error. The returned `(equal_clauses, in_clause, range_clause)` triple is discarded — the count path operates on the flat list, not the regular query path's `InternalClauses` triple — but the validation side-effect aligns the count endpoint's rejection contract with the document-query path's. **Regression tests:** - `test_count_request_with_duplicate_equality_clauses_is_rejected` (drive lib): constructs a request with two conflicting `Equal` clauses on `firstName`, runs through `execute_document_count_request`, asserts a `DuplicateNonGroupableClauseSameField` error. Without the validator call, the request would silently return a count for one of the two clauses depending on iteration order. - `test_mock_fetch_document_count_with_distinct_range_sums_entries` (rs-sdk fetch): pins the SDK seam that `DocumentCount::fetch` with `with_distinct_counts_in_range(true)` returns the per-key-sum aggregate the new dispatch produces. A regression to the always-aggregate-verifier path would fail to decode the mock-provided distinct-mode response. Tests: - drive lib: 38/38 (incl. duplicate-equality rejection) - dash-sdk fetch: 6/6 (incl. distinct-range routing) - drive-abci document_count: 9/9 - clippy clean across drive, dash-sdk
CodeRabbit review triage — pushed `05b22cd116`Two real bugs confirmed and fixed; two carryovers already tracked. 🟢 Fixed#1 — `DocumentCount::fetch` misrouted prove + distinct-range to the aggregate verifier. Confirmed live: `FromProof for DocumentCount` routed every range request through `verify_aggregate_count_proof`, ignoring `return_distinct_counts_in_range`. Server emits a `RangeDistinctProof` (`KVCount` ops) when distinct is set, NOT an `AggregateCountOnRange` aggregate — the two proof shapes embed structurally different `PathQuery`s and verification fails outright. Fixed by dispatching on `return_distinct_counts_in_range` in the range branch. When true: rebuild the same path query the SDK's `DocumentSplitCounts` verifier uses (`DEFAULT_QUERY_LIMIT` fallback + first-orderBy direction), call `verify_distinct_count_proof`, sum the verified per-key counts. The per-key counts are merk-root-bound via `node_hash_with_count`, so the sum is cryptographically committed — same forge-resistance as the aggregate proof, just expressed as a post-verification reduction. Regression test: `test_mock_fetch_document_count_with_distinct_range_sums_entries` pins that `DocumentCount::fetch` with `with_distinct_counts_in_range(true)` returns the per-key-sum aggregate from the new dispatch path. #2 — `where_clauses_from_value` skipped `group_clauses` validation. Confirmed live: the unified count parser handed the parsed clauses straight to `detect_mode` + index pickers without running through `WhereClause::group_clauses`, the system-wide validator the regular query path uses. Effect: malformed shapes the rest of the query stack rejects (duplicate Equal on same field, multiple In, multiple range, equality+In on same field) were silently accepted on the count endpoint. `find_countable_index_for_where_clauses`'s `BTreeSet` over field names and `point_lookup_count_path_query`'s `.find(...)` per index property would just pick the first clause on a duplicated field and return a wrong-but-plausible count. Fixed by calling `WhereClause::group_clauses` inside `where_clauses_from_value` after component parsing — discard the returned triple (the count path operates on a flat clause list, not the regular path's `InternalClauses`), propagate validation errors. Regression test: `test_count_request_with_duplicate_equality_clauses_is_rejected` constructs a request with two conflicting `Equal` clauses on `firstName` and asserts a `DuplicateNonGroupableClauseSameField` rejection. Without the validator call, it would silently return a count for one of the clauses depending on iteration order. 🟡 Carryovers (already tracked)#3 — `DocumentCountQuery` hard-codes `prove: true`. Acknowledged carryover. Filed #3630 tracking the no-proof decoder path. The inline comment now references it explicitly. Structural fix (either splitting transport vs domain types or adding a separate `FetchUnverified` seam) is beyond this PR's scope — coherence blocks the obvious shadowing approach. #4 — rs-sdk fetch tests stop at the mock transport. Acknowledged carryover. The reviewer's framing ("once the testnet path is available — the PR description already lists this as a follow-up assigned to @lklimek") matches the planned approach: recorded vectors against a live platform are the right shape for real verifier-byte coverage, and that work needs the testnet infra. The current mock tests pin request construction + response decoding + dispatch routing (which is enough to catch #1 above); the byte-equality contract between server builder and SDK verifier is pinned drive-side via `test_range_distinct_proof_uses_compile_time_default_query_limit_not_operator_config`, which exercises `GroveDb::verify_query` (the same primitive the SDK wraps). Test posture at HEAD
|
|
Reviewed latest changes |
QuantumExplorer
changed the title
Two consolidation changes on top of the prior v1 commit. **Response shape**: `GetDocumentsResponseV1` now mirrors every other `Get*Response` in the proto — a two-variant outer `oneof` with `result.data` at position 1 and `proof` at position 2. The non-proof result wraps an inner `oneof` (`ResultData.variant`) that switches between `Documents` (for `select=DOCUMENTS`) and `CountResults` (for `select=COUNT`). Keeps the canonical result-or-proof shape callers expect without flattening to a three-variant outer oneof. **`getDocumentsCount` endpoint removed entirely**: - `rpc getDocumentsCount` deleted from the proto service. - `GetDocumentsCountRequest` / `GetDocumentsCountResponse` messages deleted from the proto. - `CountResults` / `CountEntry` / `CountEntries` types moved into `GetDocumentsResponseV1` (the new and only home for the count wire shape). - `query_documents_count` + `query_documents_count_v0` handlers deleted from drive-abci; v1 dispatches to drive's `execute_document_count_request` directly without delegating through the now-gone v0-count layer. - `query/document_count_query/` directory removed from drive-abci. - `document_count_query` + `document_split_count_query` fields removed from `DriveAbciQueryVersions` (rs-platform-version). - `get_documents_count` trait method removed from rs-dapi / rs-dapi-client / rs-drive-abci's query service impl. - dapi-grpc `build.rs`: count messages dropped from `VERSIONED_REQUESTS` / `VERSIONED_RESPONSES`. - rs-sdk: `DocumentCountQuery` still presents the same surface but now builds a `GetDocumentsRequest::V1` with the right `select=COUNT` + computed `group_by` based on where-clause shape (preserves v0-count's implicit grouping behavior at the SDK seam). FromProof impls updated to consume `GetDocumentsResponse`. - rs-drive-proof-verifier: response type aliases updated. - rs-sdk mock harness: `GetDocumentsCountRequest` mock-loader arm removed; existing dumps for that request type need to be re-recorded against `GetDocumentsRequest` v1. The `getDocumentsCount` endpoint shipped briefly in #3623 and never had stable callers, so this consolidation is a clean pre-release removal rather than a deprecation cycle. v0 of the documents endpoint stays alive unchanged. Tests: - drive-abci `document_query` (v0 + v1): 38/38 (1 ignored). - SDK `fetch::document_count`: 6/6 (the count fetch path now exercises the v1 wire bytes end-to-end through the mock transport). - Workspace compiles cleanly across all touched crates. **Non-Rust client regen** lands in a follow-up commit (user runs the docker-based `scripts/build.sh`).
…bool The `return_distinct_counts_in_range` knob only ever lived on the v0 `GetDocumentsCountRequest` endpoint, which shipped in #3623, never had stable callers, and was fully removed (not deprecated) from the proto in PR 1 of this work. PR 2 preserved the bool on the FFI and wasm-sdk count surfaces "for back-compat" — but there was nothing to be back-compatible with, so the in-shim implicit-grouping translation is dead weight. This commit removes it. - rs-sdk-ffi `dash_sdk_document_count`: replace the `return_distinct_counts_in_range: bool` parameter with `group_by_json: *const c_char` (NUL-terminated JSON array of field names; null/empty → aggregate). Mirrors the v1 wire's `group_by: repeated string` field one-to-one. - wasm-sdk `DocumentsQuery`: replace `returnDistinctCountsInRange?: boolean` with `groupBy?: string[]`. Same one-to-one wire mirror on the JS side. - Delete `derive_group_by` helper in rs-sdk-ffi and the inline copy in wasm-sdk. No SDK-side translation table; no second source of truth for "which operators are range operators"; callsites become trivial pass-throughs. - Comment / docstring cleanup in document_count.rs and the count-fetch test to drop residual references to the legacy flag's narrative. No external callers — verified by grep against Swift / Kotlin / native / wasm-demo sources; only the auto-generated FFI header mentioned the old signature. Existing tests all still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2 participants
Issue being fixed or feature implemented
Two parallel goals on the documents-count surface, landed together because the second is impossible without the first:
GetDocumentsCount+GetDocumentsSplitCountinto one endpoint. The split semantics is now signalled by anInwhere-clause (its field is the split property, its array enumerates the values), so the second RPC was redundant.AggregateCountOnRangeproof primitive, removing the materialize-and-count cap ofu16::MAXmatching documents and getting an O(log n)-sized proof for>,<,between*, andstartsWithcount queries.The work pulled in adjacent fixes: a
range_countable: boolper-index-property in DPP + matching insert/delete walker plumbing in drive (so the storage layout actually supports range-aware counting), and an end-to-end fix forprove=true + Inafter two CodeRabbit reviewers flagged that combination as broken end-to-end.What was done?
Wire format (
GetDocumentsCountRequestV0, slot-by-slot)bytes data_contract_idstring document_typebytes whereGetDocumentsRequestV0.where)bool return_distinct_counts_in_rangebytes order_byGetDocumentsRequestV0.order_by). First clause's direction controls split-mode entry ordering AND walk order for the materialize+prove path. Replaces the originaloptional bool order_by_ascendingoptional uint32 limitmax_query_limiton no-prove paths, rejected if too large on prove pathsbool proveResponse
resultisoneof { CountResults counts; Proof proof; }.CountResults.variantisoneof { uint64 aggregate_count; CountEntries entries; }. Total-count and range-without-distinct returnaggregate_count; per-In-value and per-distinct-value-in-range returnentries. No empty-key-entry special-casing on the wire.GetDocumentsSplitCount{Request,Response}deleted entirely.Mode dispatch (drive
detect_mode)(has_range, has_in, prove, distinct)(true, _, true, true)RangeDistinctProof— regular range proof against the property-nameProvableCountTree, per-(in_key, key)KVCountops(true, false, true, false)RangeProof— grovedbAggregateCountOnRange, single(root_hash, count)(true, true, true, false)(true, _, false, _)RangeNoProof— distinct controls sum vs per-value entries(false, true, false, _)PerInValue— one entry per (deduped) In value(false, true, true, _)PointLookupProof— materialize-and-count, verifier groups by In field(false, false, true, _)PointLookupProof— same path, no grouping(false, false, false, _)Total— covering countable index, single aggregatePer-layer changes
packages/dapi-grpc— proto + autogen (JS, Python, Objective-C, web, Rust)packages/rs-dpp—range_countableper-index property (schema-level plumbing); protocol-version gates restored on v1try_from_schema; immutability check pinsrange_countableacross contract updatespackages/rs-driverange_countablestorage layout in the index insert/delete walkers — when a property's index is markedrange_countable, the value tree is aProvableCountTreeand sibling continuations under it useElement::NonCountedwrappers so they contribute 0 to the parent countbatch_insert_empty_count_tree,batch_insert_empty_provable_count_tree,batch_insert_empty_non_counted_tree_if_not_existsgrove-operation helpersdrive_document_count_querymodule split into 7 focused files (mode_detection,index_picker,path_query,execute_point_lookup,execute_range_count,drive_dispatcher,mod)verify_aggregate_count_proof,verify_distinct_count_proof) moved tors-drive/src/verify/so the prove-side path-query builder and the verifier-side reconstruction share one source of truthstartsWithsupported on the range_countable count fast pathpackages/rs-drive-abci— handler rewritten to a single drive call (execute_document_count_request); count where-clause parsing moved into drive; all per-mode dispatching now lives indrive_dispatcher.rspackages/rs-drive-proof-verifier—DocumentCountandDocumentSplitCountsretargeted at the unified response;DocumentSplitCounts::maybe_from_proof_with_split_propertycarries the In field through to per-key aggregation;into_flat_mapcollapses per-(in_key, key)entries for callers that want a flat histogrampackages/rs-sdk—DocumentSplitCountQuerydeleted; bothDocumentCountandDocumentSplitCountsnow backDocumentCountQuery;order_by_clauseson the wrappedDocumentQueryflow through the request builder and the verifier in lockstep (single source of truth for prove/verify direction)packages/rs-sdk-ffi—dash_sdk_document_countis the only count entry point (the priordash_sdk_document_split_countis subsumed by passing distinct/order/limit params);order_by_json: *const c_char(nullable JSON[{field, direction}]) replaces the earlierorder_by_ascending: i32sentinelpackages/wasm-sdk—getDocumentsCount+getDocumentsCountWithProofInforeturnMap<string, bigint>(split modes give one entry per value; total-count is a one-entry map with empty key);getDocumentsSplitCount{,WithProofInfo}deleted; TSDocumentsQueryinterface gains optionalreturnDistinctCountsInRangeand reuses the existingorderByfield for count directionbook/src/drive/document-count-trees.md— new chapter covering the unified endpoint, range modes (sum vs distinct, prove vs no-prove), proof shapes, and the no-merge compound semantics rationaleNotable in-branch fixes from review
prove=true + In→NoProofInResult): routing moved fromPerInValuetoPointLookupProofin3ef2ca3fe1, then the orderBy plumbing landed in the final proto change (bytes order_by) when the second-orderMissingOrderByForRangebug surfaced. New end-to-end testtest_documents_count_with_in_and_prove_returns_proofpins both regressionsDocumentCount::fetchalways errored): SDK now wires upverify_aggregate_count_proofend-to-end via the grovedb gate widened in grovedb#658 (rev1206049b58)How Has This Been Tested?
cargo test -p drive --lib --features=server,verify query::drive_document_count_query— 33 unit tests passing (countable index picker, mode detection, In-clause dedupe, range_countable index resolution, path-query builders)cargo test -p drive-abci --lib query::document_count_query— 8 end-to-end handler tests passing, including:test_documents_count_no_prove/_empty_result/_with_in_operator— total, empty, per-Intest_documents_count_with_prove— total + provetest_documents_count_with_in_and_prove_returns_proof— new regression pin for r3214794852 (asserts non-empty proof bytes forIn([30, 40])+order_by [["age", "asc"]]+prove: true)test_documents_count_range_query_no_prove— range happy path in summed and distinct modes plus paginationtest_documents_count_range_with_prove_and_distinct_returns_proof— distinct + prove emits per-keyKVCountopstest_documents_count_range_without_range_countable_index_returns_clear_error— InvalidArgument with picker contextcargo test -p drive-proof-verifier— new unit tests forinto_flat_map,from_verified, and theFromProofrejection footgun guard (10 tests, ~76 lines coverage)cargo clippy -p drive -p drive-abci -p dash-sdk -p rs-sdk-ffi -p wasm-sdk --lib --tests -- -D warningscleancargo fmt --all -- --checkcleanNot in this PR (tracked for follow-up)
DocumentSplitCounts::fetchround-trip with an In clause. The drive-abci end-to-end test in this PR pins the server-side regression; the SDK-vector test (replaces the deletedpackages/rs-sdk/tests/fetch/document_split_count.rs) catches DAPI-envelope and platform-version-skew issues that only show up against a live platform — needs testnet vectors. Assigned to @lklimek oncefeat/documents-countableis on testnet.Breaking Changes
Wire-format breaking:
GetDocumentsCountRequestV0slot 5 changes type fromoptional bool order_by_ascendingtobytes order_by. Pre-testnet repurpose in place, noreservedmarker.GetDocumentsSplitCountRPC removed.GetDocumentsCountResponseV0.resultreshaped intooneof { CountResults counts; Proof proof; }.Source-API breaking (SDK / FFI / WASM):
DocumentSplitCountQuerydeleted (SDK).dash_sdk_document_split_countdeleted (FFI).getDocumentsSplitCount{,WithProofInfo}deleted (WASM).DocumentCountQuery.order_by_ascending: Option<bool>deleted; direction flows viadocument_query.order_by_clauses.dash_sdk_document_countsignature:order_by_ascending: i32→order_by_json: *const c_char.DocumentsQuery.orderByAscending?removed; reuses existingorderBy?field.Checklist
🤖 Generated with Claude Code