Fix unbounded expression growth in DeduplicatedLoadSubset#1348
Merged
KyleAMathews merged 4 commits intomainfrom Mar 10, 2026
Merged
Fix unbounded expression growth in DeduplicatedLoadSubset#1348KyleAMathews merged 4 commits intomainfrom
KyleAMathews merged 4 commits intomainfrom
Conversation
…nbounded WHERE growth updateTracking() was called with the modified loadOptions (containing the minusWherePredicates difference expression) instead of the original request options. When a "load all" request (where=undefined) was made after accumulating eq() predicates, the difference expression NOT(IN([...])) was tracked instead of setting hasLoadedAllData=true. Each subsequent "load all" request compounded the expression: NOT(IN(...) OR NOT(IN(...))), producing deeply nested 193KB+ WHERE clauses that crashed Electric's parser. The fix separates loadOptions (sent to backend, may contain optimized difference query) from trackingOptions (used for tracking, preserves the original predicate). This ensures "load all" correctly sets hasLoadedAllData=true regardless of the optimization applied to the actual request. https://claude.ai/code/session_01Vp75RhjVR4tV5FdjKJbBWE
🦋 Changeset detectedLatest commit: 3a17319 The changes in this PR will be included in the next version bump. This PR includes changesets to release 12 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
More templates
@tanstack/angular-db
@tanstack/db
@tanstack/db-ivm
@tanstack/electric-db-collection
@tanstack/offline-transactions
@tanstack/powersync-db-collection
@tanstack/query-db-collection
@tanstack/react-db
@tanstack/rxdb-db-collection
@tanstack/solid-db
@tanstack/svelte-db
@tanstack/trailbase-db-collection
@tanstack/vue-db
commit: |
Contributor
|
Size Change: +6 B (+0.01%) Total Size: 93.2 kB
ℹ️ View Unchanged
|
Contributor
|
Size Change: 0 B Total Size: 3.85 kB ℹ️ View Unchanged
|
- Condense block comment explaining trackingOptions/loadOptions split - Remove duplicate inline comments at updateTracking call sites - Remove stray console.log and unused minusWherePredicates import - Fix "exponentially" → "unboundedly" in test comment (growth is linear) - Remove product-specific "sessions index page" reference from test - Strengthen expression assertion to verify full NOT(IN(...)) structure - Add sync return path test for the tracking fix - Simplify test by removing unused calls array Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
|
🎉 This PR has been released! Thank you for your contribution! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix unbounded WHERE expression growth in
DeduplicatedLoadSubsetwhen loading all data after accumulating specific predicates — prevents production 500 errors from oversized query bodies.How the bug happens
When multiple components query the same collection with specific filters (e.g.,
eq(task_id, uuid)as a user navigates between tasks),DeduplicatedLoadSubsetaccumulates those predicates into anIN(task_id, [uuid-1, ..., uuid-N])expression — this is correct and working as designed.The bug triggers when a page then loads the entire collection with no WHERE clause (e.g., a list/index page). The deduplication layer correctly computes a difference query — "give me everything I don't already have" →
NOT(IN(task_id, [...]))— and sends it to the backend. But it incorrectly records that optimized expression as what was "loaded" instead of recording that all data was requested. So the system never learns it has everything, and each subsequent unfiltered load compounds the expression further, eventually producing a 193KB deeply nested WHERE clause that crashes downstream parsers.Root Cause
When
loadSubset({})(load all data) was called after accumulating manyeq(task_id, uuid)predicates, the code computed an optimized difference query (NOT(IN(task_id, [...]))) and sent it to the backend — correct. But it then passed that same optimized expression toupdateTracking(), which meant:hasLoadedAllDatawas never set totrue(becausewherewasn'tundefined)unlimitedWherebecameIN([...]) OR NOT(IN([...]))— logically tautological but not simplifiedApproach
Separate
cloneOptionsinto two copies at the point where the difference is computed:trackingOptions: Preserves the original predicate (e.g.,where: undefined) forupdateTracking()— so "load all" correctly setshasLoadedAllData = trueloadOptions: May be narrowed with the difference expression for the actual backend requestBoth the sync return path (
return true) and async path (.then()) usetrackingOptionsfor tracking andloadOptionsfor the backend call and in-flight matching.Key Invariants
trackingOptions.wherealways equals the caller's originalwherevalueloadOptions.wheremay differ (narrowed byminusWherePredicates)loadOptions(matching must reflect what data the backend will actually return)updateTrackingreceivestrackingOptions(tracking must reflect what was requested)Non-goals
unionWherePredicatesto simplify tautological expressionsVerification
27 tests pass, including 4 new regression tests:
_loadSubsetreturnFiles Changed
packages/db/src/query/subset-dedupe.ts— Split singlecloneOptionsintotrackingOptions+loadOptions; use appropriate copy for tracking vs backend callspackages/db/tests/query/subset-dedupe.test.ts— 4 new tests covering the production bug scenario (async + sync paths, repeated loads, expression structure verification); removed debug artifactshttps://claude.ai/code/session_01Vp75RhjVR4tV5FdjKJbBWE