Fix order by with limit and where clause

[kevin-dp]

This PR fixes a problem that resulted in queries that have an orderBy, limit, and where clause to initially load too few data if the where clauses filters out some of the first limit + offset rows. Claude had a shot at fixing it in #459 but didn't do a good job. I stole the unit tests from that PR (thanks @samwillis for writing those) as they reproduced the issue and are now green in this PR.

When the query contains an orderBy and limit clause we optimize it by only loading the offset + limit first items from the range index initially (introduced in #410). Then at runtime, when new updates come in that may affect the results we send them to the query pipeline and check if the result has enough rows. If not, we load more rows from the range index.

The problem is that we should also check the result set after the initial load since where clauses may filter out some of the rows, leading to too few rows in the result. Hence, this PR introduces the necessary changes to check the size of the result after the initial load and load more rows from the range index if needed.

When introducing this fix, the problem with infinite loading loops when the collection has not enough data to fill the result reappeared. This was fixed in #450 but that solution is incompatible with the changes in this PR. Instead, i found a simpler solution which consists of not passing the callback that loads more data if we know that we already exhausted the collection.

[changeset-bot]

🦋 Changeset detected

Latest commit: a4b8401

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 8 packages

Name	Type
@tanstack/db	Patch
@tanstack/electric-db-collection	Patch
@tanstack/query-db-collection	Patch
@tanstack/react-db	Patch
@tanstack/solid-db	Patch
@tanstack/svelte-db	Patch
@tanstack/trailbase-db-collection	Patch
@tanstack/vue-db	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[pkg-pr-new]

More templates

• tanstack-start-db-projects-example
• @tanstack/db-example-react-todo
• @tanstack/db-example-solid-todo

@tanstack/db

npm i https://pkg.pr.new/@tanstack/db@461

@tanstack/db-ivm

npm i https://pkg.pr.new/@tanstack/db-ivm@461

@tanstack/electric-db-collection

npm i https://pkg.pr.new/@tanstack/electric-db-collection@461

@tanstack/query-db-collection

npm i https://pkg.pr.new/@tanstack/query-db-collection@461

@tanstack/react-db

npm i https://pkg.pr.new/@tanstack/react-db@461

@tanstack/solid-db

npm i https://pkg.pr.new/@tanstack/solid-db@461

@tanstack/svelte-db

npm i https://pkg.pr.new/@tanstack/svelte-db@461

@tanstack/trailbase-db-collection

npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@461

@tanstack/vue-db

npm i https://pkg.pr.new/@tanstack/vue-db@461

commit: a4b8401

[github-actions]

Size Change: +128 B (+0.2%)

Total Size: 64.6 kB

Filename	Size	Change
./packages/db/dist/esm/query/live/collection-config-builder.js	2.56 kB	+53 B (+2.11%)
./packages/db/dist/esm/query/live/collection-subscriber.js	2.4 kB	+75 B (+3.23%)

ℹ️ View Unchanged

Filename	Size
./packages/db/dist/esm/change-events.js	1.13 kB
./packages/db/dist/esm/collection.js	10.6 kB
./packages/db/dist/esm/deferred.js	230 B
./packages/db/dist/esm/errors.js	3 kB
./packages/db/dist/esm/index.js	1.52 kB
./packages/db/dist/esm/indexes/auto-index.js	745 B
./packages/db/dist/esm/indexes/base-index.js	605 B
./packages/db/dist/esm/indexes/btree-index.js	1.74 kB
./packages/db/dist/esm/indexes/lazy-index.js	1.25 kB
./packages/db/dist/esm/local-only.js	827 B
./packages/db/dist/esm/local-storage.js	2.03 kB
./packages/db/dist/esm/optimistic-action.js	294 B
./packages/db/dist/esm/proxy.js	4.19 kB
./packages/db/dist/esm/query/builder/functions.js	575 B
./packages/db/dist/esm/query/builder/index.js	3.79 kB
./packages/db/dist/esm/query/builder/ref-proxy.js	890 B
./packages/db/dist/esm/query/compiler/evaluators.js	1.48 kB
./packages/db/dist/esm/query/compiler/expressions.js	631 B
./packages/db/dist/esm/query/compiler/group-by.js	2.06 kB
./packages/db/dist/esm/query/compiler/index.js	2.15 kB
./packages/db/dist/esm/query/compiler/joins.js	2.36 kB
./packages/db/dist/esm/query/compiler/order-by.js	1.17 kB
./packages/db/dist/esm/query/compiler/select.js	655 B
./packages/db/dist/esm/query/ir.js	466 B
./packages/db/dist/esm/query/live-query-collection.js	333 B
./packages/db/dist/esm/query/optimizer.js	2.63 kB
./packages/db/dist/esm/SortedMap.js	1.24 kB
./packages/db/dist/esm/transactions.js	2.29 kB
./packages/db/dist/esm/utils.js	419 B
./packages/db/dist/esm/utils/btree.js	6.02 kB
./packages/db/dist/esm/utils/comparison.js	718 B
./packages/db/dist/esm/utils/index-optimization.js	1.62 kB

_{compressed-size-action::db-package-size}

[github-actions]

Size Change: 0 B

Total Size: 1.16 kB

ℹ️ View Unchanged

Filename	Size
./packages/react-db/dist/esm/index.js	152 B
./packages/react-db/dist/esm/useLiveQuery.js	1.01 kB

_{compressed-size-action::react-db-package-size}

[samwillis]

[wrong7]

I'm having this exact issue with v0.1.9 with this code @kevin-dp

const { data } = useLiveQuery(
  (q) =>
    q
      .from({ todoCollection })
      .where(({ todoCollection }) =>
        like(todoCollection.user_id, user.id)
      )
      .orderBy(({ todoCollection}) => todoCollection.updated_at, "desc")
      .limit(5),
  [user.id]
);
console.log(data.length) // 1, but the query without limit returns 60 rows.

[kevin-dp]

@wrong7 Could you try out the latest version to see if the problem persists. If it does, could you please open an issue such that we can track this bug and keep you posted when we fix it.