Move data loading concerns from join/order by to CollectionSubscription

[kevin-dp]

Currently, our optimizations for the join and orderBy operators dynamically load data directly from the indexes. However, loading data should be a concern of the collection (subscription) instead of the query operators. Hence, we introduced a new CollectionSubscription class that represents a subscription that you get from calling collection.subscribeChanges. The subscription exposes requestSnapshot and requestLimitedSnapshot methods to dynamically load data. The subscription keeps track of data it has published and turns updates into inserts (for keys that were not yet published) and filters out deletes of keys that weren't published yet.

The requestSnapshot method loads a snapshot containing all values that fulfill the where clause you provide it (if you provide any). The requestLimitedSnapshot method will only load N items (configurable) that are bigger than a minimum value (optional). This method is used by the orderBy optimization to load only the amount of values it needs.

[changeset-bot]

🦋 Changeset detected

Latest commit: fa9d49a

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 13 packages

Name	Type
@tanstack/db	Minor
@tanstack/db-ivm	Patch
@tanstack/angular-db	Patch
@tanstack/svelte-db	Patch
@tanstack/react-db	Patch
@tanstack/solid-db	Patch
@tanstack/vue-db	Patch
@tanstack/electric-db-collection	Patch
@tanstack/query-db-collection	Patch
@tanstack/rxdb-db-collection	Patch
@tanstack/trailbase-db-collection	Patch
todos	Patch
@tanstack/db-example-react-todo	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[pkg-pr-new]

More templates

• todos
• tanstack-start-db-projects-example
• @tanstack/db-example-react-todo
• @tanstack/db-example-solid-todo

@tanstack/angular-db

npm i https://pkg.pr.new/@tanstack/angular-db@564

@tanstack/db

npm i https://pkg.pr.new/@tanstack/db@564

@tanstack/db-ivm

npm i https://pkg.pr.new/@tanstack/db-ivm@564

@tanstack/electric-db-collection

npm i https://pkg.pr.new/@tanstack/electric-db-collection@564

@tanstack/query-db-collection

npm i https://pkg.pr.new/@tanstack/query-db-collection@564

@tanstack/react-db

npm i https://pkg.pr.new/@tanstack/react-db@564

@tanstack/rxdb-db-collection

npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@564

@tanstack/solid-db

npm i https://pkg.pr.new/@tanstack/solid-db@564

@tanstack/svelte-db

npm i https://pkg.pr.new/@tanstack/svelte-db@564

@tanstack/trailbase-db-collection

npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@564

@tanstack/vue-db

npm i https://pkg.pr.new/@tanstack/vue-db@564

commit: fa9d49a

[github-actions]

Size Change: +742 B (+1.08%)

Total Size: 69.1 kB

Filename	Size	Change
./packages/db/dist/esm/change-events.js	957 B	-177 B (-15.61%)	👏
./packages/db/dist/esm/collection.js	10.5 kB	-400 B (-3.67%)
./packages/db/dist/esm/indexes/btree-index.js	1.76 kB	+22 B (+1.27%)
./packages/db/dist/esm/query/compiler/index.js	2.29 kB	+19 B (+0.84%)
./packages/db/dist/esm/query/compiler/joins.js	2.5 kB	-16 B (-0.64%)
./packages/db/dist/esm/query/live/collection-config-builder.js	2.68 kB	+92 B (+3.55%)
./packages/db/dist/esm/query/live/collection-subscriber.js	1.91 kB	-489 B (-20.36%)	🎉
./packages/db/dist/esm/collection-subscription.js	1.69 kB	+1.69 kB (new file)	🆕

ℹ️ View Unchanged

Filename	Size
./packages/db/dist/esm/collection-events.js	672 B
./packages/db/dist/esm/deferred.js	230 B
./packages/db/dist/esm/errors.js	3.1 kB
./packages/db/dist/esm/index.js	1.55 kB
./packages/db/dist/esm/indexes/auto-index.js	745 B
./packages/db/dist/esm/indexes/base-index.js	605 B
./packages/db/dist/esm/indexes/lazy-index.js	1.25 kB
./packages/db/dist/esm/local-only.js	827 B
./packages/db/dist/esm/local-storage.js	2.02 kB
./packages/db/dist/esm/optimistic-action.js	294 B
./packages/db/dist/esm/proxy.js	3.87 kB
./packages/db/dist/esm/query/builder/functions.js	615 B
./packages/db/dist/esm/query/builder/index.js	3.93 kB
./packages/db/dist/esm/query/builder/ref-proxy.js	938 B
./packages/db/dist/esm/query/compiler/evaluators.js	1.52 kB
./packages/db/dist/esm/query/compiler/expressions.js	631 B
./packages/db/dist/esm/query/compiler/group-by.js	2.08 kB
./packages/db/dist/esm/query/compiler/order-by.js	1.23 kB
./packages/db/dist/esm/query/compiler/select.js	1.28 kB
./packages/db/dist/esm/query/ir.js	508 B
./packages/db/dist/esm/query/live-query-collection.js	333 B
./packages/db/dist/esm/query/optimizer.js	3.05 kB
./packages/db/dist/esm/SortedMap.js	1.24 kB
./packages/db/dist/esm/transactions.js	3.03 kB
./packages/db/dist/esm/utils.js	943 B
./packages/db/dist/esm/utils/btree.js	6.02 kB
./packages/db/dist/esm/utils/comparison.js	718 B
./packages/db/dist/esm/utils/index-optimization.js	1.62 kB

_{compressed-size-action::db-package-size}

[github-actions]

Size Change: 0 B

Total Size: 1.44 kB

ℹ️ View Unchanged

Filename	Size
./packages/react-db/dist/esm/index.js	152 B
./packages/react-db/dist/esm/useLiveQuery.js	1.28 kB

_{compressed-size-action::react-db-package-size}

[samwillis]

Currently if there is an order by on a joined collection we only perform one of the optimisations? Is it only the join lookup or the orderBy?

This requestLimitedSnapshot doesn't take a where and so it can't support both a join and a orderBy index lookup at the same time.

[kevin-dp]

Exactly. If there is both a join and an orderBy (with a limit) we optimise only the orderBy because that's the first one we check in the code. The idea is that the limit is usually rather small and so it's better to load only a limited number of rows instead of all rows that match the join.

[samwillis]

LGTM 🥳