add support for electric must-refetch

[samwillis]

this also introduces a truncate() method that a sync provider can use. There are through tests to ensure that the optimistic state is maintained after a truncate and resync.

[changeset-bot]

🦋 Changeset detected

Latest commit: 04fb63a

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 8 packages

Name	Type
@tanstack/db	Patch
@tanstack/electric-db-collection	Patch
@tanstack/query-db-collection	Patch
@tanstack/react-db	Patch
@tanstack/solid-db	Patch
@tanstack/svelte-db	Patch
@tanstack/trailbase-db-collection	Patch
@tanstack/vue-db	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[pkg-pr-new]

More templates

• tanstack-start-db-projects-example
• @tanstack/db-example-react-todo
• @tanstack/db-example-solid-todo

@tanstack/db

npm i https://pkg.pr.new/@tanstack/db@412

@tanstack/db-ivm

npm i https://pkg.pr.new/@tanstack/db-ivm@412

@tanstack/electric-db-collection

npm i https://pkg.pr.new/@tanstack/electric-db-collection@412

@tanstack/query-db-collection

npm i https://pkg.pr.new/@tanstack/query-db-collection@412

@tanstack/react-db

npm i https://pkg.pr.new/@tanstack/react-db@412

@tanstack/solid-db

npm i https://pkg.pr.new/@tanstack/solid-db@412

@tanstack/svelte-db

npm i https://pkg.pr.new/@tanstack/svelte-db@412

@tanstack/trailbase-db-collection

npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@412

@tanstack/vue-db

npm i https://pkg.pr.new/@tanstack/vue-db@412

commit: 04fb63a

[github-actions]

Size Change: +656 B (+1.09%)

Total Size: 60.8 kB

Filename	Size	Change
./packages/db/dist/esm/collection.js	10.5 kB	+656 B (+6.65%)	🔍

ℹ️ View Unchanged

Filename	Size
./packages/db/dist/esm/change-events.js	1.13 kB
./packages/db/dist/esm/deferred.js	230 B
./packages/db/dist/esm/errors.js	3 kB
./packages/db/dist/esm/index.js	1.52 kB
./packages/db/dist/esm/indexes/auto-index.js	718 B
./packages/db/dist/esm/indexes/base-index.js	605 B
./packages/db/dist/esm/indexes/btree-index.js	1.47 kB
./packages/db/dist/esm/indexes/lazy-index.js	1.25 kB
./packages/db/dist/esm/local-only.js	827 B
./packages/db/dist/esm/local-storage.js	2.03 kB
./packages/db/dist/esm/optimistic-action.js	294 B
./packages/db/dist/esm/proxy.js	4.19 kB
./packages/db/dist/esm/query/builder/functions.js	575 B
./packages/db/dist/esm/query/builder/index.js	3.79 kB
./packages/db/dist/esm/query/builder/ref-proxy.js	890 B
./packages/db/dist/esm/query/compiler/evaluators.js	1.48 kB
./packages/db/dist/esm/query/compiler/expressions.js	631 B
./packages/db/dist/esm/query/compiler/group-by.js	2.03 kB
./packages/db/dist/esm/query/compiler/index.js	2.1 kB
./packages/db/dist/esm/query/compiler/joins.js	2.31 kB
./packages/db/dist/esm/query/compiler/order-by.js	677 B
./packages/db/dist/esm/query/compiler/select.js	655 B
./packages/db/dist/esm/query/ir.js	318 B
./packages/db/dist/esm/query/live-query-collection.js	2.91 kB
./packages/db/dist/esm/query/optimizer.js	2.44 kB
./packages/db/dist/esm/SortedMap.js	1.24 kB
./packages/db/dist/esm/transactions.js	2.29 kB
./packages/db/dist/esm/utils.js	419 B
./packages/db/dist/esm/utils/btree.js	5.93 kB
./packages/db/dist/esm/utils/comparison.js	718 B
./packages/db/dist/esm/utils/index-optimization.js	1.62 kB

_{compressed-size-action::db-package-size}

[github-actions]

Size Change: 0 B

Total Size: 1.05 kB

ℹ️ View Unchanged

Filename	Size
./packages/react-db/dist/esm/index.js	152 B
./packages/react-db/dist/esm/useLiveQuery.js	902 B

_{compressed-size-action::react-db-package-size}

[KyleAMathews]

Is resetting to loading to also reset the query pipeline?

[thruflo]

LGTM 👍

[balegas]

The code looks good, but I have a couple of observations that might make or not make sense with the collection.

• The shape is not invalid at the point of must-refetch, could you mask the loading state by keeping the state of the shape while you refetch it?
• Why do you need a new loading state instead of resetting to the initial state of the shape, e.g. marking it not ready?

[samwillis]

@KyleAMathews

Is resetting to loading to also reset the query pipeline?

Good question, I don't think it will. We need a way to handle that in the live query. I think, but need to check, that the live query will not update until it goes back to ready.

@balegas

The shape is not invalid at the point of must-refetch, could you mask the loading state by keeping the state of the shape while you refetch it?

Yes, we could. The intention here was that the live query would see the loading state and not commit its changes - essentially masking it at the live query level.

I think there is a wider question here of is we want to go back into "loading" when this happen, but also if there are other reasons any other sync engine wants to go back into loading.

One thought I had was for partitioned collections, when requesting a partition that requires to be sure a new query is "read" we likely need to go back into loading. But again that maybe want to be more fine grained, so only triggering loading for live queries that span a loading partition.

My thinking here is that we are going from, everything breaks at the moment, to a first stop towards making this work. we can refine it.

[KyleAMathews]

Another way of doing this could be to delete every key and then reinsert everything?

[samwillis]

Yep, I'm now rethinking this, we should "schedule a truncate" and then do that at the bending of the next commit. It's the pattern we used in pglite.

We can go back and think about loading state later on - we need to, but no rush now.

[KyleAMathews]

Yeah, maybe even just a truncate operator that sync engines can use so:

begin()
truncate()
insert()
insert()
insert()
commit()

[samwillis]

Now has a transactional truncate()

[KyleAMathews]

Looks great!