Fix 'limitExceeded' errors related to FK constraint failures.#359
Conversation
| let rhsIndex = tablesByOrder[rhs.recordType] | ||
| else { return true } | ||
| return lhsIndex > rhsIndex | ||
| topologicallyAscending( |
There was a problem hiding this comment.
We have a bunch of places we do this sorting logic, and I needed to do it again, so I added a helper for it.
| let batchSize = 150 | ||
| let batchCount = unsyncedRecordIDs.count / batchSize | ||
| let orderedUnsyncedRecordIDs = unsyncedRecordIDs.sorted { |
There was a problem hiding this comment.
Now invoking records(for:) in batches of 150 and also topologically sorting the unsync'd records IDs before fetching so that we can get the root most records first. This logic could be beefed up to use bigger batches and catch the limitExceeded error so that we can then split the batch in half. Something worth considering later.
| errors | ||
| */ | ||
| @available(iOS 17, macOS 14, tvOS 17, watchOS 10, *) | ||
| @Test func batchAssociations() async throws { |
There was a problem hiding this comment.
This test failed before the fixes in this PR because it creates 500 reminders and syncs them to the client before the client receives the reminders list. That lead to the limit exceeded error and the system couldn't recover from that.
| privateTables: [any SynchronizableTable] = [] | ||
| ) throws { | ||
| let allTables = Set((tables + privateTables).map(HashableSynchronizedTable.init)) | ||
| let allTables = OrderedSet((tables + privateTables).map(HashableSynchronizedTable.init)) |
There was a problem hiding this comment.
Unrelated to the core of this PR, but I did want to write a test on our topological sorting of tables, and so needed the order of things to be stable.
…reeco#359) * Fix 'limitExceeded' errors related to FK constraint failures. * revert * test explanation * emulate error for modifyRecords too * fix test * rename * added a test for tablesByOrder * clean up * add test
2 participants
Right now it is possible to run into a
limitExceedederror when downloading lots of data (such as fresh install) when there are associations that have lots of data. Those associations can only sync when their parent record is available, and if the parent is not available the record ID is cached to be tried later. The problem is that when the record is tried later we load the freshest data from the server via records(for:), and that API must be batched if fetching lots of records.This PR implements that batching logic and updates the mock cloud database to reject fetching/sending data if there is too much data.
Fixes #333 / #334