Max length primary keys and checkpoints

[wuzzeb]

I am getting a max length error when writing the checkpoints to the meta instance, specifically this line in checkpoint.ts. The error is a validation error that the id exceeds the maximum length. Stepping through the code, the id is set in a call to getComposedPrimaryKeyOfDocumentData a few lines above which is setting the id to up|1 where up is the direction.

But looking at the definition of the meta instance, it is defined like so:

            id: {
                type: 'string',
                minLength: 1,
                // add +1 for the '|' and +1 for the 'isCheckpoint' flag
                maxLength: parentPrimaryKeyLength + 2
            },

But, in my own collection, I have a maximum key length of 1 (keys are only a single character),
so this is calculated to 3. But up|1 has length 4.

I see that down|1 is also a key, so the max length for the meta instance should probably be max(4, parentPrimaryKeyLength) + 2.

[pubkey]

Can you make a PR with a test case that reproduces your problem?
You can start with this template.

[wuzzeb]

I made a unit test. When single-stepping through the unit test in the debugger, I made a few more observations.

• You can bypass the error by changing the maximum length of the id key in the schema to 4. Then the error about the key up|1 doesn't happen.

• The thrown error also does not appear if the primary key is named something other than id. But I suspect this is still a problem because of what I saw when single-stepping in a debugger.

Try changing the primary key in the unit test to something else, like primaryId or something. Then, put a breakpoint at

rxdb/src/replication-protocol/checkpoint.ts

Line 118 in 3ab3124

const result = await state.input.metaInstance.bulkWrite(writeRows, 'replication-set-checkpoint');

Notice that the call to metaInstance.bulkWrite returns an error, so result is an error. The error is about up|1 too long because the maximum key length is 1 + 2 = 3. But then, getWrittenDocumentsFromBulkWriteResponse is called, and if you put a breakpoint at

rxdb/src/rx-storage-helper.ts

Line 880 in 3ab3124

const errorIds = reInsertIds ? reInsertIds : new Set<string>();

it correctly goes into the error branch there, and adds errorIds variable has up|1. But then the loop below, specifically this line

rxdb/src/rx-storage-helper.ts

Line 888 in 3ab3124

if (!errorIds.has((doc as any)[primaryPath])) {

There are now two cases:

• If the original schema has primary key id, then errorIds.has is true and nothing is pushed to ret.
• If the original schema has primary key something else such as primaryId, then the document is pushed to ret.

We then return to this line

rxdb/src/replication-protocol/checkpoint.ts

Line 124 in 3ab3124

if (successDoc) {

• If the original schema has primary key id, then successDoc is undefined (because nothing was pushed to ret) and we go to the case where we throw the error
• If the original schema has primary key something else, then successDoc is filled (but result is still an error). I suspect that the line should be changed to if (successDoc && result.success) or something like that

[wuzzeb]

Actually, maybe it is the parameter passed to getWrittenDocumentsFromBulkWriteResponse, perhaps this line

rxdb/src/replication-protocol/checkpoint.ts

Line 120 in 3ab3124

state.primaryPath,

should instead be state.input.metaInstance.primaryPath since we are trying to get whatever was written to the meta instance.