Can't use JSON fields with merge_insert

[ztorchan]

Hi.

https://github.com/lancedb/lance/blob/30dcb5b8eb01e2fd4d05824b908d4174bab549d6/rust/lance/src/dataset/write/merge_insert/exec/write.rs#L361-L373

I found that the metadata of data column field was not copied into the output schema in FullSchemaMergeInsertExec::prepare_stream_schema, resulting in metadata loss.
This results in an error when performing mergeinsert with a json column in the dataset. like:
Query Execution error: Execution error: LanceError(Schema): Attempt to project field by different types: LargeBinary and Utf8

[wjones127]

Normally, we don't care about the field metadata of data you are inserting. We will just throw it away in the end and keep the metadata that's already in the table schema.

But JSON columns are special, because we detect based on the metadata that we should convert them to a JSONB encoding. When we stripped that metadata away, we dropped the conversion from string JSON (Utf8) to JSONB (LargeBinary), and that caused the type mismatch.

[westonpace]

But JSON columns are special, because we detect based on the metadata that we should convert them to a JSONB encoding. When we stripped that metadata away, we dropped the conversion from string JSON (Utf8) to JSONB (LargeBinary), and that caused the type mismatch.

Just to be clear you are stating that the conversion should be based on the field metadata of the dataset schema right? We should always ignore the field metadata of the data itself?

[wjones127]

Just to be clear you are stating that the conversion should be based on the field metadata of the dataset schema right? We should always ignore the field metadata of the data itself?

Oh that's a good point. The consequence of this is a user should be able to do:

# Create dataset using JSON type
schema = pa.schema({'data': pa.json()})
data = pa.table({'data': [json.dumps(dict(x=3))], schema=schema)
ds = lance.write_dataset(data, "memory://")

# Insert plain JSON strings -> will be automatically converted
new_data = pa.table({'data': [json.dumps(dict(x=5))]})
ds.merge_insert(on=[...]).execute(new_data)

[Maxwell-Guo]

let me take a try ? @wjones127 @westonpace

[mackrorysd]

Looks like we can close this issue, right?

[westonpace]

Looks like we can close this issue, right?

Maybe...I'm not certain all merge-insert paths work (there are several of them now) but I am not aware of any that fail.