[Feature] Implement `DataTypeObjectSerDe::read_column_from_arrow`

[xx-derek]

Search before asking

• I had searched in the issues and found no similar issues.

Description

DataTypeObjectSerDe::read_column_from_arrow is not implemented in data_type_object_serde.h

doris/be/src/vec/data_types/serde/data_type_object_serde.h

Line 75 in 38313fc

void read_column_from_arrow(IColumn& column, const arrow::Array* arrow_array, int start,

Use case

Stream load in arrow format does not work with tables that has VARIANT columns without this function.

Related issues

Not found.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[wyxxxcat]

Can you give me a reproducible statement?

[xx-derek]

Please refer to the following steps to reproduce.

1. create test table

CREATE TABLE `demo` (
  `id` int NULL,
  `value` variant NULL
)
DUPLICATE KEY(`id`)
DISTRIBUTED BY RANDOM BUCKETS 16

2. generate data in arrow stream format

import numpy as np
import pyarrow as pa

ids = pa.array(np.arange(100))
values = pa.array([f'{{\"a\": {i}}}' if i % 2 == 0 else f'{{\"a\": \"{i}\"}}' for i in range(100)])

schema = pa.schema([
    pa.field('id', pa.int32()),
    pa.field('value', values.type)
])

with pa.OSFile('payload.dat', 'wb') as sink:
    with pa.ipc.new_stream(sink, schema=schema) as writer:
        batch = pa.record_batch([ids, values], schema=schema)
        writer.write(batch)

print('Done')

Here I'm populating json strings to value field which is VARIANT in Doris table. The type for this field in arrow structure does not matter in this case.

3. initiate stream load

curl --location-trusted -u user:pass \
-H "Expect:100-continue" \
-H "format:arrow" \
-T payload.dat \
-XPUT http://host:port/api/db_name/demo/_stream_load

The stream load operation will fail with error message "Failed to convert from arrow to block: [E3] read_column_from_arrow with type variant".

[xx-derek]

I'm working on a patch. Can I take this?

[github-actions]

We're closing this issue because it hasn't been updated in a while.
If you'd like to revive this issue, please reopen it and feel free a maintainer to remove the Stale tag!