[feature](paimon)paimon catalog supports complex types #25364

DongLiang-0 · 2023-10-12T09:45:31Z

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

DongLiang-0 · 2023-10-18T07:46:02Z

run buildall

DongLiang-0 · 2023-10-18T08:16:12Z

run buildall

doris-robot · 2023-10-18T08:53:50Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.44 seconds
stream load tsv: 577 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162160467 Bytes

DongLiang-0 · 2023-10-18T12:26:16Z

run buildall

doris-robot · 2023-10-18T13:08:59Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.46 seconds
stream load tsv: 579 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162547631 Bytes

fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonTypeUtils.java

zddr · 2023-10-20T03:22:30Z

regression-test/suites/external_table_p0/paimon/test_paimon_catalog.groovy

            def c21 = """select * from auto_bucket where dt="b" and hh="c";"""
            def c22 = """select * from auto_bucket where dt="d";"""
+            def c23 = """select * from complex_tab order by c1;"""
+            def c24 = """select * from complex_tab where c1=1;"""


24,25,26 is similar，keeping one is enough,can add some case like 'SELECT m['a'] FROM simple_map;'

Thanks for your suggestion, I have changed it

zddr · 2023-10-20T03:25:41Z

...-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonColumnValue.java

    private int idx;
-    private InternalRow record;
-    ColumnType dorisType;
+    private DataGetters record;


why change InternalRow to DataGetters

When dealing with unpackArray and unpackMap, since org.apache.paimon.data.InternalArray extends org.apache.paimon.data.DataGetters, if do not change the InternalRow to DataGetters, the force transfer will fail.

DongLiang-0 · 2023-10-20T06:21:54Z

run buildall

zddr

LGTM

github-actions · 2023-10-20T06:32:03Z

PR approved by anyone and no changes requested.

doris-robot · 2023-10-20T07:44:55Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.65 seconds
stream load tsv: 577 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162353741 Bytes

DongLiang-0 · 2023-10-21T12:09:53Z

run buildall

doris-robot · 2023-10-21T12:53:34Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.81 seconds
stream load tsv: 555 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162015400 Bytes

morningman

LGTM

morningman · 2023-10-23T09:52:01Z

Need more test case, not merge to 2.0 yet

Add Paimon complex nested type regression case. Related pr:#25364

Add Paimon complex nested type regression case. Related pr:apache#25364

…5364 #25834 (#26190)

Add Paimon complex nested type regression case. Related pr:apache#25364

…ache#25364 apache#25834 (apache#26190)

Add Paimon complex nested type regression case. Related pr:apache#25364

DongLiang-0 marked this pull request as draft October 12, 2023 09:45

DongLiang-0 force-pushed the paimon_complex branch from eef6ef7 to 583aa25 Compare October 18, 2023 07:42

DongLiang-0 marked this pull request as ready for review October 18, 2023 07:45

DongLiang-0 force-pushed the paimon_complex branch 2 times, most recently from fe3e440 to 34ff765 Compare October 18, 2023 08:16

DongLiang-0 force-pushed the paimon_complex branch from 34ff765 to 4bf6df7 Compare October 18, 2023 12:26

zddr reviewed Oct 20, 2023

View reviewed changes

DongLiang-0 force-pushed the paimon_complex branch 2 times, most recently from bf12d43 to 5e78f3a Compare October 20, 2023 06:21

zddr approved these changes Oct 20, 2023

View reviewed changes

github-actions bot added the reviewed label Oct 20, 2023

DongLiang-0 added 3 commits October 21, 2023 20:09

[feature](paimon)paimon catalog supports complex types

e56ed0b

fix

72a2af2

fix comment

1940570

DongLiang-0 force-pushed the paimon_complex branch from 5e78f3a to 1940570 Compare October 21, 2023 12:09

morningman approved these changes Oct 23, 2023

View reviewed changes

morningman merged commit 267c112 into apache:master Oct 23, 2023

morningman added dev/2.0.3 not-merge/2.0 do not merge into 2.0 branch labels Oct 23, 2023

morningman added merge_conflict and removed not-merge/2.0 do not merge into 2.0 branch labels Oct 23, 2023

DongLiang-0 mentioned this pull request Oct 26, 2023

[regression](catalog)Add test case of paimon complex type #25834

Merged

dutyu pushed a commit to dutyu/doris that referenced this pull request Oct 28, 2023

[feature](paimon)paimon catalog supports complex types (apache#25364)

309dba8

morningman pushed a commit that referenced this pull request Nov 1, 2023

[regression](catalog)Add test case of paimon complex type (#25834)

fef520c

Add Paimon complex nested type regression case. Related pr:#25364

DongLiang-0 mentioned this pull request Nov 1, 2023

[branch-2.0][catalog](paimon)paimon catalog supports complex types #26190

Merged

DongLiang-0 added a commit to DongLiang-0/doris that referenced this pull request Nov 1, 2023

[feature](paimon)paimon catalog supports complex types (apache#25364)

fa70602

DongLiang-0 added a commit to DongLiang-0/doris that referenced this pull request Nov 1, 2023

[regression](catalog)Add test case of paimon complex type (apache#25834)

b7a19e6

Add Paimon complex nested type regression case. Related pr:apache#25364

xiaokang pushed a commit that referenced this pull request Nov 1, 2023

[branch-2.0][catalog](paimon)paimon catalog supports complex types #2…

72c925b

…5364 #25834 (#26190)

xiaokang added dev/2.0.3-merged and removed dev/2.0.3 labels Nov 1, 2023

dutyu pushed a commit to dutyu/doris that referenced this pull request Nov 4, 2023

[regression](catalog)Add test case of paimon complex type (apache#25834)

ac5b6d6

Add Paimon complex nested type regression case. Related pr:apache#25364

DongLiang-0 added a commit to DongLiang-0/doris that referenced this pull request Nov 10, 2023

[feature](paimon)paimon catalog supports complex types (apache#25364)

e134a4c

DongLiang-0 added a commit to DongLiang-0/doris that referenced this pull request Nov 10, 2023

[regression](catalog)Add test case of paimon complex type (apache#25834)

9fb9538

Add Paimon complex nested type regression case. Related pr:apache#25364

seawinde pushed a commit to seawinde/doris that referenced this pull request Nov 13, 2023

[regression](catalog)Add test case of paimon complex type (apache#25834)

3b36ef7

Add Paimon complex nested type regression case. Related pr:apache#25364

gnehil pushed a commit to gnehil/doris that referenced this pull request Dec 4, 2023

[branch-2.0][catalog](paimon)paimon catalog supports complex types ap…

6687f03

…ache#25364 apache#25834 (apache#26190)

xiaokang mentioned this pull request Dec 4, 2023

Release Note 2.0.3 #27909

Closed

XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023

[feature](paimon)paimon catalog supports complex types (apache#25364)

83dea49

XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023

[regression](catalog)Add test case of paimon complex type (apache#25834)

10da329

Add Paimon complex nested type regression case. Related pr:apache#25364

suxiaogang223 mentioned this pull request Apr 17, 2024

[opt](paimon) support mapping Paimon column type "Row" to Doris type "Struct" #33786

Merged

suxiaogang223 mentioned this pull request Apr 28, 2024

[branch-2.1][opt](paimon) support mapping Paimon column type "Row" to Doris type "Struct" #34237

Closed

[feature](paimon)paimon catalog supports complex types #25364

[feature](paimon)paimon catalog supports complex types #25364

Uh oh!

Conversation

DongLiang-0 commented Oct 12, 2023

Proposed changes

Further comments

Uh oh!

DongLiang-0 commented Oct 18, 2023

Uh oh!

DongLiang-0 commented Oct 18, 2023

Uh oh!

doris-robot commented Oct 18, 2023

Uh oh!

DongLiang-0 commented Oct 18, 2023

Uh oh!

doris-robot commented Oct 18, 2023

Uh oh!

Uh oh!

Uh oh!

zddr Oct 20, 2023

Choose a reason for hiding this comment

Uh oh!

DongLiang-0 Oct 20, 2023

Choose a reason for hiding this comment

Uh oh!

zddr Oct 20, 2023

Choose a reason for hiding this comment

Uh oh!

DongLiang-0 Oct 20, 2023

Choose a reason for hiding this comment

Uh oh!

DongLiang-0 commented Oct 20, 2023

Uh oh!

zddr left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 20, 2023

Uh oh!

doris-robot commented Oct 20, 2023

Uh oh!

DongLiang-0 commented Oct 21, 2023

Uh oh!

doris-robot commented Oct 21, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

morningman commented Oct 23, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants