[Feature](Complex-type) Add struct and map type to Doris #16444

xy720 · 2023-02-06T08:37:53Z

Proposed changes

Support complex type struct and map to Doris.

Problem summary

This commit support:

1、Insert + select for struct/map type
2、Json stream load for struct type
3、m[key] function for map type

How to use:

Set the fe config to create table for struct and map type
1、admin set frontend config("enable_struct_type" = "true");
2、admin set frontend config("enable_map_type" = "true");

#16547

Co-authored-by: xy720 <xuyang25@baidu.com>
Co-authored-by: amory <wangqiannan@selectdb.com>
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>
Co-authored-by: hucheng01 <hucheng01@baidu.com>

Checklist(Required)

Does it affect the original behavior:
- Yes
- No
- I don't know
Has unit tests been added:
- Yes
- No
- No Need
Has document been added or modified:
- Yes
- No
- No Need
Does it need to update dependencies:
- Yes
- No
Are there any changes that cannot be rolled back:
- Yes (If Yes, please explain WHY)
- No

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

hello-stephen · 2023-02-06T09:51:42Z

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.54 seconds
load time: 471 seconds
storage size: 17171008864 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230206134540_clickbench_pr_90973.html

morningman · 2023-02-06T14:20:36Z

LGTM with thrift and proto.
cc @xiaokang , PTAL

github-actions

clang-tidy made some suggestions

be/src/olap/types.h

be/test/vec/jsonb/serialize_test.cpp

cambyzju

LGTM

github-actions · 2023-02-08T02:27:55Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-02-08T02:27:59Z

PR approved by anyone and no changes requested.

xiaokang

LGTM.

add some comments that can be fixed by further PRs

fe/fe-core/src/main/jflex/sql_scanner.flex

gensrc/thrift/Exprs.thrift

README.md

xiaokang · 2023-02-07T06:25:54Z

fe/fe-core/src/main/cup/sql_parser.cup

+  ;
+
+struct_literal ::=
+  LBRACE expr_list:list RBRACE


User may get confused if map literal and struct literal both use brace{}.

For now, we keep struct literal using brace{}.
After we implement struct() function, we can support paren() for struct.

xiaokang · 2023-02-07T06:37:17Z

fe/fe-core/src/main/java/org/apache/doris/analysis/StructLiteral.java

+    public String getStringValue() {
+        List<String> list = new ArrayList<>(children.size());
+        children.forEach(v -> list.add(v.getStringValue()));
+        return "{" + StringUtils.join(list, ", ") + "}";


suggest using '(' ')' to keep consistency.

May be we should support '(' ')' only in insert stmt in future.
For select stmt and other places, we keep using brace{}.

be/src/olap/rowset/segment_v2/column_writer.cpp

xiaokang · 2023-02-08T04:16:42Z

be/src/vec/data_types/data_type_factory.hpp

@@ -95,7 +97,10 @@ class DataTypeFactory {
        });


register Map to _invert_data_type_map like Array

May be we will add this part in next map pr.

be/src/vec/data_types/data_type_factory.cpp

be/src/vec/data_types/data_type_struct.cpp

be/src/vec/exprs/vexpr.cpp

…truct implement (#14545) Co-authored-by: hucheng01 <hucheng01@baidu.com>

Fix the dead loop when create column struct Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>

Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>

1、support insert null for struct type; 2、fix explain const exprs for strut type got wrong result; Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>

Add complex type map to doris on vectorized engine

…r type (#16400)

cambyzju

LGTM

github-actions · 2023-02-10T02:54:43Z

PR approved by at least one committer and no changes requested.

This commit support: 1、Insert + select for struct/map type 2、Json stream load for struct type 3、m[key] function for map type How to use: Set the fe config to create table for struct and map type 1、admin set frontend config("enable_struct_type" = "true"); 2、admin set frontend config("enable_map_type" = "true"); apache#16547 Co-authored-by: xy720 <xuyang25@baidu.com> Co-authored-by: amory <wangqiannan@selectdb.com> Co-authored-by: cambyzju <zhuxiaoli01@baidu.com> Co-authored-by: hucheng01 <hucheng01@baidu.com>

github-actions bot added area/planner Issues or PRs related to the query planner area/sql/function Issues or PRs related to the SQL functions area/vectorization kind/docs Categorizes issue or PR as related to documentation. labels Feb 6, 2023

xy720 changed the title ~~[Feature](Struct/Map-type) Add struct and map type to Doris~~ [Feature](Complex-type) Add struct and map type to Doris Feb 6, 2023

xy720 added struct-type Issues or PRs related to struct type map-type Issues or PRs related to map type labels Feb 6, 2023

github-actions bot added the kind/test label Feb 6, 2023

github-actions bot reviewed Feb 7, 2023

View reviewed changes

cambyzju previously approved these changes Feb 8, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 8, 2023

github-actions bot added the reviewed label Feb 8, 2023

xiaokang reviewed Feb 8, 2023

View reviewed changes

xiaokang previously approved these changes Feb 8, 2023

View reviewed changes

xy720 dismissed stale reviews from xiaokang and cambyzju via 7049dec February 8, 2023 07:34

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Feb 8, 2023

xy720 force-pushed the struct-map-type branch from 83a8194 to 552c49e Compare February 9, 2023 02:17

xy720 and others added 8 commits February 9, 2023 20:48

test

9464e96

save

d4b6386

[test] (test) test modify branch (#14375)

da52757

[feature](struct-type) add the class ColumnStruct and class DataTypeS…

198e0b6

…truct implement (#14545) Co-authored-by: hucheng01 <hucheng01@baidu.com>

[WIP](struct-type) support struct-type in vectorize engine (#15665)

c662151

[fix](struct-type) struct create fix and add more struct codes (#15879)

b307c4c

Fix the dead loop when create column struct Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>

[feature](struct-type) adjust the vexpr for struct type (#15669)

ca1bf0a

save (#15906)

fb718ce

xy720 and others added 19 commits February 9, 2023 20:48

save (#15908)

c9cd464

[fix](struct-type) support to remove double or single quote (#15977)

0f9be0e

[fix](struct-type) support struct type insert (#15941)

381ccb1

Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>

[hotfix](struct-type) Fix type descriptor make array load core (#16020)

aacbb79

[fix](struct-type) select of struct return wrong result (#16032)

c94ff76

Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>

[fix](struct-type) fix insert null and explain for struct type (#16054)

fbda784

1、support insert null for struct type; 2、fix explain const exprs for strut type got wrong result; Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>

[Feature](map) add map type to doris (#15966)

3d4d1a1

Add complex type map to doris on vectorized engine

[fix](map-type) fix compile error (#16413)

d8c9521

[fix](complex_type) fix clone_type_info function return default scala…

75bd917

…r type (#16400)

rebase master and fix compile error

2d99c5d

[bug](struct-type) fix be ut and format

569894f

[bug](struct-type) fix be ut and format

3084f61

[bug](struct-type) fix be ut and format

cb81a84

[bug](struct-type) fix array type miss scale

8af00eb

[bug](regression-test) fix array regression test

e6a2be5

[fix](struct-type) remove-empty-line

731d4ef

[fix] fix code for comment

df72861

[fix] fix fe ut

555cea6

fix regression-test core

ba9088c

xy720 force-pushed the struct-map-type branch from 552c49e to ba9088c Compare February 9, 2023 12:49

cambyzju approved these changes Feb 10, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 10, 2023

xy720 merged commit 1b3902b into master Feb 10, 2023

xy720 mentioned this pull request Feb 10, 2023

[Feature](struct-type) Add implicitly cast for struct-type #16613

Merged

13 tasks

xy720 mentioned this pull request Feb 24, 2023

[Roadmap] Support Struct+Map Type #16547

Closed

88 tasks

adonis0147 deleted the struct-map-type branch November 13, 2023 06:17

[Feature](Complex-type) Add struct and map type to Doris #16444

[Feature](Complex-type) Add struct and map type to Doris #16444

Uh oh!

Conversation

xy720 commented Feb 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Problem summary

This commit support:

How to use:

Checklist(Required)

Further comments

Uh oh!

hello-stephen commented Feb 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

morningman commented Feb 6, 2023

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cambyzju left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 8, 2023

Uh oh!

github-actions bot commented Feb 8, 2023

Uh oh!

xiaokang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xiaokang Feb 7, 2023

Choose a reason for hiding this comment

Uh oh!

xy720 Feb 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xiaokang Feb 7, 2023

Choose a reason for hiding this comment

Uh oh!

xy720 Feb 8, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

xiaokang Feb 8, 2023

Choose a reason for hiding this comment

Uh oh!

xy720 Feb 8, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cambyzju left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 10, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

xy720 commented Feb 6, 2023 •

edited

Loading

hello-stephen commented Feb 6, 2023 •

edited

Loading

xy720 Feb 8, 2023 •

edited

Loading