Skip to content

[opt](parquet) Support hive struct schema change #32438

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 2 commits into from
Mar 22, 2024
Merged

[opt](parquet) Support hive struct schema change #32438

morningman merged 2 commits into from
Mar 22, 2024

Conversation

@suxiaogang223
Copy link
Contributor

@suxiaogang223 suxiaogang223 commented Mar 19, 2024 •
edited by morningman
Loading

Proposed changes

Followup: #31128
This optimization allows doris to correctly read struct type data after changing the schema from hive.

Changing struct schema in hive:

hive> create table struct_test(id int,sf struct<f1: int, f2: string>) stored as parquet;

hive> insert into struct_test values
    >           (1, named_struct('f1', 1, 'f2', 's1')),
    >           (2, named_struct('f1', 2, 'f2', 's2')),
    >           (3, named_struct('f1', 3, 'f2', 's3'));

hive> alter table struct_test change sf sf struct<f1:int, f3:string>;

hive> select * from struct_test;
OK
1	{"f1":1,"f3":null}
2	{"f1":2,"f3":null}
3	{"f1":3,"f3":null}
Time taken: 5.298 seconds, Fetched: 3 row(s)

The previous result of doris was:

mysql> select * from struct_test;
+------+-----------------------+
| id   | sf                    |
+------+-----------------------+
|    1 | {"f1": 1, "f3": "s1"} |
|    2 | {"f1": 2, "f3": "s2"} |
|    3 | {"f1": 3, "f3": "s3"} |
+------+-----------------------+

Now the result is same as hive:

mysql> select * from struct_test;
+------+-----------------------+
| id   | sf                    |
+------+-----------------------+
|    1 | {"f1": 1, "f3": null} |
|    2 | {"f1": 2, "f3": null} |
|    3 | {"f1": 3, "f3": null} |
+------+-----------------------+

@doris-robot
Copy link

doris-robot commented Mar 19, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@suxiaogang223 suxiaogang223 marked this pull request as draft March 19, 2024 02:58
@github-actions
Copy link
Contributor

github-actions bot commented Mar 19, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions
Copy link
Contributor

github-actions bot commented Mar 19, 2024

clang-tidy review says "All clean, LGTM! 👍"

AshinGau
AshinGau reviewed Mar 19, 2024
View reviewed changes
@github-actions
Copy link
Contributor

github-actions bot commented Mar 19, 2024

clang-tidy review says "All clean, LGTM! 👍"

@suxiaogang223 suxiaogang223 force-pushed the hive_complex_type_change branch from 1e51e30 to a5cdbc5 Compare March 20, 2024 03:08
@github-actions
Copy link
Contributor

github-actions bot commented Mar 20, 2024

clang-tidy review says "All clean, LGTM! 👍"

@suxiaogang223 suxiaogang223 force-pushed the hive_complex_type_change branch from a5cdbc5 to ad3d61b Compare March 20, 2024 07:31
@github-actions
Copy link
Contributor

github-actions bot commented Mar 20, 2024

clang-tidy review says "All clean, LGTM! 👍"

@suxiaogang223 suxiaogang223 marked this pull request as ready for review March 20, 2024 09:14
@github-actions
Copy link
Contributor

github-actions bot commented Mar 20, 2024

clang-tidy review says "All clean, LGTM! 👍"

@AshinGau
Copy link
Member

AshinGau commented Mar 21, 2024

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 21, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Mar 21, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 21, 2024

PR approved by anyone and no changes requested.

@zy-kkk
Copy link
Member

zy-kkk commented Mar 21, 2024

run buildall

@suxiaogang223 suxiaogang223 force-pushed the hive_complex_type_change branch from 239de18 to 0f8ee1b Compare March 21, 2024 03:08
@github-actions
Copy link
Contributor

github-actions bot commented Mar 21, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Mar 21, 2024

TeamCity be ut coverage result:
Function Coverage: 35.26% (8728/24751)
Line Coverage: 27.07% (71437/263874)
Region Coverage: 26.31% (37068/140880)
Branch Coverage: 23.23% (18958/81624)
Coverage Report: http://coverage.selectdb-in.cc/coverage/239de186759e80a4fab6e31c1d5172dba18ad47a_239de186759e80a4fab6e31c1d5172dba18ad47a/report/index.html

@zy-kkk
Copy link
Member

zy-kkk commented Mar 21, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Mar 21, 2024

TeamCity be ut coverage result:
Function Coverage: 35.26% (8728/24751)
Line Coverage: 27.07% (71425/263874)
Region Coverage: 26.31% (37068/140880)
Branch Coverage: 23.22% (18954/81624)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0f8ee1b761998602b2326ebab4a9ba47725f3cc1_0f8ee1b761998602b2326ebab4a9ba47725f3cc1/report/index.html

@suxiaogang223
support alter struct type
bfd2458 
impl insert null when struct's field is missing

check field of sturct must be nullable

fix get_rep_level and get_rep_level of StructColumnReader
@suxiaogang223
add _read_column_names to StructColumnReader
04e3a36
@suxiaogang223 suxiaogang223 force-pushed the hive_complex_type_change branch from 0f8ee1b to 04e3a36 Compare March 21, 2024 06:07
@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Mar 21, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Mar 21, 2024

clang-tidy review says "All clean, LGTM! 👍"

@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Mar 21, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Mar 21, 2024

TeamCity be ut coverage result:
Function Coverage: 35.27% (8730/24750)
Line Coverage: 27.09% (71493/263910)
Region Coverage: 26.32% (37079/140885)
Branch Coverage: 23.23% (18966/81630)
Coverage Report: http://coverage.selectdb-in.cc/coverage/04e3a3699c5468a665e1f3ead99f4659d94ab9c7_04e3a3699c5468a665e1f3ead99f4659d94ab9c7/report/index.html

morningman
morningman approved these changes Mar 22, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit d584532 into apache:master Mar 22, 2024
@suxiaogang223 suxiaogang223 deleted the hive_complex_type_change branch March 22, 2024 03:24
yiguolei pushed a commit that referenced this pull request Mar 22, 2024
@suxiaogang223 @Doris-Extras
[opt](parquet) Support hive struct schema change (#32438)
2f2d488 
Followup: #31128
This optimization allows doris to correctly read struct type data after changing the schema from hive.

## Changing  struct schema  in hive:
```sql
hive> create table struct_test(id int,sf struct<f1: int, f2: string>) stored as parquet;

hive> insert into struct_test values
    >           (1, named_struct('f1', 1, 'f2', 's1')),
    >           (2, named_struct('f1', 2, 'f2', 's2')),
    >           (3, named_struct('f1', 3, 'f2', 's3'));

hive> alter table struct_test change sf sf struct<f1:int, f3:string>;

hive> select * from struct_test;
OK
1	{"f1":1,"f3":null}
2	{"f1":2,"f3":null}
3	{"f1":3,"f3":null}
Time taken: 5.298 seconds, Fetched: 3 row(s)
```

The previous result of doris was:
```sql
mysql> select * from struct_test;
+------+-----------------------+
| id   | sf                    |
+------+-----------------------+
|    1 | {"f1": 1, "f3": "s1"} |
|    2 | {"f1": 2, "f3": "s2"} |
|    3 | {"f1": 3, "f3": "s3"} |
+------+-----------------------+
```

Now the result is same as hive:

```sql
mysql> select * from struct_test;
+------+-----------------------+
| id   | sf                    |
+------+-----------------------+
|    1 | {"f1": 1, "f3": null} |
|    2 | {"f1": 2, "f3": null} |
|    3 | {"f1": 3, "f3": null} |
+------+-----------------------+
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@AshinGau AshinGau AshinGau approved these changes

@zy-kkk zy-kkk zy-kkk approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@suxiaogang223 @doris-robot @AshinGau @zy-kkk @morningman