Skip to content

[feature](struct-type) support csv format stream load for struct type #17143

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
xy720 merged 8 commits into from
Mar 1, 2023
Merged

[feature](struct-type) support csv format stream load for struct type #17143

xy720 merged 8 commits into from
Mar 1, 2023

Conversation

@xy720
Copy link
Member

@xy720 xy720 commented Feb 26, 2023

Proposed changes

Issue Number: close #16547

Refactor from_string method in data_type_struct.cpp to support csv format stream load for struct type.

Problem summary

Describe your changes.

Checklist(Required)

  • Does it affect the original behavior
  • Has unit tests been added
  • Has document been added or modified
  • Does it need to update dependencies
  • Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@github-actions
Copy link
Contributor

github-actions bot commented Feb 26, 2023

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions
Copy link
Contributor

github-actions bot commented Feb 26, 2023

clang-tidy review says "All clean, LGTM! 👍"

@xy720 xy720 added area/stream-load struct-type Issues or PRs related to struct type labels Feb 26, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Feb 26, 2023

clang-tidy review says "All clean, LGTM! 👍"

@xy720
Copy link
Member Author

xy720 commented Feb 26, 2023

run buildall

@xy720
Copy link
Member Author

xy720 commented Feb 26, 2023

run p0

@github-actions
Copy link
Contributor

github-actions bot commented Feb 26, 2023

clang-tidy review says "All clean, LGTM! 👍"

@xy720 xy720 mentioned this pull request Feb 26, 2023
Closed
88 tasks
@xy720
Copy link
Member Author

xy720 commented Feb 26, 2023

run p0

@hello-stephen
Copy link
Contributor

hello-stephen commented Feb 26, 2023 •
edited
Loading

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 33.52 seconds
stream load tsv: 461 seconds loaded 74807831229 Bytes, about 154 MB/s
stream load json: 36 seconds loaded 2358488459 Bytes, about 62 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230301033930_clickbench_pr_106098.html

@xy720
Copy link
Member Author

xy720 commented Feb 26, 2023

run p0

@xy720 xy720 force-pushed the csv-load-for-struct branch from e3e3d0a to dacc9ee Compare February 27, 2023 05:18
@github-actions
Copy link
Contributor

github-actions bot commented Feb 27, 2023

clang-tidy review says "All clean, LGTM! 👍"

@xy720
Copy link
Member Author

xy720 commented Feb 27, 2023

run buildall

1 similar comment
@xy720
Copy link
Member Author

xy720 commented Feb 27, 2023

run buildall

@xy720
Copy link
Member Author

xy720 commented Feb 27, 2023

run p0

@github-actions
Copy link
Contributor

github-actions bot commented Feb 27, 2023

clang-tidy review says "All clean, LGTM! 👍"

@xy720
Copy link
Member Author

xy720 commented Feb 27, 2023

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Feb 27, 2023

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions
Copy link
Contributor

github-actions bot commented Feb 27, 2023

clang-tidy review says "All clean, LGTM! 👍"

cambyzju
cambyzju previously approved these changes Feb 27, 2023
View reviewed changes
Copy link
Contributor

@cambyzju cambyzju left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 27, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Feb 27, 2023

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Feb 27, 2023

PR approved by anyone and no changes requested.

@xy720
Copy link
Member Author

xy720 commented Feb 27, 2023

run p0

@xy720 xy720 force-pushed the csv-load-for-struct branch from 69eeb7b to adf2faf Compare February 28, 2023 02:24
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Feb 28, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Feb 28, 2023

clang-tidy review says "All clean, LGTM! 👍"

@xy720
Copy link
Member Author

xy720 commented Feb 28, 2023

run buildall

@xy720 xy720 force-pushed the csv-load-for-struct branch from adf2faf to b190c50 Compare February 28, 2023 08:06
@xy720
Copy link
Member Author

xy720 commented Feb 28, 2023

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Feb 28, 2023

clang-tidy review says "All clean, LGTM! 👍"

@xy720 xy720 force-pushed the csv-load-for-struct branch from b190c50 to b22c713 Compare February 28, 2023 14:19
@xy720
Copy link
Member Author

xy720 commented Feb 28, 2023

xxx

@xy720
Copy link
Member Author

xy720 commented Feb 28, 2023

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Feb 28, 2023

clang-tidy review says "All clean, LGTM! 👍"

@xy720
Copy link
Member Author

xy720 commented Mar 1, 2023

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Mar 1, 2023

clang-tidy review says "All clean, LGTM! 👍"

cambyzju
cambyzju approved these changes Mar 1, 2023
View reviewed changes
Copy link
Contributor

@cambyzju cambyzju left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 1, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Mar 1, 2023

PR approved by at least one committer and no changes requested.

@xy720 xy720 merged commit 0732eb5 into apache:master Mar 1, 2023
yagagagaga pushed a commit to yagagagaga/doris that referenced this pull request Mar 9, 2023
@xy720 @yagagagaga
[feature](struct-type) support csv format stream load for struct type (…
c3dd8e9 
…apache#17143)

Refactor from_string method in data_type_struct.cpp to support csv format stream load for struct type.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cambyzju cambyzju cambyzju approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. area/stream-load area/vectorization kind/test reviewed struct-type Issues or PRs related to struct type

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Roadmap] Support Struct+Map Type

3 participants

@xy720 @hello-stephen @cambyzju