Skip to content

[feature-wip](stream-load-vec) Support stream load in vectorized engine #8709

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 32 commits into from
Mar 28, 2022
Merged

[feature-wip](stream-load-vec) Support stream load in vectorized engine #8709

morningman merged 32 commits into from
Mar 28, 2022

Conversation

@jacktengg
Copy link
Contributor

@jacktengg jacktengg commented Mar 28, 2022

Proposed changes

Issue Number: close #xxx

Problem Summary:

Implement vectorized stream load. Added fe and be configuration option enable_vectorized_load to enable vectorized stream load.

Checklist(Required)

  1. Does it affect the original behavior: (No)
  2. Has unit tests been added: (Yes)
  3. Has document been added or modified: (No)
  4. Does it need to update dependencies: (No)
  5. Are there any changes that cannot be rolled back: (No)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

mrhhsg and others added 30 commits March 28, 2022 15:22
@mrhhsg @jacktengg
Add tablet_writer_add_block into internal_service.proto
933bb77
@mrhhsg @jacktengg
Add VLoadChannelMgr/VLoadChannel/VTabletsChannel/VDeltaWriter
0cfb445
@mrhhsg @jacktengg
Add implement of tablet_writer_add_block
a3603a9
@jacktengg
stream load vectorization: vectorized broker scan and tablet sink
3b31891
@mrhhsg @jacktengg
避免 VDeltaWriter 中的 overload 造成基类的方法被隐藏
fe3f771
@jacktengg
memtable vectorization part 1
3533efa
@jacktengg
merge memtable.insert api
c542823
@jacktengg
memtable flush support vectorization
60053ab
@jacktengg
add mem_tracker for vec memtable
e1bfc69
@jacktengg
remove RowInBlock._bock
4bd19ed
@mrhhsg @jacktengg
Use virtual function to create MemTable
a0fba46
@mrhhsg @jacktengg
Fix incorrect idx value in "VDeltaWriter::write_block"
e9e1d54
@jacktengg
add test case: vectorized broker scan node and tablet sink
ea6b630
@jacktengg
vectorized stream load: fix bugs
10f294f
@jacktengg
RowCursorCell support vectorization
074599c
@mrhhsg @jacktengg
Add test case for VdeltaWriter and VLoadChannelMgr
24df714
@jacktengg
agg by vectorized method
119e602
@jacktengg
vbroker scanner handle date and datetime correctly
436de96
@jacktengg
vectorized skiplist
fb37ca8
@jacktengg
skip init_agg for dup_keys
6a8a823
@jacktengg
使用 vec agg
04dd19f
@jacktengg
RowInBlock 使用指针
f6bfc54
@starocean999 @jacktengg
add vec compaction
e795b10
@starocean999 @jacktengg
modified based on pr comments
ae50b26
@jacktengg
memtable _vflush() skip empty block
3a910e8
@jacktengg
fix compile errors
f9ab013
@englefly @jacktengg
fix compile error
f007e99
@englefly @jacktengg
replace agg by last item
6e141ae
@englefly @jacktengg
memtable mem usage
24fa448
@englefly @jacktengg
using last_value function for replace aggregation in stream load
06a2292
@morningman morningman changed the title Stream load [feature-wip](stream-load-vec) Support stream load in vectorized engine Mar 28, 2022
@morningman morningman merged commit cd10f16 into apache:stream-load-vec Mar 28, 2022
@jacktengg jacktengg mentioned this pull request Mar 29, 2022
Closed
morningman pushed a commit that referenced this pull request Apr 13, 2022
@jacktengg @morningman-cmy
[feature-wip](stream-load-vec) Support stream load in vectorized engi…
88d7d64 
…ne (#8709)

Implement vectorized stream load.
Added fe and be configuration option `enable_vectorized_load` to enable vectorized stream load.
xiepengcheng01 pushed a commit to xiepengcheng01/doris that referenced this pull request Apr 20, 2022
@jacktengg
[feature-wip](stream-load-vec) Support stream load in vectorized engi…
29dcba9 
…ne (apache#8709)

Implement vectorized stream load.
Added fe and be configuration option `enable_vectorized_load` to enable vectorized stream load.
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 28, 2022
[feature-wip](stream-load-vec) Support stream load in vectorized engi…
a6485ed 
…ne (apache#8709)

Implement vectorized stream load.
Added fe configuration option `enable_vectorized_load` to enable vectorized stream load.
HappenLee added a commit to HappenLee/incubator-doris that referenced this pull request Apr 28, 2022
@jacktengg @HappenLee @zbtzbtzbt
[feature-wip](stream-load-vec) Support stream load in vectorized engi…
ca19657 
…ne (apache#8709)

Implement vectorized stream load.
Added fe configuration option `enable_vectorized_load` to enable vectorized stream load.

Co-authored-by: jacktengg <18241664+jacktengg@users.noreply.github.com>
Co-authored-by: HappenLee <happenlee@hotmail.com>
Co-authored-by: zhoubintao <35688959+zbtzbtzbt@users.noreply.github.com>
HappenLee added a commit to HappenLee/incubator-doris that referenced this pull request Apr 28, 2022
@HappenLee @zbtzbtzbt
[feature-wip](stream-load-vec) Support stream load in vectorized engi…
86581cb 
…ne (apache#8709)

Implement vectorized stream load.
Added fe configuration option `enable_vectorized_load` to enable vectorized stream load.

Co-authored-by: tengjp@outlook.com
Co-authored-by: mrhhsg@gmail.com
Co-authored-by: minghong.zhou@163.com
Co-authored-by: HappenLee <happenlee@hotmail.com>
Co-authored-by: zhoubintao <35688959+zbtzbtzbt@users.noreply.github.com>
@HappenLee HappenLee mentioned this pull request Apr 28, 2022
Merged
HappenLee added a commit to HappenLee/incubator-doris that referenced this pull request Apr 28, 2022
@HappenLee @zbtzbtzbt
[feature-wip](stream-load-vec) Support stream load in vectorized engi…
f8bb964 
…ne (apache#8709)

Implement vectorized stream load.
Added fe configuration option `enable_vectorized_load` to enable vectorized stream load.

Co-authored-by: tengjp@outlook.com
Co-authored-by: mrhhsg@gmail.com
Co-authored-by: minghong.zhou@163.com
Co-authored-by: HappenLee <happenlee@hotmail.com>
Co-authored-by: zhoubintao <35688959+zbtzbtzbt@users.noreply.github.com>
morningman pushed a commit that referenced this pull request Apr 29, 2022
@HappenLee
[Vectorized](stream-load-vec) Support stream load in vectorized engine (
d330bc3 
#8709) (#9280)

Implement vectorized stream load.
Added fe configuration option `enable_vectorized_load` to enable vectorized stream load.

    Co-authored-by: tengjp@outlook.com
    Co-authored-by: mrhhsg@gmail.com
    Co-authored-by: minghong.zhou@163.com
    Co-authored-by: HappenLee <happenlee@hotmail.com>
    Co-authored-by: zhoubintao <35688959+zbtzbtzbt@users.noreply.github.com>
Kikyou1997 pushed a commit to Kikyou1997/incubator-doris that referenced this pull request May 9, 2022
@HappenLee
[Vectorized](stream-load-vec) Support stream load in vectorized engine (
1618d7f 
apache#8709) (apache#9280)

Implement vectorized stream load.
Added fe configuration option `enable_vectorized_load` to enable vectorized stream load.

    Co-authored-by: tengjp@outlook.com
    Co-authored-by: mrhhsg@gmail.com
    Co-authored-by: minghong.zhou@163.com
    Co-authored-by: HappenLee <happenlee@hotmail.com>
    Co-authored-by: zhoubintao <35688959+zbtzbtzbt@users.noreply.github.com>
starocean999 pushed a commit to starocean999/incubator-doris that referenced this pull request May 19, 2022
@HappenLee @starocean999
[Vectorized](stream-load-vec) Support stream load in vectorized engine (
51902c9 
apache#8709) (apache#9280)

Implement vectorized stream load.
Added fe configuration option `enable_vectorized_load` to enable vectorized stream load.

    Co-authored-by: tengjp@outlook.com
    Co-authored-by: mrhhsg@gmail.com
    Co-authored-by: minghong.zhou@163.com
    Co-authored-by: HappenLee <happenlee@hotmail.com>
    Co-authored-by: zhoubintao <35688959+zbtzbtzbt@users.noreply.github.com>
englefly pushed a commit to englefly/incubator-doris that referenced this pull request May 23, 2022
@HappenLee
[Vectorized](stream-load-vec) Support stream load in vectorized engine (
deebfd2 
apache#8709) (apache#9280)

Implement vectorized stream load.
Added fe configuration option `enable_vectorized_load` to enable vectorized stream load.

    Co-authored-by: tengjp@outlook.com
    Co-authored-by: mrhhsg@gmail.com
    Co-authored-by: minghong.zhou@163.com
    Co-authored-by: HappenLee <happenlee@hotmail.com>
    Co-authored-by: zhoubintao <35688959+zbtzbtzbt@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@jacktengg @morningman @mrhhsg @starocean999 @englefly