Skip to content

Conversation

@Kikyou1997
Copy link
Contributor

@Kikyou1997 Kikyou1997 commented Dec 5, 2023

Proposed changes

pick from master #27955

For some string value the max/min might be a very long string which might take too many memory of FE, so we truncate to 1024 chars if it's too long

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@Kikyou1997
Copy link
Contributor Author

#27955

@Kikyou1997
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.15 seconds
stream load tsv: 586 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 30.0 seconds inserted 10000000 Rows, about 333K ops/s
storage size: 17164383373 Bytes

@Kikyou1997 Kikyou1997 force-pushed the fix-truncate-min-max-2.0 branch from 48d4ec9 to f4a17cb Compare December 5, 2023 06:28
@Kikyou1997
Copy link
Contributor Author

run buildall

@Kikyou1997 Kikyou1997 force-pushed the fix-truncate-min-max-2.0 branch from f4a17cb to cc27fad Compare December 5, 2023 07:34
@Kikyou1997
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.4 seconds
stream load tsv: 581 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.6 seconds inserted 10000000 Rows, about 337K ops/s
storage size: 17163921483 Bytes

@xiaokang xiaokang merged commit 0b5b366 into apache:branch-2.0 Dec 5, 2023
eldenmoon pushed a commit to eldenmoon/incubator-doris that referenced this pull request Dec 8, 2023
eldenmoon added a commit that referenced this pull request Dec 8, 2023
* [fix](hudi-catalog) fix hudi catalog code (#27966)

backport: #27963

* [fix](Nereids): fix datetime fold-constant-be in Branch-2.0 (#27938)

fix fold-constant-be datetimev2 will lose the scale of datetime.

* [bug](function) fix compound expr coredump problem #27988 (#27989)

* [branch-2.0](cache) Fix partition cache support DATEV2  #27978 (#27990)

* [branch-2.0] delete auto partition docs (#28001)

* [fix](stats) Drop stats or update updated rows after truncate table (#27956)

1. Also clear follower's stats cache when doing drop stats.
2. Drop stats when truncate a table.

backport #27931

* [minor](stats) truncate min/max if too long #27955 (#27997)

* [agg](profile) fix incorrect profile (#28004) (#28035)

* [fix](remote-scanner-pool) missing _remote_thread_pool_max_size value #28057 (#28053)

bp #28057

* [enhance](partitionid) check partition id to avoid unexpected behavior (#28045)

* [opt](resource-tag) root and admin user can use any resource tag by default #28088 (#28080)

bp #28088

* [compile](compile option) add deflate to X86 only (#28089)

* change version to 2.0.3-rc05-var05

* [fix](Nereids): Preserve `""` in single quote strings and `''` in double quote strings. (#27959)

---------

Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com>
Co-authored-by: jakevin <jakevingoo@gmail.com>
Co-authored-by: HappenLee <happenlee@hotmail.com>
Co-authored-by: Xinyi Zou <zouxinyi02@gmail.com>
Co-authored-by: zclllyybb <zhaochangle@selectdb.com>
Co-authored-by: Jibing-Li <64681310+Jibing-Li@users.noreply.github.com>
Co-authored-by: AKIRA <33112463+Kikyou1997@users.noreply.github.com>
Co-authored-by: TengJianPing <18241664+jacktengg@users.noreply.github.com>
Co-authored-by: Mingyu Chen <morningman@163.com>
Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com>
Co-authored-by: shuke <37901441+shuke987@users.noreply.github.com>
Co-authored-by: 谢健 <jianxie0@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants