Skip to content

[fix](stats) truncate min/max if too long #27955

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Dec 5, 2023
Merged

[fix](stats) truncate min/max if too long #27955

morrySnow merged 1 commit into from
Dec 5, 2023

Conversation

@Kikyou1997
Copy link
Contributor

@Kikyou1997 Kikyou1997 commented Dec 4, 2023 •
edited
Loading

Proposed changes

For some string value the max/min might be a very long string which might take too many memory of FE, so we truncate to 1024 chars if it's too long

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@Kikyou1997
Copy link
Contributor Author

Kikyou1997 commented Dec 4, 2023

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Dec 4, 2023

PR approved by anyone and no changes requested.

@doris-robot
Copy link

doris-robot commented Dec 4, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.38 seconds
stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17163605357 Bytes

morrySnow
morrySnow previously approved these changes Dec 4, 2023
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 4, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Dec 4, 2023

PR approved by at least one committer and no changes requested.

@Kikyou1997 Kikyou1997 force-pushed the fix-truncate-min-max branch from 3ccbfae to 084ddaa Compare December 5, 2023 03:50
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Dec 5, 2023
@Kikyou1997
Copy link
Contributor Author

Kikyou1997 commented Dec 5, 2023

run buildall

@Kikyou1997 Kikyou1997 mentioned this pull request Dec 5, 2023
Merged
@morrySnow morrySnow changed the title [minor](stats) truncate min/max if too long [fix](stats) truncate min/max if too long Dec 5, 2023
@doris-robot
Copy link

doris-robot commented Dec 5, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.61 seconds
stream load tsv: 562 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17166763140 Bytes

@Kikyou1997 Kikyou1997 force-pushed the fix-truncate-min-max branch from 084ddaa to 287248d Compare December 5, 2023 06:25
@Kikyou1997
Copy link
Contributor Author

Kikyou1997 commented Dec 5, 2023

run buildall

@Kikyou1997 Kikyou1997 force-pushed the fix-truncate-min-max branch from 287248d to 6319461 Compare December 5, 2023 07:31
@Kikyou1997
Copy link
Contributor Author

Kikyou1997 commented Dec 5, 2023

run buildall

xiaokang pushed a commit that referenced this pull request Dec 5, 2023
@Kikyou1997
[minor](stats) truncate min/max if too long #27955 (#27997)
0b5b366
@Kikyou1997
Copy link
Contributor Author

Kikyou1997 commented Dec 5, 2023

run buildall

@doris-robot
Copy link

doris-robot commented Dec 5, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.18 seconds
stream load tsv: 563 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17163645033 Bytes

@doris-robot
Copy link

doris-robot commented Dec 5, 2023

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit 6319461fd59afb12dbf0f633ad1eb77216417849, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4680	4454	4464	4454
q2	365	150	158	150
q3	1464	1259	1267	1259
q4	1111	966	879	879
q5	3167	3192	3169	3169
q6	251	129	131	129
q7	1002	504	484	484
q8	2248	2225	2175	2175
q9	6697	6687	6658	6658
q10	3197	3253	3285	3253
q11	332	207	206	206
q12	358	221	218	218
q13	4593	3789	3820	3789
q14	241	208	223	208
q15	575	524	522	522
q16	443	377	385	377
q17	999	612	537	537
q18	7664	6853	6980	6853
q19	1547	2652	1393	1393
q20	523	311	340	311
q21	3087	2612	2678	2612
q22	361	298	297	297
Total cold run time: 44905 ms
Total hot run time: 39933 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4376	4379	4374	4374
q2	270	165	179	165
q3	3537	3515	3518	3515
q4	2377	2373	2380	2373
q5	5729	5749	5746	5746
q6	239	123	124	123
q7	2384	1848	1890	1848
q8	3519	3525	3546	3525
q9	9113	9045	9045	9045
q10	3909	3980	4016	3980
q11	496	377	377	377
q12	766	578	588	578
q13	4321	3570	3563	3563
q14	272	243	246	243
q15	573	522	516	516
q16	493	445	479	445
q17	1885	1829	1844	1829
q18	8700	8153	8838	8153
q19	1761	1762	1763	1762
q20	2287	1946	1953	1946
q21	6518	6159	6163	6159
q22	499	431	411	411
Total cold run time: 64024 ms
Total hot run time: 60676 ms

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 5, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Dec 5, 2023

PR approved by at least one committer and no changes requested.

@morrySnow morrySnow merged commit 7f1b558 into apache:master Dec 5, 2023
eldenmoon pushed a commit to eldenmoon/incubator-doris that referenced this pull request Dec 8, 2023
@Kikyou1997 @eldenmoon
[minor](stats) truncate min/max if too long apache#27955 (apache#27997)
8a1de11
eldenmoon added a commit that referenced this pull request Dec 8, 2023
@eldenmoon @BePPPower
@jackwener @HappenLee @xinyiZzz @zclllyybb @Jibing-Li @Kikyou1997 @jacktengg @morningman @dataroaring @shuke987 @keanji-x
[pick](branch-2.0) pick prs from branch-2.0 (#28176)
9d5f314 
* [fix](hudi-catalog) fix hudi catalog code (#27966)

backport: #27963

* [fix](Nereids): fix datetime fold-constant-be in Branch-2.0 (#27938)

fix fold-constant-be datetimev2 will lose the scale of datetime.

* [bug](function) fix compound expr coredump problem #27988 (#27989)

* [branch-2.0](cache) Fix partition cache support DATEV2  #27978 (#27990)

* [branch-2.0] delete auto partition docs (#28001)

* [fix](stats) Drop stats or update updated rows after truncate table (#27956)

1. Also clear follower's stats cache when doing drop stats.
2. Drop stats when truncate a table.

backport #27931

* [minor](stats) truncate min/max if too long #27955 (#27997)

* [agg](profile) fix incorrect profile (#28004) (#28035)

* [fix](remote-scanner-pool) missing _remote_thread_pool_max_size value #28057 (#28053)

bp #28057

* [enhance](partitionid) check partition id to avoid unexpected behavior (#28045)

* [opt](resource-tag) root and admin user can use any resource tag by default #28088 (#28080)

bp #28088

* [compile](compile option) add deflate to X86 only (#28089)

* change version to 2.0.3-rc05-var05

* [fix](Nereids): Preserve `""` in single quote strings and `''` in double quote strings. (#27959)

---------

Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com>
Co-authored-by: jakevin <jakevingoo@gmail.com>
Co-authored-by: HappenLee <happenlee@hotmail.com>
Co-authored-by: Xinyi Zou <zouxinyi02@gmail.com>
Co-authored-by: zclllyybb <zhaochangle@selectdb.com>
Co-authored-by: Jibing-Li <64681310+Jibing-Li@users.noreply.github.com>
Co-authored-by: AKIRA <33112463+Kikyou1997@users.noreply.github.com>
Co-authored-by: TengJianPing <18241664+jacktengg@users.noreply.github.com>
Co-authored-by: Mingyu Chen <morningman@163.com>
Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com>
Co-authored-by: shuke <37901441+shuke987@users.noreply.github.com>
Co-authored-by: 谢健 <jianxie0@gmail.com>
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
@Kikyou1997
[fix](stats) truncate min/max if too long (apache#27955)
111cd67 
For some string value the max/min might be a very long string
which might take too many memory of FE,
so we truncate to 1024 chars if it's too long
Jibing-Li pushed a commit to Jibing-Li/incubator-doris that referenced this pull request Dec 28, 2023
@Kikyou1997 @Jibing-Li
[fix](stats) truncate min/max if too long (apache#27955)
a233e07 
For some string value the max/min might be a very long string
which might take too many memory of FE,
so we truncate to 1024 chars if it's too long
@Jibing-Li Jibing-Li mentioned this pull request Dec 28, 2023
Merged
xiaokang pushed a commit that referenced this pull request Dec 28, 2023
@Jibing-Li
[fix](stats) truncate min/max if too long (#27955) (#29196)
388a158
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Jibing-Li Jibing-Li Jibing-Li approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.3-merged p0_b reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Kikyou1997 @doris-robot @Jibing-Li @morrySnow @xiaokang