[fix](Statistics)Drop stats or update updated rows after truncate table. #27931

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

morrySnow merged 2 commits into apache:master from Jibing-Li:stale

Dec 5, 2023

Contributor

Jibing-Li commented Dec 4, 2023 •

edited

Loading

Also clear follower's stats cache when doing drop stats.
Drop stats when truncate a table.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

Jibing-Li force-pushed the stale branch from 8ee9866 to 97d8ae3 Compare

December 4, 2023 03:30

Kikyou1997 approved these changes

View reviewed changes

github-actions bot added the reviewed label

Contributor

github-actions bot commented Dec 4, 2023

PR approved by anyone and no changes requested.

Jibing-Li force-pushed the stale branch 2 times, most recently from 95efb77 to 4841602 Compare

December 4, 2023 03:38

Jibing-Li marked this pull request as ready for review

December 4, 2023 03:54

Contributor Author

Jibing-Li commented Dec 4, 2023

run buildall

doris-robot commented Dec 4, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.33 seconds
stream load tsv: 578 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17163892350 Bytes

morningman previously approved these changes

View reviewed changes

Contributor

morningman left a comment

LGTM

github-actions bot added the approved label

Contributor

github-actions bot commented Dec 4, 2023

PR approved by at least one committer and no changes requested.

Jibing-Li dismissed morningman’s stale review via

adf5c5f

December 4, 2023 09:27

Jibing-Li force-pushed the stale branch from 4841602 to adf5c5f Compare

December 4, 2023 09:27

github-actions bot removed the approved label

Jibing-Li force-pushed the stale branch from adf5c5f to 40087be Compare

December 4, 2023 09:44

Contributor Author

Jibing-Li commented Dec 4, 2023

run buildall

Jibing-Li mentioned this pull request

[fix](Statistics)Drop stats or update updated rows after truncate table. #27956

Merged

Jibing-Li changed the title ~~[fix](Statistics)Remove stale statistics when manually analyze a truncated table.~~ [fix](Statistics)Drop stats or update updated rows after truncate table.

morrySnow reviewed

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java Show resolved Hide resolved

xiaokang added dev/2.0.4 dev/2.0.3 labels

doris-robot commented Dec 4, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.94 seconds
stream load tsv: 561 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17164012373 Bytes

Jibing-Li force-pushed the stale branch 2 times, most recently from a7482c3 to 9eeb691 Compare

December 5, 2023 02:23

github-actions bot added the meta-change label

Contributor

github-actions bot commented Dec 5, 2023

clang-tidy review says "All clean, LGTM! 👍"

Jibing-Li force-pushed the stale branch from 9eeb691 to eaef29c Compare

December 5, 2023 02:32

Contributor

github-actions bot commented Dec 5, 2023

clang-tidy review says "All clean, LGTM! 👍"

Jibing-Li force-pushed the stale branch from eaef29c to cbed274 Compare

December 5, 2023 03:15


          Drop stats or update updated rows after truncate table.

3092be6

Jibing-Li force-pushed the stale branch from cbed274 to 3092be6 Compare

December 5, 2023 03:19

Contributor

github-actions bot commented Dec 5, 2023

clang-tidy review says "All clean, LGTM! 👍"

Contributor Author

Jibing-Li commented Dec 5, 2023

run buildall

Contributor

github-actions bot commented Dec 5, 2023

clang-tidy review says "All clean, LGTM! 👍"


          Merge branch 'master' into stale

30c972e

Contributor

github-actions bot commented Dec 5, 2023

clang-tidy review says "All clean, LGTM! 👍"

morrySnow approved these changes

View reviewed changes

github-actions bot added the approved label

Contributor

github-actions bot commented Dec 5, 2023

PR approved by at least one committer and no changes requested.

Kikyou1997 approved these changes

View reviewed changes

Contributor

morrySnow commented Dec 5, 2023

run buildall

doris-robot commented Dec 5, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.71 seconds
stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17167067154 Bytes

doris-robot commented Dec 5, 2023

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 30c972e9871882df14db4661d76130c323cec6ef, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4891	4706	4718	4706
q2	363	150	160	150
q3	1479	1325	1235	1235
q4	1135	951	944	944
q5	3238	3221	3222	3221
q6	256	125	126	125
q7	1022	521	560	521
q8	2258	2250	2228	2228
q9	6860	6857	7020	6857
q10	3253	3352	3329	3329
q11	337	201	201	201
q12	354	222	212	212
q13	4662	3841	3885	3841
q14	246	216	221	216
q15	606	532	531	531
q16	445	382	388	382
q17	1036	637	610	610
q18	7897	7601	7835	7601
q19	1565	1578	1543	1543
q20	529	321	307	307
q21	3398	2918	2961	2918
q22	367	302	305	302
Total cold run time: 46197 ms
Total hot run time: 41980 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4613	4620	4622	4620
q2	301	184	201	184
q3	3721	3710	3714	3710
q4	2532	2534	2524	2524
q5	6125	6117	6122	6117
q6	240	119	121	119
q7	2638	2004	1992	1992
q8	3706	3741	3740	3740
q9	9469	9337	9409	9337
q10	4034	4104	4119	4104
q11	612	472	441	441
q12	813	664	628	628
q13	4352	3623	3615	3615
q14	272	250	240	240
q15	603	535	532	532
q16	500	470	489	470
q17	2022	1983	2013	1983
q18	9579	8843	8695	8695
q19	1744	1760	1776	1760
q20	2299	1972	1938	1938
q21	7132	6742	6751	6742
q22	626	543	569	543
Total cold run time: 67933 ms
Total hot run time: 64034 ms

morrySnow pushed a commit that referenced this pull request


          [fix](stats) Drop stats or update updated rows after truncate table (#…

aaced7f

…27956)

1. Also clear follower's stats cache when doing drop stats.
2. Drop stats when truncate a table.

backport #27931

morrySnow merged commit 02512cd into apache:master

Jibing-Li deleted the stale branch

December 5, 2023 07:00

wm1581066 added dev/2.0.3-merged and removed dev/2.0.3 dev/2.0.4 labels

eldenmoon pushed a commit to eldenmoon/incubator-doris that referenced this pull request


          [fix](stats) Drop stats or update updated rows after truncate table (a…

4a11b1a

…pache#27956)

1. Also clear follower's stats cache when doing drop stats.
2. Drop stats when truncate a table.

backport apache#27931

eldenmoon added a commit that referenced this pull request


          [pick](branch-2.0) pick prs from branch-2.0 (#28176)

9d5f314

* [fix](hudi-catalog) fix hudi catalog code (#27966)

backport: #27963

* [fix](Nereids): fix datetime fold-constant-be in Branch-2.0 (#27938)

fix fold-constant-be datetimev2 will lose the scale of datetime.

* [bug](function) fix compound expr coredump problem #27988 (#27989)

* [branch-2.0](cache) Fix partition cache support DATEV2  #27978 (#27990)

* [branch-2.0] delete auto partition docs (#28001)

* [fix](stats) Drop stats or update updated rows after truncate table (#27956)

1. Also clear follower's stats cache when doing drop stats.
2. Drop stats when truncate a table.

backport #27931

* [minor](stats) truncate min/max if too long #27955 (#27997)

* [agg](profile) fix incorrect profile (#28004) (#28035)

* [fix](remote-scanner-pool) missing _remote_thread_pool_max_size value #28057 (#28053)

bp #28057

* [enhance](partitionid) check partition id to avoid unexpected behavior (#28045)

* [opt](resource-tag) root and admin user can use any resource tag by default #28088 (#28080)

bp #28088

* [compile](compile option) add deflate to X86 only (#28089)

* change version to 2.0.3-rc05-var05

* [fix](Nereids): Preserve `""` in single quote strings and `''` in double quote strings. (#27959)

---------

Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com>
Co-authored-by: jakevin <jakevingoo@gmail.com>
Co-authored-by: HappenLee <happenlee@hotmail.com>
Co-authored-by: Xinyi Zou <zouxinyi02@gmail.com>
Co-authored-by: zclllyybb <zhaochangle@selectdb.com>
Co-authored-by: Jibing-Li <64681310+Jibing-Li@users.noreply.github.com>
Co-authored-by: AKIRA <33112463+Kikyou1997@users.noreply.github.com>
Co-authored-by: TengJianPing <18241664+jacktengg@users.noreply.github.com>
Co-authored-by: Mingyu Chen <morningman@163.com>
Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com>
Co-authored-by: shuke <37901441+shuke987@users.noreply.github.com>
Co-authored-by: 谢健 <jianxie0@gmail.com>

XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request


          [fix](stats)Drop stats or update updated rows after truncate table (a…

4b55a78

…pache#27931)

1. Also clear follower's stats cache when doing drop stats.
2. Drop stats when truncate a table.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved dev/2.0.3-merged meta-change reviewed