Skip to content

[fix](statistics)Refresh follower FE cache after alter column stats. Support alter index column stats. #31108

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Feb 20, 2024
Merged

[fix](statistics)Refresh follower FE cache after alter column stats. Support alter index column stats. #31108

morningman merged 1 commit into from
Feb 20, 2024

Conversation

@Jibing-Li
Copy link
Contributor

@Jibing-Li Jibing-Li commented Feb 19, 2024

  1. Refresh follower FE cache after alter column stats. So that follower could update the cached stats too.
  2. Support alter index column stats.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented Feb 19, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@Jibing-Li Jibing-Li marked this pull request as ready for review February 19, 2024 08:25
@Jibing-Li
Copy link
Contributor Author

Jibing-Li commented Feb 19, 2024

run buildall

morningman
morningman previously approved these changes Feb 19, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 19, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Feb 19, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Feb 19, 2024

PR approved by anyone and no changes requested.

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Feb 19, 2024
@Jibing-Li
Copy link
Contributor Author

Jibing-Li commented Feb 19, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Feb 19, 2024

TPC-H: Total hot run time: 41826 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7d86869fdc679902c73d5c8952266203d40b4896, data reload: false

------ Round 1 ----------------------------------
q1	17806	5191	5089	5089
q2	2682	141	141	141
q3	11832	1057	1052	1052
q4	4962	977	976	976
q5	7735	3249	3297	3249
q6	196	138	132	132
q7	1246	778	779	778
q8	9243	2088	2071	2071
q9	7534	6689	6680	6680
q10	8321	2657	2648	2648
q11	427	205	214	205
q12	793	328	328	328
q13	17983	3658	3701	3658
q14	295	263	252	252
q15	595	500	498	498
q16	482	415	411	411
q17	933	900	888	888
q18	7277	6690	6668	6668
q19	1555	1498	1504	1498
q20	529	357	323	323
q21	6164	3976	3943	3943
q22	401	358	338	338
Total cold run time: 108991 ms
Total hot run time: 41826 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4785	4810	4794	4794
q2	291	184	184	184
q3	3622	3587	3591	3587
q4	2532	2525	2529	2525
q5	5765	5755	5751	5751
q6	210	123	122	122
q7	2261	1660	1666	1660
q8	3008	3092	3106	3092
q9	8730	8721	8721	8721
q10	4150	4231	4244	4231
q11	528	403	380	380
q12	780	544	547	544
q13	4169	3450	3430	3430
q14	285	251	244	244
q15	631	502	496	496
q16	486	430	448	430
q17	1694	1591	1599	1591
q18	8258	7635	7622	7622
q19	1646	1647	1644	1644
q20	2122	1815	1843	1815
q21	6608	6165	6158	6158
q22	579	526	501	501
Total cold run time: 63140 ms
Total hot run time: 59522 ms

@doris-robot
Copy link

doris-robot commented Feb 19, 2024

TPC-DS: Total hot run time: 177463 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7d86869fdc679902c73d5c8952266203d40b4896, data reload: false

query1	914	352	347	347
query2	6508	1720	1619	1619
query3	6693	209	209	209
query4	23014	21063	21149	21063
query5	4253	380	454	380
query6	257	172	171	171
query7	4598	305	292	292
query8	254	195	197	195
query9	8440	2832	2817	2817
query10	417	226	222	222
query11	15108	14540	14496	14496
query12	129	83	85	83
query13	1668	418	424	418
query14	8962	7605	7584	7584
query15	211	187	192	187
query16	7327	259	248	248
query17	1415	559	527	527
query18	1957	269	263	263
query19	192	151	144	144
query20	84	81	88	81
query21	186	124	113	113
query22	5029	4824	4773	4773
query23	32440	31612	31586	31586
query24	12704	3355	3402	3355
query25	641	366	374	366
query26	1876	152	154	152
query27	3049	311	314	311
query28	6637	1869	1870	1869
query29	1132	623	606	606
query30	283	135	144	135
query31	930	749	757	749
query32	90	59	55	55
query33	724	239	226	226
query34	1093	483	486	483
query35	939	823	820	820
query36	1012	847	850	847
query37	161	61	62	61
query38	3297	3228	3151	3151
query39	1400	1334	1303	1303
query40	282	105	106	105
query41	37	34	35	34
query42	101	100	101	100
query43	480	436	445	436
query44	1061	689	698	689
query45	201	180	176	176
query46	1056	755	754	754
query47	1669	1597	1554	1554
query48	416	338	349	338
query49	1238	308	319	308
query50	779	370	387	370
query51	5336	5178	5240	5178
query52	104	88	96	88
query53	388	298	299	298
query54	289	220	224	220
query55	81	79	79	79
query56	222	200	199	199
query57	1046	931	991	931
query58	220	197	202	197
query59	2229	2081	2178	2081
query60	248	222	226	222
query61	82	87	81	81
query62	599	407	393	393
query63	318	286	290	286
query64	6375	3092	3117	3092
query65	3283	3252	3240	3240
query66	1357	345	319	319
query67	14608	14446	14433	14433
query68	3755	538	541	538
query69	475	348	359	348
query70	1249	1228	1221	1221
query71	311	255	253	253
query72	5933	2770	2627	2627
query73	709	307	312	307
query74	6850	6468	6472	6468
query75	3166	2537	2559	2537
query76	2297	1147	1209	1147
query77	332	236	230	230
query78	9349	8881	8699	8699
query79	970	523	507	507
query80	529	352	342	342
query81	434	206	200	200
query82	159	83	83	83
query83	153	118	120	118
query84	226	76	78	76
query85	1058	342	329	329
query86	296	310	305	305
query87	3430	3247	3297	3247
query88	2697	2270	2278	2270
query89	423	341	351	341
query90	2062	163	162	162
query91	155	128	121	121
query92	52	46	50	46
query93	1012	485	475	475
query94	1187	184	176	176
query95	8464	8372	368	368
query96	579	266	265	265
query97	4401	4223	4244	4223
query98	227	208	201	201
query99	1071	735	725	725
Total cold run time: 274140 ms
Total hot run time: 177463 ms

@doris-robot
Copy link

doris-robot commented Feb 19, 2024

ClickBench: Total hot run time: 30.49 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7d86869fdc679902c73d5c8952266203d40b4896, data reload: false

query1	0.03	0.02	0.02
query2	0.07	0.03	0.02
query3	0.23	0.07	0.07
query4	1.64	0.07	0.08
query5	0.49	0.47	0.49
query6	1.35	0.60	0.63
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.52	0.44	0.44
query10	0.50	0.49	0.49
query11	0.12	0.09	0.10
query12	0.12	0.10	0.10
query13	0.58	0.58	0.58
query14	0.76	0.80	0.79
query15	0.83	0.80	0.79
query16	0.32	0.33	0.34
query17	0.90	0.92	0.86
query18	0.18	0.18	0.18
query19	1.72	1.69	1.69
query20	0.01	0.01	0.01
query21	15.41	0.66	0.60
query22	3.03	4.20	1.92
query23	17.37	1.16	0.99
query24	2.25	0.27	0.50
query25	0.62	0.07	0.05
query26	0.17	0.14	0.14
query27	0.05	0.06	0.06
query28	11.89	0.85	0.83
query29	12.54	3.27	3.32
query30	0.55	0.48	0.49
query31	2.77	0.36	0.36
query32	3.35	0.48	0.49
query33	3.16	3.15	3.17
query34	15.37	4.55	4.52
query35	4.48	4.50	4.49
query36	1.08	0.97	0.99
query37	0.07	0.06	0.05
query38	0.05	0.03	0.03
query39	0.02	0.02	0.02
query40	0.18	0.15	0.14
query41	0.08	0.01	0.02
query42	0.02	0.02	0.01
query43	0.03	0.02	0.02
Total cold run time: 104.97 s
Total hot run time: 30.49 s

@doris-robot
Copy link

doris-robot commented Feb 19, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 7d86869fdc679902c73d5c8952266203d40b4896 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          60 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.2 seconds inserted 10000000 Rows, about 757K ops/s

@hello-stephen
Copy link
Contributor

hello-stephen commented Feb 20, 2024

run feut

@github-actions
Copy link
Contributor

github-actions bot commented Feb 20, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 20, 2024
morningman
morningman approved these changes Feb 20, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit 5998df3 into apache:master Feb 20, 2024
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Feb 20, 2024
@Jibing-Li
[fix](statistics)Refresh follower FE cache after alter column stats. …
890e9da 
…Support alter index column stats (apache#31108)

1. Refresh follower FE cache after alter column stats. So that follower could update the cached stats too.
2. Support alter index column stats.
@Jibing-Li Jibing-Li mentioned this pull request Feb 20, 2024
Merged
@Jibing-Li Jibing-Li deleted the alter branch February 20, 2024 03:59
Jibing-Li added a commit that referenced this pull request Feb 20, 2024
@Jibing-Li
[fix](statistics)Refresh follower FE cache after alter column stats. …
d38dcd7 
…Support alter index column stats (#31108) (#31139)

1. Refresh follower FE cache after alter column stats. So that follower could update the cached stats too.
2. Support alter index column stats.
yiguolei pushed a commit that referenced this pull request Feb 20, 2024
@Jibing-Li @Doris-Extras
[fix](statistics)Refresh follower FE cache after alter column stats. …
4aaab6f 
…Support alter index column stats (#31108)

1. Refresh follower FE cache after alter column stats. So that follower could update the cached stats too.
2. Support alter index column stats.
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@Jibing-Li
[fix](statistics)Refresh follower FE cache after alter column stats. …
b924f6a 
…Support alter index column stats (apache#31108) (apache#31139)

1. Refresh follower FE cache after alter column stats. So that follower could update the cached stats too.
2. Support alter index column stats.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.5-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@Jibing-Li @doris-robot @hello-stephen @morningman @morrySnow @xiaokang