Skip to content

[opt](rowcount) refresh external table's rowcount async #32997 #32998

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Apr 1, 2024
Merged

[opt](rowcount) refresh external table's rowcount async #32997 #32998

morningman merged 1 commit into from
Apr 1, 2024

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Mar 28, 2024

bp #32997

@doris-robot
Copy link

doris-robot commented Mar 28, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@morningman
Copy link
Contributor Author

morningman commented Mar 28, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Mar 28, 2024

TPC-H: Total hot run time: 49431 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a3c5862bceecbde408b92f90f035cdff4aee5bff, data reload: false

------ Round 1 ----------------------------------
q1	17706	4338	4384	4338
q2	2077	146	139	139
q3	10498	1924	1870	1870
q4	10319	1239	1312	1239
q5	8411	4336	3938	3938
q6	227	123	123	123
q7	2053	1624	1612	1612
q8	9532	2693	2704	2693
q9	11813	10474	10298	10298
q10	8591	3472	3454	3454
q11	422	242	235	235
q12	467	289	299	289
q13	18365	3935	4033	3935
q14	351	329	325	325
q15	494	444	460	444
q16	696	594	576	576
q17	1113	946	928	928
q18	7112	6756	6883	6756
q19	1657	1545	1532	1532
q20	555	303	306	303
q21	4446	4133	4013	4013
q22	492	391	398	391
Total cold run time: 117397 ms
Total hot run time: 49431 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4287	4268	4265	4265
q2	314	214	225	214
q3	4162	4079	4126	4079
q4	2732	2726	2721	2721
q5	7231	7140	7138	7138
q6	234	119	117	117
q7	3183	2832	2809	2809
q8	4340	4402	4422	4402
q9	16912	16883	16893	16883
q10	4192	4270	4279	4270
q11	715	672	676	672
q12	1020	844	849	844
q13	7157	3719	3687	3687
q14	437	422	412	412
q15	490	464	446	446
q16	748	713	712	712
q17	3853	3798	3839	3798
q18	8747	8607	8660	8607
q19	1688	1700	1655	1655
q20	2393	2119	2098	2098
q21	8456	8425	8447	8425
q22	1032	934	935	934
Total cold run time: 84323 ms
Total hot run time: 79188 ms

@doris-robot
Copy link

doris-robot commented Mar 28, 2024

TPC-DS: Total hot run time: 200831 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a3c5862bceecbde408b92f90f035cdff4aee5bff, data reload: false

query1	911	391	387	387
query2	6510	2153	2019	2019
query3	6915	196	197	196
query4	20911	18113	17934	17934
query5	19707	6505	6469	6469
query6	286	220	223	220
query7	4160	296	292	292
query8	262	235	266	235
query9	3183	2750	2676	2676
query10	420	277	296	277
query11	11192	10638	10579	10579
query12	117	80	69	69
query13	5581	658	657	657
query14	17711	13380	13202	13202
query15	351	241	232	232
query16	6453	268	249	249
query17	1589	1455	854	854
query18	2294	403	412	403
query19	205	143	144	143
query20	79	74	72	72
query21	185	93	93	93
query22	5235	4956	5000	4956
query23	32359	31877	31903	31877
query24	6954	6535	6509	6509
query25	522	416	404	404
query26	524	162	159	159
query27	1884	292	294	292
query28	6044	2274	2258	2258
query29	2857	2772	2728	2728
query30	241	159	160	159
query31	921	734	736	734
query32	62	60	58	58
query33	389	262	239	239
query34	847	453	470	453
query35	1108	917	918	917
query36	1395	1276	1206	1206
query37	88	58	62	58
query38	3057	2902	2923	2902
query39	1353	1304	1302	1302
query40	197	90	90	90
query41	35	33	32	32
query42	81	89	80	80
query43	605	591	560	560
query44	1123	716	736	716
query45	239	229	225	225
query46	1238	976	984	976
query47	1894	1630	1863	1630
query48	990	689	666	666
query49	603	363	364	363
query50	865	601	630	601
query51	4730	4659	4642	4642
query52	93	84	85	84
query53	442	317	316	316
query54	2641	2494	2418	2418
query55	96	78	89	78
query56	212	201	207	201
query57	1174	1151	1170	1151
query58	214	196	195	195
query59	3355	3361	3382	3361
query60	202	203	198	198
query61	86	86	86	86
query62	824	487	480	480
query63	468	326	335	326
query64	2301	1483	1441	1441
query65	3673	3553	3544	3544
query66	777	367	353	353
query67	16570	15726	14886	14886
query68	9387	691	680	680
query69	580	328	333	328
query70	1753	1358	1437	1358
query71	419	292	302	292
query72	6535	3445	3446	3445
query73	729	327	318	318
query74	6372	5870	5857	5857
query75	5232	3611	3600	3600
query76	5816	1181	1192	1181
query77	997	256	243	243
query78	12680	11797	11868	11797
query79	8553	644	650	644
query80	1483	391	390	390
query81	480	224	221	221
query82	1637	97	93	93
query83	169	132	131	131
query84	259	66	68	66
query85	849	288	279	279
query86	321	289	325	289
query87	3195	3011	3041	3011
query88	4965	2299	2297	2297
query89	375	281	294	281
query90	1974	200	202	200
query91	158	116	118	116
query92	53	49	47	47
query93	6415	600	597	597
query94	658	202	206	202
query95	1088	1045	1089	1045
query96	642	319	317	317
query97	6407	6377	6285	6285
query98	192	176	175	175
query99	3000	890	878	878
Total cold run time: 313963 ms
Total hot run time: 200831 ms

@doris-robot
Copy link

doris-robot commented Mar 28, 2024

ClickBench: Total hot run time: 31.02 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a3c5862bceecbde408b92f90f035cdff4aee5bff, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.24	0.05	0.05
query4	1.78	0.06	0.07
query5	0.54	0.53	0.52
query6	1.35	0.64	0.62
query7	0.02	0.00	0.01
query8	0.04	0.03	0.02
query9	0.52	0.47	0.49
query10	0.52	0.54	0.53
query11	0.12	0.08	0.08
query12	0.12	0.09	0.09
query13	0.61	0.62	0.60
query14	0.77	0.79	0.80
query15	0.76	0.75	0.75
query16	0.38	0.37	0.38
query17	1.01	1.01	0.99
query18	0.23	0.24	0.26
query19	1.95	1.85	1.83
query20	0.01	0.01	0.01
query21	15.47	0.57	0.55
query22	1.98	2.34	1.52
query23	16.82	1.12	1.01
query24	5.92	1.29	2.65
query25	0.33	0.12	0.05
query26	0.82	0.16	0.15
query27	0.05	0.03	0.03
query28	5.53	0.74	0.74
query29	12.65	2.41	2.35
query30	0.55	0.54	0.53
query31	2.80	0.38	0.37
query32	3.40	0.50	0.49
query33	3.07	3.05	3.08
query34	15.29	4.77	4.79
query35	4.86	4.80	4.84
query36	1.03	1.00	1.02
query37	0.06	0.05	0.05
query38	0.04	0.02	0.02
query39	0.02	0.02	0.02
query40	0.16	0.15	0.14
query41	0.06	0.02	0.01
query42	0.03	0.01	0.01
query43	0.02	0.01	0.02
Total cold run time: 102.02 s
Total hot run time: 31.02 s

@doris-robot
Copy link

doris-robot commented Mar 28, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a3c5862bceecbde408b92f90f035cdff4aee5bff with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.0 seconds inserted 10000000 Rows, about 476K ops/s

@morningman morningman marked this pull request as draft March 29, 2024 07:27
@morningman
[opt](rowcount) refresh external table's rowcount async (apache#32997)
a8c31a8 
In previous implementation, the row count cache will be expired after 10min(by default),
and after expiration, the next row count request will miss the cache, causing unstable query plan.

In this PR, the cache will be refreshed after Config.external_cache_expire_time_minutes_after_access,
so that the cache entry will remain fresh.
@morningman morningman marked this pull request as ready for review March 31, 2024 12:12
@morningman
Copy link
Contributor Author

morningman commented Mar 31, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Mar 31, 2024

TPC-H: Total hot run time: 49752 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a8c31a8196c2d7ab63f77657ad810e793489c08d, data reload: false

------ Round 1 ----------------------------------
q1	17686	4349	4352	4349
q2	2046	149	151	149
q3	10648	1853	1887	1853
q4	10301	1270	1340	1270
q5	8496	3955	3929	3929
q6	232	122	125	122
q7	2049	1617	1584	1584
q8	9309	2771	2695	2695
q9	10847	10486	10389	10389
q10	8615	3493	3470	3470
q11	432	243	248	243
q12	456	300	297	297
q13	18353	3937	3998	3937
q14	350	322	331	322
q15	524	456	450	450
q16	701	586	592	586
q17	1099	934	978	934
q18	7338	6909	6823	6823
q19	1663	1580	1593	1580
q20	500	310	313	310
q21	4430	4086	4061	4061
q22	495	401	399	399
Total cold run time: 116570 ms
Total hot run time: 49752 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4308	4262	4310	4262
q2	320	221	223	221
q3	4150	4178	4133	4133
q4	2744	2745	2741	2741
q5	7232	7162	7170	7162
q6	235	118	119	118
q7	3257	2829	2825	2825
q8	4319	4420	4429	4420
q9	17149	16933	16961	16933
q10	4233	4198	4232	4198
q11	767	668	687	668
q12	1026	834	860	834
q13	7283	3737	3697	3697
q14	459	426	416	416
q15	511	458	453	453
q16	747	701	713	701
q17	3837	3827	3859	3827
q18	8836	8596	8813	8596
q19	1708	1690	1634	1634
q20	2342	2148	2088	2088
q21	8604	8458	8505	8458
q22	1048	984	973	973
Total cold run time: 85115 ms
Total hot run time: 79358 ms

@doris-robot
Copy link

doris-robot commented Mar 31, 2024

TPC-DS: Total hot run time: 200119 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a8c31a8196c2d7ab63f77657ad810e793489c08d, data reload: false

query1	924	399	380	380
query2	6524	2053	2172	2053
query3	6943	201	195	195
query4	20895	17943	17966	17943
query5	19718	6557	6412	6412
query6	274	223	229	223
query7	4156	296	291	291
query8	261	237	236	236
query9	3189	2744	2676	2676
query10	409	284	302	284
query11	11332	10688	10642	10642
query12	125	75	77	75
query13	5580	650	639	639
query14	17566	13730	13037	13037
query15	380	228	232	228
query16	6472	270	257	257
query17	1748	1471	848	848
query18	2348	411	401	401
query19	207	138	143	138
query20	78	71	72	71
query21	181	99	90	90
query22	5191	4843	4925	4843
query23	32628	31944	32001	31944
query24	6898	6449	6489	6449
query25	531	403	417	403
query26	528	160	155	155
query27	1852	292	295	292
query28	6082	2262	2221	2221
query29	2873	2770	2830	2770
query30	243	162	159	159
query31	905	730	734	730
query32	64	60	57	57
query33	391	242	242	242
query34	850	460	480	460
query35	1109	926	948	926
query36	1233	1098	1087	1087
query37	87	66	61	61
query38	3133	2898	2935	2898
query39	1378	1317	1311	1311
query40	212	93	99	93
query41	35	33	33	33
query42	95	83	83	83
query43	615	574	554	554
query44	1189	727	714	714
query45	241	227	227	227
query46	1242	975	956	956
query47	1849	1512	1818	1512
query48	972	673	659	659
query49	623	358	366	358
query50	854	588	606	588
query51	4782	4636	4661	4636
query52	94	80	69	69
query53	442	331	320	320
query54	2664	2426	2437	2426
query55	84	83	80	80
query56	209	189	202	189
query57	1179	1108	1119	1108
query58	203	190	185	185
query59	3371	3232	3136	3136
query60	208	186	201	186
query61	84	84	88	84
query62	838	419	499	419
query63	466	333	337	333
query64	2352	1508	1343	1343
query65	3636	3521	3530	3521
query66	772	362	357	357
query67	15338	15773	15258	15258
query68	9510	668	658	658
query69	566	333	338	333
query70	1742	1251	1434	1251
query71	424	296	297	296
query72	6571	3425	3418	3418
query73	731	336	317	317
query74	6305	5827	5835	5827
query75	5394	3723	3735	3723
query76	5864	1124	1207	1124
query77	973	253	246	246
query78	12565	12510	11721	11721
query79	11870	652	602	602
query80	709	398	392	392
query81	494	225	232	225
query82	1083	96	100	96
query83	190	138	133	133
query84	260	72	69	69
query85	725	280	275	275
query86	336	296	282	282
query87	3230	3017	3028	3017
query88	5411	2309	2307	2307
query89	430	315	293	293
query90	2187	209	192	192
query91	147	115	115	115
query92	55	50	51	50
query93	6461	603	559	559
query94	873	202	206	202
query95	1107	1064	1073	1064
query96	647	325	323	323
query97	6478	6418	6359	6359
query98	182	174	169	169
query99	2897	936	927	927
Total cold run time: 316352 ms
Total hot run time: 200119 ms

@doris-robot
Copy link

doris-robot commented Mar 31, 2024

ClickBench: Total hot run time: 30.84 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a8c31a8196c2d7ab63f77657ad810e793489c08d, data reload: false

query1	0.03	0.02	0.02
query2	0.08	0.02	0.02
query3	0.25	0.05	0.05
query4	1.79	0.07	0.07
query5	0.54	0.52	0.52
query6	1.23	0.61	0.61
query7	0.02	0.02	0.01
query8	0.03	0.02	0.03
query9	0.53	0.49	0.48
query10	0.53	0.54	0.55
query11	0.12	0.08	0.09
query12	0.11	0.10	0.09
query13	0.61	0.62	0.61
query14	0.78	0.80	0.79
query15	0.77	0.76	0.76
query16	0.38	0.37	0.37
query17	1.00	1.00	1.02
query18	0.22	0.27	0.24
query19	1.86	1.86	1.85
query20	0.02	0.00	0.01
query21	15.46	0.58	0.57
query22	2.00	2.43	1.50
query23	17.47	0.99	0.89
query24	5.88	1.21	1.34
query25	0.38	0.12	0.05
query26	0.71	0.15	0.16
query27	0.03	0.04	0.05
query28	6.67	0.73	0.77
query29	12.59	2.24	2.37
query30	0.59	0.52	0.52
query31	2.82	0.40	0.38
query32	3.39	0.49	0.49
query33	3.06	3.07	3.05
query34	15.29	4.79	4.79
query35	4.85	4.84	4.83
query36	1.05	1.01	1.02
query37	0.07	0.04	0.05
query38	0.03	0.02	0.02
query39	0.02	0.02	0.02
query40	0.16	0.14	0.14
query41	0.06	0.01	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 103.52 s
Total hot run time: 30.84 s

@doris-robot
Copy link

doris-robot commented Mar 31, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a8c31a8196c2d7ab63f77657ad810e793489c08d with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       20.1 seconds inserted 10000000 Rows, about 497K ops/s

@morningman morningman merged commit 2e5fb94 into apache:branch-2.0 Apr 1, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@morningman
[opt](rowcount) refresh external table's rowcount async (apache#32997) (
53eb0e6 
apache#32998)

bp apache#32997
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@morningman @doris-robot