[enhancement](nereids) speedup sql cache with variable #37090

924060929 · 2024-07-01T10:50:14Z

Proposed changes

current sql cache will miss cache in fe and process by the fashion path cause low qps, when the variable is changed:

select *
from tbl
where dt = @dt_var  -- maybe range between 10 days

this pr support try to hit sql cache even the variable is changed.

Before this optimize: one thread qps is 300.
After this optimize: one thread qps is 3000.

refactor and use ComputeResultSet interface to return resultSet if fe can process

doris-robot · 2024-07-01T10:50:19Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

924060929 · 2024-07-01T10:55:17Z

run buildall

doris-robot · 2024-07-01T12:03:50Z

TPC-H: Total hot run time: 40349 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6a4b87f558e7f510eaed9e6a8a07afce6e9f1be0, data reload: false

------ Round 1 ----------------------------------
q1	18030	4729	4400	4400
q2	2292	208	197	197
q3	10463	1203	1182	1182
q4	10194	757	699	699
q5	7481	2692	2649	2649
q6	220	145	139	139
q7	970	628	602	602
q8	9236	2077	2083	2077
q9	8718	6503	6471	6471
q10	8935	3781	3848	3781
q11	439	233	241	233
q12	403	230	231	230
q13	18806	2956	2991	2956
q14	267	233	230	230
q15	521	481	487	481
q16	533	378	368	368
q17	967	735	771	735
q18	8107	7492	7531	7492
q19	7741	1532	1486	1486
q20	665	341	347	341
q21	5408	3261	3936	3261
q22	412	339	342	339
Total cold run time: 120808 ms
Total hot run time: 40349 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4415	4229	4272	4229
q2	366	256	266	256
q3	2966	2749	2742	2742
q4	1897	1601	1572	1572
q5	5269	5326	5290	5290
q6	225	131	135	131
q7	2147	1738	1723	1723
q8	3216	3364	3348	3348
q9	8357	8341	8324	8324
q10	3876	3661	3676	3661
q11	587	483	488	483
q12	794	613	606	606
q13	17551	2966	2958	2958
q14	289	256	264	256
q15	522	480	488	480
q16	468	412	410	410
q17	1776	1501	1478	1478
q18	7784	7539	7332	7332
q19	1691	1554	1515	1515
q20	1987	1769	1800	1769
q21	5093	4774	4898	4774
q22	595	569	541	541
Total cold run time: 71871 ms
Total hot run time: 53878 ms

doris-robot · 2024-07-01T12:14:55Z

TPC-DS: Total hot run time: 173375 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6a4b87f558e7f510eaed9e6a8a07afce6e9f1be0, data reload: false

query1	911	383	383	383
query2	6460	2354	2295	2295
query3	6656	214	219	214
query4	18853	17509	17285	17285
query5	4188	484	487	484
query6	292	188	174	174
query7	4606	298	298	298
query8	342	284	303	284
query9	8509	2392	2370	2370
query10	627	301	289	289
query11	10476	10119	10080	10080
query12	132	84	84	84
query13	1650	363	369	363
query14	10200	7074	7767	7074
query15	242	184	187	184
query16	7931	276	276	276
query17	1874	578	554	554
query18	2020	285	285	285
query19	197	157	154	154
query20	90	84	82	82
query21	210	129	124	124
query22	4294	3990	4075	3990
query23	34051	33147	33405	33147
query24	11880	2821	2865	2821
query25	662	356	374	356
query26	1774	158	158	158
query27	2948	321	318	318
query28	7229	2070	2076	2070
query29	1078	637	615	615
query30	280	151	148	148
query31	944	721	746	721
query32	118	54	53	53
query33	775	299	291	291
query34	934	466	476	466
query35	734	608	642	608
query36	1057	936	974	936
query37	208	77	74	74
query38	2916	2757	2821	2757
query39	858	784	781	781
query40	281	123	126	123
query41	53	53	58	53
query42	122	103	98	98
query43	573	547	530	530
query44	1300	722	730	722
query45	191	163	167	163
query46	1081	769	747	747
query47	1863	1778	1753	1753
query48	374	304	301	301
query49	1191	425	414	414
query50	772	394	398	394
query51	6932	6894	6834	6834
query52	106	96	92	92
query53	358	288	295	288
query54	913	448	443	443
query55	75	74	78	74
query56	288	269	275	269
query57	1180	1077	1070	1070
query58	266	246	260	246
query59	3292	3080	2986	2986
query60	326	276	276	276
query61	92	90	91	90
query62	647	437	434	434
query63	337	294	291	291
query64	9835	2246	1777	1777
query65	3178	3114	3111	3111
query66	1333	334	342	334
query67	15482	15000	14986	14986
query68	4720	543	554	543
query69	533	416	333	333
query70	1149	1143	1122	1122
query71	385	290	292	290
query72	7139	5710	5379	5379
query73	761	327	326	326
query74	5998	5494	5477	5477
query75	3371	2637	2660	2637
query76	3042	897	940	897
query77	674	297	308	297
query78	10313	10010	9831	9831
query79	2423	524	523	523
query80	1851	480	471	471
query81	586	218	219	218
query82	799	102	104	102
query83	298	180	179	179
query84	259	140	87	87
query85	2034	284	271	271
query86	486	333	338	333
query87	3303	3100	3118	3100
query88	4262	2375	2369	2369
query89	479	379	388	379
query90	1804	191	191	191
query91	132	97	100	97
query92	66	52	52	52
query93	2478	522	504	504
query94	1136	192	185	185
query95	400	321	321	321
query96	592	273	267	267
query97	3153	3023	3001	3001
query98	226	207	196	196
query99	1268	854	852	852
Total cold run time: 277767 ms
Total hot run time: 173375 ms

doris-robot · 2024-07-01T12:20:25Z

ClickBench: Total hot run time: 30.42 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6a4b87f558e7f510eaed9e6a8a07afce6e9f1be0, data reload: false

query1	0.04	0.04	0.02
query2	0.07	0.04	0.04
query3	0.22	0.06	0.05
query4	1.66	0.08	0.09
query5	0.50	0.47	0.48
query6	1.14	0.72	0.72
query7	0.02	0.02	0.02
query8	0.05	0.04	0.05
query9	0.54	0.48	0.49
query10	0.56	0.55	0.55
query11	0.14	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.60	0.59
query14	0.79	0.79	0.78
query15	0.84	0.82	0.82
query16	0.34	0.36	0.37
query17	0.95	0.95	1.01
query18	0.22	0.24	0.26
query19	1.80	1.74	1.70
query20	0.01	0.01	0.01
query21	15.46	0.77	0.66
query22	4.82	7.41	1.58
query23	18.28	1.45	1.34
query24	2.13	0.23	0.22
query25	0.17	0.08	0.08
query26	0.25	0.18	0.18
query27	0.08	0.08	0.08
query28	13.25	1.01	1.00
query29	12.60	3.36	3.32
query30	0.25	0.06	0.05
query31	2.90	0.40	0.38
query32	3.25	0.49	0.47
query33	2.85	2.97	2.88
query34	17.05	4.50	4.39
query35	4.59	4.55	4.58
query36	0.66	0.47	0.49
query37	0.20	0.16	0.16
query38	0.15	0.15	0.16
query39	0.04	0.04	0.04
query40	0.17	0.14	0.14
query41	0.10	0.05	0.06
query42	0.06	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.99 s
Total hot run time: 30.42 s

fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/ComputeResultSet.java

924060929 · 2024-07-01T14:26:11Z

run buildall

doris-robot · 2024-07-01T14:43:49Z

TPC-H: Total hot run time: 40160 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1bc3a40b7fe548e6b8105575ca3eee196039376c, data reload: false

------ Round 1 ----------------------------------
q1	18151	4494	4460	4460
q2	2076	185	187	185
q3	10450	1240	1136	1136
q4	10196	834	727	727
q5	7502	2672	2666	2666
q6	220	138	135	135
q7	965	590	603	590
q8	9219	2082	2046	2046
q9	9002	6547	6530	6530
q10	8833	3749	3739	3739
q11	470	234	240	234
q12	400	235	235	235
q13	17771	3002	2984	2984
q14	271	226	220	220
q15	521	478	485	478
q16	489	371	367	367
q17	965	609	745	609
q18	7955	7570	7463	7463
q19	6698	1462	1425	1425
q20	689	331	330	330
q21	4909	3265	3867	3265
q22	391	345	336	336
Total cold run time: 118143 ms
Total hot run time: 40160 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4362	4259	4243	4243
q2	357	257	268	257
q3	2944	2790	2737	2737
q4	1897	1600	1603	1600
q5	5244	5280	5239	5239
q6	227	132	129	129
q7	2141	1770	1751	1751
q8	3222	3322	3326	3322
q9	8295	8325	8366	8325
q10	3886	3629	3640	3629
q11	584	497	489	489
q12	784	605	589	589
q13	17488	2999	2964	2964
q14	308	263	252	252
q15	527	496	479	479
q16	469	400	413	400
q17	1788	1491	1467	1467
q18	7639	7552	7434	7434
q19	5238	1498	1538	1498
q20	2015	1775	1771	1771
q21	5001	4782	4729	4729
q22	614	554	547	547
Total cold run time: 75030 ms
Total hot run time: 53851 ms

doris-robot · 2024-07-01T14:55:16Z

TPC-DS: Total hot run time: 173458 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1bc3a40b7fe548e6b8105575ca3eee196039376c, data reload: false

query1	914	384	379	379
query2	6472	2540	2275	2275
query3	6648	206	217	206
query4	19612	17664	17468	17468
query5	4205	475	468	468
query6	263	189	164	164
query7	4597	310	293	293
query8	318	300	308	300
query9	8486	2375	2369	2369
query10	604	299	275	275
query11	10324	10040	9886	9886
query12	142	86	80	80
query13	1656	364	365	364
query14	9189	7771	7639	7639
query15	249	190	194	190
query16	7923	297	274	274
query17	1850	565	537	537
query18	2007	279	280	279
query19	212	153	161	153
query20	87	84	92	84
query21	216	132	133	132
query22	4212	4002	3955	3955
query23	33958	33069	32923	32923
query24	11957	2843	2801	2801
query25	700	381	384	381
query26	1801	158	154	154
query27	2934	315	315	315
query28	7508	2078	2064	2064
query29	1109	642	636	636
query30	285	151	150	150
query31	965	758	767	758
query32	96	55	57	55
query33	771	308	298	298
query34	915	467	486	467
query35	752	647	623	623
query36	1105	954	947	947
query37	285	77	78	77
query38	2899	2755	2755	2755
query39	893	799	800	799
query40	295	127	125	125
query41	55	57	54	54
query42	121	96	111	96
query43	606	555	520	520
query44	1213	731	729	729
query45	199	168	168	168
query46	1089	734	727	727
query47	1868	1790	1786	1786
query48	372	300	300	300
query49	1236	423	446	423
query50	769	380	383	380
query51	6993	6805	6762	6762
query52	110	92	91	91
query53	357	297	290	290
query54	903	448	450	448
query55	79	78	79	78
query56	306	295	311	295
query57	1158	1068	1065	1065
query58	262	254	361	254
query59	3617	3050	3125	3050
query60	326	278	285	278
query61	90	92	93	92
query62	660	441	436	436
query63	328	288	287	287
query64	9846	2241	1725	1725
query65	3143	3073	3078	3073
query66	1385	345	321	321
query67	15823	14932	14989	14932
query68	9135	555	599	555
query69	752	425	341	341
query70	1394	1126	1095	1095
query71	523	279	278	278
query72	9190	5539	5749	5539
query73	2188	327	324	324
query74	6004	5408	5420	5408
query75	5710	2675	2690	2675
query76	5641	957	923	923
query77	784	307	300	300
query78	10826	9829	9638	9638
query79	7866	539	524	524
query80	1878	473	467	467
query81	552	219	215	215
query82	548	104	106	104
query83	287	172	166	166
query84	270	87	86	86
query85	1161	279	304	279
query86	365	322	301	301
query87	3308	3094	3178	3094
query88	4037	2370	2358	2358
query89	536	379	382	379
query90	2060	181	183	181
query91	126	100	100	100
query92	58	51	48	48
query93	5904	515	509	509
query94	1342	194	186	186
query95	399	318	322	318
query96	620	263	264	263
query97	3190	3103	3024	3024
query98	230	208	193	193
query99	1242	829	862	829
Total cold run time: 300454 ms
Total hot run time: 173458 ms

doris-robot · 2024-07-01T15:00:46Z

ClickBench: Total hot run time: 30.62 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1bc3a40b7fe548e6b8105575ca3eee196039376c, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.04	0.05
query4	1.67	0.07	0.07
query5	0.50	0.48	0.48
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.49	0.50
query10	0.55	0.53	0.55
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.58	0.59
query14	0.78	0.78	0.77
query15	0.84	0.81	0.82
query16	0.34	0.35	0.37
query17	0.95	0.96	1.04
query18	0.22	0.24	0.26
query19	1.79	1.66	1.71
query20	0.02	0.01	0.01
query21	15.47	0.77	0.65
query22	4.24	6.83	2.18
query23	18.34	1.34	1.19
query24	2.15	0.22	0.20
query25	0.16	0.10	0.08
query26	0.26	0.17	0.18
query27	0.08	0.08	0.08
query28	13.27	1.00	1.00
query29	12.65	3.32	3.28
query30	0.25	0.06	0.05
query31	2.87	0.38	0.38
query32	3.28	0.48	0.47
query33	2.91	2.84	2.95
query34	16.99	4.40	4.45
query35	4.52	4.49	4.55
query36	0.66	0.46	0.49
query37	0.18	0.16	0.16
query38	0.15	0.16	0.16
query39	0.04	0.04	0.04
query40	0.16	0.15	0.15
query41	0.10	0.05	0.05
query42	0.06	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.47 s
Total hot run time: 30.62 s

github-actions · 2024-07-02T02:41:41Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-07-02T02:41:44Z

PR approved by anyone and no changes requested.

cherry pick from #37090

@dt

…on predicate (#37943) follow up #37090 support reuse sql cache when use variable as partition predicate and variable change: ```sql set @dt='2024-07-16'; -- create cache 1 select * from tbl where dt = @dt; set @dt='2024-07-17'; -- create cache 2, will not invalidate cache 1 select * from tbl where dt = @dt; set @dt='2024-07-16'; -- reuse cache 1 select * from tbl where dt = @dt; ```

1. current sql cache will miss cache in fe and process by the fashion path cause low qps, when the variable is changed: ```sql select * from tbl where dt = @dt_var -- maybe range between 10 days ``` this pr support try to hit sql cache even the variable is changed. Before this optimize: one thread qps is 300. After this optimize: one thread qps is 3000. 2. refactor and use `ComputeResultSet` interface to return resultSet if fe can process

924060929 added 2 commits July 1, 2024 18:46

optimize sql cache with variable

fcbdcbd

optimize sql cache with variable

b991d39

checkstyle

6a4b87f

924060929 changed the title ~~[enhancement](nerieds) speedup sql cache with variable~~ [enhancement](nereids) speedup sql cache with variable Jul 1, 2024

morrySnow reviewed Jul 1, 2024

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/ComputeResultSet.java Outdated Show resolved Hide resolved

morrySnow added the dev/2.1.x label Jul 1, 2024

924060929 added 2 commits July 1, 2024 22:24

add comment

5c08403

add comment

1bc3a40

morrySnow approved these changes Jul 2, 2024

View reviewed changes

github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 2, 2024

starocean999 approved these changes Jul 2, 2024

View reviewed changes

924060929 merged commit b7385ce into apache:master Jul 2, 2024

924060929 deleted the optimize_sql_cache branch July 2, 2024 03:08

924060929 mentioned this pull request Jul 2, 2024

[enhancement](nereids) speedup sql cache with variable (#37090) #37119

Merged

morrySnow pushed a commit that referenced this pull request Jul 2, 2024

[enhancement](nereids) speedup sql cache with variable (#37090) (#37119)

42f4271

cherry pick from #37090

morrySnow added dev/2.1.5-merged and removed dev/2.1.x labels Jul 2, 2024

924060929 mentioned this pull request Jul 16, 2024

[enhancement](nereids) speedup sql cache with use variable as partition predicate #37915

Merged

dataroaring added the dev/3.0.1-merged label Aug 5, 2024

gavinchou mentioned this pull request Aug 19, 2024

Release Note 3.0.1 #39570

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[enhancement](nereids) speedup sql cache with variable #37090

[enhancement](nereids) speedup sql cache with variable #37090

Uh oh!

924060929 commented Jul 1, 2024 •

edited

Loading

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

924060929 commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

Uh oh!

924060929 commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

github-actions bot commented Jul 2, 2024

Uh oh!

github-actions bot commented Jul 2, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[enhancement](nereids) speedup sql cache with variable #37090

[enhancement](nereids) speedup sql cache with variable #37090

Uh oh!

Conversation

924060929 commented Jul 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

924060929 commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

Uh oh!

924060929 commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

github-actions bot commented Jul 2, 2024

Uh oh!

github-actions bot commented Jul 2, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

924060929 commented Jul 1, 2024 •

edited

Loading