Skip to content

Conversation

@hubgeter
Copy link
Contributor

@hubgeter hubgeter commented Jun 3, 2025

What problem does this PR solve?

Related PR: #51329

Problem Summary:
In the previous PR #51329, global lazy materialization was implemented.
However, since RPC requests will be sent to other BE nodes during the materialization phase, the external node needs to rely on query ctx when reading the corresponding file after receiving the RPC request, but the query ctx on the BE may have been released, resulting in BE core.
Solution:
By caching some information in query ctx, RPC does not need to rely on query ctx.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hubgeter
Copy link
Contributor Author

hubgeter commented Jun 3, 2025

run buildall

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hubgeter
Copy link
Contributor Author

hubgeter commented Jun 3, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33830 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bcb1615648a7f153a2fa1abdbec96959e367b9d7, data reload: false

------ Round 1 ----------------------------------
q1	26002	5064	5019	5019
q2	1950	273	182	182
q3	10544	1235	692	692
q4	10244	1006	509	509
q5	7892	2371	2396	2371
q6	178	159	130	130
q7	973	726	612	612
q8	9305	1283	1094	1094
q9	6783	5070	5073	5070
q10	6824	2348	1897	1897
q11	493	287	275	275
q12	342	368	211	211
q13	17761	3751	3088	3088
q14	222	224	214	214
q15	561	502	496	496
q16	438	429	366	366
q17	610	847	367	367
q18	7643	7166	7182	7166
q19	1925	970	551	551
q20	332	339	225	225
q21	3710	3148	2345	2345
q22	1069	1014	950	950
Total cold run time: 115801 ms
Total hot run time: 33830 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5204	5068	5080	5068
q2	239	323	225	225
q3	2168	2668	2327	2327
q4	1334	1748	1315	1315
q5	4602	4425	4376	4376
q6	212	176	127	127
q7	2018	1931	1733	1733
q8	2567	2584	2524	2524
q9	7144	7102	7162	7102
q10	3015	3158	2733	2733
q11	572	499	483	483
q12	716	773	599	599
q13	3512	3955	3284	3284
q14	276	288	278	278
q15	547	481	482	481
q16	432	486	460	460
q17	1187	1557	1367	1367
q18	7731	7475	7227	7227
q19	800	777	887	777
q20	1967	2010	1935	1935
q21	4776	4299	4321	4299
q22	1049	1027	983	983
Total cold run time: 52068 ms
Total hot run time: 49703 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185920 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bcb1615648a7f153a2fa1abdbec96959e367b9d7, data reload: false

query1	1004	490	504	490
query2	6565	1843	1816	1816
query3	6741	233	226	226
query4	25704	23195	23303	23195
query5	4317	610	467	467
query6	309	208	215	208
query7	4622	495	283	283
query8	264	229	223	223
query9	8599	2658	2681	2658
query10	461	336	264	264
query11	15696	15148	14791	14791
query12	155	108	103	103
query13	1650	511	402	402
query14	8654	6188	6227	6188
query15	205	196	168	168
query16	7138	598	480	480
query17	1183	703	551	551
query18	1949	413	299	299
query19	186	186	164	164
query20	118	122	124	122
query21	217	131	106	106
query22	4198	4145	4109	4109
query23	33953	33180	33237	33180
query24	8444	2384	2405	2384
query25	518	460	392	392
query26	1244	267	149	149
query27	2774	506	341	341
query28	4308	2160	2132	2132
query29	773	584	441	441
query30	280	217	187	187
query31	895	834	745	745
query32	76	69	72	69
query33	567	380	337	337
query34	797	860	526	526
query35	785	806	752	752
query36	955	1010	891	891
query37	110	104	78	78
query38	4114	4135	4062	4062
query39	1474	1436	1413	1413
query40	212	120	113	113
query41	65	58	58	58
query42	130	114	110	110
query43	497	497	457	457
query44	1296	828	812	812
query45	182	174	174	174
query46	843	1023	625	625
query47	1726	1819	1705	1705
query48	388	413	317	317
query49	755	476	437	437
query50	644	695	419	419
query51	4084	4139	4006	4006
query52	112	106	100	100
query53	225	252	186	186
query54	580	577	503	503
query55	84	83	84	83
query56	308	304	309	304
query57	1148	1160	1088	1088
query58	266	248	258	248
query59	2466	2627	2556	2556
query60	325	318	289	289
query61	128	146	135	135
query62	790	719	661	661
query63	227	189	190	189
query64	4347	1118	785	785
query65	4299	4103	4188	4103
query66	1160	417	320	320
query67	15651	15771	15455	15455
query68	7783	884	535	535
query69	491	310	279	279
query70	1212	1147	1064	1064
query71	463	325	298	298
query72	5383	4777	4848	4777
query73	672	682	357	357
query74	8786	8839	8722	8722
query75	3701	3189	2681	2681
query76	3420	1202	751	751
query77	786	447	289	289
query78	10109	10227	9407	9407
query79	2722	792	584	584
query80	680	520	441	441
query81	487	256	217	217
query82	450	127	95	95
query83	271	251	230	230
query84	251	117	93	93
query85	840	354	313	313
query86	390	296	318	296
query87	4316	4550	4321	4321
query88	3859	2261	2265	2261
query89	378	315	286	286
query90	1828	210	207	207
query91	144	139	110	110
query92	78	65	59	59
query93	2296	937	590	590
query94	677	401	309	309
query95	374	292	289	289
query96	494	586	281	281
query97	2761	2785	2629	2629
query98	231	218	208	208
query99	1426	1429	1311	1311
Total cold run time: 272699 ms
Total hot run time: 185920 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.28 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bcb1615648a7f153a2fa1abdbec96959e367b9d7, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.11	0.11
query3	0.26	0.20	0.20
query4	1.59	0.20	0.18
query5	0.45	0.46	0.44
query6	1.52	0.67	0.67
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.58	0.52	0.50
query10	0.56	0.59	0.56
query11	0.16	0.11	0.11
query12	0.14	0.12	0.12
query13	0.62	0.61	0.60
query14	0.79	0.81	0.80
query15	0.88	0.86	0.87
query16	0.38	0.37	0.40
query17	1.03	1.02	1.02
query18	0.23	0.22	0.22
query19	1.91	1.84	1.78
query20	0.01	0.01	0.02
query21	15.40	0.90	0.55
query22	0.78	1.06	0.64
query23	15.08	1.38	0.60
query24	6.84	1.20	1.08
query25	0.50	0.11	0.19
query26	0.68	0.16	0.13
query27	0.05	0.05	0.05
query28	9.26	0.89	0.46
query29	12.58	4.04	3.31
query30	0.25	0.10	0.07
query31	2.82	0.59	0.38
query32	3.23	0.55	0.48
query33	3.12	3.10	3.07
query34	15.80	5.12	4.44
query35	4.49	4.51	4.48
query36	0.67	0.52	0.48
query37	0.09	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.38 s
Total hot run time: 29.28 s

@hubgeter
Copy link
Contributor Author

hubgeter commented Jun 4, 2025

run buildall

1 similar comment
@hubgeter
Copy link
Contributor Author

hubgeter commented Jun 4, 2025

run buildall

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 2.22% (1/45) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 56.26% (15077/26800)
Line Coverage 45.05% (134520/298623)
Region Coverage 44.16% (67671/153236)
Branch Coverage 38.72% (34694/89612)

@hubgeter
Copy link
Contributor Author

hubgeter commented Jun 5, 2025

run buildall

@hubgeter hubgeter changed the title [fix](topn)fix topn lazy materialize [fix](topn)fix external table topn lazy materialize cause be core. Jun 6, 2025
@hubgeter hubgeter marked this pull request as ready for review June 6, 2025 08:41
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 7, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jun 7, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jun 7, 2025

PR approved by anyone and no changes requested.

Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zy-kkk
Copy link
Member

zy-kkk commented Jun 9, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34596 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 56ae5076db00858ae9cd0f016698c625352b68a1, data reload: false

------ Round 1 ----------------------------------
q1	26342	5077	5069	5069
q2	1963	286	186	186
q3	10273	1259	732	732
q4	10247	1014	531	531
q5	7527	2396	2355	2355
q6	190	166	133	133
q7	931	743	596	596
q8	9300	1278	1082	1082
q9	6861	5116	5110	5110
q10	6895	2324	1912	1912
q11	488	290	275	275
q12	345	353	213	213
q13	17785	3723	3080	3080
q14	236	234	214	214
q15	564	484	490	484
q16	433	442	378	378
q17	597	861	363	363
q18	7685	7143	7224	7143
q19	2179	965	570	570
q20	343	341	220	220
q21	3680	3222	2977	2977
q22	1049	1009	973	973
Total cold run time: 115913 ms
Total hot run time: 34596 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5258	5273	5056	5056
q2	241	323	224	224
q3	2159	2645	2305	2305
q4	1329	1802	1488	1488
q5	4470	4435	4385	4385
q6	220	174	131	131
q7	2042	1903	1773	1773
q8	2589	2670	2538	2538
q9	7240	7195	7070	7070
q10	3034	3197	2763	2763
q11	604	523	497	497
q12	692	751	601	601
q13	3551	3889	3337	3337
q14	294	306	287	287
q15	545	484	489	484
q16	442	469	454	454
q17	1152	1535	1374	1374
q18	7801	7482	7416	7416
q19	780	830	1016	830
q20	2021	2003	1842	1842
q21	4783	4424	4480	4424
q22	1120	1090	1048	1048
Total cold run time: 52367 ms
Total hot run time: 50327 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193393 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 56ae5076db00858ae9cd0f016698c625352b68a1, data reload: false

query1	1444	1087	1066	1066
query2	6330	1855	1864	1855
query3	10991	4599	4565	4565
query4	54392	25100	23882	23882
query5	4999	541	491	491
query6	349	226	212	212
query7	4874	508	297	297
query8	283	235	221	221
query9	5510	2640	2626	2626
query10	461	349	277	277
query11	15134	15061	14931	14931
query12	166	114	106	106
query13	1037	518	417	417
query14	10036	6347	6277	6277
query15	209	216	189	189
query16	7120	695	458	458
query17	1100	751	613	613
query18	1590	408	372	372
query19	195	191	164	164
query20	126	133	120	120
query21	207	129	106	106
query22	4378	4446	4297	4297
query23	34330	33509	33609	33509
query24	6650	2415	2410	2410
query25	464	485	421	421
query26	687	275	157	157
query27	2319	504	347	347
query28	3095	2172	2168	2168
query29	594	578	454	454
query30	268	225	199	199
query31	880	875	776	776
query32	68	60	61	60
query33	453	374	310	310
query34	782	892	522	522
query35	804	834	750	750
query36	981	992	923	923
query37	118	103	80	80
query38	4333	4183	4298	4183
query39	1498	1458	1525	1458
query40	213	118	105	105
query41	62	60	59	59
query42	125	108	112	108
query43	506	513	501	501
query44	1349	852	852	852
query45	178	179	169	169
query46	867	1046	650	650
query47	1861	1869	1824	1824
query48	415	435	337	337
query49	689	502	402	402
query50	678	702	416	416
query51	4262	4227	4151	4151
query52	126	113	107	107
query53	240	259	189	189
query54	584	616	512	512
query55	87	85	79	79
query56	311	305	293	293
query57	1241	1256	1159	1159
query58	283	278	263	263
query59	2781	2881	2658	2658
query60	342	326	310	310
query61	138	127	125	125
query62	737	736	688	688
query63	227	193	193	193
query64	1827	1032	718	718
query65	4220	4162	4167	4162
query66	716	405	308	308
query67	15871	15442	15369	15369
query68	7168	891	525	525
query69	538	335	272	272
query70	1203	1116	1074	1074
query71	501	329	291	291
query72	5806	4874	5049	4874
query73	1416	733	354	354
query74	8896	9136	8887	8887
query75	3862	3265	2714	2714
query76	4100	1205	751	751
query77	616	372	286	286
query78	10181	10177	9280	9280
query79	2697	811	572	572
query80	744	508	455	455
query81	494	251	218	218
query82	435	126	90	90
query83	376	248	239	239
query84	299	107	88	88
query85	773	359	318	318
query86	374	294	273	273
query87	4371	4446	4335	4335
query88	3619	2287	2265	2265
query89	403	322	286	286
query90	1794	206	202	202
query91	144	150	112	112
query92	76	62	66	62
query93	2179	933	586	586
query94	678	408	313	313
query95	373	293	287	287
query96	485	562	287	287
query97	2688	2804	2670	2670
query98	226	209	202	202
query99	1414	1406	1249	1249
Total cold run time: 299120 ms
Total hot run time: 193393 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.85 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 56ae5076db00858ae9cd0f016698c625352b68a1, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.11	0.11
query3	0.25	0.19	0.20
query4	1.60	0.19	0.19
query5	0.43	0.42	0.43
query6	1.17	0.66	0.66
query7	0.02	0.01	0.02
query8	0.05	0.04	0.03
query9	0.57	0.52	0.51
query10	0.58	0.58	0.56
query11	0.16	0.10	0.11
query12	0.15	0.11	0.12
query13	0.62	0.60	0.60
query14	0.79	0.81	0.82
query15	0.88	0.88	0.88
query16	0.37	0.39	0.39
query17	1.03	1.04	1.04
query18	0.22	0.21	0.21
query19	1.89	1.76	1.79
query20	0.02	0.02	0.01
query21	15.40	0.92	0.54
query22	0.78	1.15	0.68
query23	14.96	1.38	0.63
query24	7.33	1.61	0.64
query25	0.47	0.19	0.07
query26	0.60	0.17	0.13
query27	0.06	0.05	0.04
query28	8.87	0.85	0.44
query29	12.53	3.93	3.27
query30	0.25	0.10	0.07
query31	2.84	0.58	0.38
query32	3.24	0.54	0.47
query33	3.03	3.16	3.04
query34	15.70	5.07	4.53
query35	4.54	4.49	4.48
query36	0.66	0.51	0.49
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.04
Total cold run time: 102.7 s
Total hot run time: 28.85 s

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 0.00% (0/42) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 56.38% (15067/26724)
Line Coverage 45.15% (134627/298208)
Region Coverage 44.24% (67739/153109)
Branch Coverage 38.81% (34736/89508)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 4.76% (2/42) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.66% (20952/26302)
Line Coverage 72.51% (216128/298061)
Region Coverage 70.66% (127269/180108)
Branch Coverage 64.37% (65866/102322)

1 similar comment
@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 4.76% (2/42) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.66% (20952/26302)
Line Coverage 72.51% (216128/298061)
Region Coverage 70.66% (127269/180108)
Branch Coverage 64.37% (65866/102322)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 9.52% (4/42) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.77% (20981/26302)
Line Coverage 72.72% (216748/298061)
Region Coverage 70.95% (127786/180108)
Branch Coverage 64.62% (66118/102322)

@zy-kkk zy-kkk merged commit ad045d9 into apache:master Jun 10, 2025
25 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants