branch-3.0: [Improve](cloud) support caching SchemaCloudDictionary in BE side #47629 #48463

github-actions · 2025-02-28T05:07:55Z

Cherry-picked from #47629

…7629) The most important changes focus on adding support for schema dictionary caching, modifying existing methods to utilize the cache, and ensuring schema consistency. ``` /* * SchemaCloudDictionaryCache provides a local cache for SchemaCloudDictionary. * * Caching logic: * - If the dictionary associated with a given key has not had any new columns added * (determined by comparing the serialized data for consistency), * the cached dictionary is directly used to update the dictionary list in the rowset meta * (similar to the process_dictionary logic in write_schema_dict). * - If new columns have been detected, the local cache is disregarded, and the updated * dictionary should be fetched via the meta service. */ ``` use SchemaCloudDictionaryCache in Backend to reduce the frequency of reading and converting SchemaCloudDictionary in MetaService

hello-stephen · 2025-02-28T05:08:01Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

hello-stephen · 2025-02-28T05:08:04Z

run buildall

doris-robot · 2025-02-28T05:20:34Z

TeamCity cloud ut coverage result:
Function Coverage: 82.13% (1062/1293)
Line Coverage: 65.66% (17605/26814)
Region Coverage: 65.13% (8675/13319)
Branch Coverage: 55.09% (4679/8494)
Coverage Report: http://coverage.selectdb-in.cc/coverage/60af2cdbf3c99fd1265bc3e40d6b6563a51bf921_60af2cdbf3c99fd1265bc3e40d6b6563a51bf921_cloud/report/index.html

eldenmoon · 2025-03-05T14:01:26Z

run buildall

doris-robot · 2025-03-05T14:19:35Z

TeamCity cloud ut coverage result:
Function Coverage: 82.13% (1062/1293)
Line Coverage: 65.66% (17605/26814)
Region Coverage: 65.15% (8677/13319)
Branch Coverage: 55.12% (4682/8494)
Coverage Report: http://coverage.selectdb-in.cc/coverage/19d014a1909c5e41f2b015f8905aa1c3b3551f93_19d014a1909c5e41f2b015f8905aa1c3b3551f93_cloud/report/index.html

eldenmoon · 2025-03-07T02:20:39Z

run buildall

doris-robot · 2025-03-07T02:36:43Z

TeamCity cloud ut coverage result:
Function Coverage: 82.13% (1062/1293)
Line Coverage: 65.70% (17618/26814)
Region Coverage: 65.19% (8682/13319)
Branch Coverage: 55.18% (4687/8494)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e6c76affd477c1ad77af6040e985257e7e12e851_e6c76affd477c1ad77af6040e985257e7e12e851_cloud/report/index.html

doris-robot · 2025-03-07T04:23:34Z

TPC-H: Total hot run time: 40308 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e6c76affd477c1ad77af6040e985257e7e12e851, data reload: false

------ Round 1 ----------------------------------
q1	17635	6812	6674	6674
q2	2074	173	158	158
q3	10666	1066	1144	1066
q4	10530	795	737	737
q5	7756	2874	2798	2798
q6	224	137	135	135
q7	992	627	611	611
q8	9361	1943	2074	1943
q9	6643	6461	6392	6392
q10	7010	2239	2347	2239
q11	458	263	265	263
q12	416	214	213	213
q13	17778	2996	2955	2955
q14	246	206	208	206
q15	503	459	463	459
q16	648	605	587	587
q17	987	616	609	609
q18	7358	6699	6720	6699
q19	1385	1089	1086	1086
q20	493	219	201	201
q21	4046	3364	3265	3265
q22	1130	1016	1012	1012
Total cold run time: 108339 ms
Total hot run time: 40308 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6597	6585	7312	6585
q2	334	238	236	236
q3	2955	2808	2970	2808
q4	2073	1816	1817	1816
q5	5805	5782	5771	5771
q6	212	130	136	130
q7	2270	1883	1861	1861
q8	3460	3625	3600	3600
q9	8750	8868	8874	8868
q10	3607	3572	3534	3534
q11	593	497	518	497
q12	851	640	622	622
q13	9415	3184	3165	3165
q14	302	265	283	265
q15	513	471	465	465
q16	690	656	638	638
q17	1865	1649	1613	1613
q18	8281	7791	7887	7791
q19	1687	1553	1648	1553
q20	2080	1889	1895	1889
q21	5631	5339	5436	5339
q22	1132	1049	1027	1027
Total cold run time: 69103 ms
Total hot run time: 60073 ms

doris-robot · 2025-03-07T04:36:17Z

TPC-DS: Total hot run time: 199196 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e6c76affd477c1ad77af6040e985257e7e12e851, data reload: false

query1	1298	917	918	917
query2	6270	2044	1994	1994
query3	10988	4670	4275	4275
query4	61524	29271	23631	23631
query5	5247	493	484	484
query6	410	190	202	190
query7	5461	325	321	321
query8	343	265	258	258
query9	8790	2677	2653	2653
query10	478	287	293	287
query11	17578	15285	15789	15285
query12	160	114	107	107
query13	1461	476	469	469
query14	10289	7580	7029	7029
query15	202	186	182	182
query16	7178	503	476	476
query17	1161	626	622	622
query18	1885	350	339	339
query19	223	180	171	171
query20	131	127	113	113
query21	207	108	107	107
query22	4664	4630	4766	4630
query23	34535	34299	33774	33774
query24	6165	2957	2881	2881
query25	543	440	428	428
query26	661	170	172	170
query27	1712	362	373	362
query28	4258	2547	2513	2513
query29	711	507	471	471
query30	248	176	166	166
query31	1005	819	842	819
query32	67	57	59	57
query33	480	308	307	307
query34	929	505	523	505
query35	902	771	736	736
query36	1103	988	981	981
query37	121	73	72	72
query38	4052	4060	4089	4060
query39	1550	1473	1472	1472
query40	208	105	104	104
query41	56	52	52	52
query42	118	110	103	103
query43	542	496	495	495
query44	1179	827	839	827
query45	190	176	171	171
query46	1153	774	719	719
query47	2045	1926	1986	1926
query48	491	387	410	387
query49	770	432	438	432
query50	845	433	446	433
query51	7450	7308	7326	7308
query52	105	93	98	93
query53	274	190	191	190
query54	584	476	474	474
query55	83	79	79	79
query56	283	272	268	268
query57	1306	1191	1172	1172
query58	231	228	228	228
query59	3304	3168	3084	3084
query60	292	266	266	266
query61	138	130	134	130
query62	789	670	679	670
query63	222	199	197	197
query64	1523	821	791	791
query65	3278	3196	3234	3196
query66	720	316	328	316
query67	15730	15565	15446	15446
query68	4243	585	582	582
query69	430	284	275	275
query70	1203	1152	1162	1152
query71	362	270	263	263
query72	6595	4377	4363	4363
query73	752	377	357	357
query74	10015	9042	9171	9042
query75	3384	2669	2702	2669
query76	1954	1061	1065	1061
query77	508	296	306	296
query78	10595	9616	9564	9564
query79	2929	629	608	608
query80	1403	457	455	455
query81	544	241	240	240
query82	999	94	95	94
query83	177	155	157	155
query84	296	86	81	81
query85	1060	369	367	367
query86	380	308	290	290
query87	4517	4359	4284	4284
query88	4693	2394	2488	2394
query89	427	303	296	296
query90	1961	193	194	193
query91	200	159	163	159
query92	67	55	56	55
query93	2612	545	554	545
query94	829	308	299	299
query95	381	272	274	272
query96	631	289	286	286
query97	3291	3150	3227	3150
query98	219	207	192	192
query99	1634	1286	1304	1286
Total cold run time: 318190 ms
Total hot run time: 199196 ms

doris-robot · 2025-03-07T04:42:00Z

ClickBench: Total hot run time: 32.62 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e6c76affd477c1ad77af6040e985257e7e12e851, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.65	0.07	0.08
query5	0.51	0.50	0.52
query6	1.13	0.74	0.74
query7	0.02	0.01	0.03
query8	0.05	0.05	0.05
query9	0.55	0.49	0.51
query10	0.57	0.56	0.57
query11	0.16	0.12	0.12
query12	0.15	0.13	0.13
query13	0.61	0.60	0.59
query14	2.73	2.87	2.75
query15	0.92	0.83	0.84
query16	0.38	0.38	0.39
query17	1.03	0.99	1.04
query18	0.19	0.19	0.20
query19	1.92	1.79	1.99
query20	0.02	0.02	0.02
query21	15.35	0.66	0.65
query22	3.93	7.70	1.84
query23	18.25	1.35	1.32
query24	2.14	0.23	0.22
query25	0.15	0.08	0.08
query26	0.28	0.18	0.19
query27	0.07	0.08	0.08
query28	13.32	0.61	0.57
query29	12.67	3.31	3.39
query30	0.25	0.06	0.06
query31	2.84	0.40	0.39
query32	3.23	0.47	0.49
query33	3.02	2.94	3.03
query34	16.85	4.54	4.51
query35	4.64	4.55	4.63
query36	0.65	0.51	0.48
query37	0.20	0.16	0.16
query38	0.16	0.16	0.15
query39	0.05	0.05	0.05
query40	0.15	0.13	0.13
query41	0.10	0.05	0.05
query42	0.07	0.06	0.05
query43	0.05	0.05	0.05
Total cold run time: 111.35 s
Total hot run time: 32.62 s

github-actions · 2025-03-11T10:07:39Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-03-11T10:07:42Z

PR approved by anyone and no changes requested.

dataroaring

LGTM

eldenmoon · 2025-03-17T07:51:36Z

run buildall

doris-robot · 2025-03-17T08:12:31Z

TeamCity cloud ut coverage result:
Function Coverage: 82.87% (1074/1296)
Line Coverage: 65.85% (17736/26932)
Region Coverage: 65.20% (8728/13387)
Branch Coverage: 55.19% (4709/8532)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1461b3d7460225525128ef395a87b2ae5b9174e9_1461b3d7460225525128ef395a87b2ae5b9174e9_cloud/report/index.html

eldenmoon · 2025-03-17T09:55:22Z

run buildall

doris-robot · 2025-03-17T10:01:06Z

TeamCity cloud ut coverage result:
Function Coverage: 82.87% (1074/1296)
Line Coverage: 65.86% (17738/26932)
Region Coverage: 65.21% (8729/13387)
Branch Coverage: 55.19% (4709/8532)
Coverage Report: http://coverage.selectdb-in.cc/coverage/01ea3264793908ee336f622240bdbc31c89f3884_01ea3264793908ee336f622240bdbc31c89f3884_cloud/report/index.html

doris-robot · 2025-03-17T11:30:50Z

BE UT Coverage Report

Increment line coverage 24.04% (44/183) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	38.87% (10172/26169)
Line Coverage	30.30% (86743/286280)
Region Coverage	29.35% (44568/151832)
Branch Coverage	25.88% (22667/87596)

dataroaring

LGTM

github-actions bot requested a review from dataroaring as a code owner February 28, 2025 05:07

dataroaring closed this Feb 28, 2025

dataroaring reopened this Feb 28, 2025

eldenmoon added the p0_b label Mar 5, 2025

Merge branch 'branch-3.0' into auto-pick-47629-branch-3.0

19d014a

Merge branch 'branch-3.0' into auto-pick-47629-branch-3.0

e6c76af

gavinchou previously approved these changes Mar 11, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 11, 2025

github-actions bot added the reviewed label Mar 11, 2025

dataroaring previously approved these changes Mar 17, 2025

View reviewed changes

Merge branch 'branch-3.0' into auto-pick-47629-branch-3.0

1461b3d

eldenmoon dismissed stale reviews from dataroaring and gavinchou via 1461b3d March 17, 2025 07:51

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Mar 17, 2025

Merge branch 'branch-3.0' into auto-pick-47629-branch-3.0

01ea326

dataroaring approved these changes Mar 24, 2025

View reviewed changes

dataroaring merged commit f4723a9 into branch-3.0 Mar 24, 2025
24 of 29 checks passed

github-actions bot deleted the auto-pick-47629-branch-3.0 branch March 24, 2025 03:09

gavinchou mentioned this pull request Apr 23, 2025

Release Notes 3.0.5 #50333

Closed

branch-3.0: [Improve](cloud) support caching SchemaCloudDictionary in BE side #47629 #48463

branch-3.0: [Improve](cloud) support caching SchemaCloudDictionary in BE side #47629 #48463

Uh oh!

Conversation

github-actions bot commented Feb 28, 2025

Uh oh!

hello-stephen commented Feb 28, 2025

Uh oh!

hello-stephen commented Feb 28, 2025

Uh oh!

doris-robot commented Feb 28, 2025

Uh oh!

eldenmoon commented Mar 5, 2025

Uh oh!

doris-robot commented Mar 5, 2025

Uh oh!

eldenmoon commented Mar 7, 2025

Uh oh!

doris-robot commented Mar 7, 2025

Uh oh!

doris-robot commented Mar 7, 2025

Uh oh!

doris-robot commented Mar 7, 2025

Uh oh!

doris-robot commented Mar 7, 2025

Uh oh!

github-actions bot commented Mar 11, 2025

Uh oh!

github-actions bot commented Mar 11, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

eldenmoon commented Mar 17, 2025

Uh oh!

doris-robot commented Mar 17, 2025

Uh oh!

eldenmoon commented Mar 17, 2025

Uh oh!

doris-robot commented Mar 17, 2025

Uh oh!

doris-robot commented Mar 17, 2025

BE UT Coverage Report

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants