Skip to content

Branch-3.0 [Fix](RowStore) fix partial columns as row store (#49542) #49609

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Mar 30, 2025
Merged

Branch-3.0 [Fix](RowStore) fix partial columns as row store (#49542) #49609

dataroaring merged 1 commit into from
Mar 30, 2025

Conversation

@eldenmoon
Copy link
Member

@eldenmoon eldenmoon commented Mar 28, 2025

cherry-pick from #49542

@eldenmoon eldenmoon requested a review from dataroaring as a code owner March 28, 2025 02:26
@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 28, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@eldenmoon
Copy link
Member Author

eldenmoon commented Mar 28, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 28, 2025

TPC-H: Total hot run time: 40021 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6db64dc4fcb7feecc9c9c5103fe29367ac887d86, data reload: false

------ Round 1 ----------------------------------
q1	17591	6664	6583	6583
q2	2084	173	170	170
q3	10666	1092	1134	1092
q4	10543	729	705	705
q5	7759	2893	2854	2854
q6	223	135	129	129
q7	980	605	617	605
q8	9368	1929	2029	1929
q9	6586	6422	6403	6403
q10	6975	2294	2273	2273
q11	471	247	248	247
q12	398	219	209	209
q13	17777	2935	3003	2935
q14	235	202	206	202
q15	499	463	451	451
q16	657	589	602	589
q17	962	575	498	498
q18	7191	6685	6737	6685
q19	1388	1088	1114	1088
q20	482	200	200	200
q21	4001	3280	3199	3199
q22	1113	1008	975	975
Total cold run time: 107949 ms
Total hot run time: 40021 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6553	6554	6542	6542
q2	327	242	227	227
q3	2921	2801	2864	2801
q4	2032	1777	1796	1777
q5	5720	5756	5697	5697
q6	209	135	131	131
q7	2260	1822	1819	1819
q8	3346	3561	3510	3510
q9	8772	8860	9007	8860
q10	3610	3611	3564	3564
q11	611	492	508	492
q12	881	619	637	619
q13	11035	3220	3160	3160
q14	295	264	280	264
q15	513	453	475	453
q16	701	677	656	656
q17	1825	1650	1589	1589
q18	8256	7831	7748	7748
q19	1665	1528	1658	1528
q20	2100	1903	1858	1858
q21	5464	5383	5421	5383
q22	1140	1047	1049	1047
Total cold run time: 70236 ms
Total hot run time: 59725 ms

@doris-robot
Copy link

doris-robot commented Mar 28, 2025

TPC-DS: Total hot run time: 197520 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6db64dc4fcb7feecc9c9c5103fe29367ac887d86, data reload: false

query1	1340	909	931	909
query2	6224	2001	1987	1987
query3	10943	4454	4404	4404
query4	65490	27796	23436	23436
query5	5015	475	443	443
query6	405	177	177	177
query7	5540	318	303	303
query8	311	223	221	221
query9	8875	2578	2580	2578
query10	452	292	259	259
query11	17205	15117	15802	15117
query12	164	105	108	105
query13	1496	451	420	420
query14	10554	7261	7641	7261
query15	206	186	182	182
query16	7177	508	497	497
query17	1072	599	609	599
query18	2083	329	318	318
query19	217	170	183	170
query20	122	115	114	114
query21	207	103	104	103
query22	4785	4768	4546	4546
query23	34963	34165	34070	34070
query24	6120	2904	2893	2893
query25	535	446	438	438
query26	651	172	173	172
query27	1848	350	357	350
query28	4229	2467	2455	2455
query29	717	474	451	451
query30	253	170	158	158
query31	1006	820	826	820
query32	72	58	58	58
query33	405	304	294	294
query34	920	516	508	508
query35	841	727	732	727
query36	1138	976	967	967
query37	125	66	70	66
query38	4071	4031	4022	4022
query39	1499	1514	1483	1483
query40	214	98	98	98
query41	52	50	48	48
query42	113	104	100	100
query43	534	487	492	487
query44	1180	814	810	810
query45	184	170	168	168
query46	1138	745	726	726
query47	2030	1921	1951	1921
query48	481	374	391	374
query49	758	402	409	402
query50	862	432	430	430
query51	7449	7231	7074	7074
query52	103	88	91	88
query53	258	186	181	181
query54	563	446	458	446
query55	78	74	73	73
query56	248	247	248	247
query57	1209	1095	1127	1095
query58	216	212	210	210
query59	3076	2961	2955	2955
query60	284	255	256	255
query61	109	110	106	106
query62	764	665	657	657
query63	217	191	190	190
query64	1471	665	648	648
query65	3262	3201	3233	3201
query66	720	310	304	304
query67	16028	15557	15643	15557
query68	4062	595	582	582
query69	425	276	270	270
query70	1140	1153	1125	1125
query71	334	255	249	249
query72	6163	4216	4038	4038
query73	745	350	363	350
query74	10204	8990	8998	8990
query75	3341	2617	2660	2617
query76	1941	1117	1005	1005
query77	508	276	271	271
query78	10724	9651	9665	9651
query79	1518	596	604	596
query80	865	437	419	419
query81	506	239	239	239
query82	1227	91	84	84
query83	242	152	152	152
query84	287	84	81	81
query85	908	296	295	295
query86	346	295	296	295
query87	4589	4244	4318	4244
query88	3894	2410	2338	2338
query89	426	301	301	301
query90	2007	182	188	182
query91	182	149	154	149
query92	67	53	51	51
query93	1976	560	557	557
query94	780	303	298	298
query95	355	259	260	259
query96	620	280	283	280
query97	3336	3195	3135	3135
query98	216	222	200	200
query99	1607	1318	1285	1285
Total cold run time: 318418 ms
Total hot run time: 197520 ms

@doris-robot
Copy link

doris-robot commented Mar 28, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 38.87% (10175/26178)
Line Coverage 30.32% (86885/286592)
Region Coverage 29.33% (44603/152084)
Branch Coverage 25.85% (22690/87764)

@doris-robot
Copy link

doris-robot commented Mar 28, 2025

ClickBench: Total hot run time: 31.93 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6db64dc4fcb7feecc9c9c5103fe29367ac887d86, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.07
query4	1.63	0.10	0.10
query5	0.52	0.52	0.51
query6	1.14	0.72	0.73
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.55	0.49	0.49
query10	0.55	0.55	0.56
query11	0.13	0.10	0.10
query12	0.14	0.11	0.12
query13	0.60	0.61	0.59
query14	2.73	2.84	2.74
query15	0.90	0.82	0.82
query16	0.38	0.39	0.39
query17	1.08	1.04	0.99
query18	0.23	0.21	0.21
query19	1.80	1.89	2.01
query20	0.01	0.01	0.01
query21	15.38	0.58	0.59
query22	2.80	2.56	1.63
query23	17.17	0.84	0.73
query24	3.20	0.97	1.99
query25	0.22	0.21	0.05
query26	0.52	0.15	0.14
query27	0.04	0.05	0.03
query28	9.45	0.54	0.48
query29	12.57	3.27	3.22
query30	0.25	0.05	0.06
query31	2.87	0.38	0.39
query32	3.23	0.46	0.45
query33	2.94	2.98	3.01
query34	16.84	4.57	4.56
query35	4.56	4.53	4.55
query36	0.66	0.47	0.47
query37	0.10	0.06	0.06
query38	0.05	0.03	0.04
query39	0.04	0.03	0.02
query40	0.16	0.12	0.12
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.98 s
Total hot run time: 31.93 s

@eldenmoon
[Fix](RowStore) fix partial columns as row store (apache#49542)
56239fd 
1. fix the row store columns comparison logic in SchemaChangeHandler.
The tests verify the correct behavior when comparing row store columns
during ALTER TABLE operations
2. fix partial columns when meet delete sign

These tests ensure that the system correctly determines when schema
changes are needed when users modify row store column properties through
ALTER TABLE statements.
@eldenmoon eldenmoon force-pushed the pick-partial-rs-fix branch from 6db64dc to 56239fd Compare March 28, 2025 03:53
@eldenmoon
Copy link
Member Author

eldenmoon commented Mar 28, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 28, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 38.87% (10176/26178)
Line Coverage 30.32% (86894/286592)
Region Coverage 29.33% (44609/152084)
Branch Coverage 25.86% (22693/87764)

dataroaring
dataroaring approved these changes Mar 30, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 0c3e73e into apache:branch-3.0 Mar 30, 2025
20 of 23 checks passed
@gavinchou gavinchou mentioned this pull request Apr 23, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@eldenmoon @hello-stephen @doris-robot @dataroaring