Skip to content

Conversation

@bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Mar 17, 2025

What problem does this PR solve?

sometimes there may be confusing but not totally random values in __DORIS_VERSION_COL__(such as 2^29, 2^29+1) on tablets which have schema change before. And this will lead to checksums of tablets inconsistent between replicas. This PR

  1. add a config to skip __DORIS_VERSION_COL__ column when calculating checksum to avoid this problem.
  2. add DCHECK when doing compaction

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@bobhan1
Copy link
Contributor Author

bobhan1 commented Mar 17, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32686 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f941903fb6c9c5ba1995a7b03a56c82b6766a65a, data reload: false

------ Round 1 ----------------------------------
q1	24296	5053	5015	5015
q2	2061	319	190	190
q3	10503	1291	715	715
q4	10238	1053	545	545
q5	7856	2456	2382	2382
q6	200	166	137	137
q7	909	753	616	616
q8	9328	1345	1129	1129
q9	4995	4815	4846	4815
q10	6822	2323	1891	1891
q11	475	273	252	252
q12	348	357	221	221
q13	17771	3715	3156	3156
q14	250	219	208	208
q15	531	483	476	476
q16	621	608	579	579
q17	593	877	353	353
q18	7027	6428	6282	6282
q19	2313	986	587	587
q20	318	340	199	199
q21	2905	2177	1946	1946
q22	1017	1004	992	992
Total cold run time: 111377 ms
Total hot run time: 32686 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5312	5143	5131	5131
q2	241	325	227	227
q3	2208	2683	2275	2275
q4	1422	1827	1408	1408
q5	4316	4285	4527	4285
q6	226	179	130	130
q7	2056	1923	1767	1767
q8	2612	2601	2655	2601
q9	7239	7149	7128	7128
q10	3010	3215	2772	2772
q11	597	498	479	479
q12	677	752	606	606
q13	3482	3964	3251	3251
q14	276	305	276	276
q15	522	474	470	470
q16	657	684	659	659
q17	1160	1645	1344	1344
q18	7809	7616	7620	7616
q19	842	850	858	850
q20	1960	1995	1881	1881
q21	5392	5037	4769	4769
q22	1072	1023	1001	1001
Total cold run time: 53088 ms
Total hot run time: 50926 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185087 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f941903fb6c9c5ba1995a7b03a56c82b6766a65a, data reload: false

query1	997	477	475	475
query2	6577	1936	1942	1936
query3	6806	216	221	216
query4	26215	23431	23183	23183
query5	4352	687	520	520
query6	331	208	200	200
query7	4611	509	290	290
query8	300	244	242	242
query9	8656	2615	2597	2597
query10	492	291	246	246
query11	15833	15506	14798	14798
query12	170	107	116	107
query13	1643	510	399	399
query14	10506	6160	6606	6160
query15	209	192	169	169
query16	7646	627	486	486
query17	1356	703	544	544
query18	1975	396	281	281
query19	182	178	154	154
query20	119	116	112	112
query21	209	124	109	109
query22	4216	4161	4123	4123
query23	33931	32926	32952	32926
query24	7049	2389	2372	2372
query25	578	453	424	424
query26	1215	276	153	153
query27	2450	465	326	326
query28	4309	2416	2425	2416
query29	707	564	428	428
query30	285	213	188	188
query31	988	868	753	753
query32	77	68	63	63
query33	561	379	317	317
query34	807	844	498	498
query35	779	824	717	717
query36	936	988	941	941
query37	121	97	70	70
query38	4156	4209	4061	4061
query39	1452	1377	1371	1371
query40	216	116	103	103
query41	67	60	52	52
query42	117	101	101	101
query43	503	505	481	481
query44	1307	793	782	782
query45	174	171	169	169
query46	835	1040	616	616
query47	1774	1794	1669	1669
query48	387	409	323	323
query49	775	506	419	419
query50	690	750	415	415
query51	4148	4161	4102	4102
query52	111	110	102	102
query53	237	268	197	197
query54	490	510	412	412
query55	78	80	84	80
query56	272	260	276	260
query57	1115	1126	1051	1051
query58	245	234	244	234
query59	2623	2721	2526	2526
query60	290	289	257	257
query61	123	123	120	120
query62	788	719	670	670
query63	223	186	192	186
query64	4252	1019	712	712
query65	4419	4329	4346	4329
query66	1034	400	304	304
query67	15614	15429	15401	15401
query68	7789	871	500	500
query69	463	357	265	265
query70	1188	1137	1124	1124
query71	418	309	282	282
query72	5620	3586	3765	3586
query73	747	733	363	363
query74	8983	9151	8923	8923
query75	3462	3170	2706	2706
query76	3400	1184	760	760
query77	788	380	335	335
query78	9946	10143	9274	9274
query79	2437	828	581	581
query80	658	516	447	447
query81	475	264	215	215
query82	518	128	104	104
query83	174	174	170	170
query84	246	91	75	75
query85	775	350	314	314
query86	370	307	277	277
query87	4414	4544	4274	4274
query88	3867	2270	2272	2270
query89	386	312	281	281
query90	1928	218	223	218
query91	142	146	110	110
query92	73	63	59	59
query93	2242	1086	576	576
query94	668	417	300	300
query95	357	266	265	265
query96	496	571	279	279
query97	3338	3361	3248	3248
query98	237	205	209	205
query99	1325	1385	1257	1257
Total cold run time: 274639 ms
Total hot run time: 185087 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.65 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f941903fb6c9c5ba1995a7b03a56c82b6766a65a, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.11	0.10
query3	0.25	0.20	0.20
query4	1.60	0.20	0.19
query5	0.60	0.59	0.61
query6	1.18	0.73	0.72
query7	0.03	0.01	0.02
query8	0.04	0.03	0.03
query9	0.58	0.53	0.50
query10	0.60	0.59	0.58
query11	0.15	0.11	0.11
query12	0.15	0.10	0.12
query13	0.62	0.60	0.60
query14	2.80	2.84	2.70
query15	0.92	0.85	0.85
query16	0.39	0.38	0.38
query17	1.04	1.04	1.09
query18	0.22	0.20	0.20
query19	1.89	1.99	1.80
query20	0.01	0.01	0.01
query21	15.34	0.93	0.54
query22	0.75	1.20	0.95
query23	14.70	1.40	0.62
query24	7.19	1.93	0.92
query25	0.44	0.21	0.21
query26	0.59	0.16	0.13
query27	0.05	0.05	0.05
query28	9.80	0.82	0.44
query29	12.62	3.93	3.33
query30	0.25	0.08	0.06
query31	2.82	0.59	0.38
query32	3.24	0.54	0.47
query33	3.02	3.06	3.04
query34	15.79	5.11	4.45
query35	4.49	4.51	4.55
query36	0.66	0.50	0.49
query37	0.08	0.06	0.07
query38	0.06	0.05	0.04
query39	0.04	0.02	0.02
query40	0.16	0.14	0.12
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.47 s
Total hot run time: 31.65 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 0.00% (0/4) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 48.47% (12982/26782)
Line Coverage 37.92% (111287/293448)
Region Coverage 36.88% (56804/154016)
Branch Coverage 32.05% (28579/89160)

@bobhan1
Copy link
Contributor Author

bobhan1 commented Mar 18, 2025

run buildall

dataroaring
dataroaring previously approved these changes Mar 18, 2025
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 18, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 32812 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f941903fb6c9c5ba1995a7b03a56c82b6766a65a, data reload: false

------ Round 1 ----------------------------------
q1	24235	5125	5108	5108
q2	2040	334	192	192
q3	10338	1253	703	703
q4	10226	1002	535	535
q5	7522	2466	2360	2360
q6	190	164	132	132
q7	902	763	600	600
q8	9302	1291	1149	1149
q9	5083	4915	4956	4915
q10	6853	2308	1863	1863
q11	473	273	249	249
q12	349	357	221	221
q13	17746	3703	3112	3112
q14	225	220	209	209
q15	523	480	494	480
q16	619	609	571	571
q17	606	877	346	346
q18	6827	6414	6349	6349
q19	1225	943	572	572
q20	324	340	199	199
q21	2804	2269	1959	1959
q22	1006	1012	988	988
Total cold run time: 109418 ms
Total hot run time: 32812 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5208	5170	5169	5169
q2	247	327	227	227
q3	2146	2686	2313	2313
q4	1462	1869	1370	1370
q5	4212	4137	4397	4137
q6	221	171	130	130
q7	1984	1912	1779	1779
q8	2627	2706	2551	2551
q9	7234	7108	7215	7108
q10	2982	3228	2701	2701
q11	572	509	519	509
q12	698	812	629	629
q13	3575	3927	3282	3282
q14	283	310	268	268
q15	522	466	475	466
q16	662	693	649	649
q17	1152	1607	1341	1341
q18	7770	7670	7598	7598
q19	898	871	964	871
q20	1993	2074	1858	1858
q21	5624	4851	4837	4837
q22	1091	1055	1001	1001
Total cold run time: 53163 ms
Total hot run time: 50794 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193552 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f941903fb6c9c5ba1995a7b03a56c82b6766a65a, data reload: false

query1	1423	1034	1035	1034
query2	6267	1969	1929	1929
query3	10982	4489	4515	4489
query4	54363	25799	23656	23656
query5	5252	576	504	504
query6	386	213	185	185
query7	5118	520	298	298
query8	343	281	243	243
query9	6453	2655	2653	2653
query10	426	308	255	255
query11	15240	15107	14980	14980
query12	164	111	103	103
query13	1165	548	416	416
query14	11197	6937	6982	6937
query15	206	207	185	185
query16	6909	657	486	486
query17	1082	735	625	625
query18	1549	412	329	329
query19	203	206	192	192
query20	123	123	127	123
query21	214	126	114	114
query22	4506	4488	4257	4257
query23	34374	33315	33450	33315
query24	5832	2414	2442	2414
query25	470	488	410	410
query26	728	277	159	159
query27	1862	514	356	356
query28	2841	2465	2493	2465
query29	582	559	434	434
query30	276	213	194	194
query31	861	903	766	766
query32	77	63	65	63
query33	464	364	301	301
query34	770	835	511	511
query35	833	842	775	775
query36	979	1024	902	902
query37	126	104	77	77
query38	4132	4424	4368	4368
query39	1510	1445	1416	1416
query40	212	125	106	106
query41	57	54	49	49
query42	124	101	103	101
query43	499	522	509	509
query44	1378	809	823	809
query45	191	176	168	168
query46	874	1069	655	655
query47	1847	1876	1790	1790
query48	415	448	334	334
query49	701	535	421	421
query50	702	773	460	460
query51	4311	4386	4241	4241
query52	112	110	98	98
query53	243	259	192	192
query54	511	519	451	451
query55	85	81	85	81
query56	295	260	305	260
query57	1145	1185	1105	1105
query58	252	236	248	236
query59	2866	2941	2605	2605
query60	295	301	261	261
query61	127	128	125	125
query62	755	744	691	691
query63	231	201	192	192
query64	1826	1053	699	699
query65	4525	4487	4330	4330
query66	736	400	305	305
query67	15691	15520	15562	15520
query68	7419	815	552	552
query69	539	317	266	266
query70	1196	1137	1159	1137
query71	531	310	285	285
query72	5600	3641	3935	3641
query73	1182	732	352	352
query74	8965	9105	8690	8690
query75	3753	3191	2716	2716
query76	4274	1241	778	778
query77	625	389	291	291
query78	10023	10187	9334	9334
query79	2846	824	585	585
query80	778	548	451	451
query81	486	255	223	223
query82	706	129	97	97
query83	295	173	157	157
query84	277	105	74	74
query85	789	359	324	324
query86	404	299	290	290
query87	4429	4428	4497	4428
query88	3331	2282	2308	2282
query89	416	309	295	295
query90	1956	211	213	211
query91	144	143	117	117
query92	80	66	55	55
query93	1292	1060	589	589
query94	692	422	307	307
query95	357	281	260	260
query96	490	558	277	277
query97	3353	3411	3310	3310
query98	229	214	219	214
query99	1447	1396	1251	1251
Total cold run time: 300198 ms
Total hot run time: 193552 ms

@bobhan1 bobhan1 force-pushed the skip-version-col-when-calc-check-sum branch from f941903 to a141d5c Compare March 18, 2025 02:37
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Mar 18, 2025
@bobhan1
Copy link
Contributor Author

bobhan1 commented Mar 18, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32576 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a141d5c2cbce772e59fea433b68e1e1c5caeb003, data reload: false

------ Round 1 ----------------------------------
q1	24179	5086	5097	5086
q2	2040	316	191	191
q3	10495	1295	697	697
q4	10233	1043	561	561
q5	7846	2422	2401	2401
q6	191	162	131	131
q7	926	762	606	606
q8	9320	1406	1120	1120
q9	4966	4895	4586	4586
q10	6832	2313	1893	1893
q11	497	276	258	258
q12	345	352	211	211
q13	17781	3686	3139	3139
q14	233	225	216	216
q15	547	472	490	472
q16	634	625	592	592
q17	589	893	350	350
q18	6964	6428	6469	6428
q19	2684	973	551	551
q20	315	350	197	197
q21	2938	2165	1941	1941
q22	1032	1013	949	949
Total cold run time: 111587 ms
Total hot run time: 32576 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5311	5165	5127	5127
q2	251	333	233	233
q3	2135	2677	2306	2306
q4	1468	1836	1353	1353
q5	4263	4167	4163	4163
q6	209	168	124	124
q7	1905	1826	1690	1690
q8	2576	2610	2451	2451
q9	6859	6823	6764	6764
q10	2907	3069	2653	2653
q11	564	510	482	482
q12	680	774	586	586
q13	3390	3711	3072	3072
q14	287	291	253	253
q15	514	472	467	467
q16	638	671	645	645
q17	1139	1568	1322	1322
q18	7408	7258	7073	7073
q19	858	826	920	826
q20	1923	1949	1783	1783
q21	5281	4779	4749	4749
q22	1081	1028	983	983
Total cold run time: 51647 ms
Total hot run time: 49105 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185554 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a141d5c2cbce772e59fea433b68e1e1c5caeb003, data reload: false

query1	1011	488	462	462
query2	6537	1984	1936	1936
query3	6797	208	214	208
query4	26286	23708	23049	23049
query5	5133	689	461	461
query6	308	205	183	183
query7	4608	495	316	316
query8	308	236	239	236
query9	8647	2616	2618	2616
query10	510	308	243	243
query11	15631	15055	15293	15055
query12	166	107	110	107
query13	1649	511	377	377
query14	11849	6543	6561	6543
query15	204	190	165	165
query16	7680	623	458	458
query17	1545	730	568	568
query18	2020	411	313	313
query19	213	187	169	169
query20	120	118	114	114
query21	210	124	108	108
query22	4136	4154	4027	4027
query23	34020	32946	32879	32879
query24	6905	2392	2411	2392
query25	552	479	424	424
query26	1229	280	158	158
query27	1995	497	328	328
query28	3921	2458	2430	2430
query29	732	584	455	455
query30	283	218	192	192
query31	955	841	770	770
query32	81	64	69	64
query33	593	377	314	314
query34	796	877	507	507
query35	758	829	759	759
query36	941	977	883	883
query37	122	103	77	77
query38	4103	4045	4228	4045
query39	1444	1374	1410	1374
query40	202	115	105	105
query41	61	52	48	48
query42	124	103	100	100
query43	508	523	484	484
query44	1298	786	776	776
query45	177	166	159	159
query46	833	1013	617	617
query47	1758	1774	1710	1710
query48	381	402	312	312
query49	770	514	415	415
query50	687	734	394	394
query51	4216	4215	4122	4122
query52	107	104	96	96
query53	234	258	181	181
query54	502	491	415	415
query55	79	78	79	78
query56	278	250	243	243
query57	1108	1150	1061	1061
query58	243	238	268	238
query59	2610	2869	2745	2745
query60	281	296	257	257
query61	123	118	148	118
query62	787	737	672	672
query63	226	192	185	185
query64	4262	1010	655	655
query65	4441	4302	4318	4302
query66	1045	400	296	296
query67	15716	15686	15295	15295
query68	8529	892	499	499
query69	479	293	256	256
query70	1229	1100	1075	1075
query71	478	300	279	279
query72	5726	3717	3982	3717
query73	771	722	349	349
query74	8979	9057	8933	8933
query75	3907	3155	2721	2721
query76	3751	1186	764	764
query77	789	441	279	279
query78	10084	10249	9292	9292
query79	3118	800	579	579
query80	611	526	440	440
query81	491	257	217	217
query82	722	122	96	96
query83	178	171	153	153
query84	249	89	76	76
query85	797	352	307	307
query86	385	328	284	284
query87	4530	4489	4291	4291
query88	3649	2272	2250	2250
query89	394	313	283	283
query90	1966	208	211	208
query91	137	142	109	109
query92	74	59	59	59
query93	1825	1024	587	587
query94	667	415	275	275
query95	357	274	262	262
query96	487	571	280	280
query97	3315	3415	3304	3304
query98	234	207	199	199
query99	1646	1394	1287	1287
Total cold run time: 278609 ms
Total hot run time: 185554 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.79 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a141d5c2cbce772e59fea433b68e1e1c5caeb003, data reload: false

query1	0.04	0.04	0.04
query2	0.12	0.11	0.10
query3	0.26	0.20	0.20
query4	1.60	0.19	0.11
query5	0.56	0.53	0.55
query6	1.20	0.71	0.72
query7	0.03	0.02	0.01
query8	0.04	0.03	0.03
query9	0.58	0.52	0.52
query10	0.64	0.58	0.57
query11	0.15	0.10	0.10
query12	0.15	0.12	0.11
query13	0.62	0.60	0.60
query14	2.66	2.82	2.82
query15	0.92	0.83	0.85
query16	0.38	0.38	0.39
query17	1.04	1.05	1.04
query18	0.21	0.19	0.20
query19	1.97	1.84	1.82
query20	0.01	0.02	0.01
query21	15.36	0.91	0.58
query22	0.77	1.17	0.63
query23	14.96	1.39	0.64
query24	7.66	0.78	0.43
query25	0.48	0.25	0.08
query26	0.74	0.17	0.14
query27	0.05	0.05	0.05
query28	8.49	0.84	0.45
query29	12.55	4.08	3.33
query30	0.25	0.09	0.06
query31	2.83	0.59	0.39
query32	3.23	0.54	0.47
query33	3.00	3.05	3.01
query34	15.82	5.11	4.49
query35	4.52	4.52	4.54
query36	0.67	0.50	0.48
query37	0.08	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.03 s
Total hot run time: 30.79 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 14.29% (6/42) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 48.88% (13093/26784)
Line Coverage 38.44% (112833/293565)
Region Coverage 37.26% (57394/154044)
Branch Coverage 32.33% (28834/89180)

@bobhan1 bobhan1 closed this Mar 18, 2025
@bobhan1
Copy link
Contributor Author

bobhan1 commented Mar 18, 2025

fix in #49193

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants