Skip to content

Conversation

@seawinde
Copy link
Contributor

@seawinde seawinde commented Aug 9, 2024

Proposed changes

if table is unique model, and create sync materialized view on it
such as mv def is as following:

        select l_orderkey, l_linenumber, l_partkey, l_suppkey, l_shipdate,
        substring(concat(l_returnflag, l_linestatus), 1)
        from lineitem_2_uniq;

if query is as following:

        select l_orderkey, l_suppkey,
        substring(concat(l_returnflag, l_linestatus), 1)
        from lineitem_2_uniq;

delete from lineitem_2_uniq_mow where l_orderkey = 1;
the result is

+------------+-----------+--------------------------------------------------------------+
| l_orderkey | l_suppkey | substring(concat(l_returnflag, l_linestatus), 1, 2147483647) |
+------------+-----------+--------------------------------------------------------------+
|          3 |         2 | ko                                                           |
|          3 |         2 | ko                                                           |
|          2 |         1 | ok                                                           |
|       NULL |         3 | ok                                                           |
|          1 |         1 | NULL                                                         |
|          1 |      NULL | NULL                                                         |
|          1 |         2 | NULL                                                         |
+------------+-----------+--------------------------------------------------------------+

the right result should be

+------------+-----------+--------------------------------------------------------------+
| l_orderkey | l_suppkey | substring(concat(l_returnflag, l_linestatus), 1, 2147483647) |
+------------+-----------+--------------------------------------------------------------+
|          2 |         1 | ok                                                           |
|          3 |         2 | ko                                                           |
|          3 |         2 | ko                                                           |
|       NULL |         3 | ok                                                           |
+------------+-----------+--------------------------------------------------------------+

the pr fix this

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions github-actions bot added the doing label Aug 9, 2024
@seawinde
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39401 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit be6cac80ab6b0a7b7b48be4fca50ce55122e9e1c, data reload: false

------ Round 1 ----------------------------------
q1	17870	4447	4489	4447
q2	3016	180	179	179
q3	11071	1160	1100	1100
q4	10654	696	734	696
q5	7694	2541	2526	2526
q6	227	141	141	141
q7	1005	609	597	597
q8	9529	1895	1928	1895
q9	8745	6547	6559	6547
q10	7033	2196	2231	2196
q11	445	245	245	245
q12	385	219	218	218
q13	18629	2967	2971	2967
q14	272	238	229	229
q15	524	477	492	477
q16	487	413	390	390
q17	970	675	724	675
q18	8094	7535	7363	7363
q19	4423	1064	949	949
q20	669	328	322	322
q21	5951	4259	4624	4259
q22	1110	983	1003	983
Total cold run time: 118803 ms
Total hot run time: 39401 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4411	4255	4261	4255
q2	377	281	265	265
q3	2831	2600	2669	2600
q4	1895	1609	1622	1609
q5	5300	5237	5289	5237
q6	222	131	129	129
q7	2055	1660	1686	1660
q8	3145	3359	3324	3324
q9	8475	8380	8441	8380
q10	3354	3139	3161	3139
q11	616	514	495	495
q12	777	589	616	589
q13	16441	3041	2973	2973
q14	308	264	285	264
q15	530	468	472	468
q16	472	433	408	408
q17	1768	1475	1508	1475
q18	7683	7504	7446	7446
q19	4348	1488	1493	1488
q20	2071	1861	1811	1811
q21	5345	5237	5038	5038
q22	1099	993	1021	993
Total cold run time: 73523 ms
Total hot run time: 54046 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 201102 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit be6cac80ab6b0a7b7b48be4fca50ce55122e9e1c, data reload: false

query1	907	370	356	356
query2	6444	2050	1921	1921
query3	6643	203	214	203
query4	28466	23198	23259	23198
query5	4308	497	497	497
query6	289	177	168	168
query7	4574	306	308	306
query8	243	211	197	197
query9	8525	2401	2356	2356
query10	543	464	417	417
query11	17938	14904	15057	14904
query12	153	97	97	97
query13	1633	377	364	364
query14	10396	7623	7545	7545
query15	261	215	240	215
query16	7754	479	503	479
query17	1706	554	539	539
query18	1956	302	278	278
query19	191	144	148	144
query20	114	102	103	102
query21	206	105	98	98
query22	4100	4095	3938	3938
query23	33891	33093	33353	33093
query24	11029	2598	2488	2488
query25	614	392	372	372
query26	1384	151	150	150
query27	2944	274	277	274
query28	7715	1964	1962	1962
query29	928	410	409	409
query30	304	149	150	149
query31	949	722	758	722
query32	101	55	54	54
query33	738	284	284	284
query34	957	466	462	462
query35	961	806	818	806
query36	1111	901	914	901
query37	160	79	81	79
query38	4242	4043	4197	4043
query39	1427	1371	1377	1371
query40	279	112	115	112
query41	46	49	44	44
query42	118	95	98	95
query43	510	483	487	483
query44	1266	742	753	742
query45	234	206	209	206
query46	1095	735	747	735
query47	1858	1723	1770	1723
query48	364	305	305	305
query49	1088	437	414	414
query50	801	404	405	404
query51	6741	6710	6730	6710
query52	107	89	87	87
query53	262	188	178	178
query54	836	453	438	438
query55	78	76	77	76
query56	268	247	258	247
query57	1139	1057	1069	1057
query58	239	233	226	226
query59	3091	2892	2730	2730
query60	279	259	265	259
query61	98	134	105	105
query62	818	653	657	653
query63	213	180	185	180
query64	9653	2286	1742	1742
query65	3190	3179	3176	3176
query66	1031	340	344	340
query67	15378	14795	14761	14761
query68	4915	535	535	535
query69	524	392	372	372
query70	1212	1127	1162	1127
query71	443	272	271	271
query72	19763	17208	17058	17058
query73	772	331	329	329
query74	9084	8712	8924	8712
query75	3452	2675	2732	2675
query76	3097	1067	956	956
query77	678	308	304	304
query78	11239	9371	8964	8964
query79	2532	513	519	513
query80	1135	485	502	485
query81	595	219	227	219
query82	789	142	133	133
query83	340	150	151	150
query84	277	83	81	81
query85	719	285	284	284
query86	462	306	272	272
query87	4799	4622	4488	4488
query88	3702	2523	2536	2523
query89	389	287	287	287
query90	1816	201	201	201
query91	123	99	97	97
query92	61	52	50	50
query93	1886	526	536	526
query94	846	297	289	289
query95	349	263	290	263
query96	608	279	285	279
query97	3208	3044	3049	3044
query98	227	206	197	197
query99	1754	1249	1250	1249
Total cold run time: 309681 ms
Total hot run time: 201102 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.12 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit be6cac80ab6b0a7b7b48be4fca50ce55122e9e1c, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.06	0.05
query4	1.68	0.09	0.10
query5	0.49	0.48	0.50
query6	1.14	0.73	0.72
query7	0.01	0.01	0.01
query8	0.05	0.04	0.05
query9	0.56	0.49	0.49
query10	0.54	0.55	0.55
query11	0.16	0.11	0.11
query12	0.15	0.12	0.13
query13	0.60	0.61	0.59
query14	0.76	0.78	0.79
query15	0.86	0.81	0.81
query16	0.36	0.37	0.36
query17	1.06	1.03	1.06
query18	0.23	0.22	0.22
query19	1.84	1.81	1.70
query20	0.01	0.01	0.02
query21	15.40	0.72	0.65
query22	4.13	7.66	2.30
query23	18.25	1.34	1.22
query24	2.17	0.23	0.22
query25	0.17	0.08	0.09
query26	0.29	0.22	0.20
query27	0.45	0.23	0.23
query28	13.22	1.04	1.01
query29	12.64	3.34	3.35
query30	0.23	0.06	0.05
query31	2.88	0.39	0.39
query32	3.27	0.48	0.46
query33	2.93	2.92	2.96
query34	16.93	4.36	4.31
query35	4.37	4.47	4.44
query36	0.66	0.48	0.47
query37	0.18	0.15	0.15
query38	0.15	0.14	0.15
query39	0.04	0.03	0.04
query40	0.17	0.13	0.13
query41	0.09	0.05	0.06
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.57 s
Total hot run time: 31.12 s

morrySnow
morrySnow previously approved these changes Aug 12, 2024
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 12, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@seawinde seawinde force-pushed the fix_sync_data_wrong_when_base_table_unique branch from be6cac8 to d70c2e3 Compare August 15, 2024 14:58
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Aug 15, 2024
@seawinde
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38569 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1ed25a98319f87d407b01a5001d428995a24b785, data reload: false

------ Round 1 ----------------------------------
q1	18623	4490	4390	4390
q2	2071	209	226	209
q3	11233	1137	1126	1126
q4	10506	772	778	772
q5	7804	2877	2873	2873
q6	276	158	161	158
q7	1017	641	644	641
q8	9398	2092	2102	2092
q9	7294	6551	6614	6551
q10	7068	2235	2277	2235
q11	486	263	260	260
q12	422	250	250	250
q13	17779	2987	3030	2987
q14	292	250	251	250
q15	551	532	506	506
q16	517	418	404	404
q17	1004	797	742	742
q18	7561	6842	6639	6639
q19	5733	1143	1147	1143
q20	715	355	375	355
q21	4094	3160	2964	2964
q22	1131	1034	1022	1022
Total cold run time: 115575 ms
Total hot run time: 38569 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4604	4300	4339	4300
q2	402	296	291	291
q3	2855	2631	2626	2626
q4	2031	1667	1682	1667
q5	5689	5657	5690	5657
q6	237	150	151	150
q7	2194	1771	1779	1771
q8	3310	3490	3475	3475
q9	8844	8864	8811	8811
q10	3599	3386	3310	3310
q11	636	540	547	540
q12	870	663	635	635
q13	17180	3183	3141	3141
q14	323	299	285	285
q15	556	510	507	507
q16	504	457	470	457
q17	1861	1591	1548	1548
q18	8192	7853	7772	7772
q19	4398	1504	1568	1504
q20	2188	1903	1927	1903
q21	13726	5393	5349	5349
q22	1180	1052	1065	1052
Total cold run time: 85379 ms
Total hot run time: 56751 ms

@seawinde seawinde force-pushed the fix_sync_data_wrong_when_base_table_unique branch from 1ed25a9 to 0139690 Compare August 16, 2024 09:47
@seawinde
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38421 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 01396907451819da716c0b9fa1a1a4ff519f59e3, data reload: false

------ Round 1 ----------------------------------
q1	17840	4441	4331	4331
q2	2048	218	222	218
q3	11725	1003	1114	1003
q4	10529	735	869	735
q5	7801	2824	2823	2823
q6	281	162	168	162
q7	1027	667	665	665
q8	9585	2120	2112	2112
q9	8771	6579	6620	6579
q10	7097	2268	2290	2268
q11	499	281	272	272
q12	432	264	264	264
q13	17788	3014	2997	2997
q14	317	261	260	260
q15	558	536	539	536
q16	532	422	411	411
q17	995	771	746	746
q18	7400	6887	6765	6765
q19	6730	1056	1063	1056
q20	710	362	358	358
q21	3934	3001	2819	2819
q22	1144	1041	1054	1041
Total cold run time: 117743 ms
Total hot run time: 38421 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4530	4293	4320	4293
q2	412	315	308	308
q3	2833	2612	2692	2612
q4	1997	1701	1736	1701
q5	5611	5733	5646	5646
q6	247	158	149	149
q7	2145	1775	1724	1724
q8	3286	3563	3449	3449
q9	8812	8755	8786	8755
q10	3561	3359	3373	3359
q11	605	525	525	525
q12	838	671	674	671
q13	16211	3127	3234	3127
q14	329	311	295	295
q15	571	506	524	506
q16	498	440	445	440
q17	1815	1546	1534	1534
q18	8134	7907	7694	7694
q19	7278	1370	1455	1370
q20	2162	1907	1901	1901
q21	14399	5255	5352	5255
q22	1211	1134	1113	1113
Total cold run time: 87485 ms
Total hot run time: 56427 ms

@doris-robot
Copy link

TPC-H: Total hot run time: 37957 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 01396907451819da716c0b9fa1a1a4ff519f59e3, data reload: false

------ Round 1 ----------------------------------
q1	17876	4613	4283	4283
q2	2057	210	206	206
q3	11863	1002	1067	1002
q4	10515	716	698	698
q5	7782	2815	2852	2815
q6	264	159	159	159
q7	1014	684	649	649
q8	9613	2090	2109	2090
q9	8692	6551	6509	6509
q10	7040	2272	2269	2269
q11	492	259	255	255
q12	416	249	255	249
q13	17960	2990	3014	2990
q14	286	252	249	249
q15	567	513	511	511
q16	511	414	399	399
q17	970	728	617	617
q18	7464	6795	6785	6785
q19	7067	977	1039	977
q20	690	356	349	349
q21	4033	2858	2957	2858
q22	1125	1057	1038	1038
Total cold run time: 118297 ms
Total hot run time: 37957 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4521	4267	4258	4258
q2	393	289	296	289
q3	2806	2659	2657	2657
q4	1975	1700	1690	1690
q5	5713	5687	5695	5687
q6	244	152	159	152
q7	2214	1774	1764	1764
q8	3303	3487	3438	3438
q9	8748	8754	8706	8706
q10	3544	3408	3337	3337
q11	655	538	524	524
q12	837	658	664	658
q13	16181	3213	3195	3195
q14	332	295	293	293
q15	558	507	511	507
q16	492	439	444	439
q17	1828	1530	1538	1530
q18	8326	8004	7851	7851
q19	7534	1593	1609	1593
q20	2162	1886	1895	1886
q21	13791	5322	5477	5322
q22	1164	1121	1086	1086
Total cold run time: 87321 ms
Total hot run time: 56862 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196595 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 01396907451819da716c0b9fa1a1a4ff519f59e3, data reload: false

query1	1381	921	876	876
query2	6553	2127	2094	2094
query3	10746	4194	3766	3766
query4	57715	23812	23320	23320
query5	5712	731	711	711
query6	546	223	209	209
query7	6358	340	346	340
query8	545	434	442	434
query9	9181	2533	2520	2520
query10	571	359	356	356
query11	15906	15059	15222	15059
query12	210	129	138	129
query13	1688	456	447	447
query14	11865	7341	7721	7341
query15	310	197	195	195
query16	7477	535	581	535
query17	1186	618	619	618
query18	1405	344	339	339
query19	303	172	170	170
query20	143	152	136	136
query21	254	142	141	141
query22	4605	4552	4404	4404
query23	34580	33862	33976	33862
query24	5717	2973	3061	2973
query25	581	429	444	429
query26	708	183	188	183
query27	1753	312	305	305
query28	3952	2196	2163	2163
query29	707	452	463	452
query30	249	201	189	189
query31	1024	882	863	863
query32	108	81	81	81
query33	512	376	353	353
query34	939	519	505	505
query35	857	802	774	774
query36	1092	1007	1001	1001
query37	152	101	105	101
query38	4009	3824	3912	3824
query39	1526	1477	1445	1445
query40	235	156	151	151
query41	139	136	139	136
query42	133	115	117	115
query43	573	525	513	513
query44	1127	784	798	784
query45	222	201	197	197
query46	1122	772	761	761
query47	1934	1801	1806	1801
query48	424	333	329	329
query49	930	581	585	581
query50	857	504	469	469
query51	6830	6753	6803	6753
query52	117	110	112	110
query53	298	227	228	227
query54	646	514	502	502
query55	89	90	89	89
query56	334	310	312	310
query57	1210	1107	1141	1107
query58	291	298	304	298
query59	3277	2871	2950	2871
query60	351	329	341	329
query61	147	151	148	148
query62	795	687	690	687
query63	257	234	227	227
query64	3296	1884	1885	1884
query65	3268	3158	3157	3157
query66	987	668	675	668
query67	15119	15080	14865	14865
query68	5170	592	587	587
query69	551	310	318	310
query70	1192	1119	1165	1119
query71	476	311	310	310
query72	7302	2287	2010	2010
query73	804	411	356	356
query74	9152	9023	8847	8847
query75	3495	2729	2764	2729
query76	2218	1054	1023	1023
query77	812	436	432	432
query78	9788	9177	9235	9177
query79	3936	552	557	552
query80	2762	608	601	601
query81	618	258	258	258
query82	741	164	158	158
query83	380	220	215	215
query84	290	94	97	94
query85	1099	362	354	354
query86	453	339	320	320
query87	4272	4201	4273	4201
query88	4703	2468	2471	2468
query89	448	329	322	322
query90	2040	227	233	227
query91	158	128	129	128
query92	88	75	74	74
query93	4230	543	545	543
query94	909	318	321	318
query95	380	296	299	296
query96	614	295	287	287
query97	3286	3087	3059	3059
query98	246	228	230	228
query99	1579	1352	1280	1280
Total cold run time: 322326 ms
Total hot run time: 196595 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.13 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 01396907451819da716c0b9fa1a1a4ff519f59e3, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.67	0.08	0.07
query5	0.49	0.50	0.47
query6	1.13	0.74	0.74
query7	0.02	0.02	0.01
query8	0.06	0.04	0.05
query9	0.55	0.49	0.50
query10	0.55	0.56	0.56
query11	0.15	0.12	0.12
query12	0.15	0.13	0.13
query13	0.60	0.59	0.60
query14	0.79	0.78	0.77
query15	0.88	0.83	0.83
query16	0.37	0.38	0.37
query17	1.04	1.04	0.97
query18	0.22	0.22	0.22
query19	1.87	1.68	1.84
query20	0.01	0.01	0.01
query21	15.40	0.66	0.66
query22	4.46	7.39	1.92
query23	18.26	1.42	1.24
query24	2.21	0.23	0.23
query25	0.15	0.08	0.08
query26	0.34	0.23	0.23
query27	0.46	0.23	0.23
query28	13.21	1.03	1.01
query29	12.64	3.30	3.29
query30	0.42	0.25	0.24
query31	2.81	0.40	0.40
query32	3.25	0.49	0.49
query33	2.93	2.94	2.92
query34	17.11	4.35	4.38
query35	4.50	4.41	4.47
query36	0.67	0.49	0.53
query37	0.21	0.18	0.18
query38	0.17	0.17	0.17
query39	0.06	0.05	0.06
query40	0.19	0.14	0.15
query41	0.12	0.07	0.07
query42	0.08	0.07	0.07
query43	0.07	0.06	0.06
Total cold run time: 110.63 s
Total hot run time: 31.13 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 20, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@morrySnow morrySnow merged commit eb67a64 into apache:master Aug 20, 2024
dataroaring pushed a commit that referenced this pull request Aug 26, 2024
if table is unique model, and create sync materialized view on it
such as mv def is as following:

        select l_orderkey, l_linenumber, l_partkey, l_suppkey, l_shipdate,
        substring(concat(l_returnflag, l_linestatus), 1)
        from lineitem_2_uniq;

if query is as following:

        select l_orderkey, l_suppkey,
        substring(concat(l_returnflag, l_linestatus), 1)
        from lineitem_2_uniq;

delete from lineitem_2_uniq_mow where l_orderkey = 1;

the result is 
+------------+-----------+--------------------------------------------------------------+
| l_orderkey | l_suppkey | substring(concat(l_returnflag, l_linestatus), 1, 2147483647) |
+------------+-----------+--------------------------------------------------------------+
|          3 |         2 | ko                                                           |
|          3 |         2 | ko                                                           |
|          2 |         1 | ok                                                           |
|       NULL |         3 | ok                                                           |
|          1 |         1 | NULL                                                         |
|          1 |      NULL | NULL                                                         |
|          1 |         2 | NULL                                                         |
+------------+-----------+--------------------------------------------------------------+

the right result should be 
+------------+-----------+--------------------------------------------------------------+
| l_orderkey | l_suppkey | substring(concat(l_returnflag, l_linestatus), 1, 2147483647) |
+------------+-----------+--------------------------------------------------------------+
|          2 |         1 | ok                                                           |
|          3 |         2 | ko                                                           |
|          3 |         2 | ko                                                           |
|       NULL |         3 | ok                                                           |
+------------+-----------+--------------------------------------------------------------+
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.2-merged p0_b reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants