Skip to content

[feature](nereids) Fix data wrong using mv rewrite and ignore case when getting mv related partition table #28678

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
seawinde wants to merge 1 commit into from
Closed

[feature](nereids) Fix data wrong using mv rewrite and ignore case when getting mv related partition table #28678

seawinde wants to merge 1 commit into from

Conversation

@seawinde
Copy link
Contributor

@seawinde seawinde commented Dec 19, 2023

Proposed changes

  • Fix data wrong using mv rewrite
  • ignore case when getting mv related partition table

this pr is depend on #28596

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@seawinde
Copy link
Contributor Author

seawinde commented Dec 19, 2023

run buildall

@doris-robot
Copy link

doris-robot commented Dec 19, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.65 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17184051365 Bytes

@doris-robot
Copy link

doris-robot commented Dec 19, 2023

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit f1a5530863b1bee3690b47bea68b7f14842d9e43, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4674	4365	4411	4365
q2	383	173	159	159
q3	1466	1248	1215	1215
q4	1114	917	918	917
q5	3154	3172	3140	3140
q6	246	127	127	127
q7	989	490	481	481
q8	2216	2213	2193	2193
q9	6681	6704	6692	6692
q10	3222	3272	3276	3272
q11	312	182	187	182
q12	350	209	203	203
q13	4536	3780	3792	3780
q14	240	216	211	211
q15	563	520	518	518
q16	433	385	383	383
q17	1006	608	595	595
q18	7275	6974	6942	6942
q19	1517	1409	1385	1385
q20	505	304	297	297
q21	3081	2637	2624	2624
q22	341	281	286	281
Total cold run time: 44304 ms
Total hot run time: 39962 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4331	4337	4341	4337
q2	266	159	166	159
q3	3527	3529	3527	3527
q4	2381	2377	2379	2377
q5	5848	5733	5730	5730
q6	239	118	124	118
q7	2394	1898	1877	1877
q8	3520	3520	3522	3520
q9	9082	9014	9016	9014
q10	3914	4000	3988	3988
q11	479	368	366	366
q12	755	586	608	586
q13	4292	3558	3552	3552
q14	279	255	258	255
q15	569	526	526	526
q16	496	442	466	442
q17	1890	1868	1875	1868
q18	8605	8325	8286	8286
q19	1764	1763	1760	1760
q20	2249	1944	1923	1923
q21	6528	6196	6164	6164
q22	499	417	442	417
Total cold run time: 63907 ms
Total hot run time: 60792 ms

@seawinde seawinde force-pushed the fix_query_rewrite_by_mv branch 2 times, most recently from 3eb4f3f to 29522b8 Compare December 19, 2023 17:07
@seawinde
Copy link
Contributor Author

seawinde commented Dec 19, 2023

run buildall

1 similar comment
@seawinde
Copy link
Contributor Author

seawinde commented Dec 20, 2023

run buildall

@doris-robot
Copy link

doris-robot commented Dec 20, 2023

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 5b7599afcf35a5ec5a67a202efccd16436071563, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4695	4430	4419	4419
q2	367	148	158	148
q3	1465	1231	1278	1231
q4	1113	907	887	887
q5	3145	3183	3176	3176
q6	249	131	127	127
q7	993	490	486	486
q8	2184	2207	2175	2175
q9	6696	6625	6683	6625
q10	3223	3280	3288	3280
q11	318	191	195	191
q12	362	216	211	211
q13	4561	3870	3783	3783
q14	244	218	216	216
q15	565	524	521	521
q16	455	399	398	398
q17	1004	574	575	574
q18	7214	7069	7042	7042
q19	1513	1396	1463	1396
q20	539	317	331	317
q21	3048	2623	2679	2623
q22	355	289	295	289
Total cold run time: 44308 ms
Total hot run time: 40115 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4399	4367	4372	4367
q2	271	167	178	167
q3	3539	3536	3514	3514
q4	2387	2368	2369	2368
q5	5748	5767	5750	5750
q6	241	120	121	120
q7	2392	1859	1910	1859
q8	3528	3520	3511	3511
q9	9017	8987	9018	8987
q10	3915	4029	4023	4023
q11	490	382	379	379
q12	777	599	608	599
q13	4299	3586	3575	3575
q14	290	255	243	243
q15	582	519	537	519
q16	537	467	494	467
q17	1870	1858	1821	1821
q18	8627	8190	8313	8190
q19	1753	1763	1751	1751
q20	2262	1951	1942	1942
q21	6595	6185	6194	6185
q22	509	452	436	436
Total cold run time: 64028 ms
Total hot run time: 60773 ms

@doris-robot
Copy link

doris-robot commented Dec 20, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.65 seconds
stream load tsv: 576 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17184123405 Bytes

@seawinde seawinde force-pushed the fix_query_rewrite_by_mv branch from a58fc0a to 8fafe23 Compare December 20, 2023 04:40
@seawinde seawinde closed this Dec 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@seawinde @doris-robot