Skip to content

[Bug](schame-change) fix wrong result after reorder mor table #29045

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
BiteTheDDDDt merged 2 commits into from
Dec 28, 2023
Merged

[Bug](schame-change) fix wrong result after reorder mor table #29045

BiteTheDDDDt merged 2 commits into from
Dec 28, 2023

Conversation

@BiteTheDDDDt
Copy link
Contributor

@BiteTheDDDDt BiteTheDDDDt commented Dec 26, 2023

Proposed changes

fix wrong result after reorder mor table

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@BiteTheDDDDt
Copy link
Contributor Author

BiteTheDDDDt commented Dec 26, 2023

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Dec 26, 2023

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Dec 26, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.32 seconds
stream load tsv: 579 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 38.5 seconds inserted 10000000 Rows, about 259K ops/s
storage size: 17184092592 Bytes

@doris-robot
Copy link

doris-robot commented Dec 26, 2023

TeamCity be ut coverage result:
Function Coverage: 36.58% (8556/23389)
Line Coverage: 28.66% (69587/242810)
Region Coverage: 27.66% (35982/130078)
Branch Coverage: 24.39% (18392/75404)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b604fdff23be63c286d583a8bc0fe0c72a0c5e87_b604fdff23be63c286d583a8bc0fe0c72a0c5e87/report/index.html

@wm1581066 wm1581066 added the not-merge/2.0 do not merge into 2.0 branch label Dec 26, 2023
@doris-robot
Copy link

doris-robot commented Dec 26, 2023

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit b604fdff23be63c286d583a8bc0fe0c72a0c5e87, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4774	4509	4498	4498
q2	365	160	159	159
q3	1472	1294	1185	1185
q4	1139	951	914	914
q5	3185	3164	3163	3163
q6	257	134	133	133
q7	1039	502	480	480
q8	2261	2275	2245	2245
q9	6742	6697	6702	6697
q10	3214	3304	3286	3286
q11	333	204	213	204
q12	365	220	209	209
q13	4535	3800	3796	3796
q14	244	213	213	213
q15	577	528	529	528
q16	437	384	385	384
q17	1052	791	621	621
q18	7111	6859	6875	6859
q19	1569	1602	1610	1602
q20	550	349	302	302
q21	3242	2751	2814	2751
q22	382	313	311	311
Total cold run time: 44845 ms
Total hot run time: 40540 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4387	4405	4384	4384
q2	274	166	171	166
q3	3515	3503	3514	3503
q4	2437	2410	2405	2405
q5	5703	5726	5719	5719
q6	248	125	127	125
q7	2372	1871	1846	1846
q8	3623	3620	3614	3614
q9	9032	9039	9013	9013
q10	3928	4026	4014	4014
q11	493	346	365	346
q12	776	592	605	592
q13	4288	3575	3550	3550
q14	284	258	248	248
q15	581	523	526	523
q16	510	445	456	445
q17	1982	1962	1947	1947
q18	8688	8203	8338	8203
q19	1812	1823	1833	1823
q20	2255	1945	1929	1929
q21	6606	6221	6230	6221
q22	563	482	465	465
Total cold run time: 64357 ms
Total hot run time: 61081 ms

@BiteTheDDDDt
Copy link
Contributor Author

BiteTheDDDDt commented Dec 26, 2023

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Dec 26, 2023

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Dec 26, 2023

TeamCity be ut coverage result:
Function Coverage: 36.58% (8556/23390)
Line Coverage: 28.66% (69587/242826)
Region Coverage: 27.66% (35984/130088)
Branch Coverage: 24.40% (18397/75408)
Coverage Report: http://coverage.selectdb-in.cc/coverage/258c521c0d52ca9b08495538503c34f9b559e5ee_258c521c0d52ca9b08495538503c34f9b559e5ee/report/index.html

@doris-robot
Copy link

doris-robot commented Dec 26, 2023

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 258c521c0d52ca9b08495538503c34f9b559e5ee, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4770	4460	4496	4460
q2	370	145	168	145
q3	1487	1368	1220	1220
q4	1150	1013	918	918
q5	3202	3163	3168	3163
q6	256	133	135	133
q7	1008	499	496	496
q8	2256	2275	2253	2253
q9	6729	6706	6694	6694
q10	3226	3286	3300	3286
q11	321	211	217	211
q12	364	207	213	207
q13	4522	3843	3812	3812
q14	239	218	212	212
q15	572	522	527	522
q16	446	384	383	383
q17	1042	817	619	619
q18	7050	6793	6945	6793
q19	1594	1592	1610	1592
q20	542	326	350	326
q21	3248	2794	2796	2794
q22	368	300	302	300
Total cold run time: 44762 ms
Total hot run time: 40539 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4433	4401	4403	4401
q2	270	161	169	161
q3	3515	3515	3487	3487
q4	2457	2424	2420	2420
q5	5736	5735	5741	5735
q6	249	125	128	125
q7	2371	1863	1864	1863
q8	3614	3607	3611	3607
q9	9013	9038	9014	9014
q10	3919	4006	4014	4006
q11	492	381	386	381
q12	786	600	605	600
q13	4282	3542	3582	3542
q14	288	258	249	249
q15	570	524	518	518
q16	494	478	470	470
q17	1962	1949	1934	1934
q18	8585	8314	8146	8146
q19	1806	1813	1836	1813
q20	2246	1946	1925	1925
q21	6601	6240	6232	6232
q22	567	460	451	451
Total cold run time: 64256 ms
Total hot run time: 61080 ms

@doris-robot
Copy link

doris-robot commented Dec 26, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.91 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 35.6 seconds inserted 10000000 Rows, about 280K ops/s
storage size: 17184007135 Bytes

@BiteTheDDDDt
Copy link
Contributor Author

BiteTheDDDDt commented Dec 26, 2023

run buildall

@doris-robot
Copy link

doris-robot commented Dec 26, 2023

TeamCity be ut coverage result:
Function Coverage: 36.56% (8555/23401)
Line Coverage: 28.63% (69588/243045)
Region Coverage: 27.64% (35980/130187)
Branch Coverage: 24.37% (18391/75476)
Coverage Report: http://coverage.selectdb-in.cc/coverage/258c521c0d52ca9b08495538503c34f9b559e5ee_258c521c0d52ca9b08495538503c34f9b559e5ee/report/index.html

@doris-robot
Copy link

doris-robot commented Dec 26, 2023

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 258c521c0d52ca9b08495538503c34f9b559e5ee, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4781	4471	4478	4471
q2	370	145	198	145
q3	1488	1288	1214	1214
q4	1146	959	914	914
q5	3310	3158	3212	3158
q6	255	132	134	132
q7	1047	501	489	489
q8	2266	2297	2258	2258
q9	6745	6720	6677	6677
q10	3222	3297	3265	3265
q11	331	192	215	192
q12	356	210	206	206
q13	4547	3803	3799	3799
q14	246	208	211	208
q15	571	532	524	524
q16	447	385	390	385
q17	1049	781	607	607
q18	7083	6775	6971	6775
q19	1593	1603	1598	1598
q20	565	327	308	308
q21	3213	2732	2712	2712
q22	377	315	305	305
Total cold run time: 45008 ms
Total hot run time: 40342 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4427	4397	4414	4397
q2	270	162	163	162
q3	3526	3506	3508	3506
q4	2440	2429	2411	2411
q5	5730	5730	5744	5730
q6	272	126	127	126
q7	2398	1825	1826	1825
q8	3615	3613	3617	3613
q9	9036	8996	8986	8986
q10	3931	4004	4028	4004
q11	492	376	372	372
q12	770	602	615	602
q13	4264	3567	3558	3558
q14	295	258	262	258
q15	576	526	519	519
q16	493	467	455	455
q17	1975	1955	1953	1953
q18	8627	8264	8106	8106
q19	1816	1830	1862	1830
q20	2266	1952	1933	1933
q21	6639	6279	6242	6242
q22	550	465	460	460
Total cold run time: 64408 ms
Total hot run time: 61048 ms

@doris-robot
Copy link

doris-robot commented Dec 26, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.42 seconds
stream load tsv: 576 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.4 seconds inserted 10000000 Rows, about 352K ops/s
storage size: 17183995575 Bytes

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 28, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Dec 28, 2023

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 28, 2023

PR approved by anyone and no changes requested.

@BiteTheDDDDt BiteTheDDDDt merged commit 118775f into apache:master Dec 28, 2023
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
@BiteTheDDDDt @HappenLee
[Bug](schame-change) fix wrong result after reorder mor table (apache…
64c865f 
…#29045)

* fix wrong result after reorder mor table

* update
@BiteTheDDDDt BiteTheDDDDt mentioned this pull request Mar 26, 2024
Merged
xiaokang pushed a commit that referenced this pull request Apr 2, 2024
@BiteTheDDDDt
[Bug](schame-change) fix wrong result after reorder mor table #29045 #…
bce3c22 
…28135 (#32839)
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@BiteTheDDDDt
[Bug](schame-change) fix wrong result after reorder mor table apache#…
bde9ac4 
…29045 apache#28135 (apache#32839)
@BiteTheDDDDt BiteTheDDDDt deleted the fix_1226 branch January 20, 2025 06:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Gabriel39 Gabriel39 Gabriel39 approved these changes

@yiguolei yiguolei Awaiting requested review from yiguolei

@HappenLee HappenLee Awaiting requested review from HappenLee

+1 more reviewer

@Mryange Mryange Mryange approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.8-merged not-merge/2.0 do not merge into 2.0 branch reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@BiteTheDDDDt @doris-robot @Gabriel39 @Mryange @xiaokang @wm1581066