Skip to content

[Feature](IP) support ipv4/ipv6 with inverted index and conjuncts for query #35734

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
xiaokang merged 3 commits into from
Jun 3, 2024
Merged

[Feature](IP) support ipv4/ipv6 with inverted index and conjuncts for query #35734

xiaokang merged 3 commits into from
Jun 3, 2024

Conversation

@amorynan
Copy link
Contributor

@amorynan amorynan commented May 31, 2024

Proposed changes

support data type ipv4/ipv6 with inverted index
and then we can query like "> or < or >= or <= or in/not in " this conjuncts expr for ip with inverted index speeding up
Issue Number: close #xxx

@doris-robot
Copy link

doris-robot commented May 31, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

morrySnow
morrySnow previously approved these changes May 31, 2024
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 31, 2024
@github-actions
Copy link
Contributor

github-actions bot commented May 31, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented May 31, 2024

PR approved by anyone and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented May 31, 2024

clang-tidy review says "All clean, LGTM! 👍"

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label May 31, 2024
@github-actions
Copy link
Contributor

github-actions bot commented May 31, 2024

clang-tidy review says "All clean, LGTM! 👍"

xiaokang
xiaokang reviewed May 31, 2024
View reviewed changes
@amorynan
Copy link
Contributor Author

amorynan commented Jun 3, 2024

run buildall

@amorynan amorynan requested a review from xiaokang June 3, 2024 01:20
@github-actions
Copy link
Contributor

github-actions bot commented Jun 3, 2024

clang-tidy review says "All clean, LGTM! 👍"

xiaokang
xiaokang approved these changes Jun 3, 2024
View reviewed changes
Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 3, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jun 3, 2024

PR approved by at least one committer and no changes requested.

1 similar comment
@github-actions
Copy link
Contributor

github-actions bot commented Jun 3, 2024

PR approved by at least one committer and no changes requested.

@doris-robot
Copy link

doris-robot commented Jun 3, 2024

TPC-H: Total hot run time: 40450 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit af29f68ed617fe5ad2e1178bfaf64a1bcf74ac13, data reload: false

------ Round 1 ----------------------------------
q1	17854	4460	4407	4407
q2	2738	199	201	199
q3	10972	1272	1183	1183
q4	11316	814	705	705
q5	7970	2767	2833	2767
q6	226	137	136	136
q7	992	609	605	605
q8	9428	2088	2084	2084
q9	9027	6560	6502	6502
q10	9037	3731	3709	3709
q11	460	247	237	237
q12	456	231	223	223
q13	17775	3007	3013	3007
q14	251	227	231	227
q15	519	479	474	474
q16	504	382	377	377
q17	967	681	784	681
q18	8139	7484	7513	7484
q19	7928	1529	1522	1522
q20	663	319	305	305
q21	5067	3276	3967	3276
q22	407	340	341	340
Total cold run time: 122696 ms
Total hot run time: 40450 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4471	4245	4239	4239
q2	373	266	265	265
q3	3012	2775	2725	2725
q4	1908	1612	1622	1612
q5	5301	5324	5319	5319
q6	221	126	130	126
q7	2101	1754	1731	1731
q8	3224	3368	3330	3330
q9	8425	8341	8378	8341
q10	3860	3745	3648	3648
q11	601	498	494	494
q12	775	613	610	610
q13	17462	2986	3010	2986
q14	304	262	271	262
q15	511	476	477	476
q16	464	421	424	421
q17	1810	1497	1457	1457
q18	7593	7619	7518	7518
q19	1663	1618	1569	1569
q20	2011	1816	1803	1803
q21	5008	4849	4796	4796
q22	618	562	553	553
Total cold run time: 71716 ms
Total hot run time: 54281 ms

@doris-robot
Copy link

doris-robot commented Jun 3, 2024

TPC-DS: Total hot run time: 172599 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit af29f68ed617fe5ad2e1178bfaf64a1bcf74ac13, data reload: false

query1	921	382	368	368
query2	6461	2568	2285	2285
query3	6658	208	207	207
query4	19399	17318	17096	17096
query5	4166	473	450	450
query6	259	195	165	165
query7	4605	331	298	298
query8	321	290	292	290
query9	8617	2366	2338	2338
query10	454	313	287	287
query11	10402	9959	10010	9959
query12	135	99	87	87
query13	1634	368	370	368
query14	9639	6468	7719	6468
query15	229	188	186	186
query16	7787	266	262	262
query17	1856	529	547	529
query18	1835	271	274	271
query19	207	149	151	149
query20	91	84	89	84
query21	210	127	131	127
query22	4466	4033	3968	3968
query23	33824	33215	33229	33215
query24	11103	2970	2887	2887
query25	624	354	367	354
query26	1374	156	159	156
query27	3002	333	324	324
query28	7694	2057	2078	2057
query29	930	616	627	616
query30	274	149	154	149
query31	969	764	727	727
query32	93	57	56	56
query33	814	274	280	274
query34	975	480	468	468
query35	762	642	603	603
query36	1082	943	927	927
query37	156	68	71	68
query38	2840	2756	2724	2724
query39	856	800	800	800
query40	210	123	124	123
query41	54	51	56	51
query42	116	97	97	97
query43	560	548	536	536
query44	1224	752	754	752
query45	194	165	169	165
query46	1082	703	709	703
query47	1889	1816	1812	1812
query48	373	315	307	307
query49	1102	401	429	401
query50	790	384	394	384
query51	6805	6677	6642	6642
query52	109	89	97	89
query53	356	291	300	291
query54	937	450	446	446
query55	71	70	74	70
query56	281	251	264	251
query57	1137	1094	1044	1044
query58	261	267	246	246
query59	3554	3243	3134	3134
query60	288	269	276	269
query61	95	90	95	90
query62	633	441	446	441
query63	330	328	304	304
query64	8945	2249	1744	1744
query65	3201	3154	3147	3147
query66	920	340	331	331
query67	15328	15140	15000	15000
query68	4604	551	542	542
query69	470	321	300	300
query70	1200	1089	1107	1089
query71	416	292	295	292
query72	7097	5720	5587	5587
query73	752	322	324	322
query74	6055	5659	5463	5463
query75	3375	2649	2676	2649
query76	2852	1004	933	933
query77	458	299	303	299
query78	10400	9932	9859	9859
query79	2881	517	509	509
query80	2275	475	480	475
query81	554	220	222	220
query82	887	117	100	100
query83	319	168	171	168
query84	262	90	95	90
query85	1719	269	268	268
query86	465	331	303	303
query87	3278	3111	3092	3092
query88	4400	2478	2440	2440
query89	473	405	392	392
query90	1790	199	271	199
query91	128	97	96	96
query92	64	50	49	49
query93	2003	519	510	510
query94	1236	188	190	188
query95	408	311	330	311
query96	585	265	268	265
query97	3216	3075	3030	3030
query98	261	216	223	216
query99	1122	826	848	826
Total cold run time: 274574 ms
Total hot run time: 172599 ms

@doris-robot
Copy link

doris-robot commented Jun 3, 2024

ClickBench: Total hot run time: 30.08 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit af29f68ed617fe5ad2e1178bfaf64a1bcf74ac13, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.66	0.07	0.07
query5	0.52	0.49	0.48
query6	1.13	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.50	0.48
query10	0.55	0.56	0.54
query11	0.15	0.11	0.12
query12	0.15	0.12	0.11
query13	0.60	0.59	0.60
query14	0.79	0.77	0.78
query15	0.84	0.80	0.82
query16	0.36	0.37	0.38
query17	0.98	1.05	1.04
query18	0.22	0.24	0.26
query19	1.88	1.73	1.75
query20	0.02	0.01	0.02
query21	15.71	0.67	0.66
query22	4.73	7.90	1.22
query23	18.28	1.39	1.32
query24	1.74	0.31	0.21
query25	0.14	0.08	0.08
query26	0.28	0.18	0.17
query27	0.08	0.08	0.07
query28	13.26	1.04	1.09
query29	12.70	3.35	3.34
query30	0.25	0.06	0.06
query31	2.88	0.39	0.38
query32	3.25	0.47	0.48
query33	2.87	2.89	2.91
query34	17.08	4.38	4.53
query35	4.52	4.50	4.48
query36	0.64	0.46	0.47
query37	0.17	0.16	0.15
query38	0.15	0.15	0.15
query39	0.04	0.04	0.04
query40	0.17	0.14	0.14
query41	0.09	0.04	0.04
query42	0.05	0.05	0.04
query43	0.03	0.04	0.04
Total cold run time: 109.92 s
Total hot run time: 30.08 s

@doris-robot
Copy link

doris-robot commented Jun 3, 2024

TeamCity be ut coverage result:
Function Coverage: 36.27% (9235/25460)
Line Coverage: 27.61% (75719/274221)
Region Coverage: 26.82% (39208/146189)
Branch Coverage: 23.55% (19865/84362)
Coverage Report: http://coverage.selectdb-in.cc/coverage/af29f68ed617fe5ad2e1178bfaf64a1bcf74ac13_af29f68ed617fe5ad2e1178bfaf64a1bcf74ac13/report/index.html

airborne12
airborne12 approved these changes Jun 3, 2024
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xiaokang xiaokang merged commit 80cc735 into apache:master Jun 3, 2024
yiguolei pushed a commit that referenced this pull request Jun 3, 2024
@Doris-Extras
[Feature](IP) support ipv4/ipv6 with inverted index and conjuncts for…
fe1a4c4 
… query (#35734)

support data type ipv4/ipv6 with inverted index 
and then we can query like "> or < or >= or <= or in/not in " this
conjuncts expr for ip with inverted index speeding up
dataroaring pushed a commit that referenced this pull request Jun 4, 2024
@dataroaring
[Feature](IP) support ipv4/ipv6 with inverted index and conjuncts for…
9e8777a 
… query (#35734)

support data type ipv4/ipv6 with inverted index 
and then we can query like "> or < or >= or <= or in/not in " this
conjuncts expr for ip with inverted index speeding up
seawinde pushed a commit to seawinde/doris that referenced this pull request Jun 5, 2024
@seawinde
[Feature](IP) support ipv4/ipv6 with inverted index and conjuncts for…
8fdd8fc 
… query (apache#35734)

support data type ipv4/ipv6 with inverted index 
and then we can query like "> or < or >= or <= or in/not in " this
conjuncts expr for ip with inverted index speeding up
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xiaokang xiaokang xiaokang approved these changes

@airborne12 airborne12 airborne12 approved these changes

@morrySnow morrySnow morrySnow left review comments

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.4-merged dev/3.0.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@amorynan @doris-robot @xiaokang @airborne12 @morrySnow @wm1581066 @dataroaring