Skip to content

[feature-wip](ranger)support datamask and row filter #32137

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 3 commits into from
Mar 18, 2024
Merged

[feature-wip](ranger)support datamask and row filter #32137

morningman merged 3 commits into from
Mar 18, 2024

Conversation

@zddr
Copy link
Contributor

@zddr zddr commented Mar 12, 2024 •
edited
Loading

Proposed changes

Issue Number: close #xxx

doris ranger support datamask and row filter

hive ranger support row filter

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented Mar 12, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@zddr zddr marked this pull request as draft March 12, 2024 11:04
@zddr zddr changed the title 1 [feature](ranger)support datamask and row filter Mar 14, 2024
@zddr zddr marked this pull request as ready for review March 14, 2024 12:41
@zddr
Copy link
Contributor Author

zddr commented Mar 14, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Mar 14, 2024

TPC-H: Total hot run time: 38381 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 04d9e659938a2116c7a4245f13c0e77ac146693e, data reload: false

------ Round 1 ----------------------------------
q1	17667	5126	4193	4193
q2	2047	153	141	141
q3	10707	1059	914	914
q4	6065	752	750	750
q5	7456	2738	2649	2649
q6	186	122	121	121
q7	1159	814	817	814
q8	9335	2036	1985	1985
q9	7097	6487	6440	6440
q10	8462	3522	3610	3522
q11	430	223	212	212
q12	704	303	304	303
q13	17773	2881	2877	2877
q14	277	244	249	244
q15	498	463	453	453
q16	470	398	392	392
q17	939	528	572	528
q18	7329	6713	6373	6373
q19	4408	1435	1492	1435
q20	555	293	276	276
q21	6689	3530	3463	3463
q22	365	296	302	296
Total cold run time: 110618 ms
Total hot run time: 38381 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4154	4055	4039	4039
q2	321	217	222	217
q3	2985	2877	2855	2855
q4	1840	1541	1498	1498
q5	5218	5258	5275	5258
q6	198	123	115	115
q7	2257	1892	1860	1860
q8	3147	3332	3274	3274
q9	8608	8591	8600	8591
q10	3755	3623	3694	3623
q11	545	435	436	435
q12	712	610	557	557
q13	16893	2851	2865	2851
q14	274	246	254	246
q15	487	457	449	449
q16	451	410	408	408
q17	1737	1503	1467	1467
q18	7615	7194	7084	7084
q19	1615	1537	1521	1521
q20	1907	1714	1725	1714
q21	4945	4661	4794	4661
q22	511	433	453	433
Total cold run time: 70175 ms
Total hot run time: 53156 ms

@morrySnow
Copy link
Contributor

morrySnow commented Mar 15, 2024

need test special column name to make sure it could work well. for example chinese name

@github-actions
Copy link
Contributor

github-actions bot commented Mar 18, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Mar 18, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Mar 18, 2024

PR approved by anyone and no changes requested.

@morningman morningman changed the title [feature](ranger)support datamask and row filter [feature-wip](ranger)support datamask and row filter Mar 18, 2024
@morningman morningman merged commit 95fcb56 into apache:master Mar 18, 2024
@morningman morningman mentioned this pull request Mar 18, 2024
Merged
morningman pushed a commit that referenced this pull request Mar 18, 2024
@zddr
[enhance](auth)row policy support catalog and match name instead id (#…
c2e155e 
…32310)

Follow up #32137

storage name instead id to meta,Prevent table deletion and reconstruction and causing ID changes
morningman pushed a commit that referenced this pull request Mar 26, 2024
@zddr @morningman
[feature-wip](ranger)support datamask and row filter (#32137)
37c8cc0 
doris ranger support datamask and row filter

hive ranger support row filter
morningman pushed a commit that referenced this pull request Mar 26, 2024
@zddr @morningman
[enhance](auth)row policy support catalog and match name instead id (#…
0a2d737 
…32310)

Follow up #32137

storage name instead id to meta,Prevent table deletion and reconstruction and causing ID changes
@zddr zddr deleted the ranger_datamask branch March 28, 2024 02:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.1-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@zddr @doris-robot @morrySnow @morningman @yiguolei