METRON-1696: Create the HDFS directory for pcap sequence files and add required privileges to metron user #1134
Conversation
…d the required previlages to metron user
|
@MohanDV thanks for the contribution! I actually have the same concern/thoughts on this PR as I made on @anandsubbu's here - #1132 (comment). I think PCAP is really trying to be a first class citizen in Metron. We had fairly manual support for PCAP before. I think if we're going to make these changes, we should make PCAP a proper component for Metron similar to [parsers, enrichment, indexing, profiler]. |
|
See comment here - #1132 (comment) I'm +1 via inspection pending a follow-on Jira being created, having it linked to this Jira, and also linked for reference in the comments on this PR. |
|
@mmiklavc can you please floor this PR, I have create follow-on Jira METRON-1709 . |
|
Just FYI, this work was already done in #1019. I do not see much value in merging this into master since it will likely be overwritten when the pcap feature branch is complete. |
|
Good catch @merrimanr When I was testing queries in some other feature branch PR's, I had to change hdfs permissions for the Storm and Metron users to be able to function on both the write and read end of things (Storm writing sequence files, MR jobs reading sequence files and writing them back out to HDFS or local FS). @MohanDV and @merrimanr Did this work for you without needing to make any additional changes? metron:metron did not allow pcap topology to write as the Storm user. |
|
Thanks for pointing this out @merrimanr, Yes this will overwrite the the #1019 . If the feature branch is not getting to master sooner then we can keep this change and rebase the feature branch else we can discard this. @mmiklavc metron:metron should allow pcap topology to write to hdfs directory. I started the topology as metron user. |
Yup @mmiklavc .. on a non-kerberized cluster, I faced the same issue. |
|
@anandsubbu @MohanDV - if we set this to metron:hadoop will that cover both cases? If it does, I would request you make this change in the feature branch. It should work for both kerberized/non-kerberized setups. |
|
@mmiklavc - in my setup, I have seen that non-kerb requires storm:hadoop permissions, while kerberized needs metron:hadoop for PCAP to write into HDFS. |
|
@mmiklavc I see the 'storm' user as the owner of pcap topology in non-kerberized setup, where as the 'metron' as the owner in kerberized setup. IMHO The pcap should have 'metron' as owner of the topology, then having the hdfs permissions set to metron:hadoop will work on both the environment. |
MohanDV
changed the base branch from
master
to
feature/METRON-1554-pcap-query-panel
|
Non Kerberized pcap topology runs under 'storm' which requires write under the 'hadoop' group so we should provide mode 0775 when the cluster is un secure and mode 0755 when the cluster is secure as the topology runs under 'metron' user. |
|
The REST component in Ambari is currently responsible for setting up Pcap HDFS directories. It looks like this PR duplicates some of that. I think we will have a dedicated Pcap Ambari component at some point so I don't think it matters that much where this setup logic lives in the meantime. However, this PR doesn't line up with the feature branch because it only creates one directory where the Pcap query feature requires 3. Also the property names in Ambari need to match REST. I would recommend reviewing how this is currently implemented in REST. Most of the work was done in #1124. Whether this ends up in the REST or parsers component doesn't matter to me as long as it functions. Also, unless we can adjust this PR to only include a couple minor changes I would strongly suggest this be tested end to end. I doubt it works in it's current state. |
|
I am fine with closing this PR as this is currently implemented in REST. When pcap gets its own start/stop script in Ambari we can move the code from REST to respective file. |
|
Should this be closed out? |
|
closing as it addressed in METRON-1738 |
|
Closed as it is addressed in METRON-1738 |
5 participants
Contributor Comments
PCAP parser fails to write pacap sequence file to hdfs on kerberized cluster due to insufficient privileges to hdfs folder for 'metron' user. Create the HDFS directory for PCAP sequence files and add the required privileges to metron user.
Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our Development Guidelines for the complete guide to follow for contributions.
Please refer also to our Build Verification Guidelines for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
For all changes:
For code changes:
Have you included steps to reproduce the behavior or problem that is being changed or addressed?
Have you included steps or a guide to how the change may be verified and tested manually?
Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
Have you written or updated unit tests and or integration tests to verify your changes?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
For documentation related changes:
Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via
site-book/target/site/index.html:Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that travis-ci is set up for your personal repository such that your branches are built there before submitting a pull request.