This repository was archived by the owner on Aug 20, 2025. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 505
METRON-947 [DO NOT MERGE UNTIL METRON-942] Stellar Management Function for Metron Parser Extension Configurations #626
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Based on Apache Nifi Nars NAR changes * new lib , rebrand to bundles from NAR * port to VFS/FileObject from File based * ability to set property values * Rework FileUtils so that you can derive and override * added initializers to set 'classes' that we care about instead of hard coding them, still needs defaults * added components nec. for integration tests ( do not want dep. on metron-* ) * VFSClassloader for NarClassLoader * Hdfs based integration test version of unpacknars tests * HDFS ( filesystem ) based fileutilities to cover for writes to hdfs, since VFS is currently R/O HDFS * modified plugin to support configuration of outputs * use class index not service loader ( both subclass and annotated supported ) Archetype * Parser Extension archetyp * incudes all configuration * creates tar.gz with bundle and configuration * class index support ( automatic generation ) Extensions * new extensions modules * parser * archetype built module for each parser type * support for configuration only parsers with tests Parsers * moved all but json, csv, grok to extensions * Bolt now loads from bundle properties Deployment * rpms for parsers * create extension directories * ambari initializes zookeeper per parser * amabri creates hdfs directories * ISSUE: Writing to hdfs Rest-API * only test against parsers in metron-parsers * still needs integration
… it is when running from archetype, causing the extension versio to be used for this dependency
…archetype to use metronVersion
Needs more tests, but this should allow implmementation of delete()
At this time, the location of the hdfs grok directory is hard coded to the default If it is changed in ambari, this will not work.
…the patterns dir in hdfs while testing
fixed bug where we were calling wrong function in ConfigurationUtils
change ConfigurationType to allow changing zookeeper root generate a new extension name, must not have 0.0.0 version at end... will be the archive.tar.gz file name, with . -> _ and dropping the archive.tar.gz this foo-0.4.0-archive.tar.gz -> foo-0_5_0 as the extension name
the call works, but I cannot get the jsonpath verification right a single statement works in a verification tool, but not here
… dispatch in mock tests to get right result
…pattern dir for hdfs Be sure to rollback on install exception, and delete grok dir for installed parsers
…en out in master. Instead - do not delete kafka topic or storm topology, since this service doesn't create them.
Needs more tests, but this should allow implmementation of delete()
…the patterns dir in hdfs while testing
Add PARSER_EXTENSION to the configuraton reads in Stellar management
6818fdf to
9f8cdb4
Compare
Contributor
Author
|
I am closing this PR, I'll re-submit when METRON-942 is landed |
10 tasks
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
METRON-777 adds the Metron Extension capability, the default parsers as extensions, and the parser archetype
METRON-942 adds metron-rest support for installing and uninstalling parser extensions, as well as the configuration primitives for extensions in zookeeper
This PR adds a Stellar Management command to retrieve those configurations.
Testing
Build and install some parser extensions
Follow the test plan for METRON-942, to the point where you have built and installed one or more parser extensions.
deploy the metron-management jar per readme
Run Stellar
[vagrant@node1 ~]$ /usr/metron/0.4.0/bin/stellar -z node1:2181Run the command:
verify that the configuration return matches what you would see in the rest client
if you executed Parser Extension Controller : get /api/v1/ext/parsers/{name} -> name = metron-parser-nice-assembly-0_4_0
For all changes:
For code changes:
[na ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
[ x] Have you included steps or a guide to how the change may be verified and tested manually?
[x ] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via:
[x ] Have you written or updated unit tests and or integration tests to verify your changes?
[ na] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
For documentation related changes:
Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via
site-book/target/site/index.html:Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that travis-ci is set up for your personal repository such that your branches are built there before submitting a pull request.