Skip to content
This repository was archived by the owner on Aug 20, 2025. It is now read-only.

Conversation

@ottobackwards
Copy link
Contributor

METRON-777 adds the Metron Extension capability, the default parsers as extensions, and the parser archetype
METRON-942 adds metron-rest support for installing and uninstalling parser extensions, as well as the configuration primitives for extensions in zookeeper

This PR adds a Stellar Management command to retrieve those configurations.

Testing

Build and install some parser extensions

Follow the test plan for METRON-942, to the point where you have built and installed one or more parser extensions.

deploy the metron-management jar per readme

Run Stellar

[vagrant@node1 ~]$ /usr/metron/0.4.0/bin/stellar  -z node1:2181

Run the command:

CONFIG_GET('PARSER_EXTENSION', 'metron-parser-nice-assembly-0_4_0')

verify that the configuration return matches what you would see in the rest client
if you executed Parser Extension Controller : get /api/v1/ext/parsers/{name} -> name = metron-parser-nice-assembly-0_4_0

For all changes:

  • [ x] Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
  • [ x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
  • [x ] Has your PR been rebased against the latest commit within the target branch (typically master)?

For code changes:

  • [na ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?

  • [ x] Have you included steps or a guide to how the change may be verified and tested manually?

  • [x ] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via:

    mvn -q clean integration-test install && build_utils/verify_licenses.sh 
    
  • [x ] Have you written or updated unit tests and or integration tests to verify your changes?

  • [ na] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

  • [ x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

For documentation related changes:

  • Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via site-book/target/site/index.html:

    cd site-book
    mvn site
    

Note:

Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that travis-ci is set up for your personal repository such that your branches are built there before submitting a pull request.

ottobackwards and others added 30 commits April 13, 2017 15:59
Based on Apache Nifi Nars

NAR changes
* new lib , rebrand to bundles from NAR
* port to VFS/FileObject from File based
* ability to set property values
* Rework FileUtils so that you can derive and override
* added initializers to set 'classes' that we care about instead of hard coding them, still needs defaults
* added components nec. for integration tests ( do not want dep. on metron-* )
* VFSClassloader for NarClassLoader
* Hdfs based integration test version of unpacknars tests
* HDFS ( filesystem ) based fileutilities to cover for writes to hdfs, since VFS is currently R/O HDFS
* modified plugin to support configuration of outputs
* use class index not service loader ( both subclass and annotated supported )

Archetype
* Parser Extension archetyp
* incudes all configuration
* creates tar.gz with bundle and configuration
* class index support ( automatic generation )

Extensions
* new extensions modules
* parser
* archetype built module for each parser type
* support for configuration only parsers with tests

Parsers
* moved all but json, csv, grok to extensions
* Bolt now loads from bundle properties

Deployment
* rpms for parsers
* create extension directories
* ambari initializes zookeeper per parser
* amabri creates hdfs directories
* ISSUE: Writing to hdfs

Rest-API
* only test against parsers in metron-parsers
* still needs integration
… it is when running from archetype, causing the extension versio to be used for this dependency
Needs more tests, but this should allow implmementation of delete()
At this time, the location of the hdfs grok directory is hard coded to the default
If it is changed in ambari, this will not work.
fixed bug where we were calling wrong function in ConfigurationUtils
change ConfigurationType to allow changing zookeeper root
generate a new extension name, must not have 0.0.0 version at end...
will be the archive.tar.gz file name, with . -> _ and dropping the archive.tar.gz
this foo-0.4.0-archive.tar.gz -> foo-0_5_0  as the extension name
the call works, but I cannot get the jsonpath verification right
a single statement works in a verification tool, but not here
…pattern dir for hdfs

Be sure to rollback on install exception, and delete grok dir for installed parsers
…en out in master.

Instead - do not delete kafka topic or storm topology, since this service doesn't create
them.
Needs more tests, but this should allow implmementation of delete()
Add PARSER_EXTENSION to the configuraton reads in Stellar management
@ottobackwards
Copy link
Contributor Author

I am closing this PR, I'll re-submit when METRON-942 is landed

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants