METRON-942 [NO MERGE UNTIL METRON-777] Rest api and configuration for Metron parser extensions

[ottobackwards]

Contributor Comments

METRON-777 introduces metron extensions and metron parser extensions. The existing parsers have been moved to be packaged and loaded as extensions.
A maven archetype for creating parser extensions has also been created.

What is missing however is the ability to install a parser extension created outside the metron tree by the archetype into a metron system.
This PR introduces a REST api to accomplish this.

Along with that, this PR introduces a parser extension configuration type, for zookeeper that serves as a registry for installed extensions
and their default configurations. The idea being that in a follow on effort, the management UI will create a new 'instance' of a parser type
through cloning, editing and deploying these configurations. The included configurations are indexing, enrichment, parser, and elasticsearch.
There may be multiple of these configurations, as each parser extension library can contain more than one parser.

NOTE: the elasticsearch template is NOT deployed to elasticsearch at this time. Any parser created will still need a separate step for this.

One outstanding question for feedback -> The configuration tracks the parsers created out of the extension, initially. In the future this list should
have new parsers added to it when they are cloned. This would mean we would want new rest endpoints to edit the configuration ( most likely api to add parser to config {name} ) added

This PR is built on top of METRON-777, and should not be merged until METRON-777 is in, and this branched is rebased on to master.

This PR also addresses METRON-947, adding Management Stellar function support for the extension configurations

USAGE

For usage, I'll cover creating a parser and deploying it... and deleting it using the rest api.

Preparation

In order to build the parser, we need to build and install the maven plugin to build the bundles, as well as the maven archetype itself.
At this time, since they are not published to apache maven, this will be a requirement.

• Build the bundle plugin

cd bundles-maven-plugin && mvn -q install && cd ..

• Build the archetype

cd metron-maven-archetypes/metron-maven-parser-extension-archetype
mvn install
cd ../..

Create and build a new parser extension

• Make a directory and cd into it
• Create from archetype

mvn -U archetype:generate -DarchetypeCatalog=local

• choose the org.apache.metron:metron-maven-parser-extension-archetype (Apache Maven Parser Extension Archetype for Metron) option
• Fill out the information (information on the parameters is in the README for the archetype)
• NOTE: at this time, the version must match the metronVersion. Maven is using version in the parent pom chain, and messing up the plugin dependency [ THIS HAS BEEN RESOLVED]
• cd into the created directory and build the parser extension

mvn package

This will result in the tar.gz being created inside the {NAME}-parser-assembly project/target directory

start full_dev

• run vagrant up from metron-deployment/vagrant/full_dev_platform

add a slot to storm

• log into ambari on http://node1:8080
• add a new slot to the storm config

Open swagger

• log into ambari on http://node1:8080
• go into the metron service | quicklinks
• start swagger with user | password

Install your parser

In swagger, using the parser-extension-controller

• drop down the POST method
• use the file selector to select the created .tar.gz
• execute
  The extension should now be installed:
• you should be able to use the other GET methods to review the generated configuration.
  • the configuration will have the extensionID to use for the other {name} operations
• you should be able to view the installed files in hdfs/apps/metron/patterns/{parsername}, hdfs/apps/metron/extensions_alt_lib/

Start the parser using rest

Still in swagger

• Use the Kafka Controller and the Storm Controller to create a {parserName} topic, and start a storm job for {parserName}
• In the storm UI you should see the storm instance for the parser
  You should be able to verify using the rest api and the various controllers that the configurations are there and the topics exist and the storm jobs as well

Uninstall your parser

• STOP and DELETE the Storm topology and Kafka Topic

In swagger, using the parser-extension-controller

• drop down the DELETE method
• fill in the name parameter with the extensionID from the config
• exectute

When complete you should see that the various configurations are all gone, along with the patterns and other things from hdfs

Because this is on METRON-777 the changed file list remains huge.
The main changes here are those in the metron-interface directories
As well as those in the metron-common configuration

In github, select all changes from 2f9cf0d onwards.

Testing Stellar

• create and install an extension as above then:

Run Stellar

[vagrant@node1 ~]$ /usr/metron/0.4.1/bin/stellar  -z node1:2181

Run the command:

CONFIG_GET('PARSER_EXTENSION', 'metron-parser-nice-assembly-0_4_1')

verify that the configuration return matches what you would see in the rest client
if you executed Parser Extension Controller : get /api/v1/ext/parsers/{name} -> name = metron-parser-nice-assembly-0_4_1

Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.
Please refer to our Development Guidelines for the complete guide to follow for contributions.
Please refer also to our Build Verification Guidelines for complete smoke testing guides.

In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:

For all changes:

• [x ] Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
• [ x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
• [x ] Has your PR been rebased against the latest commit within the target branch (typically master)?

For code changes:

• [x ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?

• [ x] Have you included steps or a guide to how the change may be verified and tested manually?

• [x ] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via:

  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  

• [x ] Have you written or updated unit tests and or integration tests to verify your changes?

• [x ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

• [x ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

For documentation related changes:

• [x ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via site-book/target/site/index.html:

  cd site-book
  mvn site
  

Note:

Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that travis-ci is set up for your personal repository such that your branches are built there before submitting a pull request.

[ottobackwards]

I am closing this in preparation for feature branch