Skip to content
This repository was archived by the owner on Aug 20, 2025. It is now read-only.

METRON-1136 Initial Merge of Metron Extensions and Parser Extensions #720

Closed
ottobackwards wants to merge 1 commit into from
Closed

METRON-1136 Initial Merge of Metron Extensions and Parser Extensions #720

ottobackwards wants to merge 1 commit into from

Conversation

@ottobackwards
Copy link
Contributor

@ottobackwards ottobackwards commented Aug 29, 2017 •
edited
Loading

This pull request will land the Metron Extensions and Parser Extensions functionality in the new feature branch.

It represents the total combined work on this functionality:

This feature branch work is now being tracked in Confluence and Jira

Status

[x] Creation of Metron Bundles ( adaptation and refactoring of Nifi NAR )
[x] Adaption of nar maven plugin to bundles maven plugin
[x] Archetype for creating parser extensions
[x] metron-parsers reduced to library for interfaces, base types, and base parsers
[x] Metron parser types moved to parser extensions
[x] Metron parser extensions packaged by rpm, installed by ambari including hdfs deployment
[x] Metron parser extensions loaded by bundle from parser bolt, from hdfs
[x] Refactored grok building to support variable metron hdfs apps root, parsers with directories of groks
[x] Rest api for parser extensions ( install, delete, list, find )
[x] Zookeeper registration for extensions and their metadata, including default configurations
[x] Stellar management for parser extension configurations
[x] Guide to adding parsers to the metron system

Currently there is no work in Metron Config UI for installing or managing parser extensions

For all changes:

  • Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
  • Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
  • Has your PR been rebased against the latest commit within the target branch (typically master)?

For code changes:

  • Have you included steps to reproduce the behavior or problem that is being changed or addressed?

see contributing PR's

  • Have you included steps or a guide to how the change may be verified and tested manually?

see contributing PR's

  • Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:

    mvn -q clean integration-test install && build_utils/verify_licenses.sh

  • Have you written or updated unit tests and or integration tests to verify your changes?

  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

  • Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

For documentation related changes:

  • Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via site-book/target/site/index.html:

    cd site-book
mvn site

Note:

Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that travis-ci is set up for your personal repository such that your branches are built there before submitting a pull request.

@ottobackwards
Squashed commit of the following:
d68019d 
commit 2b54f85
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Mon Aug 28 11:46:28 2017 -0400

    conform to logging norms

commit 2c1eab5
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Mon Aug 28 11:11:50 2017 -0400

    more test tweaks

commit 76affe1
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Mon Aug 28 10:04:49 2017 -0400

    do not need the resource copy

commit 5c38f00
Merge: dcf9cee c8e84fa
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Mon Aug 28 09:40:51 2017 -0400

    Merge branch 'master' of https://github.com/apache/metron into METRON-777

commit dcf9cee
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Mon Aug 28 09:18:25 2017 -0400

    copy resources before class

commit e18fe49
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Mon Aug 28 08:44:12 2017 -0400

    javadoc and some saftey options for ResourceCopier

commit 799b0b7
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Mon Aug 28 00:02:54 2017 -0400

    integrate with 777 + 777-grok

commit 4978863
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sat Aug 26 00:20:59 2017 -0400

    fix up imports

commit 239df5f
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed Aug 9 10:26:28 2017 -0400

    missed a fix, rename folder

commit bacd115
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed Aug 9 10:00:16 2017 -0400

    adding the metron management function for parser extension configs from METRON-947, no reason to maintain two prs

commit 94d223e
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Mon Aug 7 16:43:26 2017 -0400

    merge fixes

commit 97edf2f
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri Jun 30 16:33:24 2017 -0400

    change documenation to reflect uninstall

commit b0fc2e4
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri Jun 30 16:31:19 2017 -0400

    After merge, fix tests.  Problems re-activating kafka after it was taken out in master.
    Instead - do not delete kafka topic or storm topology, since this service doesn't create
    them.

commit a02b6fc
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Tue Jun 6 10:24:17 2017 -0400

    better readme for usage

commit beee5d9
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed May 10 09:25:24 2017 -0400

    Add endpoints to readme

commit 2d0edf6
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Tue May 9 09:50:23 2017 -0400

    Use the ambari managed spring configuration to properly get the grok pattern dir for hdfs
    Be sure to rollback on install exception, and delete grok dir for installed parsers

commit fda824e
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri May 5 10:43:25 2017 -0400

    Do not install an extension if it is already installed

commit e536adf
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri May 5 10:17:09 2017 -0400

    fix extensions alt lib path in config

commit cf3ad69
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu May 4 14:58:29 2017 -0400

    cleanup a couple of imports

commit 40048db
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu May 4 14:53:27 2017 -0400

    implementation of delete

commit 1912795
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed May 3 07:59:49 2017 -0400

    async delete call

commit 2e2c7a2
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed May 3 07:15:36 2017 -0400

    fix to return th correct httpstatus of created for install, use async dispatch in mock tests to get right result

commit a3a5d47
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Tue May 2 23:54:48 2017 -0400

    added delete test, need to implement

commit 6937811
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Tue May 2 23:44:10 2017 -0400

    fix json path issue for test of getAll

commit c283907
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Tue May 2 18:29:55 2017 -0400

    test for getall()
    the call works, but I cannot get the jsonpath verification right
    a single statement works in a verification tool, but not here

commit f942949
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Tue May 2 13:20:13 2017 -0400

    test for getAll
    change ConfigurationType to allow changing zookeeper root
    generate a new extension name, must not have 0.0.0 version at end...
    will be the archive.tar.gz file name, with . -> _ and dropping the archive.tar.gz
    this foo-0.4.0-archive.tar.gz -> foo-0_5_0  as the extension name

commit ade08a9
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Tue May 2 10:37:32 2017 -0400

    update goal comment

commit b193fe8
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Tue May 2 10:11:33 2017 -0400

    support for putting the default es templates in the extension config
    fixed bug where we were calling wrong function in ConfigurationUtils

commit e62b021
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Mon May 1 17:51:11 2017 -0400

    grok fix, after some manual testing.  need to provide an override for the patterns dir in hdfs while testing

commit 98f0c40
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Mon May 1 16:39:11 2017 -0400

    Write out files to grok.
    At this time, the location of the hdfs grok directory is hard coded to the default
    If it is changed in ambari, this will not work.

commit 098a288
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Tue Jul 4 00:46:15 2017 -0400

    track extension installs, along with metadata and defaults
    Needs more tests, but this should allow implmementation of delete()

commit 180215a
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sun Apr 30 11:22:11 2017 -0400

    Create an actual context object and build it out with more information, including bundle information
    The goal is to have enough information to fill out a configuration object that is in zk, that will allow
    for some kind of removal/uninstall of a parser

commit 7ca11b6
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sat Apr 29 13:09:38 2017 -0400

    do not  assume empty content, unless you are going to clear out everything

commit 6f52573
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sat Apr 29 00:11:26 2017 -0400

    do not need to add this, with the right intellij project ref to metron-common this works fine

commit 37def88
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri Apr 28 23:41:27 2017 -0400

    start of controller test

commit cd2f589
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri Apr 28 10:50:15 2017 -0400

    correct path to write bundle to

commit 6f9d170
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri Apr 28 10:39:31 2017 -0400

    first pass implementation, no tests

commit c0c8312
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu Apr 27 23:49:24 2017 -0400

    use context to track paths that are created, delete tmp files on exit

commit d418373
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu Apr 27 23:08:47 2017 -0400

    wire up the services we will use

commit 6e0ec09
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu Apr 27 23:00:38 2017 -0400

    start of implementation of service, validation of extensions

commit 5dc7e2f
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu Apr 27 17:02:49 2017 -0400

    create controller class shell

commit 1f71847
Merge: ae3a07a 219012b
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sun Aug 27 12:48:15 2017 -0400

    Merge branch '777-grok' into METRON-777

commit ae3a07a
Merge: a53aadc a2bae0b
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sat Aug 26 00:12:07 2017 -0400

    Merge branch 'master' of https://github.com/apache/metron into METRON-777

commit 219012b
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri Aug 25 15:21:18 2017 -0400

    remove debug and fixup docs and tests for ResourceLoader

    handle paths missing / better

commit 97cf0d7
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri Aug 25 12:08:04 2017 -0400

    cleanup SesnsorParserConfigServiceImpl

commit 4fa2423
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri Aug 25 12:01:50 2017 -0400

    cleanup BundleSystemConfig

commit 152100c
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri Aug 25 11:41:00 2017 -0400

    correct the tests

commit 5f7e52b
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri Aug 25 09:30:18 2017 -0400

    fix grok tests, add new test for save statement

commit b2eec55
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri Aug 25 07:43:51 2017 -0400

    working integegration with parser extensions as bundles and rest/ui

    cleanup and refactor todo

commit 9c7f98f
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Tue Aug 22 16:20:05 2017 -0400

    refactor hdfs fixups so they are not required

commit 5c98a15
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Tue Aug 22 02:36:33 2017 -0400

    first pass integration of rest with bundles and extensions

commit f0923a6
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed Aug 16 16:35:20 2017 -0400

    refactor hdfs and pattern loading

    Create GrokBuilder and ResourceLoader for re-usable and consistant file loading and path fixups
    tests create the correct ./target environment for rules to be loaded from

    ParserBolt injects the global configuration into the parser configuration
    Global configuration has new key for metron hdfs root

    new key is setup by ambari

    Now, give a parser configuration with a simple grokPath "/patterns/yaf"
    - it will be fixed to work in hdfs or local for tests
    - it will support yaf as a directory not just a file

commit a53aadc
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed Aug 23 02:13:43 2017 -0400

    fix test, missing comma from merge

commit 4dc1650
Merge: b25d5f1 d64afbc
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Tue Aug 22 16:25:47 2017 -0400

    Merge branch 'master' of https://github.com/apache/metron into METRON-777

commit b25d5f1
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu Aug 17 00:13:38 2017 -0400

    Changes for synchronization and other things per latest review

commit c0fd02d
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed Aug 16 16:40:37 2017 -0400

    fix indentation error

commit 8455bbe
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Mon Aug 14 19:04:25 2017 -0400

    update doc per review

commit 1394217
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Mon Aug 14 19:01:59 2017 -0400

    sigh, fix tests

commit c5f8c34
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Mon Aug 14 18:36:10 2017 -0400

    address review comments around exceptions, singltons

commit 9526fed
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Mon Aug 14 15:09:50 2017 -0400

    javadoc for extension mapping

commit d652fa0
Merge: 88c005f adb378f
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Mon Aug 14 14:54:00 2017 -0400

    Merge branch 'master' of https://github.com/apache/metron into METRON-777

commit 88c005f
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Mon Aug 14 14:50:00 2017 -0400

    per review - add comments about extensiable extensions namespaces and difference with nar

commit fd887c1
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Mon Aug 14 14:40:37 2017 -0400

    add dummy implementation of FileObject for system bundle

commit 72abee7
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sun Aug 13 13:11:26 2017 -0400

    document usage of BundleSystem

commit b672ef3
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sun Aug 13 12:50:10 2017 -0400

    javadoc additions

commit ce8d98d
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sun Aug 13 12:14:17 2017 -0400

    simplify, the extensionmanager has the classes, no need to do the init dance with the BundleThreadContextClassLoader

commit d1b9871
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sat Aug 12 22:47:23 2017 -0400

    test fix: need to clear after class

commit 5cb6bdf
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sat Aug 12 22:05:28 2017 -0400

    javadoc and formatting

commit 5bcc8c6
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sat Aug 12 22:01:45 2017 -0400

    unit test and some fixups

commit 97295d7
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sat Aug 12 21:44:32 2017 -0400

    returning a value would work better

commit 70f2106
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sat Aug 12 18:53:57 2017 -0400

    simplified the call interface for creating classes

commit 103f869
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sat Aug 12 17:38:33 2017 -0400

    we do not need to do the mapping, we are not using them

commit 2f93bfc
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sat Aug 12 17:11:16 2017 -0400

    more javadoc detail

commit 3edafdb
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sat Aug 12 16:58:09 2017 -0400

    enforce immutability in the BundleClassLoader as well

commit 4cf9cb4
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sat Aug 12 13:15:28 2017 -0400

    based on review, refactor the ExtensionManager for Immutability and Thread Safety

commit 97ef4f0
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri Aug 11 22:21:49 2017 -0400

    per review, add clarification

commit 22ba18b
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri Aug 11 22:19:13 2017 -0400

    per review fix spelling

commit 571168d
Merge: ea6240a 86b0f13
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri Aug 11 14:15:26 2017 -0400

    Merge branch 'master' of https://github.com/apache/metron into METRON-777

commit ea6240a
Merge: 8b5d665 b3148a1
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri Aug 11 08:51:50 2017 -0400

    Merge branch 'master' of https://github.com/apache/metron into METRON-777

commit 8b5d665
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu Aug 10 17:30:42 2017 -0400

    remove reference to nifi

commit 27c7de4
Merge: 0b583e0 7d55444
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu Aug 10 10:12:51 2017 -0400

    Merge branch 'master' of https://github.com/apache/metron into METRON-777

commit 0b583e0
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed Aug 9 15:36:47 2017 -0400

    per review, added some validation, also renamed variables and accessors

commit b402517
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Tue Aug 8 16:57:49 2017 -0400

    the headers, the headers

commit 6690398
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Tue Aug 8 16:52:02 2017 -0400

    add header

commit c6e9a6a
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Tue Aug 8 16:50:20 2017 -0400

    remove unused import

commit 4bad948
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Tue Aug 8 16:48:16 2017 -0400

    fix working dir reference

commit f42eef2
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Tue Aug 8 16:43:06 2017 -0400

    Changes per reviews

commit 4f8409a
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Mon Aug 7 16:09:12 2017 -0400

    fix post merge bits

commit 54602cf
Merge: 0ffac16 813adf2
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Mon Aug 7 15:35:37 2017 -0400

    Merge branch 'master' of https://github.com/apache/metron into METRON-777

commit 0ffac16
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sat Aug 5 08:47:26 2017 -0400

    remove notice about versions matching

commit b1a9644
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sat Aug 5 08:41:59 2017 -0400

    change plugin inclusion in archetype pom, this resolves requirment for version to match metron version

commit bf5ec2f
Merge: 04727ca df25d81
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sat Aug 5 08:00:32 2017 -0400

    Merge branch 'master' of https://github.com/apache/metron into METRON-777

commit 04727ca
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu Aug 3 13:30:29 2017 -0400

    update document to remove references to the working dir

commit 3212c1b
Merge: c785929 b2375a1
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu Aug 3 08:11:57 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit c785929
Merge: 4b9a124 27ab289
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed Aug 2 17:24:52 2017 -0400

    Merge branch 'master' of https://github.com/apache/metron into METRON-777

commit 4b9a124
Merge: 7a97e41 ec959d2
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Tue Aug 1 12:00:14 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit 7a97e41
Merge: afe4b9e 7b9f20c
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Tue Aug 1 09:24:49 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit afe4b9e
Merge: 5bb6a33 c99df8f
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Mon Jul 31 21:12:28 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit 5bb6a33
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sat Jul 29 23:15:16 2017 -0400

    refactor/rename VFSUtil to FileSystemManagerFactory, which is what it is

commit 2e48f08
Merge: f1cdcd7 ba46fa7
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sat Jul 29 13:00:51 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit f1cdcd7
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sat Jul 29 12:17:34 2017 -0400

    for BundleThreadContextClassloaderTest, fix last path. in travis if the folder is empty it may not be there?

commit 98545d1
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sat Jul 29 12:03:39 2017 -0400

    more travis fix attempts

commit 868cf6c
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sat Jul 29 11:45:01 2017 -0400

    attempt to fix travis only failure

commit 66a0c82
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sat Jul 29 10:46:37 2017 -0400

    Based on feedback from reviews and discussion with mattf,  this commit represent a major refactoring for the bundle functionality.
    While trying to resolve the ugliness bought about by working around VFS's read only support for HDFS, I have come up with a different approach.

    While this approach makes bundles-lib significantly different than nar-utils, it makes it much more simple, and may arguably be an approach worth
    concidering.

    The original bundles-lib maintained the nar system's method of unpacking the bundle into a 'working' directory, and then using having the classloader
    load from that directory structure.  Trying to avoid too far a departure ( at the time ) from nifi, bundle-lib maintained the 'create folders and files as nec'
    scheme found in nifi.  Since nifi is always working with local disk and File objects, this is ok. But with VFS and HDFS this did not work.  The work around
    was to create a poop static 'class cluster' type step with multiple implemenations of FileUtils ( VFS and HDFS ).

    The new approach, which is a more willing departure from the nifi approach, is to avoid having to ever do writes ( file or folder creation ) from bundles-lib at all.
    This allows for a simplier, single implementation of the file utils and the removal of the anti-pattern.  It also removes the need for a 'working' area.

    The new approach is to better utilize the capabilities of VFS such that we never have to unpack the bundle at all.  VFS can load JAR files as Filesystems. I have created
    a new Classloader ( based on the VFS classloader implemenation from the VFS project ) that is bundle aware.  It can load the bundle and it's dependencies as resources
    and make them available.  This is what removes the requirement to unpack the bundles into a working directory.

    As such, the bundle-lib has been refactored to move from the 'working directory' to only needing to work with the bundle in the lib.  I have also removed the doc directory support
    since we are not using it.

    We no longer need multiple file utils, the hdfs working directory, and other things.

    Thanks to mattf for the time and discussion.

commit 9fffd61
Merge: bbc0228 9355a04
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu Jul 27 23:32:23 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit bbc0228
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu Jul 27 19:00:19 2017 -0400

    use MessageParser for consistantcy

commit 0579750
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu Jul 27 18:40:57 2017 -0400

    remove more nar-isms.  base tests on archetype build parser projects instead of old nifi processors

commit 7c84fc4
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu Jul 27 14:16:46 2017 -0400

    refactore type property to packageType per review

commit 4650125
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu Jul 27 08:25:40 2017 -0400

    remove nar references per review

commit f84efb7
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu Jul 27 08:18:42 2017 -0400

    remove nar references

commit c4f3468
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu Jul 27 08:12:07 2017 -0400

    refactor package per review

commit fc38b3d
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu Jul 27 08:06:52 2017 -0400

    more incubator instances

commit 132763c
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu Jul 27 08:04:10 2017 -0400

    remove incubator references

commit 8f274d8
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed Jul 26 11:24:30 2017 -0400

    reformat based on review

commit a6dffb2
Merge: 0d9a43a badc6cf
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed Jul 26 11:23:37 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit 0d9a43a
Merge: fcb8ef7 309d809
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Mon Jul 24 10:05:28 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit fcb8ef7
Merge: 7ee99ef cf7043c
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu Jul 20 10:44:54 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit 7ee99ef
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Tue Jul 18 09:52:54 2017 -0400

    reverting last change, it worked in other travis repo

commit 1b095e5
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Tue Jul 18 09:49:06 2017 -0400

    try to resolve travis not building bundle

commit 06e4d76
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri Jul 14 08:58:32 2017 -0400

    need to install bundle now that we moved the other install phase

commit 3ef2059
Merge: 6d4be0f 954b5ea
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri Jul 14 08:54:09 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit 6d4be0f
Merge: b673fc7 dcf7682
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed Jul 12 10:43:40 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit b673fc7
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri Jul 7 15:59:16 2017 -0400

    fix version in new poms

commit 8b70e08
Merge: d12d3fc a8f3536
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri Jul 7 15:35:30 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit d12d3fc
Merge: efecf3a f7a94f2
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri Jul 7 12:20:26 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit efecf3a
Merge: 31d52d4 df270fe
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed Jul 5 12:56:01 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit 31d52d4
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed Jul 5 11:02:58 2017 -0400

    add documentation of multiple configurations of multiple parsers in one extension

commit 3e9c397
Merge: 8a53de8 cc7bbc9
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed Jul 5 09:10:24 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit 8a53de8
Merge: e67a55e a5b1377
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Tue Jul 4 00:43:25 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit e67a55e
Merge: 679d56f df94ed4
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri Jun 30 12:36:20 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777
    Fix integration tests to work with extension paths

commit 679d56f
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu Jun 29 20:58:54 2017 -0400

    per review, changed version in plugin to 2.4

    also updated metron version in the archetype test files

commit 2a7e8d3
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu Jun 29 07:57:58 2017 -0400

    fix extra project node per review

    add ASF headers to other poms

commit 6d98482
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu Jun 29 00:08:59 2017 -0400

    removed deployed pattern directory and common pattern files for parsers that do not use patterns

commit 2db5814
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed Jun 28 23:23:27 2017 -0400

    fix spacing, fix name

commit b8e65fb
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed Jun 28 23:11:13 2017 -0400

    deployment readme

commit 86190f8
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed Jun 28 22:19:31 2017 -0400

    readme changes per review

commit a8f34ef
Merge: b3daa9e 095be23
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed Jun 28 22:06:44 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit b3daa9e
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Tue Jun 27 18:13:28 2017 -0400

    start of deployment documenation

commit 5029033
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Tue Jun 27 13:07:08 2017 -0400

    remove reporting tag, we will inherit here - per review

commit 16f7999
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Tue Jun 27 11:29:14 2017 -0400

    more documentation work around packaging and terms

commit d4f28ed
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Tue Jun 27 08:05:25 2017 -0400

    fix readme links

commit 72eed83
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Mon Jun 26 18:15:46 2017 -0400

    start documentation of packaging and terms

commit 33682eb
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu Jun 22 09:40:26 2017 -0400

    more fge requirements

commit 269f739
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu Jun 22 09:16:28 2017 -0400

    put the fge json stuff back in the parsers that use them in test, as opposed to adding fge as a shipped dependency

commit bfafd30
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed Jun 21 16:22:20 2017 -0400

    create a testing pom project to hold testing only dependencies - per review

    remove some nar references

commit ee6fbd4
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed Jun 21 09:35:55 2017 -0400

    Per review, remove .gitignore left over from nifi plugin as separate project

commit ced6380
Merge: 9695727 b76bcd5
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed Jun 21 09:30:59 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit 9695727
Merge: 22c0cba de2c871
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Mon Jun 19 11:09:46 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit 22c0cba
Merge: 02ef46e f523c17
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri Jun 16 10:56:02 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit 02ef46e
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Tue Jun 13 14:21:34 2017 -0400

    per review, move metron-extensions under metron-platform

    associated fixes for path related things

commit a640806
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sun Jun 11 17:00:03 2017 -0400

    add readme for bundles-lib

    this readme specifically references the relationship to apache  nifi nars

    this readme is based on the nifi developer guide for nars

commit df78af1
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sun Jun 11 10:52:41 2017 -0400

    refactor based on feedback to use the metron integration test classes

    this should be OK because that jar does not pull in other metron jars and is test only

commit 9e012a5
Merge: 9099f36 30d0e2a
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri Jun 9 16:07:08 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit 9099f36
Merge: e9b8b9b 7d6121b
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri Jun 9 07:01:34 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit e9b8b9b
Merge: 350ac94 61cbab4
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu Jun 8 08:54:00 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit 350ac94
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed Jun 7 23:36:39 2017 -0400

    fix markdown so site-book does not crash

commit 145465f
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed Jun 7 23:13:50 2017 -0400

    Added a guide for adding parsers to the system

commit 572212f
Merge: 9bee17c 5b72da7
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed Jun 7 12:45:44 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit 9bee17c
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed Jun 7 09:12:55 2017 -0400

    per review: use inherited reporting in pom

commit d8b09c6
Merge: b25358f 605a3cc
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed Jun 7 09:07:42 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit b25358f
Merge: bf72681 d0e1ba5
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Mon Jun 5 21:43:47 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit bf72681
Merge: 48ce8e2 e31ed2f
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sat Jun 3 08:58:45 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit 48ce8e2
Merge: 23e2a0b 85872bd
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu Jun 1 11:34:03 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit 23e2a0b
Merge: f12922d 61105c7
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed May 31 10:20:37 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit f12922d
Merge: 0dd8cf8 a2634f3
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu May 25 22:01:56 2017 -0400

    Merge branch 'jz-metron-777' into METRON-777

commit a2634f3
Merge: 0dd8cf8 2e66ea3
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu May 25 22:00:07 2017 -0400

    Merge branch 'METRON-777' of https://github.com/JonZeolla/metron into jz-metron-777

commit 0dd8cf8
Merge: 382a29e 08f95dc
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu May 25 21:58:05 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit 2e66ea3
Author: Jon Zeolla <zeolla@gmail.com>
Date:   Thu May 25 18:07:23 2017 -0400

    Trivial documentation changes

commit 90a51de
Merge: 7ef929a 382a29e
Author: Jon Zeolla <zeolla@gmail.com>
Date:   Thu May 25 17:52:55 2017 -0400

    Merge branch 'METRON-777' of https://github.com/ottobackwards/metron into METRON-777

commit 382a29e
Merge: eb2062b e219731
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Mon May 22 08:01:38 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit eb2062b
Merge: b73f35f 356881a
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Mon May 22 07:58:50 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit b73f35f
Merge: 97c1be1 ce381f3
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri May 19 11:33:30 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit 97c1be1
Merge: 225fb4e be03076
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Tue May 16 22:22:33 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit 225fb4e
Merge: 5565ad6 345eba8
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri May 12 14:12:03 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit 5565ad6
Merge: ab0aad8 e375936
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed May 10 15:47:58 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit ab0aad8
Merge: 6c0201b 8532e28
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed May 10 08:57:21 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit 6c0201b
Merge: e3b71ae e08b7d4
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Tue May 9 09:55:31 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit e3b71ae
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sat May 6 12:08:58 2017 -0400

    use simple json and not the JSONUtil Instance, there are issues in storm loading

commit 1e0e305
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri May 5 14:18:32 2017 -0400

    update archetype based on changes to loading and configuration

commit a3e6355
Merge: 5f74fc3 716bda3
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu May 4 23:42:54 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit 5f74fc3
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu May 4 16:25:18 2017 -0400

    be sure to clear out before and after, we cannot assume order of maven build

commit 995a4d7
Merge: de48845 38d26d4
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu May 4 09:46:58 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit de48845
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed May 3 22:40:55 2017 -0400

    second attempt to fix plugin errors with jacoco

commit 9cafe97
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed May 3 22:18:29 2017 -0400

    specify the plugin to fix dependency problem

commit fd021e4
Merge: ac92d7b 1d27a32
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Wed May 3 10:34:31 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit ac92d7b
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sun Apr 30 09:44:58 2017 -0400

    instead of hard-coding in metron-parsers-extensions, overload in the archetype to use metronVersion

commit e21efb8
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sun Apr 30 09:16:41 2017 -0400

    make dependency explicit, I think parent version is not what we think it is when running from archetype, causing the extension versio to be used for this dependency

commit 158f463
Merge: c25585d 47e5aa7
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sat Apr 29 08:58:41 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit c25585d
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sat Apr 29 00:07:11 2017 -0400

    use the configured extension list, do not hardcode

commit bc7cc2d
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sat Apr 29 00:05:41 2017 -0400

    use correct testing bundle

commit 7760425
Merge: c18cbe3 5dd8788
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Thu Apr 27 06:45:37 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit c18cbe3
Merge: e8caaf5 41b5b10
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Sun Apr 23 12:14:15 2017 -0400

    merge apache/master

commit e8caaf5
Merge: 850154c e6992d5
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Mon Apr 17 08:19:02 2017 -0400

    Merge remote-tracking branch 'apache/master' into METRON-777

commit 850154c
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri Apr 14 08:55:37 2017 -0400

    fix package names

commit 864d320
Author: Otto Fowler <ottobackwards@gmail.com>
Date:   Fri Mar 17 00:56:49 2017 -0400

    Metron Extension system

    Based on Apache Nifi Nars

    NAR changes
    * new lib , rebrand to bundles from NAR
    * port to VFS/FileObject from File based
    * ability to set property values
    * Rework FileUtils so that you can derive and override
    * added initializers to set 'classes' that we care about instead of hard coding them, still needs defaults
    * added components nec. for integration tests ( do not want dep. on metron-* )
    * VFSClassloader for NarClassLoader
    * Hdfs based integration test version of unpacknars tests
    * HDFS ( filesystem ) based fileutilities to cover for writes to hdfs, since VFS is currently R/O HDFS
    * modified plugin to support configuration of outputs
    * use class index not service loader ( both subclass and annotated supported )

    Archetype
    * Parser Extension archetyp
    * incudes all configuration
    * creates tar.gz with bundle and configuration
    * class index support ( automatic generation )

    Extensions
    * new extensions modules
    * parser
    * archetype built module for each parser type
    * support for configuration only parsers with tests

    Parsers
    * moved all but json, csv, grok to extensions
    * Bolt now loads from bundle properties

    Deployment
    * rpms for parsers
    * create extension directories
    * ambari initializes zookeeper per parser
    * amabri creates hdfs directories
    * ISSUE: Writing to hdfs

    Rest-API
    * only test against parsers in metron-parsers
    * still needs integration
@ottobackwards ottobackwards mentioned this pull request Aug 29, 2017
Closed
@nickwallen
Copy link
Contributor

nickwallen commented Aug 30, 2017

Thanks, @ottobackwards . This looks ready to commit to our new feature branch. +1

asfgit pushed a commit that referenced this pull request Aug 30, 2017
@ottobackwards
METRON-1136 Metron Extensions System and Parser Extensions Feature Br…
ffcb91e 
…anch (ottobackwards) closes #720
@ottobackwards
Copy link
Contributor Author

ottobackwards commented Aug 30, 2017

ffcb91e

asfgit pushed a commit that referenced this pull request Aug 30, 2017
@ottobackwards
METRON-1136 Metron Extensions System and Parser Extensions Feature Br…
5f7454e 
…anch (ottobackwards) closes #720
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ottobackwards @nickwallen