METRON-671: Refactor existing Ansible deployment to use Ambari MPack

Upgrading.md

@@ -152,3 +152,11 @@ When using Ansible to deploy the latest Metron bits to an existing installation,
 #### Workaround
 Set the `topology.classpath` property for storm in Ambari to `/etc/hbase/conf:/etc/hadoop/conf`
 
+## 0.3.1 to PLACEHOLDER
+
+### [METRON-671: Refactor existing Ansible deployment to use Ambari MPack](https://issues.apache.org/jira/browse/METRON-671)
+
+#### Description
+Since the Ansible Deployment uses the MPack, RPMs must be built prior to deployment. As a result,
+[Docker](https://www.docker.com/) is required to perform a Quick-Dev, Full-Dev or Ansible deployment.
+This effectively limits the build environment to Docker supported [platforms](https://docs.docker.com/engine/installation/#platform-support-matrix).

metron-deployment/README.md

@@ -1,97 +1,51 @@
 # Overview
-This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster, Metron services, or both using ansible
-playbooks. These playbooks currently only target RHEL/CentOS 6.x operating
+This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating
 systems.
 
-In addition, an Ambari Management Pack can be built which can be deployed in conjuction with RPMs detailed in this README.
+Installation consists of -
+- Building Metron tarballs, RPMs and the Ambari MPack
+- Deploying Ambari
+- Leveraging Ambari to install:
+  * The required Hadoop Components
+  * Core Metron (Parsing, Enrichment, Indexing)
+  * Elasticsearch
+  * Kibana
+- Starting All Services
 
 ## Prerequisites
 The following tools are required to run these scripts:
 
 - [Maven](https://maven.apache.org/)
 - [Git](https://git-scm.com/)
-- [Ansible](http://www.ansible.com/) (version 2.0 or greater)
-
-Currently Metron must be built from source.  Before running these scripts perform the following steps:
-
-1. Clone the Metron git repository with `git clone git@github.com:apache/incubator-metron.git`
-2. Navigate to `incubator-metron` and run `mvn clean package`
+- [Ansible](http://www.ansible.com/) (version 2.0.0.2)
+- [Docker](https://www.docker.com/) (Docker for Mac on OSX)
 
 These scripts depend on two files for configuration:
 
 - hosts - declares which Ansible roles will be run on which hosts
 - group_vars/all - various configuration settings needed to install Metron
 
-Examples can be found in the
-`incubator-metron/metron-deployment/inventory/metron_example` directory and are a good starting point.  Copy this directory
-into `incubator-metron/metron-deployment/inventory/` and rename it to your `project_name`.  More information about Ansible files and directory
-structure can be found at http://docs.ansible.com/ansible/playbooks_best_practices.html.
-
+For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below.
 ## Ambari
-The Ambari playbook will install a Hadoop cluster with all the services and configuration required by Metron.  This
-section can be skipped if installing Metron on a pre-existing cluster.
-
-Currently, this playbook supports building a local development cluster running on one node but options for other types
- of clusters will be added in the future.
-
-### Setting up your inventory
-Make sure to update the hosts file in `incubator-metron/metron-deployment/inventory/project_name/hosts` or provide an
-alternate inventory file when you launch the playbooks, including the
-ssh user(s) and ssh keyfile location(s). These playbooks expect two
-host groups:
-
-- ambari_master
-- ambari_slaves
-
-### Running the playbook
-This playbook will install the Ambari server on the ambari_master, install the ambari agents on
-the ambari_slaves, and create a cluster in Ambari with a blueprint for the required
-Metron components.
-
-Navigate to `incubator-metron/metron-deployment/playbooks` and run:
-`ansible-playbook -i ../inventory/project_name ambari_install.yml`
-
-## Metron
-The Metron playbook will gather the necessary cluster settings from Ambari and install the Metron services.
+The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana.
 
-### Setting up your inventory
-Edit the hosts file at `incubator-metron/metron-deployment/inventory/project_name/hosts`.  Declare where which hosts the
-Metron services will be installed on by updating these groups:
-
-- enrichment - submits the topology code to Storm and requires a storm client
-- search - host where Elasticsearch will be run
-- web - host where the Metron UI and underlying services will run
-- sensors - host where network data will be collected and published to Kafka
-
-The Metron topologies depend on Kafka topics and HBase tables being created beforehand.  Declare a host that has Kafka and HBase clients installed by updating these groups:
-
-- metron_kafka_topics
-- metron_hbase_tables
-
-If only installing Metron, these groups can be ignored:
-
-- ambari_master
-- ambari_slaves
-
-### Configuring group variables
-The Metron Ansible scripts depend on a set of variables.  These variables can be found in the file at
-`incubator-metron/metron-deployment/inventory/project_name/group_vars/all`.  Edit the ambari* variables to match your Ambari
-instance and update the java_home variable to match the java path on your hosts.
-
-### Running the playbook
-Navigate to `incubator-metron/metron-deployment/playbooks` and run:
-`ansible-playbook -i ../inventory/project_name metron_install.yml`
+Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2.
 
 ## Vagrant
-A VagrantFile is included and will install a working version of the entire Metron stack.  The following is required to
-run this:
+There are current two Vagrant modes, full-dev and quick-dev. Full-dev installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure.
+Quick-dev re-installs the core Metron Services (Parsing, Enrichment, and Indexing)on a pre-built instance.
+Use quick-dev for testing out changes to core Metron services.
 
-- [Vagrant](https://www.vagrantup.com/)
-- Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is
+### Prerequsities
+- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+)
+- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is
 installed
 
-Navigate to `incubator-metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`.  This also provides a good
-example of how to run a full end-to-end Metron install.
+### Full-Dev
+Navigate to `incubator-metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`.
+
+### Quick-Dev
+Navigate to `incubator-metron/metron-deployment/vagrant/quick-dev-platform` and run `vagrant up`.
 
 ## Ambari Management Pack
 An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update.

metron-deployment/amazon-ec2/conf/defaults.yml

@@ -17,19 +17,12 @@
 ---
 # which services should be started?
 services_to_start:
-  - elasticsearch
   - pcap-service
-  - kibana
   - yaf
   - snort
   - snort-logs
   - bro
   - pcap-replay
-  - yaf-parser
-  - bro-parser
-  - snort-parser
-  - enrichment
-  - indexing
 
 # the ami for cent6 by region
 amis_by_region:

metron-deployment/amazon-ec2/playbook.yml

@@ -26,12 +26,12 @@
     - include: tasks/create-vpc.yml
     - include: tasks/create-open-inbound-security-group.yml
     - include: tasks/create-open-outbound-security-group.yml
-    - include: tasks/create-hosts.yml host_count=1 host_type=sensors,ambari_master,metron,ec2
+    - include: tasks/create-hosts.yml host_count=1 host_type=sensors,ambari_master,ec2,monit
     - include: tasks/create-hosts.yml host_count=4 host_type=ambari_slave,ec2
-    - include: tasks/create-hosts.yml host_count=1 host_type=ambari_slave,metron_kafka_topics,metron_hbase_tables,metron,ec2,pcap_server
-    - include: tasks/create-hosts.yml host_count=1 host_type=ambari_slave,enrichment,metron,ec2
-    - include: tasks/create-hosts.yml host_count=2 host_type=search,metron,ec2
-    - include: tasks/create-hosts.yml host_count=1 host_type=web,metron,ec2
+    - include: tasks/create-hosts.yml host_count=1 host_type=pcap_server,monit,ec2
+    - include: tasks/create-hosts.yml host_count=1 host_type=ambari_slave,enrichment,metron,ec2,zeppelin
+    - include: tasks/create-hosts.yml host_count=2 host_type=ambari_slave,search,ec2
+    - include: tasks/create-hosts.yml host_count=1 host_type=ambari_slave,web,ec2
   tags:
     - ec2
 

metron-deployment/amazon-ec2/run.sh

@@ -58,15 +58,10 @@ echo "=============================================================" >> $LOGFILE
 echo "Launching Metron[$ENV] @ $NOW"... >> $LOGFILE
 $DEPLOYDIR/../scripts/platform-info.sh >> $LOGFILE
 
-# build metron
-cd ../..
-mvn package -DskipTests -PHDP-2.5.0.0
-RC=$?; if [[ $RC != 0 ]]; then exit $RC; fi
-
 # deploy metron
 cd $DEPLOYDIR
 export EC2_INI_PATH=conf/ec2.ini
 ansible-playbook -i ec2.py playbook.yml \
-  --skip-tags="solr, sensor-stubs" \
+  --skip-tags="quick_dev,sensor-stubs" \
   --extra-vars="env=$ENV" \
   $EXTRA_ARGS

metron-deployment/extra_modules/ambari_cluster_state.py

@@ -385,9 +385,9 @@ def blueprint_var_to_ambari_converter(blueprint_var):
         this_host_map['hosts'] = this_host_list
         host_map.append(this_host_map)
     blueprint = dict()
+    blueprint['configurations'] = blueprint_var['required_configurations']
     blueprint['host_groups'] = new_groups
     blueprint['Blueprints'] = {'stack_name': blueprint_var['stack_name'], 'stack_version': blueprint_var['stack_version']}
-
     return blueprint, host_map
 
 from ansible.module_utils.basic import *