METRON-235 Expose filtering capability for PCAP via CLI tool

[mmiklavc]

In the process of testing with Vagrant, but wanted to get this in front of people for review.

Relevant Jira:
https://issues.apache.org/jira/browse/METRON-235

In the process of upgrading to Kibana 4, we lost our about to query/filter pcap results. This PR exposes the 2 methods for filtering PCAP data, fixed parameters and the "Stellar" query language, via a command line tool. The tool is executed via ${metron_home}/bin/pcap_query.sh

Note: I also tweaked the mem settings for Ambari mapreduce to get around persistent OOM errors - mapreduce container sizes (mapreduce.[map | reduce].memory.mb) have been increased to 1.2 GiB, mapreduce.[map | reduce].java.opts have been increased to 1 GiB.

PR Testing Instructions:

Get PCAP data into Metron: Install and setup pycapa - the instructions below reference/mirror those in PR-93

1. Install the pycapa library & utility $ cd /opt/pycapa/pycapa && pip install -r requirements.txt && python setup.py install
2. (if using singlenode vagrant) Kill the enrichment and sensor topologies via for i in bro enrichment yaf snort;do storm kill $i;done
3. Start the pcap topology via /usr/metron/0.1BETA/bin/start_pcap_topology.sh
4. Start the pycapa packet capture producer on eth1 via /usr/bin/pycapa --producer --topic pcap -i eth1 -k node1:6667
5. Watch the topology in the Storm UI and kill the packet capture utility from before when the number of packets ingested is over 1k.
6. Ensure that at at least 2 files exist on HDFS by running hadoop fs -ls /apps/metron/pcap
7. Choose a file (denoted by $FILE) and dump a few of the contents using the pcap_inspector utility via /usr/metron/0.1BETA/bin/pcap_inspector.sh -i $FILE -n 5
8. Choose one of the lines and note the source ip/port and dest ip/port as well as the protocol.
9. Note that when you run the commands below, the resulting file will be placed in the execution directory where you kicked off the job from.

Fixed filter

1. Run a fixed filter query by executing the following command with the values noted above (match your start_time format to the date format provided - default is to use millis since epoch)
2. /usr/metron/0.1BETA/bin/pcap_query.sh fixed -st <start_time> -df "yyyyMMdd" -sa <address> -da <address> -sp <port> -dp <port> -p <protocol_num>
3. Verify the MR job finishes successfully. Upon completion, you should see a file named with the current datestamp in your current directory, e.g. pcap-data-20160617160549737+0000.pcap
4. Copy the file to your local machine and verify you can open it in Wireshark

Query filter

1. Run a Stellar query filter query by executing a command similar to the following, with the values noted above (match your start_time format to the date format provided - default is to use millis since epoch)
2. /usr/metron/0.1BETA/bin/pcap_query.sh query -st "20160617" -df "yyyyMMdd" -query "ip_src_addr == '192.168.1.1' and ip_src_port == '49197' and ip_dst_addr == '192.168.1.2' and ip_dst_port == '80' and protocol == '6'"
3. Verify the MR job finishes successfully. Upon completion, you should see a file named with the current datestamp in your current directory, e.g. pcap-data-20160617160549737+0000.pcap
4. Copy the file to your local machine and verify you can open it in Wireshark

References:

• METRON-119 Move PCAP infrastructure from HBase #93 (comment)
• https://github.com/apache/incubator-metron/tree/master/metron-sensors/pycapa

[cestella]

Is this a short option as well as a long option? All of the other CLI programs use both, so we should conform.

[cestella]

Overall outside of the things I mentioned, this looks really great. Solid contribution @mmiklavc. The only other thing that I would add is to please document this as part of the README.md in pcap-backend A simple description and the possible options for the CLI in both modes would be sufficient.

[cestella]

Can we conform to the naming convention around src/dst addr/port and change:

• srcAddr to ip_src_addr as the long option
• dstAddr to ip_dst_addr as the long option
• srcPort to ip_src_port as the long option
• dstPort to ip_dst_port as the long option

[james-sirota]

I agree with Casey. We need more docs on the CLI. What can I query for? Which commands should I run to validate this?

[cestella]

I thought these were long args now, how does - work as opposed to --?

[cestella]

+1 from me

[dlyle65535]

+1, looks good!