METRON-183 Allow the simple hbase enrichment adapter and simple threat intel adapter to use multiple column families

[cestella]

Allow the simple hbase enrichment adapter and simple threat intel adapter to use multiple column families. As it stands, the table and column family used are configured at topology submission time rather than via zookeeper. It makes sense to allow different enrichment types to be associated with different column families so that column family specific configuration, such as retention policies, can be used per enrichment type.

[merrimanr]

+1

[james-sirota]

The docs should probably be moved out of common to the enrichment topology

[james-sirota]

What am I supposed to do to validate this? I see there is now a CF field in the enrichment and threat intel loaders and it bulk loads HBase with that CF. Is that it?

[cestella]

The way to validate this is to

• Create a second column family on the enrichment HBase table, say cf1
• Push some enrichment data into the table in that column family. Something similar to the malicious IP's from the blog post. I'd suggest calling the enrichment type malicious_ip
• Modify the configs for one of the topologies you want to test in /usr/metron/0.1BETA/config/zookeeper/enrichment to add a config section under threatIntel marking that malicious_ip should come from column family cf1 like the following

{
      "index": "bro",
      "batchSize": 5,
      "threatIntel": {
        "fieldMap": {
           "hbaseThreatIntel" : [ "ip_dst_addr" ]
        },
      "fieldToTypeMap": {
        "ip_dst_addr" : [ "malicious_ip" ]
        },
      "config" : {
          "typeToColumnFamily" : {
                        "malicious_ip" : "cf1"
                                 }
                }
      }
 }

• Run some data through and ensure that enrichments still exist.

[cestella]

As it stands, we have the docs for the configs with where the configurations live. I think you are absolutely right that we need to move them closer to the place where they are used, rather than where the configuration objects are defined.

[james-sirota]

+1. I was able to put the enrichments into different column families and have the message enriched appropriately