METRON-1158: Build backend for grouping alerts into meta alerts

[justinleet]

Contributor Comments

The motivation for this is also in the ticket, but the idea is that we want to be able to group alerts into a meta alert that lives and is queryable alongside the alerts from the standard sources. These would be manually created, particularly from the UI, as a result of investigations and slicing and dicing done there.

A REST API endpoint allows for the creation of a meta alert and another allows for retrieving all meta alerts associated with a given alert (by GUID). The first is obviously directly useful, and the second is more useful for testing and validation, but may be useful in other contexts (e.g. finding events that tend to get swept up into sets of investigations).

The various search endpoints should work as expected, and be able to query on the metaalerts index.

A test plan is incoming, and a couple more things need to be checked out a bit more thoroughly, but it should be pretty set for review. If anyone is interested, it should be easy enough to spin up, use the REST APIs (both the couple small meta alert ones along with the other search ones) and interrogation of ES to see what's going on. It's not completely transparent, because searches against the child alerts of meta alerts need to be done against the alert.<field>, e.g. "query": "alert.guid:35adafd5-ffd1-4b80-806b-91d336b87220 OR guid:35adafd5-ffd1-4b80-806b-91d336b87220" Pretty sure I made a mistake when testing this, because it worked fine in the example provided in the comments.

The basic test plan outline is going to be along the lines:
Spin up full dev
Create a couple meta alerts through the REST API.
Ensure they come back as expected from queries.
Ensure they link up appropriately as seen from the other endpoint.
Ensure the other endpoints still work as expected (e.g. search should still work, etc.)

Documentation also needs some updating and will be added (but I'll obviously take into account anything anybody wants as I write more of it).

Notes

• Given that these are manually created, a single index is used, rather than a timed index like for the standard alerts
• Creation of meta alerts takes a map from GUID -> Index. This is so we can do a multiget on the items, rather than trying to search through every index for them.
• A list of groups is provided as metadata for the UI to be able to reproduce whatever slicing and dicing was used to group these alerts into a metaalert. This could be extended to take arbitrary metadata, but that seems like followon. It's possible we may want to do something akin to METRON-1114: Add group by capabilities to search REST endpoint #702, but I'm not familiar enough with it to say what the benefits of any synergy are.
• Unfortunately, sorting has to be done on a common field. This means the metadata index is required, at index time, to populate the common field. In our case, we use the standard threat score field. By default, the meta alerts will use the sum of the child alerts threat scores, but this can also be set via configuration to "median", "min", "max", "average", and "count". All values are calculated and stored in the event that they have other interest, but can't comingle with standard alerts.
  • It's possible that scripted sorting could be used to get around that limitation, but that's a potentially much larger change that interferes with existing sorting and requires appropriate construction of the script to handle. Plus, scripting generally isn't enabled by default for security reasons, iirc.
• The templates have an "alert" nested field added to them. To my incredible annoyance, this version of elasticsearch will fail on any searches where "alert" is missing rather than just consider it not found. Even though it's in a "should" clause. This is improved on in 5.x, where there's a param to avoid failure. The fields should be removed again as part of that upgrade.

Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.
Please refer to our Development Guidelines for the complete guide to follow for contributions.
Please refer also to our Build Verification Guidelines for complete smoke testing guides.

In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:

For all changes:

• Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
• Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
• Has your PR been rebased against the latest commit within the target branch (typically master)?

For code changes:

• Have you included steps to reproduce the behavior or problem that is being changed or addressed?

• Have you included steps or a guide to how the change may be verified and tested manually?

• Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:

  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  

• Have you written or updated unit tests and or integration tests to verify your changes?

• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

• Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

For documentation related changes:

• Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via site-book/target/site/index.html:

  cd site-book
  mvn site
  

Note:

Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that travis-ci is set up for your personal repository such that your branches are built there before submitting a pull request.

[justinleet]

Create Meta Alert

The first thing to do is to build and deploy full dev with this PR.

Afterwards, we're going to set up a bit of base data.
Retrieve the current list of indices so we know where to put our data

curl 'node1:9200/_cat/indices?v'
health status index                     pri rep docs.count docs.deleted store.size pri.store.size
green  open   snort_index_2017.09.06.14   1   0        130            0    180.9kb        180.9kb
green  open   bro_index_2017.09.06.14     1   0        160            0    564.3kb        564.3kb
green  open   .kibana                     1   0         52            0     71.2kb         71.2kb
green  open   metaalerts                  1   0          6            0     62.3kb         62.3kb

In this case, we care about snort_index_2017.09.06.14 and metaalerts. To make our lives easier, we'll add a couple of stripped down messages to the our snort index (Make sure to sub in the correct index name):

curl -XPUT 'node1:9200/snort_index_2017.09.06.14/snort_doc/snort_test_1?pretty' -H 'Content-Type: application/json' -d'
{
  "msg": "snort test alert",
  "ip_dst_port": "8080",
  "ethsrc": "0A:00:27:00:00:00",
  "protocol": "TCP",
  "source:type": "snort",
  "ip_dst_addr": "192.168.66.121",
  "ip_src_addr": "192.168.66.1",
  "threat:triage:rules:0:score": 10,
  "timestamp": 1504708744000,
  "threat:triage:rules:0:reason": null,
  "threat:triage:score": 10,
  "is_alert": "true",
  "ip_src_port": "50187",
  "guid": "snort_test_1"
}
'

curl -XPUT 'node1:9200/snort_index_2017.09.06.14/snort_doc/snort_test_2?pretty' -H 'Content-Type: application/json' -d'
{
  "msg": "snort test alert 2",
  "ip_dst_port": "8080",
  "ethsrc": "0A:00:27:00:00:00",
  "protocol": "TCP",
  "source:type": "snort",
  "ip_dst_addr": "192.168.66.121",
  "ip_src_addr": "192.168.66.1",
  "threat:triage:rules:0:score": 10,
  "timestamp": 1504708744000,
  "threat:triage:rules:0:reason": null,
  "threat:triage:score": 10,
  "is_alert": "true",
  "ip_src_port": "50187",
  "guid": "snort_test_2"
}
'

At this point, we'll group these alerts together. In Ambari, go to Metron -> Quick Links -> Swagger UI and go to the "Meta Alert Controller".
Click on the create endpoint, and we'll want to send this request (but with the appropriate index subbed in):

{
  "groups": [
    "group_one",
    "group_two"
  ],
  "guidToIndices": {
    "snort_test_1":"snort_index_2017.09.06.14",
    "snort_test_2":"snort_index_2017.09.06.14"
  }
}

Validate that the metaalert was created and looks good:

curl 'node1:9200/m*/_search?pretty'
{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : "metaalerts",
      "_type" : "metaalert_doc",
      "_id" : "d414a9f2-7a03-4e47-9108-16b6fc800dd3",
      "_score" : 1.0,
      "_timestamp" : 1504725383439,
      "_source" : {
        "average" : 10.0,
        "min" : 10.0,
        "median" : 10.0,
        "alert" : [ {
          "msg" : "snort test alert",
          "threat:triage:rules:0:reason" : null,
          "ip_dst_port" : "8080",
          "ethsrc" : "0A:00:27:00:00:00",
          "threat:triage:score" : 10,
          "is_alert" : "true",
          "protocol" : "TCP",
          "source:type" : "snort",
          "ip_dst_addr" : "192.168.66.121",
          "ip_src_port" : "50187",
          "guid" : "snort_test_1",
          "ip_src_addr" : "192.168.66.1",
          "threat:triage:rules:0:score" : 10,
          "timestamp" : 1504708744000
        }, {
          "msg" : "snort test alert 2",
          "threat:triage:rules:0:reason" : null,
          "ip_dst_port" : "8080",
          "ethsrc" : "0A:00:27:00:00:00",
          "threat:triage:score" : 10,
          "is_alert" : "true",
          "protocol" : "TCP",
          "source:type" : "snort",
          "ip_dst_addr" : "192.168.66.121",
          "ip_src_port" : "50187",
          "guid" : "snort_test_2",
          "ip_src_addr" : "192.168.66.1",
          "threat:triage:rules:0:score" : 10,
          "timestamp" : 1504708744000
        } ],
        "max" : 10.0,
        "threat:triage:score" : 20.0,
        "count" : 2,
        "guid" : "d414a9f2-7a03-4e47-9108-16b6fc800dd3",
        "groups" : [ "group_one", "group_two" ],
        "sum" : 20.0,
        "status" : "active"
      }
    } ]
  }
}

Note that both sub alerts are present, the various counts are filled in, a GUID has been given specifically to this meta alert, etc.

Searching

Searching from the REST API works mostly as expected. After the above data has been created, use the search endpoint to run this query:

{
  "from": 0,
  "size": 2,
  "indices": [
    "*"
  ],
  "query": "guid=snort_test_2",
  "sort": [
    {
      "field": "_timestamp",
      "sortOrder": "ASC"
    }
  ]
}

The result should look similar to the following, and have the messages we'd expect (one from snort and one from metaalerts):

{
  "total": 2,
  "results": [
    {
      "id": "snort_test_2",
      "source": {
        "msg": "snort test alert 2",
        "threat:triage:rules:0:reason": null,
        "ip_dst_port": "8080",
        "ethsrc": "0A:00:27:00:00:00",
        "threat:triage:score": 10,
        "is_alert": "true",
        "protocol": "TCP",
        "source:type": "snort",
        "ip_dst_addr": "192.168.66.121",
        "ip_src_port": "50187",
        "guid": "snort_test_2",
        "ip_src_addr": "192.168.66.1",
        "threat:triage:rules:0:score": 10,
        "timestamp": 1504708744000
      },
      "score": 1,
      "index": "snort_index_2017.09.06.14"
    },
    {
      "id": "d414a9f2-7a03-4e47-9108-16b6fc800dd3",
      "source": {
        "average": 10,
        "min": 10,
        "median": 10,
        "alert": [
          {
            "msg": "snort test alert",
            "threat:triage:rules:0:reason": null,
            "ip_dst_port": "8080",
            "ethsrc": "0A:00:27:00:00:00",
            "threat:triage:score": 10,
            "is_alert": "true",
            "protocol": "TCP",
            "source:type": "snort",
            "ip_dst_addr": "192.168.66.121",
            "ip_src_port": "50187",
            "guid": "snort_test_1",
            "ip_src_addr": "192.168.66.1",
            "threat:triage:rules:0:score": 10,
            "timestamp": 1504708744000
          },
          {
            "msg": "snort test alert 2",
            "threat:triage:rules:0:reason": null,
            "ip_dst_port": "8080",
            "ethsrc": "0A:00:27:00:00:00",
            "threat:triage:score": 10,
            "is_alert": "true",
            "protocol": "TCP",
            "source:type": "snort",
            "ip_dst_addr": "192.168.66.121",
            "ip_src_port": "50187",
            "guid": "snort_test_2",
            "ip_src_addr": "192.168.66.1",
            "threat:triage:rules:0:score": 10,
            "timestamp": 1504708744000
          }
        ],
        "max": 10,
        "threat:triage:score": 20,
        "count": 2,
        "guid": "d414a9f2-7a03-4e47-9108-16b6fc800dd3",
        "groups": [
          "group_one",
          "group_two"
        ],
        "sum": 20,
        "status": "active"
      },
      "score": 1,
      "index": "metaalerts"
    }
  ]
}

[merrimanr]

Is this a mistake?

[justinleet]

Absolutely a mistake. Thought I'd fixed that, but apparently not.

[merrimanr]

You might want to consider moving this to TestConfig instead. The REST layer depends heavily on the InMemory components and reuses them across all the integration tests. You are breaking the pattern although I understand why because setting up InMemory components is usually done in @BeforeClass in other modules. This would allow other controller integration tests to leverage this infrastructure without having to do anything.

The SearchController uses a mock ES backend but, in my humble opinion, it's hard to maintain and is unnecessary. Would be nice to switch that to what you've done here someday.

[justinleet]

This actually is the metron-elasticsearch integration testing, so it's outside the scope of the REST layer. This is pretty much the pattern that exists within that module.

The MetaAlertControllerIntegrationTest does use the mock ES backend (InMemoryMetaAlertDao maintains the mock and delegates to it similar to how the actual DAOs work)

[merrimanr]

You are correct. I read this wrong and thought it was the REST controller test.

[merrimanr]

Great job on this. I spent a lot of time testing and had trouble finding anything wrong. The one thing I did notice was that the "timestamp" field was missing from documents in the metaalerts index. This is the field we commonly use for sorting so we might want to consider adding it or switching our default sort field to "_timestamp" instead.

[justinleet]

@merrimanr I added the timestamp field in the meta alert create with current timestamp. Should take care of it lining up with the other sources.

[merrimanr]

Tested this again in full dev and now the default sort is working as expected (due to timestamp being added). +1 from me. Nice job.

[justinleet]

@merrimanr I added in one fix for some typing on meta alert updates and also merged in master and the group stuff

[justinleet]

@merrimanr Are you still +1 after the most recent changes before I commit this?

[merrimanr]

I'm still +1